How We Test

Test devices of cooltechzone.com
6 devices which are used for VPNs testing

To choose the best VPN services, we have developed and use our own methodology, and here's why:

Most of the articles we have seen on the Internet don’t provide any confirmation of the reliability and security of services. There wasn’t even a trivial comparison of VPN providers’ performance on different platforms

We decided to do even deeper research:

  • Our own software for testing. We have developed special software to test VPN apps under actual usage conditions, where disconnections from the Internet are possible, devices change their locations and connect to various networks.
  • Traffic interception. We use Wireshark, a powerful network activity analyzer, to intercept traffic for Windows, Mac, and Linux.
  • All devices. We use 5 devices and one virtual machine:
  1. Notebook HP
  2. MacBook Pro
  3. Huawei Mate smartphone
  4. Huawei tablet
  5. iPhone X
  6. VMware 15        
  • All OSs. We test VPN software on all the five major OSs - Windows 10, macOS 10.13.6, two versions of Android (6 and 10), iOS 13, Linux Ubuntu 18.
  • Speed Test on 1 Gb tunnel. We set up an Internet server, which we use to measure the speed of VPN services.
  • VPN Spiral. We have combined all these issues, added our experience and developed our own method of analysis, testing and ranking of VPN services -“VPN Spiral”. This method helped us create dozens of reviews and rankings that we are proud of.

15 steps of VPN services researching using the VPN Spiral Method

Almost every step can be correlated with a spiral coil which is twisted along a diminishing radius. As a result, by the end of the research, just a given number of VPN services that meet one or another spiral parameter (VPN requirements) remains there.

1. Making a List of relevant VPNs

Making a list of VPNs

We are aware of more than 100 VPN services and are constantly updating this list using the following sources:

  • advertising
  • news
  • Reddit, Twitter, and other resources where VPN topics are discussed
  • search engines result on specific queries
  • other sites with VPN reviews
  • Google Play Market and Apple App Store
  • personal communication

After that, we check out if the services work and if it allows you to log in without an invite.

2. Fraud Check (for new services)

We don’t research those VPN providers that do not have a clear monetization or funding model.

If the service is paid, then it must accept payment through its website or application. At the same time, the site or service through which payment is performed should not be blacklisted (it should not be in databases with authentically fraudulent resources).

If the service is free, then we check how advertising works or look for other funding sources. For example, VPN Gate is ad-free and funded by National University of Tsukuba, Japan.

Funding of VPN providers research

We study the Privacy Policy and the Terms of Usage concerning users’ data processing both with paid and free providers.

We reject those providers which use suspicious methods of money-making, in order to protect users from fraudulent and other criminal schemes.

3. Signing up, payment, soft downloading

Each VPN ends up in our computers and other devices in the same way that in any ordinary user’s computer:

  1. Signing up on the site, if there is a website
  2. Payment by a credit card or PayPal

PayPal receipt for Private Internet Access

PayPal receipt for StrongVPN

  1. Downloading applications or configurations for third-party VPN clients
  2. Setup and running of apps or configure a third-party client on all available platforms

At this stage, only those providers that have functioning software or correct settings for the required platform remain (for example, VPN for Mac).

4. Researching and comparing functions and features on each platform

Researching of VPN services features

At this stage, we get acquainted with the declared features, settings and prices of a VPN service. Information is taken from its official website as well as from its applications for each OS.

Each function that may affect a VPN performance is flagged for further testing and comparison with other services.

At this point, we select those VPNs that will suit a particular ranking (for example, VPN for torrenting, cheap VPN, etc.).

5. Security and Data Protection Tests

Wireshark for traffic analysis

The most important parameter for each VPN service is the ability to reliably hide the user's location and protect the traffic. It is necessary for a VPN to hide the addresses of visited sites and prevent IP leaks.

To achieve the most reliable results, we intercept and analyze traffic at the output of a tested device (a computer with Windows or Mac, a virtual machine with Linux).

Thanks to an in-depth traffic analysis, we can identify with 100% accuracy whether unprotected data is being sent to the network, whether the addresses of visited sites will be visible to the ISP or to the government firewall, and much more.

At this stage, we do not filter out VPN services, but we report about identified bugs in our future ranking. Besides, we indicate how to enhance security using the settings, if it is possible.

6. IP Leak Test done on special services

IP Leak Test on public services
The image was taken with the help of ipleak.net

Even though we usually identify most of the security problems when analyzing network activity, we additionally carry out a classic test – we check for WebRTC, IPv6 and DNS leaks using public services (ipleak.net, whoer.net, etc.)

  • WebRTC leak allows in some cases to find out the user’s IP address with a VPN enabled.
  • IPv6 leak may expose the user's current location if a VPN is connected to the classic IPv4 which is used most often.
  • DNS leak may reveal the addresses of visited sites and the user's country (in some cases, it may lead to the location disclosure).

We filter out those VPNs that allow all the three leaks.

7. Mobile apps test under actual usage conditions

Cool Tech Realtime IP Logger

We test mobile applications for the ability to protect user’s data under actual usage conditions when a device can connect to different hot spots, switch between Wi-Fi and the cellular Internet, and some other situations.

At such moments, a VPN is vulnerable and, if no additional measures are taken, allows leaks of locations and insecure data.

We check all possible combinations of unexpected connection failures with a VPN server and register them with Cool Tech Realtime IP Logger, software created by cooltechzone.com.

8. Speed Test

Speed Test
The image was taken with the help of speedtest.net

The test of the maximum possible speed of VPN services is carried out using the speedtest.net service. We test the fastest VPNs which provide speeds above 100Mbps using a special Internet server connected to a 1Gb tunnel.

9. Streaming and Torrenting Tests

Test for streaming support

We also test some of the most popular and solicited VPNs features:

  • Streaming. Bypassing the geo-restrictions of Netflix, Hulu, HBO, and other video services, including on TV platforms;
  • Torrenting. Safe use of torrent clients, access to free video content distributing via P2P technology (Popcorn Time, Kodi addons, and others).

10. Analysis and verification of VPN connection settings

If a VPN provider has any settings concerning the VPN connection aside from the choice of countries, then we check them out and analyze their work.

In some cases, we give recommendations for their correct use for certain purposes.

11. Applications Quality and stability Testing

Not all providers have their own software (applications).

At the same time, not all applications are safe and stable.

Using our experience and the results of previous tests, we select the most convenient and reliable providers. Having their own applications would be an asset, but getting into the top cannot be guaranteed.

We prioritize those VPNs that provide reliable and high-quality user’s protection, have a minimum of advertising (for free VPNs), are convenient to use, and support as many platforms as possible.

12. Analyzing of Advanced Functionality

Analyzing of the advanced functionality of VPN apps

Many VPN services are expanding their basic functionality, adding a wide variety of additional features. Here are just a few of them (we have seen more than 20 useful technologies):

  • Kill Switch
  • VPN Obfuscation
  • Double VPN
  • Split Tunneling
  • Share VPN

There is a direct connection between the level of a VPN service and their advanced functionality. We check out additional functions and prioritize more functional providers, if their other capabilities are at a high level, as well.

13. Battery Load Testing

Power consumption Test

In 2019, we introduced an additional power consumption test for VPN applications for mobile platforms cause a VPN requires additional computing resources.

Devices runtimes are estimated approximately.

14. Feedbacks Analysis

We tune in to the real users’ experience.

There are no VPN providers which review we would write without taking into account the users experience.

To collect additional information and verify our assumptions, we use:

  • Reddit
  • Twitter
  • trustpilot.com

We also developed our own methodology for filtering out “customized” posts and comments.

15. Press Analysis

Press analysis

The history of VPN services (or the history of the companies that own them), as well as current events, affect the selection of providers in a substantial way.

We study the press and follow the publications in the accounts of about 20 popular VPN services to be aware of the VPN market changes.

  • As a rule, the research of each VPN takes three days, in rare cases, it takes up to a week (we use some VPNs for our purposes, including outside of working hours).
  • ALL paid services were purchased in the same way as other users do - via a credit card or PayPal. We, like everyone else, buy, restore, use trial versions, switch to paid ones, etc

What makes our work different from others?

* We have developed and use our software to test VPNs under actual usage conditions.

* We intercept and analyze traffic coming to the network from test computers during VPN performance (on three OSs).

* In each review, we describe the features of VPN applications on every platform, not just the entire service taking all things together.

We believe that the Internet lacks accurate information about VPN services. There are dozens and hundreds of reviews of their features but there is no current evidence of their safety and verification of their performance under real (often difficult) conditions. Moreover, there is no comparison of their work on different platforms. For example, you can find a mention of a DNS leak of a provider but it does not say that this leak is on Windows only, and all other applications of the service are completely safe.