Written Mar 6, 2017
In June of last year, I published my first VPN review on That One Privacy Site. Just previous, I had announced how I wanted to conduct this feature – namely eliminating bias, and including transparency regarding the selection of the services to be reviewed. I began using random.org’s “trails” feature which made public the services that were chosen using said website’s number generator. Early on, this solution proved useful, as anyone could get on that site as they wished, and could see the last 10 services that were rolled. Thus, it was easy to provide transparency when a service was selected.
However, during the selection process for the last several reviews, I’ve experienced a problem. When I roll a service that has already been reviewed, I re-roll and tweet out that I’m doing so, again, to be transparent. As the number of services reviewed grows, this method becomes less viable, because the rolls in the trail start becoming more bogged down with duplicates, making:
“The last 10 rolls for services to be reviewed”
…become…
“The last 2-5 rolls for services to be reviewed, mixed with other meaningless rolls”
I also wish for my reviews to be as relevant to people’s interests as possible. This has led me to consider other possibilities of conducting this selection. After some thought, I believe I have found a workable solution that will allow a selection process relevant to my reader’s interests, while keeping bias out of the equation and maintaining transparency.
Here is how I will conduct the VPN review selection from now on:
- When I am ready to begin a new review, I will send out the link to a form where anyone who wishes may nominate a service of their choice to be reviewed – this call will be sent out via twitter, diaspora, and gab.ai.
- Services nominated must be currently on the VPN Comparison Chart and can’t have been reviewed by me for at least one year (if you’re curious about a date that a given service has last been reviewed, each has a “written/last updated” section at the top).
- Duplicates will NOT be accepted and will be removed just prior to selection as will any non-response.
- random.org will continue being used to select the service and record the “roll” for people to independently verify – instead of the number on the chart, the number selected will simply correlate to the row number in the spreadsheet – which can be found here for anyone to view.
- The selected service will be highlighted and will remain on the sheet for anyone to verify, until the next review selection begins and the sheet is cleared – Respondents are also able to see summary charts and text responses from the form side, to verify, if they wish.
After some considerable thought, I believe that these changes will help the VPN reviews I perform to be more focused on services people are interested in learning more about, while maintaining transparency and negating bias.
Lastly, nominations for the next review are open as of now, and will close in approx 48 hours. The form to nominate a service can be found here (I will also send this out to the channels mentioned above).
If you like the project and find my work useful, please consider donating – your generous contributions help pay for the hosting, tools, and time I need to do my research and keep the data fresh.