Does Online Panel Quality S*ck?

A huge topic for Market Research & Insights professionals is data quality, and in most cases, a key component of survey research data quality is driven by sample quality.

The catch? Many of us don’t have our own pristine, in-house sample sources. Sure, some research teams primarily do research with customer populations (where a private community or panel may be the primary sample source), and some have access to other specialty sources. But for many U.S.-based survey researchers, a common sample source is online panels, available from over 20 providers.

Need to collect data from 2,500 SUV owners, 500 ferret enthusiasts, or 100 IT managers? There’s a panel for that. If you have never ordered from a panel company, the scenario is basically this:

• Imagine you are now the Insights Manager at Fabulous Ferrets, a nationwide chain of ferret grooming salons (Fido would be jealous), and you are preparing for a survey of current ferret owners. Unfortunately, you do not have a customer list, so you decide to check out online panels. Could one of them get you access to this special population? Ferret owners are a niche population—so if you are planning a 10-minute online survey, you could be looking at a whopping $15 per complete (this is a completely fictional example, of course).

Online Panel Companies

Online panels are a big business. One could argue that some of the largest market research companies in the entire industry are the panel companies. Panel provider Dynata is rumored to have revenue of over $400 million a year, making it easily one of the top 20 “market research” companies, though smaller than industry giants Nielsen, Ipsos and GfK.

A lot of money is being made by a lot of panel companies. And any survey researcher who has been in the biz will tell you: respondent quality varies. But by how much? Sure, we all know you get what you pay for, but how bad can it really get?

According to online panel expert Ron Sellers, pretty bad.

In August 2020, Ron published his latest update on panel quality, “Still More Dirty Little Secrets of Online Panels.” In his most recent analysis of online panel quality, in which he tested response quality online panels, Ron found some disappointing results. One stunning example: in five of the ten panels tested, his team had to toss 46% of the responses due to strong indicators of data quality issues and even fraud.

Ouch. Many survey researchers rely on online panel sources.

So here’s a tip: when it comes to online panels, you really do get what you pay for. That panel provider that comes in at a 30% lower cost per complete than other suppliers? Yeah, there’s a reason for that.

Ron published his recent online panel quality analysis findings, and you can get his report here. And you can also enjoy my recent interview with Ron—I hope you enjoy our conversation. 

Ron welcomes your questions and comments. You can contact him directly:

RELATED: Be sure to check out Instructor Jeff McKenna’s Sampling Practicum for some great content on sampling, including panels. Be sure to check out the video lectures to have Jeff teach you information all survey research professionals need to know about sampling techniques overall, as well as sample source issues (like online panels).

ALSO RELATED: I also recently interviewed another sample expert on online panel quality, Andrew Moffatt of OpinionRoute about current and evolving challenges with online panel quality, including fraud, and the fixes that may be on the horizon (this guest lecture is available to Research Rockstar students in the Portal).