Online surveys and research panels go together like…well, milk and cookies. If they are both high-quality, yummy snack. If either is poor quality, the experience is ruined.
These days, there is a lot of awareness of online panel quality issues. So what does a market research buyer need to know?
Here are 5 things you need to be aware of to find the best panel sources for your needs, and mitigate potential risks. Why is this so important? Because panel quality varies. A lot. You can’t assume all research panels are the same.
- Some panel companies focus on selling to market research agencies—not folks looking to do research in-house. So don’t be surprised if you find some agencies have a sales process that feel awkward to you.
- Panel suppliers have different processes for validating the authenticity of their members. Some do a better job of weeding out “incentive hunters” and “professional respondents” than others. A key concept here is “digital fingerprinting.” For an excellent introduction, I strongly recommend this very easy-to-read article, “Digital Fingerprinting and Sample Quality,” from Simon Chadwick as a guest blogger on the QuestionPro blog.
- Panel suppliers have varying policies about research participation for their panel members. In some panels, a member can take 10 surveys a day. In others, it is restricted to a few per month. If you want to know more about some of the brutal realities, read the new and controversial report, “The Dirty Little Secrets of Online Panels”, thanks to the research of Ron Sellers from Grey Matter Research. Read the article for instructions on how to get the full document.
- Just because a panel company allows you to use a very long, onerous questionnaire with their panelists, doesn’t mean it’s a good idea. They just make more money that way. For more on this and related points, read Jeffrey Henning’s excellent article, “Data Quality & Validation Lessons from Panel Professionals.”
- Some panel companies aggregate multiple panels. Some panel companies sell access to their own panels. But there are a few that aggregate, and de-duplicate, across multiple panels. These aggregators can be handy, especially if you are doing research with a hard-to-find population.
OK, those 5 points are key. But if you can handle a little more, I also recommend reading these 3, short articles for more tips and practical insights into online market research panels:
- From Forrester’s Reineke Reitsma, on Online Panel Quality.
- From ESOMAR, the wonderful “26 Questions” to ask before you buy online sample.
- From the QuestionPro blog, Steven H. Gittelman wrote a short blog about how Microsoft has dealt with buying online panel sources.
Look, I am not trying to scare you off. But there are risks, and by being informed you will be able to make better decisions about which panel companies to work with. There are plenty of them out here—but you do need to be vigilant to find one that will best meet your needs.
[Subscribe by email to get the latest blogs as they are published. Click here: Yes, Send me Blog posts by email!]
6 comments
Good advice, Research Rockstar! Having spent 5 years working in this arena and now working at the ‘receiving end’, your tips resonate well with me. It is definitely worth shopping around when it comes to panel suppliers!
Whilst we all need to earn a living, it is a good idea to go with a supplier that thinks of their panel as more than a ‘database’ or a ‘commodity’ or a ‘muster’ of ‘respondent. If a supplier thinks about their panel as a collective of ‘people willing to share’ – give them a big green tick!
It helps if your supplier give prioritis(z)es your research objectives above invoicing – first things first!
One that shows a committment to ‘research on research’ would also be key in my books. If your supplier is willing collaborate and share their knowledge with you and if they can demonstrate an in-depth understanding of how people behave when participating in online research, you know you are on a good wicket! If a prospective supplier declines on an offer to run a long, complex and/or onerous survey for you – take that as a positive sign! Your client will thank you!
Finally, here’s a link to a useful site:
http://www.research-voice.com/
(Whilst it is an initiative set up by a global provider, it does provide a useful space for though and discussion)
Once again, Rockstar – thanks for your useful tips!
Thanks–and nice to meet you!
My pleasure – thank you! And nice to meet you too!
Dear Research Rockstar,
Thank you for the great set of articles for in-house researchers in your blog! Many things for us, as a research software company, to look at with the client’s eyes.
We have been always defending the point that Research should be accessible to everyone. The Thing number 1 in the current post sounds like a warning…
“Some panel companies focus on selling to market research agencies—not folks looking to do research in-house. So don’t be surprised if you find some agencies have a sales process that feels awkward to you”. Can’t agree more, knowing how the sample procurement process usually works.
Not having an easy access to the key component of online research, which is access to consumers or the target groups (without paying exorbitant fees), makes it too hard or impossible for the client-side folks to do research on their own. It shouldn’t be that way, but first the research industry has to embrace DIY research and make most out of it. Make it easier for in-house researchers to find and buy sample rather than have them put up with unfriendly sales processes.
http://www.linkedin.com/groups?groupDashboard=&gid=3713287
Good points Kathryn. However sourcing panels in emerging markets has a few other considerations. Such as understanding the cultural nuances in these markets. This impacts questionnaire design and thus the research perception in developed vs. developing world.