I am stunned at how many experienced market researchers don’t bother pre-testing before they start data collection for survey projects.

Stunned.

It is the market research equivalent of a pilot who decides not to bother with the pre-flight checklist before takeoff.

I have had two recent experiences where I had seasoned researchers working with Research Rockstar clients, and they had assumed pre-tests were not required.  Really? That’s the assumption? I wonder how many pilots assume pre-flight checklists don’t apply to them.

There are certainly varying opinions about many market research best practices, but this really shouldn’t be one of them. Unless the survey research you are doing is a tracking study or an ongoing transactional study (in these cases the questionnaire has been tested, standardized, and assessed over time), pre-testing is critical.

Semantics: Pre-testing or Soft Launch?

I use the phrase “pre-test” and that is what I teach in Research Rockstar classes on project management and questionnaire design. Some people use the term “soft launch.” I am not hung up on the language, but there are some elements that are required in professional research regardless of your preferred lexicon:

• Collecting responses from real research participants. A pre-test is not asking your Uncle Stan to take your survey and give you feedback. Sure, get Stan’s feedback—but before the pre-test, not in lieu of it. A real pre-test needs to be done with people from the actual sample source.
• Using the final questionnaire. The pre-test must be done with the final instrument. Not a draft you know you will be editing anyway.
• Using the intended data collection methodology. If it is an online survey, collect it online. “Phone” testing an online survey isn’t a true pre-test. Maybe it can be a pre-pre-test. For example, if you need to get feedback on answer options for a particularly jargon-full questionnaire, fine, do some phone work so you can find out how people are responding to answer options and wording. But that is not a pre-test.
• Analyzing the results. It isn’t a pre-test if you don’t actually look at the results. There are several things we look for in a pre-test, but the most important one for many people is survey duration. This is a huge market research budget consideration—and can either hurt or help. So why not be precise? Especially for researchers who work with panel providers.  What if you told your panel provider the average duration would be 10 minutes, but your pre-test says 7? That’s real savings for you.

Pre-testing: Is Your Questionnaire Cleared for Take-off?  

For every 10 projects I pre-test, I may only make post-pre-test changes in three of them. Seven go forward, no changes needed. But the three that do get changes? Those are important. I have had pre-tests catch duration issues, programming logic errors, drop-out risks, and more. So yes, even though I have been doing this for 25 years, I still do pre-tests. Does it mean I don’t ever make questionnaire mistakes? Sure I do (in fact I had a doozey just recently, which I will post about soon). But pre-testing minimizes my risks.

Bottom line? Pilots have a re-flight check list that has 50 or more steps. We researchers don’t have quite that many on our pre-launch list, but pre-testing should be right at the top.

[Interested in more free market research tips and news? Want it delivered right to your inbox? Sign up for our newsletter here.]