A surgeon wouldn’t begin surgery without first ensuring that their medical instruments were sanitized. And most professional researchers wouldn’t launch a new survey without first testing their instruments (the questionnaire). But these days there is a lot of temptation to skip the pre-test in order to shave off a day or three from a tight project schedule. We all know the reality: market research & insights professionals are under pressure to deliver results faster, and often more iteratively, than ever before. Timelines for preparing deliverables are short, and sometimes corners are cut. Sometimes that “corner” is pre-testing.
No, not the pre-test! There are other, better ways to save time without compromising known market research best practices, like the pre-test. Unless the survey research you are doing is a tracking study or an ongoing transactional study (where the questionnaires have already been tested and standardized), pre-testing is critical to your survey research’s successful outcome. Not convinced? Let’s review what pre-testing really does for survey research data quality.
Pre-testing or Soft Launches for Survey Research
I use the phrase “pre-test,” as that is the phrase used in Research Rockstar classes such as Market Research 101 and Questionnaire Design 101: A 10-Step Process. Some people prefer the term “soft launch.” The phrase “soft launch” is often used by tech companies, referring to a limited roll-out phase of a new product that comes after Beta testing but before a full market launch.
I am not hung up on the language, but there are four minimum elements that are required in professional market research pre-testing, regardless of your preferred lexicon:
- Collect “real” data. Collecting responses from real research participants is essential. A pre-test is not asking your Aunt Sally to take your survey and give you feedback. Sure, get Aunt Sally’s feedback—but before the pre-test, not instead of it. A real pre-test needs to be done with people from the actual sample source.
- Use the final questionnaire. The true pre-test should be done with the final instrument—not a draft you know still needs work.
- Use the intended data collection methodology. If it is an online survey, collect the pre-test responses online. “Phone” testing an online survey isn’t a true pre-test. Maybe it can be a pre-pre-test. For example, if you need to get feedback on answer options for a particularly jargon-full questionnaire, great, do some early phone work so you can find out how people are responding to answer options and wording. But that is not a pre-test.
- Analyze the pre-test results. It isn’t a pre-test if you don’t actually look at the results to identify opportunities for improvement. There are several things we look for in a pre-test, but the most important one for many people is survey duration. This is a huge market research budget consideration—and can either hurt or help. If you work with panel providers, you especially want to have an accurate estimate of survey duration. What if you told your panel provider the average duration would be 12 minutes, but your pre-test says 8? That four-minute difference represents real savings for you.
Pre-testing: Is Your Questionnaire Prepped for Surgery?
In reality, you may find that for every five questionnaires you pre-test, you may only make post-pre-test changes to two of them. Three may be good to go, as is, no changes needed. Great! But the two that did need optimization? Those are important. I have had pre-tests catch duration issues, drop-out risks, confusing scale labels, and truly egregious programming logic errors. Had these things gone unchecked, the research’s data quality would have been notably compromised.
Bottom line? Yes, we are under pressure to deliver market research studies quickly. And the temptation to cut corners is real. But cutting the pre-test puts data quality at risk. And like our friend the surgeon, we don’t want to cure the disease and kill the patient.
