Survey Design 2011: NEW REALITY, NEW RULES

Let’s get a few facts on the table, shall we?

1. We all have non-researcher friends, colleagues and acquaintances who are writing their own online surveys using one of the many free or low-cost tools.
2. Despite all of the well-publicized voices shouting, “The survey is dead!”, “DIY” survey volume continues to surge.
3. We may tire of hearing, “The survey is dead!”, but we all know that the role of the survey isn’t what it used to be, and we really aren’t that sad about it. Indeed, many of us now spend more time on non-survey methods, either to develop best practices or for “live” projects.
4. Short surveys are better surveys.
5. New tools and technologies create possibilities for survey design and data analysis that never existed before.

So, what does this hodgepodge of facts mean?

The role of the survey is changing. Who creates them, how they are designed, when they are used, and their role relative to other methods is all shifting, fast. To remain effective in this new reality, survey research best practices must change, and the time is now.

New Rules of Survey Design

Some of the New Rules are driven by length, others by what is feasible given recent advances.

Then: Three to five screening questions were acceptable.

Now : One or two. Better yet, none—just collect the profiling data from your MROC database or panel provider.

Then: 15 to 20 minutes average duration was the old “short” compared to the 40 to 60 minute monsters of the old days.

Now: Ten minutes or less.

Then: Multiple questions per screen to minimize number of total screens.

Now: Shorter surveys make it worthwhile to exploit the visual appeal of one question per screen.

Then: Two to three open-ended questions at most, due to both respondent compliance and analysis headaches.

Now: Wildcard. We still want to avoid demanding too much “work” from our respondents, but these questions have become much easier to analyze. Whenever an open-ended question could replace a question with a long list of answer options, it’s worth a look.

Then: Survey design and programming were done by a professional researcher, going through multiple iterations, often taking days if not weeks.

Now: Basic survey design can be easily done by any motivated professional with the help of training, survey templates, or other support.

Then: Surveys designed with the assumption that they are the primary data source for an overall project.

Now: Surveys designed with the context that the survey is only one source of data, and is likely to be used in combination with multiple data sources.

Then: Surveys designed with the assumption that there would be one mode of data collection (typically phone, online or paper).

Now: Surveys designed for multiple modes of data collection, possibly including mobile.

I am generalizing here, and there are plenty of “buts” and “exceptions.” The reality, however, is that the landscape of survey design has changed. We can’t let yesterday’s best practices hold us back today.

Bottom Line

Those of us who remember spending hours (even days) to craft what can only politely be described as visually bland surveys can be forgiven a secret twinge of envy when we see DIYers creating visually stunning surveys in under an hour. But we have to get over ourselves; new tools are in, and the old rules are out.

[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]