Survey Data Quality Concerns? Mixed-mode Surveys May Be Your Answer

Kathryn Korostoff
October 16, 2019

Survey researchers have several data collection options, even if many of us default to online surveys these days. Still, in professional market research, once a given survey project’s objectives and sampling plan are finalized, the next decision is to select the data collection mode that will be the best fit. Often, we assume one mode, such as online data collection for some, phone for others. However, there are other options, and cases where more than one mode may be optimal.

What is a Mixed-mode Survey?

Mixed-mode surveys (also known as multi-mode surveys) combine data collected via, you guessed it, more than one mode. Most commonly, it combines telephone and online data collection. But technically, it can combine any of the following data collection modes:

Face-to-face (mall intercepts, event intercepts, etc.)
Interactive voice response
Mail
Online
Telephone (landline)
Telephone (cell phone)
Text messaging

In reality, mixed-mode surveys are fielded in a few different ways. In some cases, multiple modes are executed simultaneously. That is, you could be collecting data via online and phone at the same time. But some researchers take a sequential approach, where modes are introduced as a way to reach non-responders. Here’s a simple example:

Another scenario is to let the respondents choose. That is, the researcher sends an invitation (perhaps by email or postal mail) that offers mode choices. The participant can then choose to respond via online, phone or other method offered. In this case, the participant gets a say in the choice, and the researcher also gets some nice data about which subgroups choose different options.

Advantages of Mixed-mode Surveys

The potential benefits of mixed-mode data collection are:

Higher Response Rates
- The researcher can potentially make participation more appealing either by targeting specific groups with the expected likely mode preference or by offering choices (personalization).

Better Data Quality due to Representivity Improvements
- Mixed-mode survey research may boost representivity, by reaching people who may be hard to reach through a single mode. For example, surveying a given target population with a combination of email/online, telephone and postal mail may give greater coverage of the researcher’s population of interest, rather than online alone.

Years ago, when researchers did more phone and in-person data collection, another benefit of mixed-mode was potential time and cost savings. Unfortunately, that is no longer the case. If your standard is email invitations and online data collection, mixed-mode will usually be more expensive and time-consuming. Still, these are important options to consider in the interest of improving surey data quality. “Mixed-mode surveys…combine the best of all possible worlds by exploiting the advantages of different modes to compensate for their weaknesses.”[1]

Disadvantages of Mixed-mode Survey

Following are five disadvantages of this method:

Response Effects
- Survey questions received via different modes are processed in different manners by the respondents. As a result, a respondent might answer the same questions differently if by phone vs online, for example. In one study by Gallup, phone respondents gave more positive feedback than did online survey respondents: which data was more accurate? Theories exist, but this is an effect that may vary by topic and population.
Analysis Complexity
- If data is collected through more than one mode, how do we adjust the data analysis for the known response effect variations? Do we always assume that the more honest answers are from online surveys (which are more anonymous than phone surveys)? Start by analyzing the results by data collection mode: do you see differences? And if so, will you adjust the data? Do you need to perform some weighting? Some survey researchers implement very significant processes to adjust blended data as precisely a spossible.
Question & Scale Wording Variations
- How we word a question for telephone data collection can be different than how we word it for an online survey. Knowing that the data will be consolidated, do I compromise on “ideal” mode-specific wording to reduce number of wording differences?
Duplicate Responses
- Might one person answer the phone survey AND the web survey? Will we be able to prevent this, and if not, to identify those records?
Time
- It takes time to create/program multiple survey instances (such as phone and online). It also takes time to clean and combine data (and likely debate with team members about best ways to do the combining—a huge topic beyond the scope of this article, see the Recommended Reading at the end of this article).

So yes, mixed-mode surveys have some challenges. The question is: what’s the value of potentially better data from more engaged participants? Might these benefits outweigh the costs? For market research and insights professionals interested in improving survey data quality, it may be an option worth testing.

How common it is?

In “commercial” market research, online surveys alone are obviously far more common than mixed-mode, but mixed-mode is rising. A lot of anecdotal evidence suggests mixed-mode approaches are becoming more common. Why does this seem to be happening? Many survey researchers who have embraced (or at least experimented with) mixed-mode surveys have done so to try to address data quality concerns due to declining survey response rates and sample source quality concerns. Of course, mixed-mode has always been more common in public health and government research.

Want more? Recommended reading

Mixed-mode survey research is a big topic. We prepared a compendium (free, of course) of related research on research, which you can get here. In addition, here are some additional articles that will help you further your expertise on this subject: