The struggle (for data quality) is real. And a large contributor to data quality risks in online surveys is, you guessed it, survey design. So how do we ensure that we are designing questionnaires that will collect quality data? A great start is by avoiding known, common design mistakes.
If you have been doing professional survey research for even just a few months, you likely know that some of the most common survey design mistakes include having surveys that are too long, too onerous, or that have questions that encourage bias (of which there are many types). But what are the next most common mistakes? Here they are:
- Overusing open-ended questions. Very often, when we’re writing a questionnaire, we want opportunities to discover new things or we are unsure about which discrete answer options to offer. Our solution? Open-ended questions. For example, we might ask people about unaided brand awareness; that is, “When you think of Product Category A, what brands come to mind?” Or maybe “What else can our company do to improve your satisfaction with our products?” Or perhaps you ask a closed-ended question followed by a list of possible answers, including an “Other. Please specify:_________” option. Having one, two or maybe even three questions that require a typed response is reasonable. However, once you start to ask more than that, it becomes too onerous, and few respondents are willing to type that much. The undesirable result? You end up with a lot of blank or “garbage” results (people typing in random letters just so they can progress to the next screen). Plan of action: Choose wisely and use open-ended questions judiciously.
- Using jargon. If you’re doing a survey project, there is a good chance you have a great deal of knowledge in a particular product category, industry, or topic area. As an expert, you have likely developed a specific language for talking about relevant issues. It’s very easy for subject matter experts (SMEs) to forget that other people simply don’t use the same language to discuss the same topics. Using too much jargon turns people off and can lead to dropouts. Even worse, if they don’t know what a term means, they will guess, which can really impact data quality. We have to be vigilant when we’re creating surveys to use friendly language. Plan of action: Go for the lowest common denominator in terms of who’s going to be taking your survey and only use language that they are likely to use.
- Combining two questions in one. Commonly referred to as “double-barreled questions”, this often-well-intended attempt at being concise just ends up hurting data quality. Example: “How satisfied are you with the look and navigation of our new website?” (followed by a 5-point satisfaction scale). Without separating the concepts of look and navigation, it is impossible for respondents to accurately describe how they truly feel about either concept. And they may feel frustrated that they can’t tell you the truth, which may be that they are very satisfied with one, but not the other. Plan of action: Break these questions into two separate items: “How satisfied are you with the look of our new website?” and “How satisfied are you with the navigation of our new website?”
Great Survey Design for Great Data Quality
While there are many factors that impact online survey data quality, questionnaire design is a big one. The three common mistakes above can impact data quality and are easily avoided. Better still, by avoiding them, you’ll make a great impression on your survey participants with your professionalism and attention to detail.
