Open-ended Questions: The Home Exercise Machine of Survey Research

Americans happily buy $1,000+ pieces of home exercise equipment, with the best of intentions. But what happens when the slick new gear is unpacked and assembled? Nothing. Many home exercise machines are unused. Stationary bikes become expensive clothing racks. Elliptical become awkward home décor.

In survey research, we also waste an “expensive” item: the open-ended question.

Most online surveys rely heavily on “closed-ended” questions. For example, we often pose questions that are followed by five-point scaled responses (such as “very unlikely” up to “very likely”), or that allow the participant to “check one” or “check all that apply” from a list.

In contrast, an open-ended question is followed by a text box, in which the participant can type an answer. An example of this is, “Please describe your most recent dining experience at our restaurant,” or “What’s most important to you when purchasing frozen pizza?” How about: “For your next trip to a grocery store, what department will you visit first, and why?”

But sometimes we market researchers put open-ended questions in our questionnaires, and then neglect the results because analyzing them is too time-consuming. Despite all of the advances in marketing research software, the tools available for automating the coding and analysis of open ended questions are still underwhelming.  Indeed, even now in 2016, many experienced researchers use manual processes.

However, even just one or two carefully constructed open-ended questions can give you a wealth of information. It’s worth the pain, if they are well-designed, relevant questions.

Open-ended questions are especially useful for three types of market research goals:

1. When you want to discover something: A great discovery question is, “What else can we do to improve your experience with our product?” or “What do you like best about X?” and “like least?” Simple questions like these can yield delightfully unexpected themes.
2. When you want to hear people describe their opinion or attitude in their own words: What words do they use to describe their satisfaction or experience with something?
3. When you want to measure awareness: Open-ended questions are a great way to measure awareness with the highest bar: unaided awareness. What is their top-of-mind awareness for a given product category? For an awareness question, do they name your brand? Do they name brands that you don’t think of as even being in your competitive set?

Yes, open-ended questions are “expensive.”  We are spending our participants’ time, and goodwill, by asking them to comply with these requests. We don’t want to annoy our respondents by asking for too much effort (yes, typing is effort), only to ignore their inputs.

Like the well-intentioned home exerciser, we want to avoid making the investment, only to waste it.

Yes, analyzing open-ends is hard work. So set realistic expectations: don’t put five open-ends in your questionnaire, if realistically you will only have time to analyze two.