As questionnaire designers, how do we maximize survey content while keeping them reasonable for our respondents? By making smart scale choices. And one of the common choices we have to make is among available options for measuring priorities. In various market research applications, we need to gather priorities. Just a few of the many common examples:
- Product features that need to be prioritized in terms of customer preference
- Marketing messages that need to be prioritized by relevance to a given target market
- Perceived brand traits that need to be prioritized by believability
- Behaviors (such as activities) that need to be prioritized based on enjoyment
When we need to include prioritization in our survey instruments, we have several options. Two that are easy to implement in any online survey platform are the tried and true rank order and constant sum scales. But how to choose even among these two? First, let’s review their structures and then consider the trade-offs.
Rank Order Questions Measure Priorities
Rank order questions allow respondents to order answer options based on their personal preference. This allows respondents to evaluate multiple row items (such as ice cream flavors, as in our example) in relation to one prioritization metric (such as importance, believability, or as in this case, preference). This commonly used scale option is illustrated in Table 1.
Please rank the following ice cream flavors in order of your preference for them, from 1 to 5, where 1 is most preferred and 5 is least preferred.
Why use rank order questions?
- Respondents find them fairly easy. Rank order questions are familiar to many people, making it easy for participants to use.
- Survey programmers like them. They are easy to program, and available in all survey platforms.
- Researchers like them for:
- Clarity. Data from rank order questions show objective, clear priorities.
- Analysis. Rank order questions identify priorities by identifying order of preference, which can be considered in aggregate or by sub-group (such as by market segment or demographic subgroups)
- Ease of reporting. The “winner” can be reported using both means and frequencies to tell the complete story.
The challenge for survey researchers:
- Rank order questions generate ordinal data, not interval, which has analytical limitations. We will have measured priorities, but we don’t know the “distance” between them. Is the #1 item twice as important as the #2 one? Or just a bit more? Is the “last item” less important than the others, or is it totally unimportant? We don’t know.
- There is a limit on the number of items we use. For example, you wouldn’t want to ask your respondents to rank 20 flavors. It’s considered best practice to limit the list to 5-7 items and to make sure to employ randomization, limiting first-order bias. Some researchers will use as many as ten items, but in these cases, they often ask participants to rank order their “top 3”, not the whole list of ten.
Constant Sum Questions Measure Priorities & Weight
An under-used but effective alternative to using rank order questions, Constant sum questions also measure priorities. In this scenario, respondents indicate their priorities by allocating 100 “points” (other units can be used, but “points” is most common) to indicate their responses. Each numeric entry is summed and the total displayed to the respondent. Table 2 shows an example of a constant sum question. In this example, the constant sum question instructs the participant, “Please distributes 100 points among the following 5 items, based on their importance to you when buying ice cream. An item that has no importance to you should be given 0 points. The total must equal 100 points.”
Why use constant sum questions?
- It allows for precise discrimination amongst items. I know that not only is chocolate #1, I know by how much it leads the #2 flavor.
- It allows for ties. If the respondent honestly feels that 3 items are equal, they can give them equal “weight.”
- Like rank order questions, this scale is also objective in nature. It carries little risk of subjectivity bias.
The challenge for survey researchers:
- This method requires greater cognitive effort from participants as it involves asking people to do math. So be sure to use your survey platform’s features to aid the participant. As Julie Worwa, an instructor with Research Rockstar, suggests, “Make sure your survey programmer shows the running total to help your respondents know how many points they have left to allocate.”
- Just like rank order questions, with constant sum there is a limit on the number of items that you want to use. Many researchers limit the list to 4 or 5 items, as the total “100” is easily divisible for 4 or 5.
Conclusion
There are benefits and limits with both rank order and constant sum question types. If you want a simple rule of thumb, you will usually want to choose rank order questions when you need priorities but not the “weight” per priority. Rank order questions require less cognitive effort than constant sum, so if you can keep it simple for your respondent, that is always best. But if you need the extra granularity of constant sum, go for it—just keep it simple. And if you need to prioritize more than 5 or 10 items? Consider other options, such as using two questions (break a long rank order list into two shorter ones) or consider other options such as MaxDiff.
