Part of being a good market researcher is the ability to determine when the research conclusions we start to draw reflect reality, and when they’re only coincidences. A few shiny data points can easily tempt us to jump to conclusions too quickly—before we dig deeper.

I’m reminded of raising a teenager, who reported that she “just had to have” a certain pair of shoes. They were stylish. They were unbelievably comfortable.  And all her friends were getting them. In good parental fashion, I struck a deal. We’d wait a month until report cards, and if her grades met expectations, we’d consider the purchase of $100 shoes.

A month later the grades were fine, but the shoes were no longer desired. Turns out nobody was wearing them anymore because they were too tight, fell apart and looked funny. Yet had we gone with her initial “analysis”, we’d likely have concluded that this was a great product — and we’d have been wrong. So how do we avoid the trap?

Market Research & Insight

As researchers we have to keep ourselves honest.  A few coincidences do not a key finding make, and I’ve seen some frightening cases of “conclusions” based on weak data analysis and flimsy proof points. This isn’t value—it’s hyperbole.

During analysis we can’t assume that “interesting patterns” reflect a broader reality, nor force a conclusion where none is justified.

Here are two best practices to help practice the discipline of meaningful insight creation:

Challenge your own conclusions

If you’re doing a research study and the data seems to perfectly support all your hypotheses, you need to ask yourself whether your research or the process was biased.  Perhaps your research was intended to confirm things your organization already knew. But if you’re hoping for some real “aha” points, this may not be what you want.  If your research results wrap up too neatly, look closer.  Is there an opportunity to challenge any of these results?  I’ve done several customer loyalty studies that at face value gave very positive results—customers were broadly satisfied and loyal. Digging deeper, though, we learned that important segments were loyal, but due to existing contracts and cost-of-switching issues — externals that dramatically changed the story.

One way to challenge your conclusions is to seek contradictions.  Look at how specific key attitudes and patterns vary by subgroup.  Examine differences by gender, age and geography, or if it’s a B2B study, by job function, industry and company size. Does the story still hold?  Sometimes the obvious is masking something that might require a bit more intellectual rigor.  As researchers, it’s our job to be disciplined and get beyond face value.  What’s the story behind the story?

Validate your results

We have to validate our data, especially when the research conclusions seem too-good-to-be-true. Whenever possible, validate by comparing with other data sources. You want to determine if relevant external data points also show consistencies about the customer attitudes and behaviors you’re examining in this particular study. A couple of examples:

• In a recent survey project, the client used social media monitoring to find customer stories and anecdotes that reflected behaviors measured in the survey project. Not the most rigorous methodology, perhaps, but it did provide real-world examples.
• In many studies, I have had clients doing online surveys conduct a small number of follow-up IDIs with respondents to access some of the “why” context.

Market Trend or Coincidence?

Sometimes as researchers we get so excited about our data that we jump.  But remember, a few coincidences don’t justify a conclusion.  We have to make sure that the story we’re sharing can be confirmed, replicated, and demonstrated with solid proof points. Otherwise we must present those conclusions as hypotheses, or “directional”. That’s not necessarily a bad thing, but it is different, and we have to have the discipline to recognize the difference. Otherwise we’re just peddling bad shoes.

[Want access to more market research articles and training materials? Sign up for the Research Rockstar newsletter: SIGNUP]