Getting a 16-Word Survey Wrong [a Special Guest Post by Jeffrey Henning]

I’m a fan of Google Consumer Surveys’ limitation on question length. Google limits you to questions of no more than 125 characters long, primarily – I believe – for a better experience for readers of the sites of its publisher partners (see In Search of an Answer).  The Google UI does point out that “Longer questions and answers reduce the quality of responses” and advises “Keep questions short and simple.”

Sadly, I recently encountered a Google Consumers Survey question that proved even short questions can go badly awry.

Respondents were asked “Will the $95 IRS penalty motivate you to shop this October for an Obamacare health plan?” and could choose “Yes,” “No,” and “Not Certain.”

Every day I quickly review about a dozen recently published surveys to choose a few to write about on Researchscape.com, setting aside ones like this with obvious errors. Good questions are hard to write, and I’ve written my share of bad ones. This question has quite a few problems:

1. The “$95 IRS penalty” is incomplete – The penalty is $95 per individual or 1% of household income, whichever is greater. By understating the penalty, this most likely depressed “Yes” votes.
2. The “in October” is overly specific – Some may purchase insurance in advance of October to avoid the penalty. This probably depressed “Yes” votes.
3. Obamacare is a loaded term – Originally coined as a pejorative by Republicans opposed to the Affordable Care Act, it has only gradually come to be used by Democrats. I’ve used Affordable Care Act in healthcare surveys I’ve conducted for precisely this reason. I’m not sure of the effect this would have on the overall results; it might increase nonresponse, for instance.
4. “Obamacare health plan” is too narrow – The intent was to find out if people would buy insurance because of the penalty, but they do not have to buy insurance from ACA health exchanges, as this wording may have implied. This also most likely depressed the “Yes” votes.
5. The sample is wrong – The question is asked of everyone, when it really should be asked of those currently without health insurance with a different question with new wording for those who currently have insurance. The user should have added a screening question with Google Consumer Surveys’ “Pick audience” functionality.

Whew, that’s a lot of mistakes for 16 words and 88 characters!

Questionnaire writing can be incredibly demanding, even when that questionnaire is only 16 words long.

Speaking of healthcare, maybe more Do It Yourself researchers should have their questionnaire’s vitals checked out by SurveyMedic?!

[This Guest Post was written by Jeffrey Henning of Researchscape. Be sure to follow him on Twitter at @JHenning]