When surveys behave badly … and what to do about it
Through a special arrangement, presented here for discussion is an excerpt of a current article from the Joel Rubinson on Marketing Research Consulting blog.
Bad survey results are toxic. They lead to false beliefs, inaccurate measurement and bad consumer segmentation that injure rather than nurture innovation and media strategies. "Fixing" a survey starts with good survey design principles, but it doesn’t end there. In the digital age, we need to "think beyond the survey" and integrate behavioral information.
Here are three reasons that surveys behave badly and what you can do about it.
Problem: Telescoping. Surveys always elicit overstatement on brands bought over the past year, past three months, etc., leading to inaccurate estimates of market penetration and misidentifying users.
The fix: Anchor in behavior. Minimize telescoping by first asking about longer timeframes, then following up with the shorter timeframe you are interested in. To remove discrepancies, calibrate (rim weight) the data on known brand incidence. Selected suppliers now offer the ability to survey respondents whose behavior is known from scanned purchases, frequent shopper data matching, etc. creating a true single source approach. In subscription or retail businesses, you can also survey your own customers and match in transaction and clickstream data.
Problem: Random answering of attitudinal questions when beliefs are weakly held. Often attitude and usage studies ask questions that the respondent doesn’t really know how to answer, but guess what? They answer the question anyway! When you conduct consumer segmentation off of such data, the segments seem to make sense, but, sadly, via a test-retest reliability experiment, you might find that the same respondent only has a 50 percent chance of falling into the same segment the second time around. Such segmentation is practically useless for media targeting.
The fix: Anchor segmentation in digital and shopping behaviors. Survey people on fewer but meaningful statements and about media and purchase behaviors that are known. Create segments that are maximally different in purchasing behaviors and media habits as well as attitudes. That way, key segments can be targeted and personalized differently in digital media and will hold up over time. Finally, did you know that you can now listen to social media specifically for a segment of interest? The technology now exists to do profile matching to accomplish this.
Problem: Inaccurate recall of time spent on various media behaviors. In my experience, people underestimate the time spent watching TV and over-estimate the time spent on various digital behaviors, notably social media.
The fix: Go single source. Conduct survey research among people whose behaviors are metered or recorded in some way. An award-winning study I consulted on "Seven Shades of Mobile" did this; we surveyed people whose smartphone behavior was being metered.
The new news? In a digital age, think beyond the survey. Conduct surveys among those where clickstream behaviors, transaction and shopping behaviors, social media profiles or Facebook likes are measured.
- When surveys behave badly … and what to do about it – Joel Rubinson on Marketing Research Consulting
What do you see as the most common causes of inaccuracies in surveys? Will the use of social media and digital technologies enhance survey effectiveness?