Observing Consumers Beats Listening to Them
By
Tom Ryan
Nate
Bolt, co-founder and chief executive of user research firm Bolt Peters,
argues that while tech entrepreneurs are encouraged to listen to potential
customers, observing is a much better tool to see whether a new product
will succeed in the market.
“The
main problem with opinions is self-reporting bias,” wrote Mr. Bolt in a
column in the venture-capital blog, VentureBeat. “Opinions
are often inconsistent with behaviors or other attitudes, especially when
discussing hypotheticals.”
As
an example, he noted that Clippy, Microsoft’s animated paperclip helper,
came about after the company’s researchers found that when asked, users
roundly agreed that they wanted help working with their documents.
“But
once people started actually using it in the real world, they hated it
— it might be one of the most hated features in the history of computing,”
said Mr. Bolt.
Mr.
Bolt offered three ways tech entrepreneurs can gauge whether people would
use a service:
- Test ideas early by watching behavior: He
suggests having eight people interact with a prototype or even wireframes
or design makeups. In the tech world, a number of websites (Chalkmark,
Pidoco, Balsamiq, etc.) allow companies to easily test prototypes. Wrote
Mr. Bolt, “You can still ask all your needy questions about what
they think after the session — just don’t take those too seriously.” - Get all stakeholders to watch the research: Technical
and business constraints obscure the basic question of whether the interface
is any good. - Use unorthodox methods: Mr.
Bolt noted that Apple claims it doesn’t conduct user research but releasing
products in generations provides the company with loads of reviews, task-specific
complaints, crash reports, customer support issues, and Genius Bar feedback. “It’s
audacious, large-scale behavioral research,” Mr. Bolt wrote. Similarly,
long-beta testing periods have helped launch services such as Gmail.
Mr.
Bolt concluded, “Do whatever you need to do to understand how people
use your product. If it’s a device meant to be used in cars, watch people
use it cars; if it’s a video game, avoid sterile lab environments. Just
don’t ask perfunctory, cookie-cutter survey questions to your potential
customers, and expect that to ensure your product’s usefulness.”
Discussion
Questions: What are the merits of observation versus listening in testing
new products? What are some ways researchers can incorporate observational
research alongside listening techniques?
We’ve been making this point for years, which is why our VR research has been so popular. Yes, it’s great to listen to your audience, but you should also take what they say with a grain of salt–the social psychology literature is replete with reasons why people would say positive things about your product or service.
Note that this is applicable to any industry, not just tech. Highly appropriate to the CPG business, for example.
One of the issues with listening is what you hear depends on the wording of question and the tone, delivery, etc, of the researcher. The same could be true for observing–the environment, the mere fact someone knows that they are being observed, etc, can influence behavior.
That being said, I admit to being a fan of real world observation. We did this first in a small store format in 1990 and while we were likely not the best at interpreting all the behaviors we saw, we did learn a lot and made significant changes in the layout and merchandising of the store.
Why not do both? Frequently listening to existing customers can uncover opportunities for new products or for product improvements. Depending on the cost of going to market, the most requested items can be incorporated directly into the product or service or can be tested by groups of users, as suggested by Mr. Bolt.
Observing people is more effective than listening to them in regard to what goes on in the marketplace. Just ask Elin Woods.
It sounds like a case of “observe first, listen second”–and I agree. I’ve sat behind the glass at focus group sessions, and even the most well-prepared session can go off the rails at the drop of a hat, and you’ll never get to what you’re looking for. But the Apple example underscores the point–throw it out there (set expectations that it’s early generation), and then watch what they do with it, and then listen to their feedback.
This is exactly why I am amazed when I hear retailers say, “We’re not doing X (X=some consumer-facing technology investment) because we don’t see the ROI.” Well, you won’t, unless you get it out there and see what consumers do with it.
Retail in general, and grocery retail in particular, have a unique opportunity to “observe” customers: retailers and their vendors are able to watch what customers buy. More than usage patterns, stated preferences, or anything else, actual purchases is the strongest indicator of what your shoppers care about.
When it comes to determining what people will do, don’t listen to the words. Listen to the actions.
If you ever took a course or listened to a program on communication, you will remember the widely quoted research on communication. It talked about the fact that the message is not in the words you use or the tone of voice but over 55% of the message is conveyed by the gestures (actions) of the speaker.
Actions speak louder and more truthfully than words.
Not every company is Apple and can afford to throw it out there and see how it works. It’s not about one size fits all.
Research must be formed on the basis of what will bring the necessary information. Even observation can be misguided if the form of observation influences the results or doesn’t capture the context of the customer experience.
Professionally trained researchers understand the dynamics of what is real and what is not. Combining tools of observation (and there are many) along with interviewing techniques (everything from structured to unstructured to in-depth) can provide answers and insights that guide successful new product launches.
I understand that it’s “newsworthy” to start off by knocking a well respected approach to research. However, “newsworthy” is not necessarily accurate or even helpful. There are many techniques and methodologies in the research world that can be part of a well designed program for new product development or an evaluation of existing products seeking opportunities for enhancement.
If you think that any doctor can perform your operation or any X-ray technician can interpret the findings, you probably expect the same with research. But I suspect you ask around for the best doctor and have a specialist help with X-ray interpretation. Doesn’t it follow that you would follow the same process when it comes to backing your new product launch with research?
Since we work in the technology field, we have had to use user observation and can attest to its importance in the research mix. For technology, we are often offering use cases or user interfaces not seen by consumers before, so focus groups or questions often cannot get the feedback needed. On the other hand giving items or models or mockups to consumers and observing their reaction is very useful.
This also extends to more basic ethnographic work where one can observe the “pain points” and gaps that we want technology to fix or bridge. Asking consumers about things they have not experienced often leads to useless input. However, as in all things balance is what is important as one has to scale observation with some quantitative work to see.
There’s a reason that people read the Daily Racing Form (which has been published since 1894!) when they bet on horses: past behavior is a great and typically the best predictor of future behavior.
Observing behavior can take many forms but in our world there is an abundance of behavioral data (transactional, online, etc.) that leads to the information that drives valuable insights. Of course this applies to services like retail and telecom as much as it applies to products like consumer electronics. The challenge is always to make the data usable.
There are times where observing behavior is impossible without actually putting something in (the real) market. Loyalty programs are prime examples and for this reason, we always recommend piloting live tests (even through loyalty promotions or scaled down programs) versus simply relying on conventional research.
The world, and especially marketing, needs more fact-based evidence.
From the retail end of it, I like to see how the customer interacts with the product on the shelf. Packaging and labeling are so important these days that vendors lose sight of how the product sits on the shelf. Does it display well? Does it scream out to be picked up by someone passing by? How does it perform in a promo situation such as a dump bin or endcap?
Vendors should work with their retail partners to get new products into test labs and pilot stores to see how they do in real world conditions (this note goes out to the good people of L’Oreal and their Studio line of products!).
I am truly amazed at the headline of this article. ‘Listening’ vs. ‘Observation’ is a totally false choice…one does not have to choose between one or the other and it would be ridiculous to do so since both activities have value. Generally speaking, ‘listening’ helps you understand ATTITUDES while ‘observing’ helps you understand BEHAVIOR…and understanding BOTH is important!
For example, you could be in a focus group, ask how many people are interested in buying a product being investigated, and count the 7 out of 10 hands raised. Should you pay attention to 7 out of 10 who indicate they’re interested in buying the product? Of course not! You need to observe behavior to gain a meaningful assessment of how many customers you will have. What you DO care about, however, is learning WHY the 7 say they would be interested…AND learning WHY the 3 others AREN’T interested. There is enormous value in understanding how the consumer thinks about products or services!
To say that ‘observation’ beats ‘listening’ is simply naive. There are valid reasons to do both.
As the great Yogi Berra is quoted; “Sometimes you can observe things just by watching.” The focus group and the virtual store environment are (to me) always suspect as the participants tend to want to please. As has been often repeated in this blog, what consumers say and what they do are not necessarily in sync.
In my previous life as a Division Manager, I spent a lot of time in the stores, just watching customers. What they put in their carts, how much they put in their carts, how they visited the shelf. Often I would engage them and offer to help make a decision. (Like I was being a helpful store employee). I did the same thing when I was a Store Manager. It builds an intuitive base that often challenges the academic research.
It is always more reliable to observe what people do than to ask them. The single greatest flaw in “ask” research, is the presumption that the researcher knows what the question should be. This means that, to a greater or lesser extent, ask research always provides a look in the mirror for the researcher.
Historical researchers have noted this problem, (self-referential results,) and describe it as the researcher looking into a deep well to perceive the past. They see a reflecting pool at the bottom of the well, and often inadvertently describe their own image as the past.
Of course, that blurs the line between ask and observe, and points up our self-referential perception of the world. Maybe ask research simply exacerbates the problem. In any event, the soul of science is observation.
All points well taken. When it comes to understanding consumers, observational research is a very powerful tool. There is a growing acceptance of ethnography–qualified observers viewing consumers where they use products–home, office, leisure. There are a wide range of usage tests for products and packaging that are used by researchers to determine if the product performs to expectations. Often it’s a great idea, but….!
In all cases, we need to consider the experience, not only the consumer perception of it. Watching consumers work with new packages and products helps us understand the features and benefits of true interest that will be of value in considering repeat purchases. Watching shoppers in the aisle–how they scan, where they go, what they pick up–all have information that is useful for retailers.
I’m not exactly sure what he means by “self-reporting bias,” but I think the largest effect here is that consumers just don’t know how they will really behave until they get a chance to really behave. Sure, there is some bias in people saying the answer they think the researchers want to here (the classic problem with public health researchers asking people about their risky behaviors). But I think that is secondary. Most consumers have good intentions–they want to report how they will behave, but they just don’t know.
Bottom line: real-world testing is the only way to truly understand what consumers will buy and use.
Every consumer will tell you they want variety in the stores. Invariably they don’t buy variety. In a typical supermarket, 85% of the SKUs move less than a case per month per store. It’s crazy to only listen to consumers, and not try things and see how they actually react. Keep up the experimentation.
Observing consumers is far more useful than listening to consumers for many reasons. For any number of human reasons people often do not discuss accurately what drives their own buying decisions. Truth is, many consumers do not even know what drives his or her purchasing decisions. Stand in an aisle at the supermarket to watch and observe. Say nothing. Ask nothing. You will learn a lot.
This discussion calls to mind an interesting company, Affinova (www.affinova.com), that seeks to straddle the “Listen vs. Observe” divide when it comes to new product development. They take various features and design aspects of a new product or new packaging, and then their system uses evolutionary algorithms to mix and match the different features to create hybrid designs, which consumer panels then rate against one another.
It’s not quite as pure a test as actually putting the hybrid products on a shelf next to one another and seeing what shoppers buy, but because it’s presenting each product as a whole, and it’s asking consumers to rate them against each other, it is far more concrete than a focus group discussion would be.
Nothing substitutes for real life experience. A focus group cannot capture the essence of a harried consumer rushing through the store anxious to be on their way. Watching consumers in the actual environment will give an accurate portrayal on how consumers interact with products. They say one thing and do something else
It’s very similar to sustainability issues. Consumer say they support all the green initiatives but when it comes to the pocketbook and making a choice most consumers opt for the more economical option, green or not.