Braintrust Query: Are Mystery Shops ‘Constructively Negative’?

Commentary
by Kerry Colligan, Integrated Marketing Manager, Second to None

A
study published in the November Journal
of Marketing
attempts to answer
two important questions applicable to mystery shopping programs:

1. Are
shoppers who expect to evaluate service delivery inclined toward “constructively
negative” evaluations regardless of the actual experience?

2. What
can be done to reduce the negative bias of survey respondents?

In
brief: 1) No. 2) Give them something to think about.

According
to the study, when cognitive loading precedes the customer experience
it reduces negative bias. We know that when customers expect to complete
a survey after visiting a store, they have a natural tendency to focus
on negative aspects of the experience in part because those aspects are
more easily identified. It’s easier to conclude “My coffee is cold” than
it is to conclude “The associate far exceeded the service standard.”

However,
when shoppers are asked to pay attention to specific details about an
experience beforehand, they are less likely to identify negative aspects
of the experience when they occur. Their focus is more positive.

What’s
more, the study found that “shoppers do not ‘fabricate’ negative evaluations
just to comply with their task. … [T]hey report such evaluations only
if they are able to gather supporting evidence during the shopping experience.”

Consider
the impact of cognitive loading on:

Shopper
training and preparation:
Well-trained
mystery shoppers should arrive on-site with a high degree of cognitive
load that results in a more balanced evaluation. Conversely, poorly trained
or unprepared mystery shoppers are more likely to over-report the negative
in an effort to fulfill their obligation to report. Thus, ‘constructively
negative’ evaluations should be more prevalent among poorly trained shoppers.

Customer
experience program goals:
Shoppers are more likely to identify negative
aspects of the experience if their brains aren’t busy processing shop scenarios
and service procedures. If your goal is improved training and employee appreciation
(i.e., incrementally positive outcomes), you’ll want shoppers loaded to identify
the subtle differences between four-star and five-star service.

Discussion
Questions: What can be done to reduce the bias of mystery shoppers (perceived
or actual)? How do the conclusions in the Journal of Marketing study compare to your own mystery shopper research experiences? What
do you think of the value of mystery shoppers overall?

BrainTrust

Discussion Questions

Poll

20 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Bob Phibbs
Bob Phibbs
14 years ago

We’ve used Nsite Mystery shopping service for thousands of shops. The results of a mystery shop are only snapshot, a moment in time.

The number one thing business owners tell me is, “I just need more customers.” Wrong; you need more customers to return. You can’t attract everyone in a two-mile radius to try your store, deliver lousy results, and expect to get “more bodies in the door.” You can burn through a whole neighborhood with bad word-of-mouth; and, without mystery shoppers, you’ll never know it.

And please, drop the idea that mystery shops are a way to “spy” on employees for compliance; that’s only what they’ll think if you don’t present it correctly. You certainly don’t need a mystery shop to justify your desire to fire someone.

I wrote extensively about this on my blog recently.

Kevin Graff
Kevin Graff
14 years ago

I’m not a clinical psychiatrist so I can’t comment on this mysterious ‘cognitive loading’ thing. However, the value of mystery shopper programs in my mind continues to decrease with each passing year as more and more retailers adopt online customer surveys. My clients report increased benefits including many more surveys completed, input from real customers, not fake ones, and the chance for ongoing dialogue with these customers.

Most retailers don’t conduct enough mystery shops each month to make them effective for evaluation purposes (however, a lot make the mistake of using them that way). At best, mystery shops give you a ‘snapshot’ that can be used for coaching purposes.

Sorry, but I think mystery shopping is on the way out the door within the next few years for most retailers.

Ian Percy
Ian Percy
14 years ago

There is no such thing as an “observer” as physicists found in the famous split screen studies with waves and particles. We always influence the observed so the goal of getting rid of bias is as futile as “Let’s leave personalities out of it.”

I’d like to see a whole new way for stores to evaluate the experience they offer customers. For example from an ‘energetic’ perspective. Does the display table at the entrance “add” or “deplete” energy? Same thing with the lighting. Paint color on the walls. Employee behavior. Quality of printing on the receipt. Size of changing room. Quality of product.

Looking at whether price adds or depletes energy is the most interesting question. Contrary to popular opinion drastic cuts in price don’t necessarily add energy to the buying experience and, I suggest, just as often deplete it. But it is weird that a Dollar Store can have more energy than a store on Rodeo Drive.

When someone comes away from the buying experience with more energy than they began with, they’ll be back for a fill-up.

Camille P. Schuster, Ph.D.
Camille P. Schuster, Ph.D.
14 years ago

As with anything requiring people for implementation, training is a key issue. As with any marketing research, determining validity and reliability of results is critical. One aspect that is related to both is how you set the frame for the mystery shoppers: what are they doing, what do they look for, what is the purpose, what will they be expected to do after visiting the store? Without addressing all of these issues, the results are not necessarily valid.

Marge Laney
Marge Laney
14 years ago

Unfortunately, mystery shopping, like almost everything else that has to do with the customer touch area of in-store experience is Retail 1.0. How can one moment in a store, once a month if that, give any kind of actionable insight or data? Technology platforms that give the customers access, the associates visibility and control, and management actionable BI is what is needed to effectively manage and accurately measure in-store performance. I agree with Kevin, mystery shopping is on its way out.

Doron Levy
Doron Levy
14 years ago

Every major client I have worked for uses mystery shops. It’s a fact of life in the retail world. I was always trained to use shop results as a unscientific gauge of what’s going on in the store. Depending on how well the shopper form criteria is worded, you can get a good picture of the level of service in your store. Data on items such as greetings, line wait times, overall image and friendliness are indispensable to store and district managers.

And as for bias, effective shops will utilize yes and no questions as opposed to asking for comments. This really takes the subjectivity out of it. Managers want to know in the simplest terms if customers are being greeted at the door and are being invited to return. Shops that require extensive comments and critique are not providing essential information to ops personnel.

Steve Montgomery
Steve Montgomery
14 years ago

“Mystery shops” can cover a wide range of activities from age verification to compliance checking for merchandising, etc. As used in the article, it applies only to customer service surrounding the purchase experience. I agree that it reflects only a moment in time but given enough moments in time, you can form a picture of what is occurring at your sites. Will it be replaced by the use of additional technology? Perhaps, but until it is, I believe it can perform a valuable service.

James Tenser
James Tenser
14 years ago

It seems reasonable to conclude that some degree of response bias is present among observers who are tasked with evaluating retail service encounters. Of course they will discover where things go wrong–isn’t that what we’re paying them for?

Let’s stipulate that mystery shoppers are trained to be hyper-observant. Of course they will detect more problems than an average shopper. Some are more objective than others, but in general they are less likely to behave as naive customers in the service environment. This is an unavoidable observer’s bias, that I agree, metaphorically resembles the Heisenberg principle in physics that states how the act of observing a particle influences its state of being.

Where very simple observations are gathered, such as the yes/no questions also mentioned above or audit facts like “is the display up?”, there is some hope of an objective read. Purchasers of mystery shop research have a responsibility to understand the limitations of its findings, and research vendors have an ethical obligation not to oversell its validity.

Ryan Mathews
Ryan Mathews
14 years ago

My vote goes to hiring ethnographic researchers rather than mystery shoppers. That way you get a sense of the collective flow of a store and not reportage on an individual transaction.

Ralph Jacobson
Ralph Jacobson
14 years ago

Here’s an anecdotal example: I know a neighbor who works as a mystery shopper. She always finds real-world issues that would be so easy for the retailer to correct, but leave such a bad taste in her mouth. Many issues she sees are deal breakers that make her choose another store to shop in the future.

Regardless of how many retailers increase their usage of surveys, mystery shoppers see the problem in real time and the retailer needs to see that. The truth hurts and the retailer should take action with what these people see.

Eliott Olson
Eliott Olson
14 years ago

Mystery shopping is a salve for poor hiring, training, leadership and effective supervision that might cover up the sore but it will not cure the illness.

Kathy Doering
Kathy Doering
14 years ago

This is such a great discussion. I am pleased to see it. Many times mystery shopping is overlooked when talking about market research.

I have been in this business for over a decade and have seen many changes. Mystery shopping will be changing in the years ahead as well. I agree that ethnographic research is going to become much more important to retailers down the road. But I see it more along the lines of complimenting mystery shopping, not replacing it.

The other aspect to mystery shopping that is very important to retailers is selling, up selling, cross selling, suggestive selling, etc. Mystery shopping is the best practice to uncover this. Retailers many times miss fantastic opportunities in this area and the MS scores always reflect it. Customer feedback surveys will not give you this information. At least not accurately.

Timing issues are another area of importance. I hate when I take a survey and they ask me how long I waited in line. I wish I could say that I have no idea. Guessing is the best I can do and research shows that is usually way off from actual time. The average customer doesn’t time their wait in line.

If anything, I see mystery shopping growing stronger in the years ahead. Thanks for the great posts!

Michael Tesler
Michael Tesler
14 years ago

I have worked in and with many, many stores both small and large and in most all cases, it has been “management on the floor’, team atmospheres. My approach my seem naive, particularly for the battle-scarred retail veteran that I am, but when management is constantly on the floor and always interacting in a positive way with well paid, highly trained and motivated staff, there are no mysteries, and management has total understanding of their customers and what the store needs to do to improve and adapt. This knowledge is gotten through first-hand communication with front-line staff and customers plus on-the-floor observation.

Nothing works better than “hands on management” and nothing works worse or destroys morale more than sneaking around to find out what is going on or depending on low paid, clueless outsiders to tell you what is going on in your own business.

Tom McGoldrick
Tom McGoldrick
14 years ago

The idea of cognitive loading to reduce negative bias in mystery shoppers is a very important tool for collecting reliable information. Mystery shops can be a valuable tool to determine if operational processes are being followed and to check for compliance. However, mystery shoppers are not customers and can have little to say about customer satisfaction.

Cognitive loading does reduce bias but it also moves a mystery shopper even farther away from a customer’s frame of mind. Cognitive loading helps train the mystery shopper to identify the aspects of service that have been predetermined as important. However, it does not tell you if you are measuring the correct aspects of service.

I do not intend to impugn the value of mystery shopping, but I do think it is important to distinguish it from customer satisfaction research. They are both valuable tools, but tools intended to answer different questions. Mystery shopping tells you if your operational plan is being well executed. Customer satisfaction research tells you if you have the correct plan.

Anne Bieler
Anne Bieler
14 years ago

Mystery shops do try to gather as many data points around the store experience, particularly useful for large, multi-chain organizations like Quick Serve restaurants, etc. Having a trained observers to see if the plan delivers at a number of sites is one way to monitor the business. Valuable, but not designed to be a way to get to deeper insights that identify great customer service.

Satisfaction surveys, social media posts, and lead consumer panels represent another valuable source of insight into consumer thinking–people who are emotional about products and services will contribute here. Ethnography is a great tool to understand consumer usage and thinking around the products and services, but requires good design and execution to find the valuable insights in the data.

John McNamara
John McNamara
14 years ago

Of course they’re biased but the results are still useful when comparing against last year and comparable stores. It’s up to the retailer to figure out what the shoppers should be looking at and set up a checklist appropriately.

Nick Samson
Nick Samson
14 years ago

I couldn’t agree more that EXISTING customers offer retailers the greatest opportunity for new revenue. However, mystery shoppers cannot tell you how to tap this opportunity and no matter how hard you try, it will never be more than a minuscule snapshot amongst thousands of transactions. Only your existing customers can inform you about key indicators such as loyalty, satisfaction, likelihood to defect (to a competitor), etc.

In the past 14 months, the world has changed and so have customers–so the ways in which you learn from them must also change. There are only 4 ways to influence your customers to generate more revenue and that has to be your new mantra: (1) customer conversion, (2) average purchase amount, (3) retention rate and (4) visit frequency.

This is where your research, marketing and operational efforts should be focused.

Ron Welty
Ron Welty
14 years ago

Mystery shopping research is a proven, fact-based form of research that forward-thinking retailers who “get it” when it comes to the customer experience utilize heavily. If it wasn’t, why would so many of the world’s leading brands utilize it? Many of our colleagues in the market research world simply don’t understand it, many are so-called “survey experts” hiding in their offices behind their beloved statistics and with possibly self-serving agendas, and as with anything that’s not understood, it’s easy to take pot-shots at it rather than gaining a better understanding and finding the true value of it.

Well-designed-and-executed mystery shopping programs return many times their investment, but they are NOT easy or cheap to do. They require complete commitment and “talking the talk” from top management; top-level decision-making and program responsibility; investing time to conduct intensive and intelligent due diligence to select a provider that meets a companies’ specific needs; and, not the least, having a commitment of hiring good people and investing in their success.

Unfortunately, we too often see retailers who fail on these key success requirements, so it follows that they will not have success with mystery shopping research. Our industry bears some guilt here as well: many people see it as a low barrier-to-entry business, invest so little that they have nothing to offer but a low price; they do not or cannot properly educate their potential clients on the appropriate usage of this research, cannot fulfill their commitments to clients, and are simply incapable of meeting these companies’ complex needs.

No single form of customer-centric research can make or break a retailer. In a perfect world, mystery shopping research will be teamed with other research (surveys, focus groups, social media, etc.) to gain a broader understanding of what’s happening at the customer “moments of truth,” so that better decisions can be made about whether those actions are as-desired or, if not, what to do about it. Too often, companies portray it a zero-sum game: “you can get 50 surveys/month for the same price as one mystery shop.” Well, would you like 50 apples or 1 orange? It’s just not an accurate comparison, but many of those who have a stake in selling surveys have attempted to portray it that way. There are numerous examples of companies that utilize both, as well as those who dropped mystery shopping for surveys, and then went back to mystery shopping when they discovered the differences.

I’m happy to report that, while we’ve taken our lumps like most everyone in this recession, the mystery shopping industry is still thriving and growing. We will continue to evolve in both our service enhancements and new innovations, but we’re here to stay. Retailers who view mystery shopping as a strategic investment in understanding and improving the customer experience will always out-perform those who do not.

Dawne Richards
Dawne Richards
14 years ago

Years ago, as a college student, I spent two years mystery shopping various retail outlets, primarily a fast-food chain. The information we were required to provide was as scientific as possible: We ordered very specific items, had a stopwatch to time the transaction, a thermometer to report the temperature of the food, and a form that was clear and specific as to the information requested. “Comments” were optional. There are several differences between then and now: 1) The money to do the shops was provided up front; 2) I was paid for my time (including travel time) and mileage; 3) I worked for a well-established research firm. Sadly, today’s mystery shoppers are often people looking for a way to get a free oil change. They are not researchers at all. In addition, my belief is that, if done properly, mystery shopping is an important component of retail service–I now have college-age daughters working in retail, and they behave not only because they have been raised to do so (I hope!), but also because they have been trained by their employers to believe “Treat every customer well, because any one of them could be a mystery shopper.” So I still see value in this, but not if it continues to be devalued by our unwillingness to pay the true cost of such a service.

Doug Pruden
Doug Pruden
14 years ago

I would rather have staff trained to treat every customer well because those customers are ultimately the ones paying their salaries. But if that is too macro a view and too long term for most employees, then I suppose treating every customer well because they might be a mystery shopper (and tied to some kind of short-term penalty or reward) can have a desirable impact on staff behavior.

Mystery Shopping really should be all about measuring compliance with a set of very carefully defined standards. Asking shoppers to think like a customer and allowing them to make subjective decisions and ratings of how they are being treated is a big mistake. One great use of Mystery Shopping could come in conjunction with other research and planning. Once more, in-depth customer experience research has identified the issues most critical to retaining and developing existing customers, and management has made the changes in process, policy, training, etc., that are most critical, then Mystery Shopping can dispassionately visit the stores and tell those back at headquarters whether their new direction has been enacted in the field (with some nice black and white standards for the Mystery Shoppers to measure).