NEC’s NeoFace technology – Source: NEC
Do the benefits of using facial recognition in retail outweigh the risks?
Facial recognition technology has many practical applications, including law enforcement, airport security and retail loss prevention to name a few. And while these use cases seem reasonable to most people, not everyone is enamored with facial recognition technology or how it’s being used.
Thus far there has been very little legislation regarding the use of facial recognition, but that’s changing. On May 14, the city of San Francisco passed the Stop Secret Surveillance Ordinance that bans city agencies including law enforcement from utilizing facial recognition technologies. The legislation doesn’t apply to businesses, but one has to wonder if this is only a matter of time.
In March, a bipartisan bill was introduced in the U.S. Senate to strengthen consumer protections by prohibiting companies that use facial recognition technology from collecting and resharing data for identifying or tracking consumers without their consent. Illinois made it illegal to collect biometric data without consent in 2008.
These legislative actions along with the heightened sensitivity to privacy resulting from the introduction of the General Data Protection Regulation in Europe may prove to be problematic for retailers, including Walmart and Walgreens, that are actively experimenting with facial recognition technologies.
Walmart has been testing facial recognition to improve customer service by trying to recognize if a shopper is “unhappy”. If that determination is made, then customer service can intervene and turn a bad customer experience into a good one — so the theory goes.
Walgreens is testing sensors that detect shoppers and cameras that scan their faces to estimate their gender and approximate age for delivering targeted messages on their experimental soft drink coolers outfitted with digital displays.
While many retailers experimenting with facial recognition insist they are not storing facial images or using them for any purpose except to help deliver a better shopping experience, many questions remain. How are facial images stored? Who has access to them? Are shoppers notified in advance? Can shoppers opt out?
Notwithstanding all the interesting ways facial recognition could be used by retailers and regardless of how well intended they may be, it’s still mostly uncharted territory, with serious potential pitfalls retailers should be wary of.
- San Francisco becomes the first US city to ban facial recognition by government agencies – The Verge
- Thousands Of Stores Will Soon Use Facial Recognition, And They Won’t Need Your Consent – BuzzFeed
- Will facial recognition tech make for happier customers at Walmart? – RetailWire
- Walgreens tests tech that sort of recognizes you in-store – RetailWire
Discussion Questions
DISCUSSION QUESTIONS: Are the benefits of facial recognition technology more trouble than they are worth for retailers? Do you think that legislation will be passed within the next five years regulating how businesses, including retailers, use facial recognition technologies?
Poll
BrainTrust
Dr. Stephen Needel
Managing Partner, Advanced Simulations
John Karolefski
Editor-in-Chief, CPGmatters
I’m hard pressed to come up with a use for facial recognition in retail that will overcome the creepiness factor or even pay out. How many people does Walmart think will be turned into happy shoppers? Does it matter – what are they going to do, not shop at Walmart any more? I’m betting the cost of losing some shoppers will never exceed the cost of the system.
Expect more and more legislation claiming your face is your property, not a retailer’s, and opt-out won’t be sufficient – you will have to actively opt in.
The use of facial recognition in retail stores is fraught with dangers. There are far too many things that can and will go wrong. This is another of the ideas that falls into the category of “just because you can doesn’t mean you should.”
I agree with Stephen’s statement that if done, it should on the basis of opt in rather than forcing people to opt out.
I’m sure legislator panic driven by sinister perceptions and lack of factual knowledge will drive more laws coming onto the books. However, that’s how most laws get passed, right?! With GDPR impacting the whole world more and more, I believe the challenges of compliance may outweigh the potential benefits for the moment. I would think retailers will find a permissible way to have shoppers opt in without too much inconvenience very soon, though. I do believe data analytics from this tech is valuable.
The risk-reward equation doesn’t pan out. Until we uncover a “killer app” for this technology that offers big benefits, retailers should stay on the sidelines (but continue to monitor closely).
While facial recognition may create creative marketing and sales opportunities, the risk of alienating customers may outweigh the benefits. Just because you can, doesn’t mean you should.
Regarding regulation, I suspect that we will see legislation that will require retailers to be fully transparent and inform customers that they are being filmed and offer them a way to opt in or opt out.
With pending regulation a big unknown today, it is risky for retailers to invest in this technology that can and will change in an instant. I believe we will continue to see tight regulatory limits on privacy and investments in this type of technology would be a highly speculative.
The problem with facial recognition isn’t so much that a camera might identify how thirsty I am walking by a cooler. Have at it. The problem is that the technology can be refined via retail but will not stay contained within retail. When retailers adopt these types of innovations, the assumption is that they are cooking them up themselves. The reality is that they are partnering with third parties that have a vested interest in expanding the technology across many industries, and for many purposes. Amazon is going after facial recognition, and not because it dreams of a day when my facial expressions will determine my level of interest in an algorithmic recommendation. Amazon is courting (and has already won) government contracts. The ability to identify a single face in a crowd. That type of thing. Focusing on the trees ignores the forest.
First, let’s understand the distinction between facial recognition and detection algorithms. Recognition is matching a specific individual to an existing database and uniquely identifying an individual. Detection is specific enough to identify a human face – period. This is often referred to as anonymous video analytics, the key word being anonymous.
In today’s volatile retail landscape, marketers and merchants are frantically searching for methods to make brick-and-mortar stores more relevant. Focusing on experience, providing shoppers with localized inventory, personalized experiences and a reason to shop a physical store (should!) drive retailers and brands. As a pioneer in the use of anonymous video analytics at retail, I have learned and experienced the good, the bad and the ugly with video analytics. All too often its use is directly related to monetization masked by claims of enhancing the shopping experience.
Video analytics can be a powerful tool that, when used appropriately, can bring tremendous value to the shopper, the retailer, and brand. And yes, it can be worth the trouble if, and only if, there is a meaningful objective that truly helps all of the stakeholders — especially the shopper.
Location beacons already know if I am in the vicinity and then what aisle I am in and then my dwell time in any given location. Now they want to know what kind of mood I am in? I’m told I continually manifest RBF, which is short for I always frown. And now a computer is going to try to cheer me up. I don’t think so.
It takes a special kind of guy to admit he has RBF – ha!
LOL! A momentary respite from RBF.
Stop 10 customers in a store, tell them that their face is being analyzed, and watch them walk right out the door.
I agree that sentiment and theoretical marketing use cases may not outweigh the risk, and transparency is critical. Filter to an initial set of scenarios that aren’t considered “creepy.” Start slow, with safety/security and ease of transacting in mind (rather than leaping to how to cross-sell/upsell) and the technology may be more readily welcomed.
Consider:
Let’s say grocers polled their shoppers and asked if they would mind if facial recognition was used to improve customer service. My hunch is that 90 percent or more of them would say NO WAY. The customer is always right. Or am I wrong?
Although most of us agree that marketers and retailers are taking responsible steps, there’s no doubt facial recognition can be abused. Just have a look at how the Chinese government is monitoring the activities of their citizens.
The time is now to build safeguards into facial recognition systems. An analogy would be the nuclear power industry. While mostly safe, we now know how great a potential for disaster is inherent in the technology. Scientists from the onset worked to build in safety features and the government mandated certain assurances.
It’s important to get out in front of the potential problems because a leak of facial recognition data (to sinister parties) could be devastating. Technology like this can do great good, but we need to be smart about it.
As a consumer, this technology is terrifying. If I’m unhappy in a Walmart, there’s a good chance that my mood has nothing to do with the shopping experience, and the last thing I would want is a computer trying to read my emotions.
However the creepiest part of this technology is the consent element. How will shoppers know that retailers are using facial recognition technology? Where’s the option to opt out?
I have no doubt that, if used responsibly, video technology could provide a better shopping experience. But I don’t think consumers trust corporations to use technology responsibly, nor should they.
Facial recognition is quickly emerging right along with voice commerce as one of the latest commerce innovations. However the use cases outlined in retail, especially by Walmart, are a little concerning. Information is flowing so freely in these relentlessly connected digital days, that we tend to lose touch with what privacy settings we have set to keep us comfortable. It’s an evolutionary cycle, and we are all getting used to this new world.
While there is a significant use case for airports or other high-security settings for facial recognition, is detecting whether or not a shopper is “happy,” “pleased” or content in a big box shopping setting going to impact the customer experience, or drive increased sales?
Just let customers shop as they always have, without the unnecessary thought that their facial expressions are being watched.
Seems there are some misconceptions about “facial recognition,” or at least, the many benefits the broader AI concept of Computer Vision for Brick and Mortar Retail. The example provided in this article about detecting customer “moods” is not “facial recognition,” rather, “mood detection” of anonymous people in a store at any given time.
Unless the retailer has built up individual’s “faces” to “match” up against, say, like potentially a mug shot from previous shoplifting events (facial recognition) — measuring “moods” or “gestures” anonymously are in fact very promising areas to continue to explore.
Tell me this. If a store/customer service manager could be provided a simple real-time prompt of nothing more than “a sad female, 39 yrs old, at entrance B” — think that could provide retailers with competitive differentiation without piercing customer privacy?
Suggest being a bit cautious here about potentially conflating facial recognition with the broader streaming video analytics. Facial recognition by itself is a relatively small value-add here in the West in comparison to the much broader topic of Computer Vision and Streaming Video Analytics — which will be and (perhaps the most) critical component of retail stores of the future, considering the many more applications from a merchandising compliance, out-of-stock, friction-less checkout use cases that all depend on computer vision and streaming analytics.
Just did a talk at Data Works Summit in Washington DC this week on the many use cases and business impact Computer Vision IS already addressing for Retail Leaders. Here’s a link to the presentation if anyone’s interested.
Good points Brent. However, I’m curious about the example you provided — exactly what would store personnel do if they knew there was “a sad female 39 yrs old, at entrance B”? The shopper may appear “sad” (assuming the algorithm accurately assessed the sentiment), but there’s no way to know why this person is sad — it may have nothing to do with the store experience.
In this one use case of CV, the point is to identify an opportunity and provide some intelligence to “engage” a customer, rather than to ignore, or stand behind a CS desk, or even provide front door “greeters” that simply provide canned and meaningless “welcome” chants at the front door (as is the brick and mortar experience today).
Facial recognition technology is fraught with risks. Sharing the data would be absolutely a no-no, but even using it to recognize shopper characteristics will make shoppers nervous. They will not know what the data is being used for and trying to explain it is a problem.
People are getting more and more concerned about the use of their data, especially facial recognition on their phones, so this will become even more of an issue for shoppers over time. Just the fact that legislation is being considered and passed tells that citizens are already concerned about this technology.
As with any technology, the customer has to see the value proposition. Despite all the noise in the media focusing on the fears around government usage, we have seen a steady stream of huge consumer-focused brands carefully deploying and testing it, getting positive customer feedback, and then rolling it out wide (Citi, Delta, JetBlue, Amazon, just to name a few). The latest is cruise line Royal Caribbean: “The typical wait time to board is 10 minutes with a mobile boarding pass; less if the passenger opts into facial recognition by uploading a “security selfie.” Before those additions, he says the typical wait time was around 90 minutes.
It depends, of course, on what it’s being used for: targeting ads or sending assistance to the unhappy seems harmless enough (though I don’t doubt someone, somewhere will be manage to be upset by it). But the more the technology moves from identifying “types” of people to trying to ID specific people, the more it will run into problems. We haven’t seen a lot of practical demonstrations yet, but I’m dubious that it can work as well in real life as it can in a development facility somewhere … and the shortfalls will be quickly and amply publicized.
Facial recognition is a solution looking for a problem at retail. Just because we have a technology doesn’t mean that it is an appropriate tool for use at retail. Also, this article clearly doesn’t understand how facial recognition works and the requirements for it. Close proximity is a requirement, and familiar alignment of the face needs to be a key co-factor in facial recognition. Being able to do this with customers who are not purposely allowing this is nearly impossible with today’s facial recognition technology (imagine doing this with your phone). So far, the benefits are minimal and with this the risks are even fewer.
Some form of “facial recognition” is already used in retail to prevent retail loss, as mentioned in the article. The question is, can this technology be extended to improving customer experience in the store. I ‘m hard pressed to understand how a store manager walking to a customer, who is already, according to the algorithm, “sad,” address the customer. “It appears you are sad, how can I can help you?” How creepy is that?
Gauging the shopper’s mood is definitely the next frontier in retail, but I do not believe reading moods using facial (or mood) recognition algorithm is the right way to go about it.
Facial recognition runs the risk of alienating more shoppers than it wins. This is especially true in retail — how many people want a store associate to come up to them to try to cheer them up, with little to no context about what is going on and why they’re having that conversation to begin with? The sales won with facial recognition won’t outweigh the risks, at least not for a while.
There is a real confusion in this article between facial recognition and facial profiling. What is legislated against is recognition. I absolutely agree my face, for what it’s worth, is my property. I cannot be traded for others benefit without my permission.
However what is then described is profiling. We do this by measuring a face dimension set and comparing it to a known set of metrics. From this we are able to capture gender, an age estimate, engagement, and emotional state. We can then make decisions based on the observations. For example, are we marketing to the right audience? Does the shop window need a refresh? Changing the content on a digital medium based on the gender and or age of the audience present, or imagine this — turning off the screen because nobody is there!
I support profiling as a measurement system. I do not support obtrusive recognition and resale of my face!