Why are retailers struggling with Big Data?

Sep 12, 2016

Brian Kilcourse

Through a special arrangement, what follows is a summary of an article from Retail Paradox, RSR Research’s weekly analysis on emerging issues facing retailers, presented here for discussion.

When RSR recently asked retailers about the status of capabilities related to capturing and analyzing Big Data, we found that no capability was reported to have been “implemented” by up to 50 percent of retailers and less than a quarter claimed to be “satisfied” with what they had implemented.

The industry has a “half empty or half full” problem. The optimistic view is we’re in the very early days when it comes to truly leveraging non-transactional data. The pessimistic view is that retailers are just too slow to change.

Whether slow or fast, retail remains a fundamentally reactive business. While heavily depending on the quality of the data (both internal and external), Big Data promises to virtually eliminate the lag time to reaction by predicting events before they occur. Beyond internal transactional data, it can take in consumer path-to-purchase and social data, as well as exogenous data like weather, competitive, consumer psychographic profiles, calendared events (such as concerts or sporting events), and manufacturer and supply chain data.

So will Big Data cause retailers to change their model? In a way, the opportunity that Big Data represents might be distracting retailers from the real question: “What value are you trying to deliver?”

Retailers need to answer these questions:

  • What level of intimacy is needed to be relevant? Some needs are highly predictable, others much less so. The granularity of customer-specific data needed to make a highly relevant offer is closely related to the specificity of the need (for example, offering basic commodities doesn’t require much customer specificity);
  • What level of relevance is needed to be relevant? This is closely related to the level of customer “intimacy” required, but relates to the granularity of the offer itself. Extreme localization isn’t necessary for every brand;
  • How much data do you really need? How much analytical sophistication do you need to apply what you know to be relevant to customers?

Getting an answer to “What value are you trying to deliver?” has to come first and you don’t need technology to answer it. But you may need technology to deliver it and do need technology to tell you whether you are getting the job done in a way that will satisfy customers and generate profits for you. That’s when your version of Big Data becomes important.

DISCUSSION QUESTIONS: Are retailers failing to leverage Big Data due to caution around technology investments or something more? How does industry thinking have to change to drive more relevancy with consumers?

"What retailers really lack are the systems and technology required to mine disparate data and turn it into actionable intelligence."
"It's simple: retailers are obsessed with one and only one thing — operational efficiency."
"Data and analytics have been around for years. Retailers have under-invested in technology and it is catching up with them in a big way."

Join the Discussion!

26 Comments on "Why are retailers struggling with Big Data?"

Notify of

Sort by:   newest | oldest | most voted
Frank Riso

I do not think the issue is their technology investments. Lets face it, there is a lot of data out there and there are not many solutions that can take the data and provide the retailer with the necessary direction to take. We know a lot of information, but what we do with it is the issue. We store it, we print it out but we should have software that tells us what we should do and what we should not be doing to meet the expectations of our customers. The industry needs to spend more time knowing what the data means for it to be more relevant.

Doug Garnett

Good points, Frank — especially taking time to think about what might be relevant.

I think we also need to be open to the possibility that Big Data’s value is incremental and relatively small. I’ve worked extensively with big data analyses for quite a few years and find they tend to identify interesting things of low impact.

At this point, articles like this are based on the assumption there’s considerable benefit to be had. But is there? Even Target’s mythological pregnancy detector … interesting. But did it matter?

Tom Brown
1 year 8 days ago

Software isn’t the answer. You need a top notch data analyst to make predictions. Data only shows what happened in the past, and while it does give indications about what the future may hold, the future is very difficult to predict.

Peter Fader

It’s simple: retailers are obsessed with one and only one thing — operational efficiency. Big Data can be somewhat helpful in this regard but the real payoff from it, as noted in Brian’s introductory comments, is in different kinds of personalization/customization. Retailers are notoriously bad at doing this (because these activities go against the grain of operational efficiency).

Until retailers are willing/able to change their core strategies in a fundamental manner, they will never reap the rewards of Big Data.

Dr. Stephen Needel

The assumption in the article is that retailers need to be using Big Data — I’m not sure that assumption is true. Retailers need to be asking questions, and the answers to some of those questions may be in Big Data or they may be in small parts of the data. Retailers may be struggling because they think you have to analyze all of the data you have — that’s not true.

Chris Petersen, PhD.

I think that term Big Data can be very misleading. Retailers have been drowning in a mountain of data for years. Retailers do not lack data, they need intelligence.

What retailers really lack are the systems and technology required to mine disparate data and turn it into actionable intelligence. If a customer makes a purchase in store, that’s on the POS system. If they use their loyalty card, that’s on a separate system and if they use their smartphone in-store to order something else online, that’s most likely stored somewhere else. None of this comes together in an integrated system to enable a total view of the customer and their “lifetime value.”

The single biggest reason retailers are failing to leverage Big Data is that legacy systems were built to manage individual sales transactions. Today’s critical questions revolve around the total customer relationship. Until retailers are able to build and analyze CRM data, they will never be able to answer questions about “intimacy” and “relevancy” across the many channels that omnichannel consumers use today.

Sterling Hawkins

Disparate data is a huge issue in being able to leverage big data, mostly due to the lack of a cohesive data strategy and the inflexibility of the POS. Technology tends to be stacked incrementally vs. integrated to get something into market at the lowest possible cost. The tradeoff is that over time, it becomes unwieldy and nearly impossible to manage. Many retailers are in need of an overhaul or a reset to really be able to create actionable intelligence — especially with the new, more flexible vision of the POS we’re seeing come to market.

Kim Garretson

VentureBeat.com last August detailed the number of new companies launching around Big Data (more than 800), and the story included this: “Demonstrating value is also costly. It turns out that marketers are spending well over a third of their budgets (on average) on analytics. This in spite of the report finding that levels of confidence in analytics’ ability to generate insight are mediocre, at best.”

I think there is another reason why retailers are failing to leverage Big Data. If you boil it down, the data is backward-looking, based on what happened in the past. So using that data to predict the future and adjust operations to these predictions is, in effect, making guesses on the future. That’s why the use of first-party data where each individual consumer gives retailers their consent and criteria to market to them in the future is winning every time against Big Data guessing.

Lyle Bunn (Ph.D. Hon)

Data is a by-product of everything historic and knowing history allows us to be predictive. Data is also an asset which, if not applied or leveraged, is useless — lying around neglected like so much junk in a yard. Data leverage takes a giant step forward when it is somebody’s job to interpret what it says. As this falls to the Customer Experience (CX) group within marketing or the merchant, the dots can get connected to inform stocking, promotions and other key functions. In-store digital signage, for example, with its nimble messaging capability leverages data insights and even real-time data from applications (e.g., inventory, pricing, point-of-sale, weather, etc.) for product promotion.

Dick Seesel

I’ve long noticed a conflict between the work done by line and staff managers. In the case of retailers, the merchants and store management would constitute “line” functions; areas such as HR, IT, marketing and finance tend to be viewed as “staff” and therefore lower on the organizational totem pole. This cultural disconnect isn’t new, but the problems it can cause are now underlined by the massive growth in available data.

There needs to be a stronger commitment to data science, not just IT, among retailers. Colleges (such as Marquette here in Milwaukee) are developing new data science majors — separate from both computer science and math — because of the growing demand for this skill set. But the key to effective data science for retailers is to make it actionable, not to provide another reason for “analysis paralysis.”

Adrian Weidmann

It would be simple if it was just reticence to invest in technology. While that is certainly in play, I believe it’s the lack of perspective to interpret what the data is saying. In many cases the data clearly suggests that the marketing or merchandising initiative being measured simply doesn’t work. Many marketers would like their creativity and artful experiments to remain cloaked in subjective ambiguity. Remember Got Milk!? A great award-winning campaign that did little to increase the consumption and hence sales of milk. Data is just numbers unless you can interpret and decipher those numbers into useful insights. More importantly you need the courage, fortitude and willingness to respond to what the numbers are telling you — no sacred cows! Brand marketers and merchants want and need accountability for their budgets. Retailers will no longer be able to drop those investments — coop funds, MDF, slotting fees, etc., to the bottom line. Accountability and transparency are required to meet the expectations of today’s shopper.

Roger Saunders

Retailers, too often, begin their work with a snapshot in time as they are looking at data. They’ll find greater benefit in making use of longitudinal views.

The consumer is an intelligent being. Consumers make purchase decisions based on both emotional and rational states. Numerous social and economic factors, media forms, competitive opportunities and life events will influence those decisions. By listening to and having SMART data, not merely mounds of data, from a longitudinal analysis of their shoppers — what, why, when and how they are doing within their stores and at competitors’ stores — leads to faster, better decisions.

Beyond longitudinal insights, retailers have to be willing to evaluate whether they are getting the results from current data sources. If they are not, they have to break from defending those data sources just as they would if a merchandising line, price point, or store plan-o-gram weren’t delivering the results they needed.

Stay focused on the consumer — what and why she is shopping your store and what and why she is shopping at competitors’ stores.

Tom Redd

Big Data is a marketing fluff term for granular. Retail has finally gotten to the point of granular data management. The priority that is paying off for many retailers with granular data management is analytics. Simple, live data-sourced analytical reporting that leverages the intelligence of today’s supporting technologies — from optimization to advanced forecasting. This way retailers at all levels can see a common, single version of the truth and some ideas on how to act on it.

Retailers that do not take advantage of this initial step will end up with issues on inventory, pricing and workforce resourcing. Data and knowledge is the element retailers need as they grow. The time is now to leverage these technologies — but in a manner that does not mean a huge investment with years to wait for returns.

Ross Ely

It’s interesting that the article focuses on “non-transactional data” such as weather, consumer psychographic profiles and events such as concerts. I agree with retailers’ perceptions that it can be difficult and expensive to garner meaningful insights from disparate data such as these.

Retailers should first focus on understanding insights from their transactional data. Retailer POS systems deliver tremendous amounts of data which can be analyzed to understand shopping patterns and preferences. There are many easy-to-use and low-cost tools available for transactional data analytics, and retailers should master this dataset before they try to analyze non-transactional data.

Ralph Jacobson

There are several challenges for retailers to overcome, which leads to the current situation found in the research. First, there is structured and unstructured data to capture. This comes from both internal (call center, customer reviews, etc.) and external (social, local events, news, weather, etc.) sources. The challenge is not only to identify, because 80 percent of data is literally invisible to the vast majority of retailers’ systems, but also to capture and effectively analyze. So, to view this “dark data” you need the right tools (which are available in the marketplace today, by the way). Additionally, you need to get key lines of business (marketing, ops, merchandising, etc.) to agree on what your objectives are. This is the first piece that many retail organizations struggle to define and use to guide strategic execution. Keeping the consumer in focus and understanding which specific shopper persona you are targeting will help clarify your objectives.

Peter Charness

“Retailers” is a pretty broad group. If you divide the world into retailers who have frequent interactions with their customers (grocers, convenience, drug store) vs. infrequent transactions (specialty apparel, etc.), then the types of data available are quite different and “best practices” haven’t really emerged. I think the challenge may well be that for most retailers gaining benefit out of Big Data is a unique and personal practice, so the learning curve is steep all-around.

Herb Sorensen
The Big Data retailers are managing is not the Big Data everyone is thinking about. The bottom line reality is that perception of who retailers are and what they do is grossly distorted. For self-service retail — and most is at least partially self-service — RETAILERS ARE MERCHANT WAREHOUSEMEN who rely on UNPAID STOCKPICKERS, aka shoppers. As has been called out by others here, they manage operations and the supply chain, NOT the shoppers. There were two major forces that drove Tesco UP the global retail rankings in the ’90s. One was changes in UK realty laws giving them more control over their RE future and the other was dunnhumby, which managed the kind of Big Data this discussion is presumably about. Notice that TESCO did not manage that Big Data but did what big retailers do — managed the contribution of dunnhumby just as they did other major suppliers. Wake up! (Many dying brick-and-mortar retailers have not.) Why would a brick-and-mortar retailer manage the shopper and their experience? They know little or nothing about the efficiency of those shoppers. It’s like when George told his buddy Harry that the two problems afflicting America are ignorance and apathy and Harry… Read more »
Lee Kent

While I understand that retailers might tend to focus more on using Big Data to achieve customer relevancy, aren’t we missing a possibly more important role that Big Data plays? That of helping retailers to become truly digital.

In these times, retailers are no longer telling customers when, how and where to shop or fulfill orders. No, customers are telling retailers and retailers who don’t understand this and don’t have a way of keeping up with the changing attitudes, wants and needs of their shoppers will be left behind. Not only because their e-commerce platforms aren’t adequately using Big Data but also because the real-time data needed to drive business decisions is missing or lacking. With no digital strategy in place, these retailers are missing opportunities to transform business functions required to stay ahead of their customers.

Ken Cassar

Brick-and-mortar retail (where 90 percent of sales still occur) had always been challenged by the fact that with all of the customer data available, there weren’t good options to use that data while customers were in the purchasing moment. Personalized direct mailers were the only targeted tactic available to micro-target shoppers before they shopped. Coupons on the back of receipts were too late to impact today’s shopping trip, so it didn’t make a lot of sense to make huge investments in being a world-class Big Data enterprise.

With the near ubiquity of smartphones, though, it finally makes sense for retailers to make this a strategic priority. The half of brick-and-mortar retailers that haven’t mastered Big Data can be forgiven for not anticipating ubiquitous mobile devices, but we are now at a juncture where there is no excuse: The data exists (if you’ve got a loyalty program), the technology exists and the medium exists. Retailers ignore this at their own peril.

Patricia Vekich Waldron
Patricia Vekich Waldron
1 year 9 days ago

Data and analytics have been around for years. Retailers have under-invested in technology and it is catching up with them in a big way. Analytics has moved from descriptive, to predictive, to prescriptive to cognitive computing. The advent of new cognitive computing technologies running on the cloud can bring retailers into the 21st century. These systems understand, reason and learn over time. That’s what retailers need to do to understand consumers — both longer-term trends as well as real-time decisions.

Peter Sobotta

Patricia, I totally agree. Retailers that failed to make appropriate investments are realizing their mistake. Catching up will be very costly, if not impossible for some.

Doug Garnett

I remain skeptical of the return on large investments in big data. Especially if we pin its value on “relevancy.”

In digital advertising, the increase in use of ad blockers came from advertiser moves to “relevancy.” And in digital, clearly now relevancy means “relevant to the advertiser” not “relevant to the consumer.”

The mere concept of relevancy breaks down when we realize that it involves the marketer leaping to broad conclusions that algorithmically lump you (the individual) with a bunch of other people by assuming this bit of data we have has a single meaning.

It’s likely that relevant … isn’t.

Bill Hanifin

The potential of Big Data is enormous and retailers (or other marketers) who seek to grapple with the entirety of that potential may fail to execute well in any area.

I am an advocate of collecting just the data that you plan to use in a thoughtful manner. Identifying specific objectives and customer behaviors needed to reach those objectives leads to creation of a short list of data-enabled offers that can be measured for success.

In easy language, keep it simple, narrow the field of what data you collect and make sure you execute well on specific promotions, campaigns, or offers. Success in small steps creates the foundation for a richer embrace of the potential of Big Data.

Vahe Katros

Reading this a bit late but it’s a good piece with great comments. I think Tom Redd’s comment about granularity was interesting since it made me ask: how does data eventually drive outcomes. Sure, real time data on an advertising platform (oh and retail site) like Amazon leads to relevant displays, for most retailers this is overkill and the outcome that is more important is redesigning business models and workflows to win people over on service relevance.

ps: The loop back to Tom’s comment was when I saw the same word – “granular” – in this piece from McKinsey – that you can access – here.

Larry Miller
1 year 7 days ago

Great article, great comments, but I sense that we are still spinning and spinning around “data” and “analytics” and suggesting that we need more data analysts/scientists to decipher the “data” and “predict” from data to create more data, but not enough “SOLUTION.” I offer that we must remember that the “data” results from “behaviors” and “practices” and that to truly gain value we must return to the behavioral causes and effects that produced the data.

New dimensions in plain english analytics can tell the retailers (on a day-to-day basis) what the data means, what behaviors and practices likely caused the data anomalies and what to do (now) to fix it. How to fix it is the key! We have to get the decoded and deciphered information into the hands of trained and knowledgeable store team leaders who are closest to the point of correction and who can act quickly. The solution is not in the data but in the actions and practices people take to correct everyday store challenges to improve profitable selling. Always remember: People solve, not data. People using smart, plain english BI is the real key to improving performance. THANK YOU! I appreciate the opportunity to comment.

Cynthia Holcomb
Retailers are struggling with Big Data because agnostic systems, that eliminate subjectivity, do not exist to process disconnected data points into “customer” relevant intelligence. We retailers still use subjective, anecdotal evidence to make expensive business decisions. Yes, the retail industry does need to change its thinking. It is not about the technology, yet technology is the only vehicle that can process and humanize the massive amounts of data collected by most large scale retailers. Missing in today’s retail thinking is a cognitive approach, rather than the current computer science, linear approach to processing Big Data. As retailer, I was taught to “listen” to the customer. How can we “listen” to millions of customers at once? Embracing the business of retail using 21st century tools. Being open to taking a look at or at least trying to understand new ways of thinking. Embracing the development of new technologies designed to process sensory based, Big Data in new cognitive based processing systems. You know what sold; now you know why, specifically…. As an example, we retailers have reams of recent customer purchase history. A plethora of rich, sensory driven, customer product feedback, hiding in plain site in Big Data. Using recent purchase… Read more »
"What retailers really lack are the systems and technology required to mine disparate data and turn it into actionable intelligence."
"It's simple: retailers are obsessed with one and only one thing — operational efficiency."
"Data and analytics have been around for years. Retailers have under-invested in technology and it is catching up with them in a big way."

Take Our Instant Poll

How far is retail from tapping the customer-specific data needed to create highly relevant offers?

View Results

Loading ... Loading ...