Big Data is all that, except when it isn’t

I’ve got a problem with Big Data. It’s not that I don’t believe that insights gathered from a variety of data points can put retailers in a better position to serve me, their customer. It’s just that while retailers may be saying they are doing a better job of using my personal data to reward me for past behavior and to anticipate my future needs, it’s not so clear to me. I certainly have not seen enough to make me want to share more information that I currently am.

So, what’s wrong with Big Data? Here are two quick takes.

1. Insufficient data: Retailers are making recommendations and presenting offers based on an incomplete set of data. Using my REDcard data, for example, may give Target information to tailor offers that I would find a value. But what about those products I purchase from other retailers instead of buying at Target or on target.com? What about purchases I make at Target for which I pay cash or use another card? The chain doesn’t know what it doesn’t know, and fact-based decision-making only goes as far as the facts will take you. In most cases, it only takes you part way.

Big data

2. Analytical inertia: Many retailers simply aren’t using data to move the needle. While IT pros often talk about the advancements they’ve made in analytics, a precious few businesses are actually using the insights gained in strategically significant ways, from what I’ve observed. I’ve heard from some that this is a management and cultural issue. The data says do A, but the CEO says B is the answer. Others have told me that chains simply do not have the capacity (human and/or technological) to properly analyze the data captured. This has been an issue since the creation of the UPC code. Wouldn’t you have thought we’d be further along by now?

I don’t know if retail can close the gaps I’ve identified as well as others I’ve missed. Maybe some have figured it out already. Perhaps others are close. It could be that Big Data provides some good answers, just not all of them. Without the hype that has been attached to Big Data since the beginning, that may be good enough. Maybe it will have to be.

BrainTrust

"Then there is Staples, who recently asked me to write a review of a PAD OF PAPER I bought. In short, personalization and Big Data have realized their potential in that many encounters are now automated and automatic."

Al McClain

Co-Founder, RetailWire


"I know of many retailers using so-called Big Data to improve store clustering, customer segmentation, assortments, allocation, promo/price optimization, etc. Where things get tricky is applying it down to the individual customer level."

David Dorf



"If retailers truly want to extract value from Big Data they had best begin by abandoning everything they think they know about data accumulation, analysis and activation."

James Tenser

Retail Tech Marketing Strategist | B2B Expert Storytellingâ„¢ Guru | President, VSN Media LLC


Discussion Questions

What is your take on the advancements (or not) retailers are making in the use of data capture and analysis? Is it all leading to significantly improved customer experiences down the road, or something less?

Poll

23 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Dr. Stephen Needel
Dr. Stephen Needel
8 years ago

I wrote about his issue at length for ESOMAR in 2013. The problem is that we’ve put the cart before the horse, IMHO. The data itself is just data and often has lots of problems, including the one noted. The belief that the data will actually tell you anything is misguided — you have to ask it questions. Lots of companies have invested heavily in kitchen sink approaches — let’s throw everything into an analysis and see what pops out.

I advocate the reverse. Ask specific questions of the data and then make sure you have the right data to answer the question. A retailer might want to look at co-purchasing, either at the transaction or customer level, to see if there are synergies between businesses. Think peanut butter and jelly, or cake mix and frosting, both pairs being purchased by the same people or at the same time. If the former, promote at the same time. If the latter, promote only one of the pair at any one time (because they’ll buy the other anyways). Just one example.

Al McClain
Al McClain
8 years ago

This is a timely topic for me. With several bank and credit cards accounts (personal and business) at a large U.S. bank, I had a situation this morning where the bank told me I suddenly cannot use their online bill pay to pay a bill for one of their own credit cards, which I have been doing for five years. They say it could be that their system is screwing up, or it could be I need to use another browser or it could be … whatever. They are terribly sorry for the inconvenience but try again later or call back later …

As happens with many retailers, this is a case of too many tech/data moving parts and interactions and too many customer service reps who are just programmed to give pat answers. On the other hand, I do find Amazon recommendations to be getting better (although perhaps that’s because I am giving them more information), but their customer service isn’t working as well as they have apparently offshored many of their call centers. Then there is Staples, who recently asked me to write a review of a PAD OF PAPER I bought.

In short, personalization and Big Data have realized their potential in that many encounters are now automated and automatic, making it easier and more efficient for the companies but less helpful for the average customer, in many cases.

Tom Redd
Tom Redd
8 years ago

Many retailers are making huge advancements by leveraging Big Data with the right tools and user interfaces that help decision-makers. Big Data must be coupled with the right decision-making tools — sometimes that might mean a simple-to-use front-end or user interface or an API into the retailer’s forecasting toolset or apps.

Big Data needs integration into the decision process. I am familiar with one major apparel retailer that focused their Big Data efforts just on inventory performance — real-time. They had a user interface for the merchants that let them see where the demand was and where the inventory was. This help them re-balance the next replenishment efforts and assured that items most in demand would be in stock.

Simple? Not across multiple countries and thousands of stores in real-time.

Big Data becomes big knowledge with the right platforms and tools.

Ryan Mathews
Ryan Mathews
8 years ago

Ah … George has repurposed that old Aristotelian argument about potentiality and actuality and somehow applied it to 21st century retailing.

Bravo!

As Aristotle told us, potentiality mist precede actuality, i.e., it must be possible for a thing to happen before it can, in fact, happen.

So, in the case of “Big Data” its potential is clear even — as George correctly points out — if its potential is far from being realized.

Too often we forget that individual datum, and therefore data, are both content and value neutral. Put another way: absent an effective contenting vehicle, information has no intrinsic value. And so, while we can take some measure of pride in our ability to aggregate some data, we are woefully lacking when it comes to exploiting the promise of “Big Data.”

It isn’t capture that’s critical, it’s context.

As to the future, if we keep confusing the ability to aggregate information with actual information, we aren’t going to make much progress.

Hopefully, more people will begin to understand the paradox George has surfaced — the more data we capture, the less valuable it is to us and — ultimately — the less likely people are going to be to share it in the absence of tangible benefits.

Gene Detroyer
Gene Detroyer
8 years ago

The problem with Big Data is that people try to analyze it just as they did with “small data”. You can’t. It is too big. It has to be used very differently. You can’t look at the numbers uniquely. There are too many. One must find a different way to analyze the data. It has be done graphically.

There is a great book that uses Big Data graphically, The Human Face of Big Data. Also, there is a group in Sweden that is spectacular with its graphic interpretations of Big Data, www.gapminder.org. Try it out. It is so much fun to use.

Big Data is not going to tell big retail how to specifically sell to John Smith. Big Data is going to tell big retail where the business is and where it is going. Then the retailer has to make decisions.

Ron Margulis
Ron Margulis
8 years ago

This is the classic “the companies you’d expect to be leading innovation are the ones at the forefront with using Big Data and analytics” scenario. With its use of dunnhumby, Kroger is certainly leading the way with taking large amounts of data and converting it into actionable knowledge on tactics like targeted promotions and price optimization. Macy’s is deploying big analytics to not only engage with the customer better, but to improve the potential of new items. And there are many more examples.

One of the other things I’ve recently heard a lot about is retailers using the data to avoid mistakes, either their own or others. Strategic and competition units within retailers are reviewing what’s gone wrong in the past and creating models to ensure the problems aren’t repeated, and even trying to reverse engineer them to see why they failed.

I do understand George’s reservations about Big Data, though. More than once I’ve thought that even with all the data that, say, dunnhumby collects there might still be an analyst sitting in a locked room behind the “black box” at Kroger editing the findings.

Roger Saunders
Roger Saunders
8 years ago

George has identified two essential steps that Retailers have to take. That will entail stepping back from protecting the system that they have in place.

  1. In terms of insights, George’s perspective is spot on. Target, and too many other retailers have superb insights into what consumers do in their stores, but they are clueless as to what consumers are doing away from their stores. The data scientists that are being hired by retailers have to keep in mind the second portion of their titles — “scientists.” That will require them to explore and test more readily, as opposed to solely defending the collection methodology of data used in the past.
  2. With the wealth of multiple data streams available to data scientists, there is limited need to permit analytical inertia to creep into the mix. Analytics is not a black box service. By letting today’s computers cross and append diverse data sets — proprietary, custom, syndicated, government, industry — retailers will serve themselves well by uncovering the new “new,” and provide themselves with a path that only they can see and hence follow. That offers them the advantage of SPEED to market.

It has never been about BIG DATA. We have had “green bar” sheets and their updated cousins for more than 40 years. It has always been about SMART DATA. Today’s retailer, manufacturer, financial concern, etc., have the assets to support these needs. Insights plus analytics equal better, faster decision making.

Lee Kent
Lee Kent
8 years ago

It is not so much about Big Data as it is figuring out the right data. Have you read Malcolm Gladwell’s Blink? A great read about applying knowledge and wisdom to data. Then culling it down to the right data to support your actions.

But that’s just my 2 cents.

Li McClelland
Li McClelland
8 years ago

They merely need better collection, mining and utilization of highly personal Big Data? Oh please. This is the shiny object excuse too many companies now use to explain their own time-tested retail failures: not supplying adequate or respectful customer service, not having shelves/warehouses adequately stocked and having slow or aggravating checkout procedures.

Sarah Engel
Sarah Engel
8 years ago

I have worked with senior retailers every day for years, and never once have I heard one of these leaders say, “I need more data.” In fact, they are drowning in data. Merchants, eComm directors and marketers head into the dreaded “Monday reporting meeting” with their own spreadsheets and analytics/BI reports, disconnected from one another in every way. Rather than being ready to take the actions that will improve their customer experience and answer their customers’ true desires, they are into Wednesday or Thursday before they know what truly happened the week before. They are then left arguing their own theories of what action should be taken to solve for their individual KPIs, in their finite area of the business.

Big data is only useful when it is actionable. When it is connected across channels and in every area of the customer’s shopping experience. When retail teams are looking at one “source of truth.” When they know on Monday morning where disconnects are happening between marketing and inventory, sales and returns, etc., and when they have actions prescribed and ranked by how much each one is worth to the business to solve for. Only a well-run and data action-focused retail organization can truly anticipate and solve for customers’ desires, and therefore earn their willingness to share additional data.

For example: Sephora knows that I put three items in my cart a week ago, never purchased, but then bought them in store when I happened to be in the neighborhood. Because their loyalty program is valuable enough to me, I am happy to share my customer data in all three channels I shop: store, mobile app and ecommerce. So the “come back and finish your purchase” emails cease. They stop spending money retargeting me. They know who I am, where I am in the purchase journey and what will motivate me. And like most consumers… I’ll give them my customer data any day, if I get what I want, when I want it, in the exact channel I want to shop at that specific moment in time.

Gordon Arnold
Gordon Arnold
8 years ago

There is nothing more profound as a simple solution which has been properly prepared, implemented and maintained. In addition to redundancy and authentication problems there is, as the discussion implies, a matter of material relevance which is taken into the now and future decision making processes for marketing strategies.

Is the data we are collecting coming from asking the right questions in the proper order? I think not, for if it were, the data we own would be seen as important instead of just big. This in turn drives us to collect mountains of data that is less reliable causing retailers to know almost nothing about 21st century shoppers. The opportunity is to know exactly what we need from an inquiry and what questions to ask and the order of asking to get this information.

The first step is to identify all intuitive data/information and the unsupported assumptions that key executives carry with them into the company’s business decisions. The next step is to “learn how to learn” with the modern tools that are both available and reliable. The tedium and strain we face in creating a winning project or program assures success with little or no wasted time and money.

Ralph Jacobson
Ralph Jacobson
8 years ago

To George’s first point, some retailers are grabbing some great insights via partnerships with other merchants of all types to share data via common loyalty platforms and other vehicles. Further, there are now more capabilities available to leverage online chatter in new, innovative ways that helps fill the gaps in knowledge of a particular retailer. Putting all of these channels together develops a much more complete picture of the shopper.

Point #2 is all about data being the new “oil,” however it is only as good as your tools are to derive actionable insights from the data. E.g., a barrel of crude oil is not very useful to run a car, unless you have the tools to refine it. The software in the marketplace today, along with the still critical human interaction element can grab those usable insights to make decisions based less upon anecdotal knowledge and more on science. That is the key.

Christopher P. Ramey
Christopher P. Ramey
8 years ago

It’s not the data. It’s the interpreters, and managers who don’t manage it on a granular basis. Big data is nothing. Teasing one-to-one insights out of it is everything.

Michael Day
Michael Day
8 years ago

If you look at where retail is going, and the data-driven capabilities required to operationally enable the retail enterprise of now and in the future, the continued convergence of web and stores, the connected customer—technology enabled to buy anytime and anywhere, Internet of Things for retail (starting to move along now past the hype stage, to in-store operational deployment by some Fortune top 10 Retailers), it’s not a big leap to conclude that Big Data and Analytics are here, and here to stay, in our wonderful world of retail.

According to a recent survey fielded by Forbes Insights and Teradata, 78% percent of the 316 companies surveyed see effects of data-driven insights on revenues. Retailers, in particular, see an ever greater effect on their bottom line, with 31% of retailers seeing a revenue increase of more than 3% from data analytics insights, compared to 29% for other industries.

Other retail industry findings from the survey:

  • Data analytics is a key CEO focus for 37% of retail CEOs, versus 17% of CEOs in other industries
  • Data and analytics has transformed the way 45% of retailers do business, compared to 39% for other industries
  • 33% of retailers, data analytics is the single most important way to gain a competitive advantage, versus 33% for other industries
  • Retailers show a greater likelihood of investing in data acquisition, analytics technology and talent than other industries
  • And retailers also use and acquire more data and data sources then other industries

Retailers are beginning to realize both the potential and the high stakes when it comes to deploying and leveraging data and insights. That said, many retailers are still searching for the right “data strategy” for their business, and the right investments required to realize the full potential of Big Data and Analytics. At this point, for many out there, it really does come down to senior retail leadership leading the way forward for their organizations.

David Dorf
David Dorf
8 years ago

I know of may retailers using so-called Big Data to improve store clustering, customer segmentation, assortments, allocation, promo/price optimization, etc. Where things get tricky is applying it down to the individual customer level, and although great strides have been made, it’s still immature.

I often get targeted for things I’ve purchased as one-off gifts, or items I’ve already purchased. The data set for an individual interacting at a specific retailer is actually quite small and therefore it difficult to derive meaningful intent.

We can already collect lots of general social data, so now we need to add purchase and behavior data from across all retailers. Then we’ll have enough data to form an accurate picture of individual customers. Maybe the next generation of payment solutions will help.

Kenneth Leung
Kenneth Leung
8 years ago

The issue from my point of view is how do you act on the data? Knowing and being able to respond is two different things. A lot of emphasis is made on getting insight from Big Data, but the ability to execute and deliver value to customer and retailer is still missing for the most part. To approach Big Data, the question should be “what is the problem I am trying to solve” and see if the data can guide actions, and then measure the results/accuracy. I see sometimes Big Data is “Let’s see what the data tells us and then we figure out what to do with it” which is the wrong approach.

James Tenser
James Tenser
8 years ago

If retailers truly want to extract value from Big Data they had best begin by abandoning everything they think they know about data accumulation, analysis and activation. The shopper data generated within the confines of the store are certainly very large, but the shopper data generated outside the store are far greater in magnitude and significance.

It’s time to stop contemplating our (data) navels folks. We need to lift our heads and start gazing into the (data) clouds, if we want true insights into the Big Picture.

Big Data is certainly about volume and variety, but most importantly it is about velocity. To cope with ever-increasing speed, we’ll need to stop attempting to measure the molecules in the data flow and start monitoring its temperature. In this context it is very often not the data itself that yields the insights, but the data about the data.

All metaphors aside, I am at my core, a stores guy. I believe strongly that retailers must learn to offer relevant experiences and solutions to shoppers. Sensing their in-store behavior is a place to begin segmentation and targeting. But to what end?

If the retailer’s goal is simply to extract more shopper marketing funds from its suppliers with claims of robotic relevance, I submit that this is ultimately a soul-less and bankrupt strategy.

If the goal is to continually adjust stores, communications, shopper experiences and offers to match shoppers present and anticipated needs, that may well be the golden path to Big Data nirvana.

Doug Garnett
Doug Garnett
8 years ago

I think there are core analytic/scientific problems with the theories of Big Data.

Observed data (or found data, which is what Big Data is in this context) has exceptional weaknesses. In many ways, big data enthusiasts have gotten away with ignoring those weaknesses with the idea that “but there’s so much data, how can it not reveal huge insight?”

And yet, that’s a fallacy. We’ve known for decades that more data doesn’t mean better action. It just means more data.

But the core challenge in the fact that it’s observed. And observations never reveal motivation.

Here’s a simple (not retail) example. A friend of mine worked for an accounting software company. They hired a Big Data firm to do all their prospecting. The firm analyzed the observable data and sent them the most active web folks around their product as valuable prospects.

Except, the motivation for all that web action was that these were unemployed accounting people trying to learn about the software in order to get a job—they weren’t even close to prospects.

Shift to retail: You observe a bunch of things about people and decide “everyone fitting this profile should get a personalized email encouraging them to do XYZ”. But… Each and every one of the people fitting that profile might have had entirely different motivation for taking those steps. Your team must ASSUME motivation in order to hopefully do something good.

Except, it doesn’t. Only in rare instances are those assumptions correct (and we all know what happens when you ASSUME)…

Here’s one post I wrote about how weak observed data is.

I don’t, in fact, think there’s a vast untapped strategic advantage lying hidden in big data “if only retailers would change their ways.” Big data offers smallish incremental advantages. Each company must decide whether those are worth the investment needed. In many, many cases they aren’t.

Peter J. Charness
Peter J. Charness
8 years ago

Hmmm quick quiz. If you could choose only one of:

a) A quant and a seriously awesome collection of all kinds of great data (in all sizes and shapes)

or:

b) A great buyer who understands the customer, fashion trends, and how to build a great collection for sale.

Which one would you put your money behind?

The great ship Panacea sailed some time ago. The correct answer IMHO is c) a well informed buyer… data and context.

Happy Friday.

Arie Shpanya
Arie Shpanya
8 years ago

Big data is like the Goldilocks story: you don’t want too much or too little, but just the right amount. Once you have that right amount it’s a matter of using the right tools and devoting the right team to analyze it properly. Next, a retailer needs to have a flexible enough strategy to implement the takeaways they gained.

I think that the potential of big data is vast, but it’s about finding the Golidlock zone for each individual retailer.

Currently retailers are doing their best to make the most out of big data, but there’s only so much they know about their customers. As they gather more and figure out how to analyze it more effectively, I see big data having a huge impact on improving the retail experience for shoppers and retailers.

Shep Hyken
Shep Hyken
8 years ago

Big Data spots trends. It doesn’t create a personalized experience. Something I refer to as “little data” or “micro data” drills down to the individual customer. That is where a customized experience—in your store and in your marketing—begins.

Kevin Sterneckert
Kevin Sterneckert
8 years ago

I see that many retailers are beginning to understand the importance of using more data to make better decisions. The first hurdle seems to be recognized by many, however, there is chaos across the industry on just how big data should be used.

It’s been my experience that retailers do not need more data or more insights, in truth, I can not recall a retailer ever telling me they needed more of either. Retailers are looking for insight driven recommendations.

If a retailer believes the net result of big data are “better insights,” I believe those who do will struggle to find measurable value in what they have invested big money to do with big data. However, if retailers are driving to recommendations that are delivered by technologies that leverage big data combined with advanced prescriptive analytics…now this is a winning equation.

When the data of an entire enterprise is used and potentially combined with other sources of data outside of an organization (like syndicated assortment/sales data) recommendations can then include the impact to all aspects of a retail organization. Finance, marketing, merchandising, stores, digital channels, the supply chain, etc. all provide a view of the business. Combine these views and you certainly have big data, however, the combined view also provides a holistic understanding of the constraints and conditions that exist and must be reconciled as organizations strive to be better at serving the customers needs than their competitors.

When big data is harnessed to recommend better decisions, retailers will dramatically improve. If retailers continue to think about big data delivering only insights and require decision makers to understand the insights and then formulate actions, the resulting decisions will only be as good as the talent making the decision.

Matt Talbot
Matt Talbot
8 years ago

The advancement—and opportunity for advancement—retailers are making by leveraging “big data” will prove to be some of the greatest benefits to consumers, since the ability to purchase goods/services from a connected device. There are few other plays in retail (at the moment), which lead to such massively beneficial impacts as collecting and leveraging “big data.”

This is evidenced and there is a very little debate, that “big data” is a macro driver of change to the consumers’ benefit in many industries already (transportation, healthcare, telecom, etc.).

Retail will absolutely rise to the opportunity to realize the benefits of “big data” for consumers, if for no other reason than because it is so commonplace (and beneficial) in nearly every other transaction in their lives that they’ll demand it.