Big Data quandary

Why Small Data is the new Big Data

Presented here for discussion is a summary of a current article published with permission from Knowledge@Wharton, the online research and business analysis journal of the Wharton School of the University of Pennsylvania.

In his new book, “Small Data: The Tiny Clues That Uncover Huge Trends,” Martin Lindstrom argues that Small Data explains the why behind what Big Data reveals.

“The issue right now is that the corporate world has become completely blinded by Big Data,” Mr. Lindstrom recently said on Wharton Business Radio, SiriusXM channel 111. “But it’s very, very hard to describe emotions using data.”

In his estimation, Big Data is all about finding correlations, while Small Data is about finding the causation — the reason why.

Using Snapchat and Post-It notes as examples, Mr. Lindstrom estimates that of the top 100 biggest innovations of recent times, around 60 to 65 percent are based on Small Data. In his book, he interviewed 2,000 families in more than 77 countries to get Small Data clues to how they live.

“You have to remember that Big Data is all about analyzing the past, but it has nothing to do with the future,” said Mr. Lindstrom. “Small Data, which I define as seemingly insignificant observations you identify in consumers’ homes, is everything from how you place your shoes to how you hang your paintings. I call those the emotional DNA we leave behind ourselves. … You need the hypothesis first before you start to mine it and find correlations.”

To acquire Small Data, companies have to “embed themselves into the community.” The simplest way is spending time in consumers’ homes, but most CEOs and senior managers are too “reliant on sitting in meetings.”

At retail, it’s easier because consumers are in stores. He noted that he once visited the founder of IKEA, Ingvar Kamprad, at the chain’s Stockholm offices. He was told the owner was at his “usual spot,” which turned out to be behind a cash register at a nearby store.

“I said to him, ‘Why are you doing that?,’” recalled Mr. Lindstrom. “He said, ‘Because this is the cheapest and the most efficient research ever. I can ask everyone why they choose it and why they didn’t choose it.’ This is the essence of how good business leaders are.”

BrainTrust

"Big Data has become the path of least resistance. You can look at Big Data numbers, draw a conclusion and justify it to whoever requires justification."

Joan Treistman

President, The Treistman Group LLC


"I’ve always been a proponent of learning both the "why" and the "why not" behind data trends. Without both, trends can lead you down the wrong path or one that suddenly turns into a blind alley. Motive is key to understanding behavior."

Dan Raftery

President, Raftery Resource Network Inc.


"Fast Data is overlooked in this post and is of paramount value. Whether it’s big or small, if your data is slow, inaccurate or unorganized, it’s impossible to extract maximum value from."

Matt Talbot

CEO & Co-Founder, GoSpotCheck


Discussion Questions

DISCUSSION QUESTIONS:
Are retailers “completely blinded by Big Data” and consequently missing out on insights from smaller one-on-one conversations and observations? What lessons should retailers glean from Mr. Lindstrom’s take on Small Data versus Big Data?

Poll

19 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Dr. Stephen Needel
Dr. Stephen Needel
7 years ago

Once again, Lindstrom’s ability to interpret correctly is called into question. He likes one-on-ones and observations because it keeps you from having to do real research. Big Data is hardly as pervasive or as entrenched as he’d like us to believe. More important, Big Data, analyzed and used appropriately, can be invaluable. His assumption that you need to do Small Data before Big Data is going to be right sometimes, but his assumption that hypotheses don’t already exist is incorrect.

Adrian Weidmann
Adrian Weidmann
7 years ago

Big Data in many ways has been the “shiny new object” that we’re chasing to unlock an easy passage to the consumer’s wallet and during the chase we’ve been blinded by the endgame rather than observing the details along the way. The founder of an RFID-based start-up used Little Data to describe the focus on the core principle that retail is essentially a consignment business. With this as a foundation — developing a business solution that facilitates this reality based on Little Data — did a shopper remove the product from the shelf and leave the store with it in her shopping bag? If the answer is yes then you can tally a sale as well as supply chain management and then monitor what, where and when to improve your product mix and address localized trends. The visualization and response to Little Data opens the aperture to the predictive modelling and recommendations found within Big Data.

Understanding and responding to the Little Data is often common sense but certainly not common knowledge.

Chris Petersen, PhD
Chris Petersen, PhD
7 years ago

It all depends what you are solving for. Said another way, it depends upon the kinds of business questions you are trying to answer.

If you are Amazon and trying to solve where to best place inventory for quick shipping, then predictive analytics using Big Data will probably provide the best answers.

If you are trying to understand why consumers are not buy smart-home IoT products, then the best answers will be found observing consumers in their homes trying to figure out how to make them work.

Joan Treistman
Joan Treistman
7 years ago

Big Data has become the path of least resistance. You can look at Big Data numbers, draw a conclusion and justify it to whoever requires justification. I agree with Mr. Lindstrom about risks associated with that research approach. But it seems Mr. Lindstrom suggests ethnography (going into the home, standing at the cash register) as the way to insure appropriate and reliable strategic insights are gathered. I think that’s risky as well.

From a practical perspective many retailers don’t have the budgets (or won’t allocate the investment) for broad scale ethnographic initiatives. Mr. Lindstrom’s study of about 2000 households in 77 countries represents about 26 families per country. I don’t believe that any retailer should rely on results of 26 observations to develop strategic plans.

An integrated approach which includes stating an objective, developing hypotheses, investigating them and going back to validate them, will lead retailers to a broad array of research approaches. I’m not saying ignore Big Data and I’m not saying ethnographies can’t be useful. I am saying there is no one-stop data gathering when it comes to making informed decisions. There’s a whole host of research approaches in the universe. Some are easily accessible to all and some require professional guidance to determine a methodology. Some require huge investments and others are nimble.

I do agree that relying on one approach, whether it is Big Data or some other form of research (big or small), can prohibit the retailer from uncovering strategic insights which will be most useful.

Ken Morris
Ken Morris
7 years ago

I believe Small Data is the key to unlocking the power of Big Data. I developed a loss prevention product called XBR, now sold by Oracle, in the mid-’90s. We developed the application based on a premise based on observation of theft at store-level (Small Data). We developed the hypothesis (54 basic theft scenarios) and then leveraged our version of Big Data, 12 to 36 months of a retailer’s store level transaction logs to mine, find correlations, develop pattern recognition, identify and remove thieves from a retail chain. This is just one example of harmonizing Small Data with Big Data for retail innovation. Mr. Lindstrom is spot on with his Small Data observation.

Anne Howe
Anne Howe
7 years ago

Big Data is the new black, but it does tend to hide emotional insights, and this area is where Martin Lindstrom shines. I’ve been previously involved with Lowes Foods and witnessed some of the very dramatic changes his work inspired in stores to add multiple areas of emotional engagement.

I’ve witnessed shoppers in many stores over the last 20+ years and can attest that shoppers truly engage with the remodeled Lowes Foods stores in ways that data can’t reveal.

I’ve also used Big Data to compare category growth trends in re-modeled stores vs. existing stores. The data gave hard facts in support of investing in more re-models.

Martin’s work may not please everyone all the time but his innate understanding or human emotion through human interaction could inspire change for many many boring retail environments in the world today.

Patricia Vekich Waldron
Patricia Vekich Waldron
7 years ago

The data debates are far from over! The dimensions of Big Data (volume, variety, velocity and veracity) can be used for many purposes and across time frames (past, real-time and predictive). There are many technologies, processes and analytics techniques that can tease insights out of data.

To turn these insights into action a retailer must have a strategy for accessing and managing data, automating analysis, taking advantage of the newest capabilities like cognitive computing and giving everyone access to information at the point of decision.

Charles Whiteman
Charles Whiteman
7 years ago

Big Data is getting big attention because it generates big cash for the big companies that sell the tooling needed to accumulate, store, crunch and interpret it.

It’s probably true that some retailers are over-investing in this tooling because of all the Big Data marketing and media coverage (the “Hype Cycle”).

We do a ton of work aggregating and analyzing user behavior across markets. The way Big Data factors into what we do is:

  1. We aggregate lots of data along the right dimensions to get a baseline (aka, Big Data).
  2. This baseline exposes specific exceptions (what Lindstrom would call Small Data).
  3. Explaining these exceptions ideally yields a testable hypothesis.
  4. Creating the test and confirming the hypothesis = new insight.

As Lindstrom points out, item 3. is almost always done by an expert with the ability to perceive “why.” The promise of Big Data is systems that dynamically perform steps 3. and 4., solving the problem without having to first understand “why.”

This is where, at least today, the reality of Big Data dramatically lags in its promise.

Michael Day
Michael Day
7 years ago

For starters I have to disagree with Mr. Lindstrom’s premise: “You have to remember that Big Data is all about analyzing the past, but it has nothing to do with the future.”

Really? Since when is understanding and leveraging insights gleaned from big data “all about analyzing the past”?

Operationalizing predictive and prescriptive analytics to forecast better and react better, understanding how to optimize Big Data to meet the new requirements of retail’s always-connected customer and the demands of modern retail supply chain execution, etc. This is not “all about analyzing the past.” It is not only about forecasting what will happen, but it is also about giving the retailer options to act on what the data says is going to happen.

Understanding the insights from and leveraging Small Data can be important to retailers of course, but this is not an either/or value proposition. It is about having an integrated data strategy and leveraging both Small Data and Big Data to meet customer expectations and drive operational excellence, etc.

Graeme McVie
Graeme McVie
7 years ago

In most situations you arrive at the best outcomes by combining art and science. It’s important for retailers to speak directly with shoppers to gain insights into the “Why behind the Buy” and “Small Data” as described can enable this aspect of the learning process. However, only taking a “Small Data” approach is very limiting because it’s not scalable and could potentially be misleading due to audience bias. It is essential to combine the best of the art and science approaches to arrive at optimal outcomes.

The distinction made between Big Data and Small Data also misses a key aspect of this conversation. Retailers are sitting on a fantastic data asset in their sales and customer data, which would not always be described as Big Data as it doesn’t include the unstructured data that is often found within the definition of Big Data. This sales and customer data can yield valuable causal insights that can be used in a scalable way to truly understand how customers react to price changes, promotions, assortment decisions, etc. The insights from this approach often benefit from an explanation of why shoppers are price sensitive to one item versus another, for example, and these explanations are best obtained by speaking directly to shoppers.

Brian Kelly
Brian Kelly
7 years ago

As students and practitioners of retail, when we cross a threshold we know if the store manager is spending time on the floor or in the office. The better store has the manager on the floor. The better chains suck this local knowledge up to HQ, actively discuss and analyze it so that relevant programs return to the store.

Ethnographies are helpful and like any methodology the role of them within the research mix needs to be carefully determined.

I prefer the CEO who spends time on the floor away from the coterie of sycophants that distract and filter the reality of the store experience. A CEO working at POS is an interesting idea. I think it adds more value than its sitting in the home of a single shopper. I agree with Angela Ehrendts, who states that the primary target is the store front liner. CEOs who are active in stores raise the bar on customer care because field word of mouth is powerful.

So I like this formal recognition of “Small Data” and its role within and in context of “Big Data.” As long as one voice doesn’t mislead.

As we like to say today and in the future, “retail ain’t for sissies!”

James Tenser
James Tenser
7 years ago

“Small is the new big?” Well a glib title does not a complete thesis make.

Of course it’s important to get your specifics correct when aggregating a big data trove. Debates about the tension between qualitative observation and the massive crunch of quantitative data are nothing new, however.

In my experience, qualitative investigation reveals the fundamental research questions that may be investigated using quantitative methods. Big data adds something different, however — it consists of massive, continuous, expanding flows of structured and unstructured information generated not from research studies, but aggregated from numerous interactions with systems that operate every minute.

So POS and frequent shopper data qualify under this definition. So do social media data flows, online interactions, and signals emanating from the Internet of Things. Understanding the individual motivations behind a shopper’s decision to buy grows ever more complex and subtle as we attempt to incorporate those factors into our insight-generating apparatus.

When it comes to data, Big isn’t just a dimension, it’s data in a new forms, with new traits and new rules. Traditional analytical methods no longer apply. I observe many advocates of analytics for analytics’ sake, but the real breakthroughs come as we learn to extract and respond to Big Data insights to tune our business systems for competitive advantage.

Lee Kent
Lee Kent
7 years ago

Obviously both are needed. Is one more important than the other? I agree with Chris, it all depends on what you are solving for. Does Big Data have nothing to say about the future? Hogwash!

Big Data is filled with insights. The trick is gleaning out the most relevant indicators for your business. Making Big Data small and meaningful.

And that’s my 2 cents.

Vahe Katros
Vahe Katros
7 years ago

Qualitative, Ethnographic, Experiential, Human-Centered, or even walking the mall with Les Wexner are already part of the retail tool set so we need to look for the small data in Small Data.

We are in a time when Big Data is running the table and this is a great title for (small data why: “selling books”) connecting with CEOs who intuitively value the importance of the gut. There are many examples of how being there can help capture the events taking place in the “empty space” or capture the “unmeant needs.” And it is in those (sparse data) nuggets where the future can be found.

When the World Trade Center Towers came down, the intelligence community did some soul searching and admitted that SIGINT (Signals Intelligence) vs. HUMINT (Human Intelligence) had become the dominant way of tracking events. So this is good. Design research or whatever you want to call it can use a cheerleader to direct energy and budget towards making observational and experiential research part of the Business DNA especially in a world of rapid innovation.

The real issue becomes developing methods and leadership around how to manage all of the insights — or as Steve Jobs once said:

“People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of the things we haven’t done as the things I have done. Innovation is saying no to 1,000 things.”

Oh, “Just one more thing” from the master of small data:

“You know, I can remember the only new car my father ever bought. How proud he was. The way it smelled. Kind of like the inside of a Pullman car.” – Lieutenant Columbo

Shep Hyken
Shep Hyken
7 years ago

I can’t imagine that retailers are “completely blinded” by Big Data. The problem retailers have (and any company) is having too much data and not being able to do anything with it. The key is to get the right data. It helps spots the trends needed to make good decisions.

The Small Data is important for a more customized experience. It can be small based on the region, the city, the suburb, the neighborhood and even the individual customer. Using the data properly can help a retailer create a better customer experience.

Carlos Arámbula
Carlos Arámbula
7 years ago

I believe retailers need to utilize both. Think of it as qualitative and quantitative data, each has a purpose and each will provide specific answers to help retailers and manufacturers develop strategy.

Unfortunately, we have seen evidence over the years where retailers and manufacturers over-state one vs the other and make puzzling decisions (i.e. New Coke, Fresh & Easy, and JCPenney).

Dan Raftery
Dan Raftery
7 years ago

I’ve always been a proponent of learning both the “why” and the “why not” behind data trends. Without both, trends can lead you down the wrong path or one that suddenly turns into a blind alley.

Motive is key to understanding behavior.

Matt Talbot
Matt Talbot
7 years ago

Both Small Data and Big Data have their place and provide value to the retail industry. However, Fast Data is overlooked in this post and is of paramount value.

Whether it’s big or small, if your data is slow, inaccurate or unorganized, it’s impossible to extract maximum value from. However, if you’re able to receive retail data as it happens in a clear and concise manner, you can take immediate action. By doing so, you can correct small issues before they become big ones, quickly double-down on successful promotions and make a host of other business-improving decisions right away.

In summary, both Big Data and Small Data can be very valuable depending on the use case, but if that data isn’t instantaneous, much of that value is lost.

Doug Garnett
Doug Garnett
7 years ago

YES. Most retailers (and really most marketers) have become blinded by the theories of “big data” — there’s a lot of money being spent to convince them it will solve all their problems. And it won’t.

Yet I’m really disappointed these folks call everything else “small data.” Because individual interviews and focus groups, for example, aren’t about data — their about understanding. Developing a sense of what makes people tick, revealing the hidden things that can’t be seen in data.

We need to return to our research training. Big data is generally (a) secondary data and (b) quantitative. That means it can’t reveal a great many things that are meaningful and that we need to accept its limitations as secondary data.

I’ve written that we need look no further than paleontologists and the errors of their theories based on “big data” to see how much that’s very important ISN’T possible to see in big data.