Where have all the retail data scientists gone?

Through a special arrangement, what follows is a summary of an article from Retail Paradox, RSR Research’s weekly analysis on emerging issues facing retailers, presented here for discussion.

At its recent annual analyst meeting in Steamboat Springs, CO, SAS referred multiple times to four key user groups. To paraphrase a bit, the four are business management (basically, deciders), business analysts (the people who prepare the analysis that helps deciders decide), IT staff (which keep the infrastructure running), and data scientists (who basically make the data useful and easily consumable by the business analysts).

Data scientists? I can honestly say that I have never met a data scientist in retail. They are a class of user that appears to have all the frequency of unicorns and dragons in the retail enterprise.

In theory, data scientists help business analysts model and understand their data. When data scientists exist, data can actually be democratized (as in pushed closer to the front line), because it is the data scientists that ensure that the data is described, staged, and modeled in a way that makes it more accessible to casual users. Casual users best understand the problems they are trying to solve, and data scientists best understand the opportunities and constraints of the data that is available to provide insight into how to best solve problems. Only when the best of both worlds is combined can real insights be gained — and acted upon.

So why are there no data scientists in retail? First, they tend to be expensive resources. Retailers are competing with investment banks, pharma, insurance companies and telcos — industries that have known the importance of data actually for a lot longer than retail has.

Second, historically, the only data that was important to retailers was product data. Specifically, sales data (demand) and inventory levels (supply). You don’t need data scientists for these well-understood types of data.

But a lot of new kinds of data have been added in the last decade or so, most notably a vast expansion in types of customer data — and huge growth in the importance of that data to the enterprise. Top it all off with the extremely messy realm of sentiment and social media, and all customer data challenges become compounded.

And then there’s the notion that sales don’t just happen in stores. The relationship between influence and sales has become very messy as new channels and touchpoints are added.

Will retailers recognize and respond to the need for this layer of resource within the enterprise?

BrainTrust

Discussion Questions

Is the lack of strong data analysis talent a big hurdle for retailers trying to capitalize on data insights? Can retailers compete with investment banks, pharma and other industries for data scientists? What other resources or alternatives may be available to retailers?

Poll

19 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Zel Bianco
Zel Bianco
10 years ago

This is a subject I have been screaming about for a number of years. Finally, it is subject that the industry is starting to recognize as a big problem. Business Intelligence and data mining have been around for some time, but the issue has always been that the biggest, most expensive BI solutions are one size fits all. This has resulted in very poor results and disappointment to organizations that have made significant investments without realizing an appropriate ROI.

You must give the “power” users in an organization different tools than the casual user. You have to build a center of excellence where the analysts and data scientists who understand the data and at least recognize what the business users need and the issues they are trying to solve. This group produces the data analysis and more importantly, the format that allows the more casual Line of Business user to use that analysis to make better and more insightful business building recommendations.

It is not only retailers that have this hurdle, it is many manufacturers as well. I would say that the manufacturer community has a leg up in this area and has been working with loyalty card, web and other qualitative data and in many cases, combining that data with the typical demand data for better insights. I also firmly believe that this is an area where the manufacturer can take back some control and have a bigger seat at the table with their retail partners to even the playing field!

Retailers cannot be experts in every category and they certainly cannot be experts across all the data that is available to mine for greater sales. The data scientist is just a new name for the super power users in every company. The real issue is there are not many of them, and that is why a Center of Excellence makes so much sense, because when a data scientist builds a compelling analysis, it should be done in a way that it can be shared with others throughout the organization so if that person leaves or “gets hit by a bus,” it can be picked up by others.

Camille P. Schuster, Ph.D.
Camille P. Schuster, Ph.D.
10 years ago

Retailers are not the only industry faced with the issue of not being able to find enough data scientists to hire. McKinsey’s projection is that there will be about a 40-60% shortfall of qualified people for these positions by 2018, so the issue is not going away anytime soon.

Retailers, in my experience, have difficulty competing for people to hire for these positions for two reasons. One is because most people do not think of retailers analyzing this data. Most people I talk with assume that retailers outsource the data analysis so they are not the ones doing the hiring. The other one is that universities are just now gearing up to develop programs addressing this issue. Retailers can get engaged with the conversations that are taking place about what programs need to be developed and what kind of positions need to be filled.

Dr. Stephen Needel
Dr. Stephen Needel
10 years ago

My experience with retailers has been that they would prefer their vendors to do a lot of this work for them, and to do it for free. Good analysts aren’t cheap and retailers haven’t ponied up the way these other industries have.

Adrian Weidmann
Adrian Weidmann
10 years ago

I’ve been pressing for the use of quantitative data capture and analysis at retail for the past 10 years.

The challenge is twofold. The first historical challenge has been the lack of understanding of the power of data insights. The marketing departments will rely on their research departments who manage the traditional exit surveys and standard metrics such as number of transactions, average basket size, revenue per square foot, etc. Many retail marketers believe they already have the data they need. Even the data they do get is often not taken seriously enough by retail marketers. Retailers SHOULD be seeking deeper shopper behavior insights and developing recommendation engines to keep pace with the expectations of the digitally empowered shopper and online channels.

The second historical challenge is retail marketers struggle with leveraging the insights derived from quantitative data to design and implement innovative ideas and solutions that would be valued by their shoppers and customers. All too often, retail marketers take the safe way: “This is the way we have always done things” and it’s proved to be successful for all these years. The digital revolution has created a seismic shift in the status quo and it is imperative that retailers take quantitative data seriously!

Retailers can and should compete to secure business relationships with organizations that offer quantitative data science and shopper marketing design services. An external agency can dramatically increase the use of data and more importantly successfully implementing valued solutions. An external agency can overcome internal cultural indifference and status quo.

Chris Petersen, PhD
Chris Petersen, PhD
10 years ago

The simple answer is no … not right away. Bricks and mortar retailers are steeped in a history of managing product data for demand and supply.

In recent interviews with 15 technology based retailers worldwide, all but one had separate staff and organizations for online ecommerce and stores. Yet in every country, consumers are rapidly turning to online shopping where data is king.

True online giants like Amazon and Alibaba definitely value “data scientists.” As many as 10% of Amazon’s web pages are A/B tests at any given time. 

Omni-channel is the new normal. To compete, bricks and mortar stores must merge their channels, data, and the ability to execute where the consumer wants to shop, when they want to shop.

The future of retail is Big Data, and that will require “data scientists,” or whatever retailers choose to call the talent who has the ability to turn data into business intelligence.

Marge Laney
Marge Laney
10 years ago

Nikki’s right, data scientists are hard to come by, are expensive, and haven’t been utilized by the retail community. The attitude has been, “we understand supply and demand, so why bother?” Well it turns out there’s a lot more to what drives demand and effects supply.

As Nikki points out, sales don’t just happen in stores! What happens between the time a customer enters a store and leaves with or without a purchase has been a murky mystery. Unlike the online shopping experience where the customer’s every click is captured and analyzed, offline has been a challenge, but the same insights are there! They’re simply much harder to get!

New camera and sensor technologies are shedding light on the in-store experience and challenging the old in-store experience paradigm. The trouble is that even though retailers intuitively know that what they are being shown is useful and can help them increase KPIs they need proof aka ROI!

Enter the data scientist who can take the data generated by these new technologies and definitively say; “If you do X, conversion et al, will increase by Y’” That’s actionable and powerful!

Do I think retailers need to employ data scientist, absolutely! But first, I believe that the suppliers of the technologies that provide solutions should put their data under the analytical microscope of a data scientist. Providing the retailer proof that their solution will produce the results as advertised should be the price of admission to begin the discussion.

W. Frank Dell II, CMC
W. Frank Dell II, CMC
10 years ago

Retailers are clearly at a disadvantage by not having data scientists. Just look at how little is being gained from the frequent shopper data. Some larger chains have been outsourcing and they are showing better results. Having the computer storage and processing power is not an issue today. Having the people that can think and understand is. The future is to link the inside company information with the outside world. This will be needed to understand cause and effect relationships.

Vahe Katros
Vahe Katros
10 years ago

Retail is a great industry for these types of people. In the bubble of Silicon Valley, Walmart had one of the larger and more prominent billboards on highway 101. It basically said: “If you think you’ve seen big data, you should see Walmart data.”

Resources are always limited in the early stage of the next big thing – that’s where consulting companies make their living. I wonder why SAS is not cultivating that channel?

Regarding SAS, that seems like a company dealing with upstream packaging for end users. When I think of the folks you describe, I think of people who come from the world of Cloudera and Hadoop. Why doesn’t SAS do what Cloudera did and have their own version of Cloudera’s “Certified Professional Data Scientist” program? They can spin it for retail – which they know so well.

Dimitris Tsioutsias
Dimitris Tsioutsias
10 years ago

Not all retailers are created equal. E-tailers like Amazon are well ahead of the curve. Others who have seen an erosion in their brick-and-mortar business have started, belatedly, jumping onto the wagon. Again, it’s a matter of pervasive business culture, short- vs. long-term investment and focus/commitment (a fundamental change in approach vs. a fad) — and of course, as the article well indicates, of money.

Ryan Mathews
Ryan Mathews
10 years ago

There isn’t a shortage of analysts, there’s a shortage of analysts who want to work in retail and an even more pronounced shortage of retailers who fully understand their value.

As far as “alternate resources” go — what about a “home-grown” program in which promising young associates have their educations paid for in exchange for a commitment that they will spend several years with the firm?

Bryan Pearson
Bryan Pearson
10 years ago

The demand for data analytics will surely affect retailers. Among the figures cited: 97 percent of companies with revenue of more than $100 million are pursuing expertise in business analytics, according to Forrester Research. Yet the data analytics field is forecast to fall short of professionals by 2018, with an estimated 190,000 qualified data scientist positions left vacant at that time.

As a marketer, this worries me. But as a parent, I see it as a sign of opportunity. We should encourage our kids to pursue a future in analytics. Think about it: In the coming years, the industry’s capabilities will flourish. More data will be available, and both consumers and organizations are going to have escalating expectations in terms of the benefits they receive from this information.

Today’s college-bound populace will define the future of data analysis, as well as the responsibility required in managing this information. But who better to understand the importance of responsible data use and privacy than today’s graduates, who have matured in an environment where seemingly every move is tracked, if not voluntarily shared through social media?

Younger people are generally more accepting of data sharing, and therefore have a more nuanced appreciation of its implications.

In 10 years, we’ll look back at today’s data collection processes and think we were just bumbling through the process. I hope. But the deciding factor between where we are now and where we want to be exists down the hall, under the pile of clothes on your teenager’s bed.

Matthew Keylock
Matthew Keylock
10 years ago

The retail sector has arguably the most valuable data out there: the knowledge of what people actually buy.

Accessing value from the data is growing in importance for retailers. We have seen many struggle to compete with prices, assortments and convenience offered by e-commerce or data-savvy players that derive value from data to personalize their proposition and/or maintain low prices. Data skills including analysts and scientists are critical for liberating this value from the data.

I concur with the comments that retailers should be building internal capabilities in this space. In the “Big” and “Open” data world, it is also going to be important for retailers to use external providers who have access to more and different data too, and likely a richer talent pool and better experience and know-how in turning data into value.

With both an internal and external science function the challenge is to ensure they are complementary and aligned in approach, definitions and direction. This is a leadership and organization challenge that few retailers are equipped with the knowledge or skills to address right now.

If your data scientists don’t experience the right leadership, variety of work and ability to impact the business, the good ones will either not join the business in the first place or won’t stay long. There are plenty of other better opportunities for them out there right now!

Retailers should make sure the leadershp and framework for success is in place first. Otherwise they will hire badly, experience high churn, gain no value and generally perceive data science to all be a waste of time and money in retail.

Given the fantastic data retailers have, if they get these foundations in place, there is no reason why retailers can’t compete for great science talent.

Lee Kent
Lee Kent
10 years ago

Being a “closet” data scientist myself, I’m lovin’ this conversation. I HEART data and can spend hours on end playing with it. And yes, retail needs to employ data scientists. Especially in this day and age.

It’s not all about the product, it’s about the customer, who they are, what they look like, how they shop, how they spend, when they spend, where they spend, and the list goes on.

Customers are demanding that retail know more about them, however, they get a little creeped when retail starts predicting stuff about them or wants to follow them around. But, I digress.

What we really need to keep in mind is the point that Malcolm Gladwell makes in his book, “Blink.” There can be lots of data but without the proper “wisdom” behind it, you may not win the race.

Retail can pull data scientists right out of school and at a reasonable price, but they need to look within for the “wisdom” to guide them. I’ll bet, if they look, the wisdom is already there. And that’s my 2 cents….

Ralph Jacobson
Ralph Jacobson
10 years ago

Another great perspective from Nikki! As more and more robust tools enter the marketplace to perform data analyses, the propensity to hire capable experts in this field dwindles. This is not a good reason for this trend, however, many companies are relying more heavily on insights captured by these technologies, and they view this as enough expertise, without the need to hire strong data scientists. The challenge comes in attracting the talent to the retail industry. No easy answers to that one.

Mark Price
Mark Price
10 years ago

The lack of strong data analysis talent is a huge hurdle for progressive retail marketers who are trying to move their organizations towards fact-based decision-making and data-driven marketing.

Retailers traditionally have not paid well enough, nor have the problems they want solved interesting enough, to command attention from the best data scientists. Those data scientists are choosing between retail and banking, government, dotcom, etc. — places where they would be treated as rock stars. To be a bit player in retail is just not that interesting to them.

For retailers to gain respect and the talent necessary to succeed, data scientistis must be moved closer to the decision-makers — the CMO, VP of Marketing, VP of Sales, etc., and must have access to show their results and see them acted upon.

John Boccuzzi, Jr.
John Boccuzzi, Jr.
10 years ago

Hard to argue with Zel’s comments. SAS did a nice job breaking down and defining the four groups. I believe the retail world could and should do more to attract Unicorns (Data Scientists). Someone needs to boil down the terabytes of data into bytes that can be used by analysts and decision makers. This takes smart people and great software solutions.

Ed Dunn
Ed Dunn
10 years ago

I believe Walmart has some of the talented and admired data experts in the field who regularly teach the rest of us the tricks of the trade. I remember at a tech conference once, everybody wanted to sit next to the people from Benton, Arkansas to talk shop.

The best resources are going to come from e-commerce talent, to incorporate an omni-channel strategy.

Herb Sorensen, Ph.D.
Herb Sorensen, Ph.D.
10 years ago

The vast majority of profits for self-service retailers come from leveraging their suppliers – NOT THE SHOPPERS!!! Not long ago there was a new book out of Wharton and Harvard, subtitled, “The new science of retailing.” Surprise! It’s all about supply chain management. Get a clue: NO ONE is minding the store, as far as shoppers are concerned. You don’t need any data analysts for shopping, when you don’t have any shopping data. PRODUCT DATA IS NOT SHOPPING DATA – except incidentally. So says the shopper scientist, focussed first on PRODUCING shopper data. 😉

Alexander Rink
Alexander Rink
10 years ago

Is the lack of strong data analysis talent a hurdle for retailers? I would say the proof is in the pudding, the pudding being Amazon. It is not unreasonable to conjecture that Amazon’s success is in large part due to their focus on data and analytics. Can retailers compete with iBanks? Probably. Although it certainly plays a large role, money isn’t everything when it comes to choosing work and I’m sure there are data scientists out there who don’t want to live in the traditional iBanking cities, or who prefer different kinds of challenges.

As for alternatives, as some of the other comments have mentioned, many retailers are relying on third-party solution providers to perform the data analysis for them. This is likely the best approach, as the solution providers tend to focus on specific areas and can attract the top talent to work on the challenging problems in their chosen area of expertise.