When will predictive models become more predictable?
Photo: Getty Images/halbergman

When will predictive models become more predictable?

Retailers and brands that used predictive models were forced to constantly retrain and make adjustments around their machine learning tools as historic data became unreliable during the pandemic. While the “new normal” is becoming more evident as vaccines roll out, changes in buying patterns are still volatile enough to continue to frustrate forecasting efforts.

“The question is: How quickly can we adapt and retrain ML models?,” said Dan Simion, VP of AI & analytics for Capgemini North America, in a recent interview with VentureBeat. “Not only do the models need to be rebuilt or redesigned based on new data, but they also need the right processes to be put into production at a pace that keeps up. Until there is some sort of stability, it will continue to be difficult for organizations to identify consistent trends.”

A recent Wall Street Journal article said the pandemic accelerated moves to lean more heavily on real-time live data, including website activity, to drive predictive software, although some companies abandoned pre-pandemic predictive models altogether.

“The problem in highly volatile times is that sales data of last week is not a good predictor of sales in the following week,” Alex Linden, research vice president at information technology research and consulting firm Gartner, told the Journal.

For its part, Walmart in a shift that predated the pandemic by several months, began tracking longer-term trends over several years and decreased its emphasis on the most recent year. Ravi Jariwala, a Walmart spokesman, told the Journal the lengthier data set “helped us normalize the data and more accurately perform demand forecasting.”

In a column for the Harvard Business Review, Angel Evan, president at Particle Inc., a data analytics firm, stated that the pandemic-driven challenges facing historic data should serve as a reminder that customer data is not limited to point-of-sale transactions. He wrote, “Retailers should consider data to be any information that is relevant to their customers’ behavior that can be ethically collected, organized, and studied for insights that decision-makers can rely on.”

BrainTrust

"Investing in predictive models is necessary, but this must go hand in hand with other initiatives – such as improved inventory management."

Jennifer Bartashus

Senior Analyst, Bloomberg Intelligence


"If adaptation to emergent market conditions requires that the model be re-constructed by human beings, that’s a huge watch-out."

James Tenser

Retail Tech Marketing Strategist | B2B Expert Storytelling™ Guru | President, VSN Media LLC


"The enterprise’s brand promise and entire customer experience ultimately hinge on decisions made at this level."

Gib Bassett

Director, Solutions Marketing with Alteryx


Discussion Questions

DISCUSSION QUESTIONS: What challenges continue to limit the use of predictive models in supporting forecasting? Has the pandemic offered any lessons around the reliance on historical data or the capabilities of machine learning?

Poll

22 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Mark Ryski
Noble Member
2 years ago

Predictive models are only as good as the assumptions used to build the algorithms and the quality of the underlying data to inform them. As the article points out, all bets were off given the impact of the pandemic. Prediction and data will continue to be extremely important as retailers re-calibrate post-pandemic, but the bigger lesson should be that retailers shouldn’t blindly believe that they can “algorithm their way to success.” The physical world is extremely complex, and so even accounting for all the variables that can impact sales demand is a challenge. The best advice for retailers is to view trends from numerous data vantages, and then connect the views to make decisions – judgement will always be required.

Bob Amster
Trusted Member
Reply to  Mark Ryski
2 years ago

I could not agree more. One can predict very accurately in a steady state, however we live in a world of ever-changing forces and conditions. Under that reality, predictive models will only work so far. Beyond that, intuition and real-life experience will drive the decisions to be made.

Dr. Stephen Needel
Active Member
2 years ago

The over-reliance on data trends is what dooms a lot of these models. They are not modeling how people think or shop, they are modeling data. When the data gets weak, like during a pandemic, a flood, bad weather, or a stock market correction, then the models get weak – they can’t represent the underlying reasons for the data. Remember that machine learning is stupid – the machine is looking for patterns the user has identified and told the machine “this pattern means that.” At this stage of development, one relies on AI or ML at their own risk.

Ken Morris
Trusted Member
2 years ago

Today, real-time data is a foreign concept to analytics companies. They work on yesterday’s — or even last week’s — data. We need to understand what is happening now in order to capture the KPIs that influence our customers. Predictive models only work if the inputs paint an accurate, real-time picture of activity. Retailers can have all the tools needed to build this ideal “nervous system” if they’re willing to commit. What are the tools? Shopper-driven mobile checkout, RFID throughout the supply chain, sensors to monitor security and receiving activities, and a way to centralize all these IoT inputs in real time. Only then will data analytics and machine learning deliver the goods: accurate predictive models.

Also, items like supply chain disruptions and global pandemics need to be considered in scope for analytics. Mostly, we need to learn from our recent experience and weave it into our go-forward strategy.

David Weinand
Active Member
Reply to  Ken Morris
2 years ago

Ken, great post. I can’t really add much other than to say that predictive analytics models should be looking at social sentiment to learn of trends and get closer to real time.

Liza Amlani
Active Member
2 years ago

The biggest challenge that limits the use of predictive models in supporting forecasting is people.

Retailers and people just don’t like change. We are creatures of habit and if technology is going to change the way you work and retail roles, there will be resistance. Predictive analytics and data can help retailers in combination with creativity and human judgement to make the best decisions for their customers. We need to implement more data driven decisions across retail and remove some of the guesswork.

Suresh Chaganti
Suresh Chaganti
Member
2 years ago

There are multiple aspects to this, but just looking at the models – the key is in understanding what problem you are solving. The more specific you get, the more predictable the model will be. Demand forecast at a category level will be less useful when ordering individual SKUs. Demand at the regional level will be less actionable for a store.

It boils down to this: Business leaders need to have clarity on what question they are trying get a predictive insight on and how they are going to take action on it. And they have a huge role in championing the usage.

Each such question becomes an exercise in analyzing data, understanding the causal factors, testing models for efficacy and creating mechanisms for the measurement of true efficacy. Data scientists need to have a good sense of knowing the changes in the underlying business landscape, and continually test whether models are holding well.

The pandemic is an anomaly. But the underlying principles of how it should be done, or good data science, do not change.

Melissa Minkow
Active Member
2 years ago

There’s a tension between real-time and short-term data potentially being too volatile, and long-term data potentially being too irrelevant. Both approaches offer challenges in accuracy, but the more data points that are leveraged, the closer retailers can get to precision. We now have access to so many data sets beyond just sales – we have demographic, psychographic trends, to name a few. Smart retailers will create models that can digest a wide swath of data sets to mitigate as much risk of inaccuracy as possible.

Jeff Sward
Noble Member
Reply to  Melissa Minkow
2 years ago

Yes! Data has a “shelf life.” Or, data on season-less basics has cumulative meaning over the long term. Data on the hottest current fashion has limited value. It might be a best seller, but can you reorder and get delivery before the item or trend fizzles out? And then does the data have any meaning next season or next year?

Oliver Guy
Member
2 years ago

My first work on retail forecasting was in 1994 using Fourier analysis within Manugistics for shower gels in UK supermarkets. Forecasting has become much more advanced since then – the power of computing being a major enabler. But advancement has created complications – for example choosing how best to forecast – not only what statistical technique to use but from where to apply seasonality and how to account for historical events. Then there are other factors like human faith in the forecast, the interaction of other merchandising factors – price, promotion, assortment, substitutions and so on. Source data remains a big challenge – and the pandemic impact certainly made people question the validity of historical data.

Challenges still exist around forecasting. For example, where to forecast – in the past you had to forecast at the POINT OF TRANSACTION – historically this was store level because that was the place where the stock existed and demand manifested itself. Along came e-commerce and you had something much bigger from a transaction perspective. Today is different – you need to forecast for the POINT OF POSSESSION – the location or channel where the consumer actually takes possession of the item – otherwise you cannot plan optimal inventory placement. This is a great subject worthy of discussion.

Andrew Blatherwick
Member
2 years ago

The pandemic did make forecasting more complex but it did not make it valueless. In fact, the use of more granular data has helped several retailers get through this very difficult time. It is true that typical demand forecasting using only recent sales data is not going to deliver but more comprehensive forecasting that looks at longer time spans, uses different data inputs like website hits and local data at store level can provide invaluable insights even in times of high volatility.

What is most important is that the forecast is not used in isolation. It needs to be incorporated into a full supply chain optimization solution that looks at forecasts, supplier performance, customer behavior and data sets as outlined above at the lowest granular local level. Where to hold safety stock and what to hold will be decided using many inputs and factors. How to flow inventory through the different channels of the supply chain efficiently is paramount in these times and intelligent solutions are the best place to evaluate and manage that process. Using manual processes and yesterday’s sales will only lead to massive swings in the supply chain, out-of-stocks and overstocks and very inefficient supply chains. Do not throw the baby out with the bathwater — panic is no way to run a modern retail business.

Jennifer Bartashus
2 years ago

Investing in predictive models is necessary, but this must go hand in hand with other initiatives – such as improved inventory management to really deliver the benefits they can offer. Otherwise, it doesn’t matter how good you are at predicting trends and volumes needed if goods can’t be easily found or ordered by consumers. Transparency throughout the entire supply chain will also be key.

Chuck Ehredt
Member
2 years ago

Predictive analytics has existed for decades and they work increasingly well when the data fed into them is clean and the parameters are adjusted over time based on reality. The problem we see every week is that companies have not been able to get all the relevant data into one place so it can be used appropriately. This not only affects purchasing, but personalization. Since I´m involved in innovation, I sign up for just about every new service to learn how they work. But even after months of engaging with a new service, only one out of 100 will start to call me Chuck (rather than Charles). The problem is not with the models, but with too many people introducing anomalies and too few taking responsibility for proper execution.

Jeff Sward
Noble Member
2 years ago

Predictability does not lend itself to a one-size-fits-all solution. The model I have come to adapt in the apparel business has four levels of predictability, ranging from basics, to key items, to trend, to novelty/fashion in descending order of predictability. In the most stable of times the predictability of trend and novelty/fashion product is going to be limited. That’s the nature of the beast. It’s all about data + design. Data is a huge factor at the basics level. And data has a very small role at the novelty/fashion level. It could have a larger role if — big if — the supply chain could respond to best sellers in season. When you are developing and buying product six to 12 months out, the data on basics is a heck of a lot more useful than the data on novelty/fashion. That’s when the novelty/fashion level is pure guess work. When I worked in China we could test and have almost any product on the floor in three to six weeks. All of a sudden data has a huge role in the novelty/fashion level of the business. When you are only buying three to six weeks out it is impossible to get into too much trouble. So it become a function of all three variables. Data + design + supply chain = level of predictability.

Ryan Mathews
Trusted Member
2 years ago

Predictive models are excellent tools for telling you what happened in the past, but are severely limited in terms of their ability to live up to their name, i.e., “predict” demand. At best, many of these models become self-fulfilling prophecies. Ship inventory in anticipation of demand and that’s what you are going to sell since that’s all you have to sell. If COVID-19 has taught us anything, and that’s a dubious assumption, it ought to be that the future is inherently unpredictable. Predictive models assume continuity – that the future is somehow a linear projection of the present and, more critically, the past. That may help you maintain sales or score incremental gains, but it’s not a path toward exponential returns.

Venky Ramesh
2 years ago

It’s great to see a quote from my friend Dan Simion. I agree with him – the entire approach to planning demand needs to be rewired. In the past, planning systems have provided a functionality called “suppression” that allows forecasters to suppress a portion of history specific to a location that has seen a temporary disruption so that the algorithms don’t use that data for projecting out, instead, using data prior to and after the event. However, the pandemic has brought in a massive change in consumer behavior, consumption pattern, and shopping journeys. Past data is highly questionable under today’s circumstances. CPG and retail companies need to figure out how to leverage real-time data more effectively in their planning processes – and get towards a more autonomous supply chain.

Gib Bassett
2 years ago

What challenges continue to limit the use of predictive models in supporting forecasting? I think models are already used widely, the question is more about improving their utility and value to the business. To that end, it’s about relying less on internal and historical data, and more on blending those sources with external and real time factors that might influence demand even before customers express intent.

Has the pandemic offered any lessons around the reliance on historical data or the capabilities of machine learning? Clearly yes, relying on backward looking data creates a blind spot that hurt many companies and even those with strong analytics capabilities struggled to adapt their operational supply chain ML systems. The assumptions on which they were based didn’t recognize the huge variance in data versus what they were trained on when faced with mass panic buying.

The net of all is that disruption is going to continue into the foreseeable future, so you must improve, but this presents an opportunity. Demand forecasting should arguably become a core analytic competency subject to the same test and learn cultural iteration of any transformational AI use case. The enterprise’s brand promise and entire customer experience ultimately hinge on decisions made at this level.

Peter Charness
Trusted Member
2 years ago

I’m starting to see the glow on AI/ML solutions fading a bit, as their seemingly (and over hyped) unbounded potential to solve every and any problem has met reality. There are certainly high benefit use cases for predictive/ml models, just not all things to all people. When prediction is just not possible, retailers also need to have the ability to be agile and quickly react to new trends/challenges. I think a balanced and realistic approach is the winner here.

James Tenser
Active Member
2 years ago

In the Incredible Dissolving Store, predictive models can handcuff decision making by locking the organization into a finite set of assumptions based on inward-looking, historical trends. If adaptation to emergent market conditions requires that the model be re-constructed by human beings, that’s a huge watch-out.

Business forecasting remains an essential discipline, but errors can become systematic and recursive — as when poor inventory data results in worse space, assortment, and pricing decisions, which in turn undermine the integrity of the demand signal.

There is a role here for AI and machine learning to enable better and faster decision making. Models based on data-mining (versus human designed “algorithms”) have potential to enable faster response to market shifts and more accurate predictions.

The critical factors for forecasting may lie outside the enterprise. One may imagine that the critical indicator for a merchandising decision may be a trend detected on social media. Much more predictive than sales history from your POS.

Craig Sundstrom
Craig Sundstrom
Noble Member
2 years ago

When? 2 years, three months, 6 days…? (OK, OK, but it’s hard to pass up a joke about asking us to predict about “predictive models.”)

Unfortunately, those hoping for Delphic truths to emerge from the Pandemic are likely to be disappointed: no model can account for the infinite number of possible things that can happen in the world, and even with good models one is frequently left with choices that are small in number but mutually exclusive … what’s the best path when you have a 50% chance of boom, and a 50% chance of a depression?

Doug Garnett
Active Member
2 years ago

This is quite concerning. Manufacturers usually are quite tuned into their product’s sales across a set of stores. Retailers would be far more successful to demand that manufacturers keep track and somehow participate in stocking. That would solve their internal manpower problem.

The idea of turning it over to machine learning has many problems. For example, machine learning doesn’t reveal “why” an answer is given only that the answer is given. That makes these algorithms brittle — incapable of handling a rapid change.

It is also worth noting that retail acts as an ecosystem — not an engine. In an ecosystem, unusual change is not predictable based on anything. (Proven fact in scientific work.) Yes, there are situations we know we can NEVER predict with or without AI.

And that brings us to Wiener’s Law #19: Digital devices tune out small errors while creating opportunities for large errors. Success with ML likely means generally stable inventory times with incredibly unstable times every now and then.

Radim Jung
Radim Jung
2 years ago

The issue here is that demand forecasting trends seem to be quite obsessed with AI/machine learning principles. But they are (pandemic or not) mostly effective with big data and truly have value over fast-moving everyday goods like milk, or bread. If you apply these algorithms to businesses that only sell a couple of items for each SKU, then they will have trouble forecasting. This is not to say anything about traditional (good old math) algorithms, which still work just fine. So for any forecasting to work, it needs to incorporate both principles and test itself on the results to understand when AI should be used and when it should be avoided. If such principles are in place then the pandemic cannot stand in the way of accurate forecasting.