Will a data scientist shortage hurt Big Data’s promise?

Through a special arrangement, presented here for discussion is a summary of a current article from the blog of LiftPoint Consulting.

Data scientists are a new breed on the marketing team, with expertise across a diverse collection of areas, each of which can have a positive impact on marketing initiatives, separate and combined.

Yet the McKinsey Global Institute predicted that "by 2018 the United States could face a shortage of between 140,000 to 190,000 people with deep analytical skills, as well as a shortage of 1.5 million managers and analysts who know how to use the analysis of Big Data to make effective decisions."

A survey by Robert Half Technology concurs, suggesting that "most companies aren’t maximizing their data collection and don’t have the people in place to do so."

These hard-to-find people can be your tour guides through the complexities of data. They combine computer science, statistics, math, and business skills with creative problem solving and clear communication to help you make marketing sense of all that data.

In recognition of their value, marketing and analytics are already merging in the C-suite, with CMOs increasingly aligned with their CIOs. In 2012, only 36 percent of CMOs said their CIO was a critical partner; by 2014, that percentage grew to 51 percent, according to "The Evolved CMO In 2014," a joint research project by Forrester Research and Heidrick & Struggles.

Data scientist shortage

There is urgency too. Forrester Analytics’ State of Customer Analytics 2014 report concluded that analytics is no longer an option, but a necessity for any organization to compete. Every organization in every industry needs a senior-level data specialist on their marketing team.

Data scientists can translate all the analytic mumbo jumbo into concepts and theories that non-analytic marketers can understand. They recognize that a given business question exists inside the context of a given company and industry, and the nuances those outside influences play on the technical work of solving the business problem.

Many firms outsource their data scientist needs if they can’t afford or can’t find the correct skill set. The benefit that an external data scientist brings is a perspective on marketing from outside your company ("best practices") that can be leveraged to save time and improve results for your team.

The days of marketing as a "creative" fraternity are over. Today’s marketers need a data translator to help question, discover, interpret and, ultimately, succeed in today’s data world. The era of data scientists has arrived.

BrainTrust

"For what it’s worth our data shows the exact same thing — a shortage of talent. I think this puts pressure on the vendor community to provide easy-to-consume data. They can hire the talent and make it easier to use."

Paula Rosenblum

Co-founder, RSR Research


"There are plenty of vendors that will slice and dice a retailer’s data and provide reports ad nauseam. The problem comes when the retailer tries to use the data to improve and change outcomes. That’s where that special person who, as Mark points out, "can translate all the analytic mumbo jumbo into concepts and theories that non-analytic marketers can understand" shines."

Marge Laney

CEO, Alert Tech


"We must be careful here. As Mark and Marge point out, the person needed to handle this data is not just a data person. Maybe not even a data person."

Gene Detroyer

Professor, International Business, Guizhou University of Finance & Economics and University of Sanya, China.


Discussion Questions

How important is it for retailers and brands to leverage data scientists? What are the barriers that can prevent a data scientist from succeeding on a retail marketing team?

Poll

14 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Paula Rosenblum
Paula Rosenblum
8 years ago

For what it’s worth our data shows the exact same thing — a shortage of talent. I think this puts pressure on the vendor community to provide easy-to-consume data. They can hire the talent and make it easier to use.

Marge Laney
Marge Laney
8 years ago

There are plenty of vendors that will slice and dice a retailer’s data and provide reports ad nauseam. The problem comes when the retailer tries to use the data to improve and change outcomes. That’s where that special person who, as Mark points out, “can translate all the analytic mumbo jumbo into concepts and theories that non-analytic marketers can understand” shines.

I would argue that the person described in the article is not a pure data scientist. Data scientists focus on the details of data sets and methodology. The person Mark describes lives between the data scientist and the non-data scientist as the embodiment of an unusual mix of computer science, statistics, math and business skills with creative problem solving and clear communication skills.

I think they are the new CMO.

Zel Bianco
Zel Bianco
8 years ago

I don’t think anyone would question that analysis is necessary, and as the article states (and we can all see as we go about our daily lives) most data is not being fully utilized. Data scientists are essential, as are tools and education to help non-analytic marketers and business users better understand and interact with the information available.

Gordon Arnold
Gordon Arnold
8 years ago

It is my opinion that big data is a runaway nightmare. The problem is redundancy caused by a never-ending supply of incompatible data file structures that must be polled separately for information reporting purposes. An example of this is demonstrated by many of the internet search engines that report identical information offerings/statistics and count them as separate or new. Some search engines have filters that are automatic or manual used to reduce multiple reports of a single event and the confusion users are left with. This process only adds to the need for more support and system resources. The proposition that business needs more data techs is further demonstration that the problems we have with big data are only getting more out of hand in an industry that was founded to put an end to repetitiveness and increase productivity.

Cathy Hotka
Cathy Hotka
8 years ago

I participated in a Wharton School of Business conference this week, and the importance of data science was a common theme.

There’s a huge disconnect in this country. We’ve gold-plated our colleges and now 80 percent of students need financial assistance. We need to make it easier for our students to to acquire the skills necessary to make money in the knowledge economy.

Vahe Katros
Vahe Katros
8 years ago

At first I thought of the movie The Imitation Game and imagined retail offices taking on a look like Bletchley Park, then I thought of a near future with an abundance of unemployed data scientist-lites who heard this article’s call to action only to be later displaced by improved tools, then I wondered if McKinsey used data science to construct this “click-bait” scenario. Then I realized that I was in trouble and took my medication. Do I get the job?

Gene Detroyer
Gene Detroyer
8 years ago

We must be careful here. As Mark and Marge point out, the person needed to handle this data is not just a data person. Maybe not even a data person. We are educated to analyze, but handling big data by analysis will just get you bogged down in the trees and you will miss the entire forest. Big data is a picture of a dynamic system at work.

The person who will see it the best will be a synthesizer, not an analyzer. They are hard to come by.

Graeme McVie
Graeme McVie
8 years ago

A shortage of data scientists and managers who can use the analytical outputs will hold back retailers from realizing their full potential. I’ve been working with retailers for the past 15 years to deploy advanced analytics to help with decision making. Going into most engagements most retailers said their biggest concern was whether the analytics would work in their businesses. There was never any doubt that the analytics would work but my biggest concern was the ability of the organization to digest the outputs and use them on a consistent basis to make more analytics-driven decisions.

There’s a big difference between using analytics to identify value and working with the organization to execute decisions that will capture value. Retailers will have to invest in hiring personnel who can use analytical outputs and then ensure they provide the right training to managers on the underlying analytical principles (so managers understand where the analytical outputs are coming from) and on how to use the insights to make better day-to-day decisions.

Bryan Pearson
Bryan Pearson
8 years ago

The role of Chief Data Officer might become one of the most important positions in the c-suite.

Gartner predicts 25 percent of all large global companies will have someone dedicated to the role of data oversight by next year. Here are five facts Gartner shares about these CDOs:

1) The role is growing to the power of 10: More than 100 chief data officers—that specific job title—work at large organizations today. That’s more than double the number Gartner counted in 2012.

2) Finance and government lead the way: The role of the chief data officer is most prevalent among banking, government and insurance entities, in that order. That said, Gartner is now seeing an uptick of CDOs in other industries.

3) Most are in the United States: While there are now chief data officers in more than a dozen countries, most of them—65 percent—are in the United States. The United Kingdom accounts for 20 percent.

4) New York and the capital corner the U.S. market: More than a quarter of U.S. chief data officers work in New York or Washington, DC, not surprising since the government is a leading employer. Increased regulatory scrutiny is fueling the trend.

5) Women are major players: More than 25 percent of chief data officers are women. That compares with just 13 percent who are chief information officers. (I may have a talk with my daughter.)

These five facts converge to point out one truth: Data is among the most valuable of corporate assets today. It should be treated with the same care and scrutiny as product inventories and SG&A expenses.

By appointing a chief data officer, organizations are shifting the corporate conversation from the analytics group to the boardroom, and eventually to the front lines where consumer is. When it comes to corporate assets, nothing beats that.

James Tenser
James Tenser
8 years ago

There’s a special breed of folks out there who can’t necessarily do the math, but who can interpret findings and explain their significance to others.

Marketing decision-makers need people like these on their teams, I’d argue, even more than they need pure data scientists.

Many Big Data analytics are—or will soon be—pre-packaged. Database design innovation and new analytic tools are arriving thick and fast. Expertise in how to design the query will be increasingly at a premium in this era. That takes business acumen, supplemented by an understanding of data technology.

The new crop of Marketing MBAs can certainly write some code. That skill should help them work smoothly with the new crop of data scientists who plug them into the Big Data flows behind their dashboards.

Lee Kent
Lee Kent
8 years ago

Yes, retail needs data analysts, but may not so much data “scientists.” An entry level data manipulator who knows how to smash and mash data but more importantly, the wise old sage who knows how to match said results to the corporate mission.

If you have read Malcolm Gladwell’s “Blink,” you know what I’m talking about. Success is in finding those 4 or 5 key “realities” that will drive your business.

Let me add that I am not talking about using data to personalize marketing messages. That is just mapping. I am talking about using data to drive the business. Who really are the best customers and what makes them so. Is it really the top spenders or have we overlooked the value of influencers?

Aside from my passion for customer experience, this is the job I always wanted…but decades too early. Ahhhh to be young again.

And that’s my 2 cents!

Bill Davis
Bill Davis
8 years ago

Very important and yes, this will adversely impact the retailers and brands that don’t secure qualified personnel. The three biggest barriers to be overcome, IMO, are:

1) senior management making decisions because of what they think as opposed to what’s really happening
2) being able to change how things operate today as well as establishing and adhering to a roadmap for how to achieve the desired future state
3) finding a balance between data, intuition and common sense

Retailers will have to address each of these three items, and likely more, before data science is more widely accepted in driving business decisions.

Kai Clarke
Kai Clarke
8 years ago

The McKinsey Global Institute? How many studies does this data embrace? What about other resources? The nebulous information and data referenced here is skeptical at best. Cannot believe these conclusions from dubious positions.

Michael Day
Michael Day
8 years ago

Big data’s promise has a bit of its sheen taken away by the absence of not just data scientists, but really any other agency that is not available in full supply to take advantage of it. Data scientists, in theory, have the skill sets, the knowledge and the experience to agglomerate data from a variety of sources and of many types, cleanse the data and then apply a combination of analytics techniques to gain insights. It takes a special blend of characteristics to do even a passably decent job, let alone a stellar one, to derive these insights.

When data scientists are in short supply, it reduces the likelihood that non-data scientists can apply their innate skills to obtain insights in due time. If they do derive these insights, there is a strong likelihood that some serendipity would’ve come into play. In either circumstance, this shortage does not bode well for organizations to take advantage of their large stores of data in a timely manner and address their myriad use cases (e.g., customer churn, fraud, channel attribution, network security).

Ultimately, failing to address these use cases in a timely manner has a negative impact on a number of crucial success metrics such as revenue, sales, customer satisfaction, employee retention, reputation loss, price-demand mismatch and so on