If you read business or tech publications, you’ve probably heard about the ‘explosion of data in the business world’. There has certainly been no lack of voices shouting about it from every rooftop:
That a claim has become clichéd does not, however, make it inaccurate. It is true that the internet, digitization, storage and other technologies have contributed to data becoming far more abundant in a 2017 business compared to a similar shop in 1997. What’s more, today’s workers are far more technologically-inclined than their predecessors, and have more digital tools at their disposal. Talking about data has itself become more pervasive and ingrained in corporate culture.
Then there is more data, and everyone is talking about it; but is this talk actually adding value to the organization? To me, the answer is yes, but not to the extent others would have you believe. The current model of ‘data-driven business’ still has some major flaws, and its application often results in discussions where fancy graphics hide a lack of valid statistical evidence.
The bad news is that contrary to what tech marketers might have you believe, there is no software that currently solves this problem. The good news is that we are rapidly approaching a future where AI-based tools actually will be able to drive more intelligent and meaningful use of data in the workplace.
The Last Mile
In transportation planning there is a lot of discussion about the “last mile” problem: there could be a very effective system for transporting people (or goods) long distances, such as high-speed trains; but the last part of the journey, from train station to home or office, tends to be slower and inefficient.
Analytics suffers from a similar problem. We have optimized everything about the way data is stored, processed, and accessed. An executive who wants data, has data. But that last, crucial mile – from data to the revered actionable insight, is still far from effective. It still relies on line-of-business workers who usually have little to no formal training in data to look at the numbers, identify the ones that matter, and understand how they correlate.
Some of the people engaged in this impromptu data analysis might have a knack for it; most do not, and the results are as one might expect. Yes, the business world is talking about data; but that conversation often lacks direction, expertise and context.
Signal from Noise
But wait, you might be saying, isn’t that what data visualization and analytics tools are for? Unfortunately, this type of software doesn’t usually make business users better at analyzing data. Instead, these tools are meant mostly for different, albeit important, tasks:
- Back-end tools extract, collect, and query data – which are IT-oriented tasks that business workers never needed to deal with in the first place.
- Data providers, such as my own web data as a service startup, can enrich existing organizational information with data from external sources.
- Front-end tools let users create interactive dashboards; alas, a visualization is just a visualization, and only means as much as the data that is being presented. For a more provocative take on this, check out Martin Butler’s Visual Analytics – Just a Dumb User Interface.
Unlike Mr. Butler, I believe that these are all amazing developments and immensely useful; but none of them magically transform a sales team leader into an experienced data analyst. I’ve yet to encounter a system that can effectively identify business questions, meaningful KPIs and the factors driving them. The exceptions are either ‘self-service’ tools that actually rely on extensive work by the vendor’s implementation and services team, or software that comes with a pre-existing data structure that is managed on the vendor’s side, such as Google Analytics.
Still Expensive and Cumbersome
There is yet another elephant in the room: all of these data tools tend not to come cheap, and you still won’t get very far with them if you don’t know what you’re doing.
For the first point, consider IDC’s prediction that worldwide revenue for big data and analytics will grow to $187 billion in 2019. Needless to say, that expected revenue is going to come from businesses purchasing the software. Regardless of whether this specific figure is accurate, anyone who is involved in business software purchasing can tell you, tearfully, about how much money is being spent on data, and how meager the returns often are.
As for the second part, think about your favorite analytics or reporting tool of choice. Did someone need to write a bunch of SQL (or similar scripting) code to get it to work properly? In 9/10 cases the answer is yes. The entirely self-sufficient business worker is all but a myth in most organizations.
In other words, the business world is spending a ton of money, and employing a ton of technical expertise, in order to have people who don’t really understand data and statistical analysis talking about numbers that are often inaccurate or statistically insignificant.
Not All is Lost
This conclusion might seem bleak, but I do believe there is hope for better, more accurate use of data in the business world, thanks largely to expected developments in artificial intelligence.
To avoid creating more hype, I should clarify that AI will also not be the magic bullet it is made out to be. An AI-driven data analytics system will not give you crisp, instantaneous action items such as “increase marketing spend in China” at the flip of a switch. It will not ask your questions or run your business for you.
However, what AI can do, is take on some of the heavy duty of creating complex statistical models based on a large amount of data points. It can apply learning algorithms to adjust the weight given to specific sources of information, and it can potentially identify patterns and correlations that are independent of the biases the user came in with (Rita Sallam of Gartner has written similarly about the concept of augmented analytics).
The Next Frontier
The next step would be adding external data: whereas currently the business world is having enough trouble getting a handle on the data it already has, a well-designed predictive analytics model would not shy away from adding new data sources, and in fact would try to ‘ingest’ as much as it possibly can. Many of our clients use Webz.io to generate training datasets for their AI algorithms – the results I’ve seen are very impressive.
This opens new possibilities for combining external and internal data for more sophisticated analysis. At Webz.io we are seeing this with web data – which is sprawling, highly disparate, and constantly changing. An executive or even a business analyst can typically do very little with this type of data; but by applying artificial intelligence algorithms, companies like IBM Watson are using this data to drive smarter analyses in specific scenarios, based on a wealth of internal and external information.
Cautious Optimism
While the present of ‘data-driven’ is far from the ideal picture that others might try to paint, the future looks much more promising. Once pointed in the right direction, AI can potentially bridge the gap between the questions a non-technical individual has about his or her business, and their ability to actually answer these questions using the data at hand; and it could enrich business decisions with a stream of data from external sources.
For now, we shall wait, see, and beware of anyone promising an easy fix when it comes to data.
