While structured web data presents exciting possibilities in many fields of endeavor – including finance, cyber-security, artificial intelligence and more – the market for data extraction platforms is still fairly young. Only a handful of companies are providing online data at scale, and unlike other technologies which are covered extensively by analysts and professional publications, very little research into what drives buyer and vendor behavior in this sphere exists.
Although we definitely have a horse in this race, it was fascinating to see the results of a survey we recently ran. As part of our ongoing efforts to improve our own platform, we surveyed tens of thousands of people who have signed up to use Webz.io over the past few years, and received hundreds of responses that provide some interesting insights into which companies are looking for a web data collection service, and what they’re looking to get.
Here are the key takeaways:
Buyers come from a diverse range of industries, but the usual suspects are the majority
First, who actually answered the survey? Which types of companies are looking for web data solutions?
Industry Breakdown
No big surprises here: data analytics and digital solution providers lead the pack, with the rest spread evenly among a wide range of specializations – including finance, cyber-security, and education. Still, interesting to see the high diversity within this group, demonstrating how access to crawled web data is becoming a common necessity across many industries, perhaps as a side effect of the increasing reliance on data and analytics.
More data and less technical barriers are on everyone’s wishlist
What would you change about how you currently collect data?
When we asked people what they were unhappy with in their current data collection, the most common answer was that they need more data. This is reasonable, considering the ever-expanding nature of the web and the challenges associated with acquiring data from disparate sources. The technical challenges associated with extracting, parsing and indexing the world wide web also scored high on the list of buyer pain points.
Homegrown solutions are still common practice
What do you currently use to collect your data?
A majority of respondents reported currently using internally developed solutions to acquire data from the web, and another large group currently relies on scraping tools. This makes sense since, as mentioned, it’s a fairly new market; so far, companies have mostly been relying on patchwork scrapers written by their own developers on an ad-hoc basis. As online data collection becomes an integral part of more business processes, we expect to see a shift towards solution providers.
To understand the difference between these approaches, you can read read our article about how companies extract data from websites.
Price and data quality are the most important factors in the buying decision
What matters most to you when choosing a data collection solution?
When it comes to purchasing a these solutions, it turns out that if there’s one thing that’s more important than having a lot of crawled data, it’s the quality of what you get. A whopping 39% reported this was the most critical part of their buying decision, with other notable factors including pricing and the volume of available data.
The Bottom Line: The Market is Maturing
The ability to transform online content into structured data streams was traditionally seen as a niche requirement for very specific industries, such as web monitoring, or as a tool employed mostly by industry giants such as Google and Microsoft. However, as data analytics tools and skills proliferate in both new and traditional industries, web data is enjoying growing mainstream appeal.
Today’s buyers understand the most important factors in a web data solution are breadth of coverage, quality of data and ease of use, along with sensible pricing; and while many still rely on home-grown solutions, their limitations in these areas are quickly becoming clear.
These trends are expected to continue well into the future, where the web is likely to play an even larger part in the lives of even more people, along with the increasing prominence of analytics and data science.
Do you have a different take on the survey results? Tell us in the comments! Or learn how web data can help your business by scheduling a talk with one of our experts.
