In countless ways, data is the fuel that drives business today. There are media monitoring and media intelligence solutions that analyze billions of online news data points and synthesize insights into content performance, industry and social trends, and brand equity. There are Large Language Models (LLMs) – machine learning models that power solutions like ChatGPT and are trained on massive amounts of news data. And there are thousands of other applications like risk management and financial monitoring solutions that are driven by data.
It’s no secret that the Internet is basically one giant news dataset and that it’s free. In some cases, free news datasets can indeed be sufficient for specific, ad-hoc purposes.
Yet it’s important to keep in mind that a key challenge facing organizations today is not too little data. In fact, it’s quite the opposite. The challenge is that too much noisy and messy data make scaling actionable insights challenging.
This means that the question data stakeholders need to ask is not how we get more data. Rather, it’s how do we get the data we need to produce the financial, media, reputational, market, sentiment, regulatory, and other insights that will drive our business forward?
In this post, we’ll examine how organizations can generate more, faster, and better insights – with free news datasets or with a paid news API?
What is a free news dataset?
A free news dataset is just that: a dataset available without charge that consolidates news data from around the web, often covering a wide range of different news sources, languages, countries, and categories.
Free datasets offered by commercial data providers like Webz.io are used by leading organizations and universities around the world for predictive analytics, risk modeling, NLP, machine learning, sentiment analysis, and more. There are also open-source datasets offered by nonprofits like Common Crawl – a repository of non-curated web crawl data going back to 2008 that contains petabytes of data obtained from billions of web pages with trillions of links
What is a news API?
An API (Application Programming Interface) is a tool that enables different types of software to exchange information and data. A news API is how applications can communicate with various commercial online news sources. Some news APIs are specific to a news site. All the big online news providers have them: NYT, Bloomberg, BBC, The Guardian, and more. These APIs allow applications to scan, extract, analyze, and enrich data from their particular news source, then use that data for a wide range of purposes.
There are also news APIs that offer news data feeds at scale – from millions of sources (like Webz.io). Powered by AI, these advanced news APIs use Natural Language Processing (NLP) and Machine Learning (ML) to recognize categories, sentiments, topics, persons, dates, events, and other parameters in data collected and parsed data from news websites. This data is then tagged with contextual metadata and delivered in a standardized format, which software can use.
By using a news API, companies can access more relevant live data, more efficiently. This drives actionable insights, which facilitates better decision-making. Using a news API, a monitoring company would be able to, for example, better advise clients to discontinue working with an existing supplier, invest in a new company, or run a PR campaign in reaction to backlash against a new product.
Three initial questions to ask when choosing
When choosing between a free dataset versus a news API, first ask yourself:
- Is the data live or historical? To capture a true perspective, data needs to be fresh – ideally in real-time. Many free news datasets offer only limited historical data, whereas a news API can enable a live stream of fresh data.
- Is the dataset downloadable or continuous? The datasets offered by many free dataset providers are finite and downloadable only and thus not updated to-the-minute. A news API offers access to feeds – with fresh news data supplied to applications as it is obtained.
- Is the data structured or raw? Free datasets often have a lot of noise and unwanted content – offensive content, data from spam websites, raw HTML and code, boilerplate text (navigation menus, error messages), Lorem Ipsum text, and duplicate content. A news API from a reputable, professional provider will eliminate most noise.
Which option is right for you?
There are many advantages to using a free news dataset – most notably, the cost. Also, these datasets are immediately and readily accessible. Free datasets are a good fit for very specific use cases, and for companies that have a technical team capable of building and maintaining an infrastructure that can scale despite messy or limited datasets.
At the same time, like in any domain, ‘free’ often carries a price tag. It’s important to understand the options available and the cost-benefit implications before choosing to base key business decisions on a free news dataset.
For example, for large language model training, it is quite possible to use a dataset from Common Crawl. The dataset is, of course, free. However, in general, data teams spend nearly 40% of their time cleaning and preparing the data for AI or ML models. Keep in mind that these are high-salaried, in-demand professionals spending nearly half their time on the data equivalent of manual labor. What’s more, over 80% of the data used to train GPT-3 (the tech behind ChatGPT) came from Common Crawl – and one estimate put the overall cost of scraping the data, hosting the data files, and manually cleaning the data at some $400,000. That’s a rather steep price tag for a free dataset.
Advantages of a news API
Choosing an advanced news API offers numerous advantages, including:
- Broad coverage – Free news datasets tend to be limited in scope (often drawing on just a handful of publications) in comparison to the coverage offered by a paid news API. This limited scope of data can skew insights and negatively affect decision-making accuracy.
- Better scalability – It is very difficult to scale with free news datasets. True insights are generated at scale. Structured data helps scaling faster (otherwise they’d need to clean the datasets on their own – which is costly and time-consuming. A top-tier news API will automatically discover and classify new sources of relevant data while enabling granular data analytics with adaptable and automated classification.
- Better customizability and granularity – Free news datasets are generally supplied ‘as is’, meaning there’s no option to filter and adjust the datasets to unique organizational needs. News APIs offer advanced filters and enrichment layers to ensure that datasets delivered are exactly to spec and scale.
- Higher quality – Data professionals spend over a third of their time cleaning and preparing data. Many free news datasets are imperfectly structured or contain noise. Noisy data is a huge resource consumer, and this takes time and energy away from the goal of data: creating insights. A news API provides structured web data that facilitates actionable insights at scale.
The bottom line
Choosing the right news dataset source can make or break the quality and value of the insights created by the application using the data. There is plenty of free information of varying quality and accuracy online, and in some cases this level of quality is sufficient. In other cases, it’s worth examining a curated, scalable, customizable, and timely option like Webz.io’s News API.
Ready to generate more accurate and actionable insights from news data? Talk to one of our experts today!
