The vast majority of us use the web every single day – for news, shopping, socializing and really any type of activity you can imagine. But when it comes to acquiring data from the web for analytical or research purposes, you need to start looking at web content in a more technical way – breaking it apart into the building blocks it is composed of, and then reassembling them as a structured, machine-readable dataset.
In this post we’ll be covering three of the basic steps of transforming textual web content into data – regardless of what data extraction technique you choose to apply in order to do so. This should also help you understand some of the terms you commonly hear thrown around in discussions about web data, starting with…
1. Crawling
A web crawler (or spider) is a script or bot that automatically visits web pages, typically creating a copy of their contents for later processing. The web crawler is what actually grabs raw data from a webpage – the various DOM elements behind what the end-user sees on his or her screen – which is a necessary prerequisite to any further action we would like to take with this data.
Basically, it’s a bot that browses the web and clicks on the metaphorical ctrl+a, ctrl+c, ctrl+v buttons as it goes along.
Crawlers are used by search engines, research companies, and data providers, making them quite common around the web.
Typically, a crawler would not stop at one webpage, but instead crawl through a series of URLs before it stops, based on some predetermined logic – for example, it might follow every link it finds and then crawl that site, then follow the links it finds there… however, you would generally want to crawl in a smarter way, prioritizing which websites you crawl vs the amount of resources (storage, processing, bandwidth, etc.) that you can devote to the task.
2. Parsing
In general, parsing means extracting the relevant information components from a dataset or block of text, so that they can later be easily accessed and used for additional operations.
In the context of web data, your crawlers would generally grab a whole bunch of HTML – which includes the text that the user sees along with a lot of instructions meant for the browser regarding how to render the page, and various related information sent from the website to the end user.
To turn a webpage into data that’s actually useful for research or analysis, we would need to parse it in a way that makes the data easy to search, categorize and serve based on a defined set of parameters. For example, here is how Webz.io would parse a message board post:
Wait, so what’s scraping?
Scraping, as it is commonly used, is basically the first two steps of this process – crawling a webpage and extracting the data. The difference here is mainly in terminology rather than essence.
3. Storage and Indexing
Finally, after you’ve gotten the data you want and broken it into useful components, you would want to find a scalable way to store all the extracted and parsed data in a database or cluster, and then to create an index that will allow end users to find relevant data sets or feeds in a timely fashion.
In Webz.io’s case, since we deal with fairly massive amounts of web data (as a result of the high level of coverage we provide), and because our clients naturally expect constant uptime, we manage the hardware infrastructure in-house – running on our own server farm rather than AWS, Azure or similar. For indexing we use ElasticSearch.
The Result: Neatly Organized Data, Ready for Analysis
To summarize: transforming a piece of web content into machine-readable data is a 3 step process. You have to send your bots to crawl the data; then you have to parse it into logical components; and finally you need to find a scalable way to store and index the data so that it’s easy to search and retrieve specific information from it.
If you’re trying to extract data from a single website, or even a hundred websites, this is simple enough – but to do it at scale requires some expertise and understanding of the technological hurdles and optimizations available at each of these steps.
To learn more about web data extraction, download the complete guide right here.
