What is the Omgili Bot, and why is it Crawling Your Website?

What is the Omgili Bot, and why is it Crawling Your Website?

Hi there. If you’re reading this, it’s probably because you’ve run into Omgilibot – perhaps in your web analytics or server logs (user agent: omgili/0.5 +https://omgili.com) – and turned to Google to…

3 Predictions for Web Data in 2018

3 Predictions for Web Data in 2018

2017 was a turbulent year: With Donald Trump shaking up the American political system, cryptocurrencies causing riptides throughout financial markets, and advancements in artificial intelligence sparking both anticipation and anxiety in the…

Was Black Friday Worth It? The Crawled eCommerce Data Answer

Was Black Friday Worth It? The Crawled eCommerce Data Answer

Now that we’ve had a chance to recover from the Black Friday and Cyber Monday crazes, which chalked up record sales this year, it’s time to ask the inevitable question: was Black…

Richer Media Analysis with Broadcast News Transcripts

Richer Media Analysis with Broadcast News Transcripts

Learn how to use broadcast news transcript to enrich your media analysis.

3 Steps to Turn Webpages into Machine-Readable Data

3 Steps to Turn Webpages into Machine-Readable Data

The vast majority of us use the web every single day – for news, shopping, socializing and really any type of activity you can imagine. But when it comes to acquiring data…

Quick Guide to News APIs

Quick Guide to News APIs

All you need to know about News APIs.

How To Get The Data Your AI Application Needs

How To Get The Data Your AI Application Needs

The more data you have available to use, the more you can reduce uncertainty in the functioning of your machine learning program, whether for training or for deriving insights. Learn how web data can help.

There's An Exciting New Way to Access Dark Web Data

There's An Exciting New Way to Access Dark Web Data

For the past few months we’ve been rolling out a new feature that’s already garnering a lot of excitement. Now that it’s officially live and actually being used by our customers, it’s…

3 Ways to Use eCommerce Product Data for Market Research

3 Ways to Use eCommerce Product Data for Market Research

The web is an invaluable source of data when it comes to competitive intelligence, research and creating a go-to-market strategy. A simple Google search will reveal incredible amounts of public information about…

How to Extract Data from Websites: Scraping Tools, DIY or DaaS

How to Extract Data from Websites: Scraping Tools, DIY or DaaS

This is part 2 of our guide to web data extraction. Read part 1 to learn about the questions to ask before you start, or download the complete Web Data Extraction Playbook…