Web Data

Hi there. If you’re reading this, it’s probably because you’ve run into Omgilibot – perhaps in your web analytics or server logs (user agent: omgili/0.5 +https://omgili.com) – and turned to Google to…

This is part 2 of our guide to web data extraction. Read part 1 to learn about the questions to ask before you start, or download the complete Web Data Extraction Playbook…

The following is an excerpt from our new Web Data Extraction Playbook. We’ll be publishing the second part next week, or you can grab the full guide here. The internet has become…

When is it okay to grab data from someone else’s website, without their explicit permission? A new ruling by a federal judge in California might have dramatic implications on this question, and…

Competitive programming competitions, commonly referred to as Hackathons, offer a great opportunity for new talent to show what they can do. Much like professional sports, industry leaders send recruiters to scout out…

Last February, co-authors Leiff Azopardi and James Maxwell completed the latest edition of their book Tango with Django. It presents an excellent step-by-step approach to learning Python on the popular Django framework…

Sifting through millions of posts on review sites presents both a massive undertaking and an incredible opportunity for influencer marketing. Some of the most successful app makers are capitalizing on that oppotunity. Use…

Sentiment classification is a fascinating use case for machine learning. Regardless of complexity – you need two core components to deliver meaningful results; a machine learning engine and a significant volume of…

We’re used to getting questions about accessing structured web data. But recently, we’ve been fielding a different kind of use case.  Researchers and scientists have been asking about data citation conventions and how…

In a technologically driven environment, the temptation to develop a proprietary web crawling solution is virtually irresistible. Our latest report examines the true cost of computing and software development resources required to deliver a data…

In our new report, we deconstruct the all-too-familiar race to achieve 100% coverage of the web. Data acquisition efforts usually rely on one of three approaches – build an internal web crawling…

The analysis you provide is only as good as the raw data you start with. Although data from the open web is often perceived as a commodity, not all crawled data is…