Ran Geva

Web Data

Tiny basic multi-threaded web crawler in Python

If you need a simple web crawler that will scour the web for a while to download random site’s content – this code is for you. Usage: [crayon-676d05cd00e84830317351/] Where https://cnn.com is your seed site. It could...

Web Data

How we quadrupled the performance of Elasticsearch

Well, that’s a misleading title. We actually quadrupled the performance of our brand monitoring alert system that uses Elasticsearch’s Percolator, but that would have been a much longer title. Some background Buzzilla...

News API

Webz.io Tip: Search for top performing (viral) posts

Here at Webz, our crawlers download millions of posts a day from millions of sources. When searching for web data among these many sources, you may want to limit your results to...

Web Data

Building a Better Search Query

Many factors can affect streaming data relevancy. When the data you consume isn’t ordered by relevancy, rather by the time it was crawled, getting the relevant posts is essential. I would like...

Web Data

Webz.io Tips & Tricks: Content Marketing & SEO

I would like to share with you 2 simple tips about how to leverage Webz.io to promote your website, product or service organically.

Web Data

Webz.io Tips & Tricks: Search for Reviews

Are you looking to focus your data search specifically on consumer generated reviews? Here are a couple of simple Webz.io tricks that might help: Limit your query to specific sites You can...

Web Data

Vertical Aggregation and Pattern Matching Crawlers

After bashing various crawling techniques, I would like to describe the technique we use here, at Webz.io, a technology that was developed over the past 8 years. Our crawlers were developed with...

Web Data

Crawling Horrors – Computer Vision Crawlers

So if RSS Crawlers are bad, Browser Scraping isn’t efficient, what about computer vision web-page analyzers? This technology uses machine learning and computer vision to extract information from web pages by interpreting...

Web Data

Crawling Horrors – Browser Scraping

In my previous blog post, I wrote about RSS crawlers, and why they don’t really work. In this post I want to discuss the technique of using a headless browser to parse...