To crawl or not to crawl, that is the question

In order to write an efficient crawler, you must be smart about the content you download. When your crawler downloads an HTML page it uses bandwidth, memory and CPU, not only its…

Building a Better Search Query

Building a Better Search Query

Many factors can affect streaming data relevancy. When the data you consume isn’t ordered by relevancy, rather by the time it was crawled, getting the relevant posts is essential. I would like…

Webz.io Tips & Tricks: Content Marketing & SEO

Webz.io Tips & Tricks: Content Marketing & SEO

I would like to share with you 2 simple tips about how to leverage Webz.io to promote your website, product or service organically.

Webz.io Tips & Tricks: Search for Reviews

Webz.io Tips & Tricks: Search for Reviews

Are you looking to focus your data search specifically on consumer generated reviews? Here are a couple of simple Webz.io tricks that might help: Limit your query to specific sites You can…

Vertical Aggregation and Pattern Matching Crawlers

Vertical Aggregation and Pattern Matching Crawlers

After bashing various crawling techniques, I would like to describe the technique we use here, at Webz.io, a technology that was developed over the past 8 years. Our crawlers were developed with…

Crawling Horrors – Computer Vision Crawlers

Crawling Horrors – Computer Vision Crawlers

So if RSS Crawlers are bad, Browser Scraping isn’t efficient, what about computer vision web-page analyzers? This technology uses machine learning and computer vision to extract information from web pages by interpreting…

Crawling Horrors – RSS Crawlers

Crawling Horrors – RSS Crawlers

One of the fastest, simplest and unfortunately wrong ways of extracting content out of a website, is by reading its RSS feeds. I will show you how its done and why it’s…