How to Automate Supply Chain Risk Reports: A Guide for Developers
Do you use Python? If so, this guide will help you automate supply chain risk reports using AI Chat GPT and our News API.
On this page
Tiny basic multi-threaded web crawler in Python
Tiny basic multi-threaded web crawler in Python
If you need a simple web crawler that will scour the web for a while to download random site’s content – this code is for you.
Usage:
|
1
|
$ python tinyDirtyIffyGoodEnoughWebCrawler.py https://cnn.com
|
Where https://cnn.com is your seed site. It could be any site that contains content and links to other sites.
My colleagues described this piece of code I wrote as “Dirty”, “Iffy”, “Bad”, “Not very good”. I say, it gets the job done and downloads thousands of pages from multiple pages in a matter of hours. No setup is required, no external imports, just run the following python code with a seed site and sit back (or go do something else because it could take a few hours, or days depending on how much data you need).
tinyDirtyIffyGoodEnoughWebCrawler.py
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
import sys, thread, Queue, re, urllib, urlparse, time, os, sys
dupcheck = set()
q = Queue.Queue(100)
q.put(sys.argv[1])
def queueURLs(html, origLink):
for url in re.findall(”‘<a[^>]+href=[“‘](.[^“‘]+)[“‘]’”, html, re.I):
link = url.split(“#”, 1)[0] if url.startswith(“http”) else ‘{uri.scheme}://{uri.netloc}’.format(uri=urlparse.urlparse(origLink)) + url.split(“#”, 1)[0]
if link in dupcheck:
continue
dupcheck.add(link)
if len(dupcheck) > 99999:
dupcheck.clear()
q.put(link)
def getHTML(link):
try:
html = urllib.urlopen(link).read()
open(str(time.time()) + “.html”, “w”).write(“” % link + “n” + html)
queueURLs(html, link)
except (KeyboardInterrupt, SystemExit):
raise
except Exception:
pass
while True:
thread.start_new_thread( getHTML, (q.get(),))
time.sleep(0.5)
|
Features:
- Multi-threaded – for fastness
- Duplication elimination (kinda) – for link uniqueness
- Saves both source and its link – for the purpose it was built
- FREE
Enjoy,
Ran
Subscribe to our blog for more news and updates!
Read Up
How to Automate Supply Chain Risk Reports: A Guide for Product Managers
Use this guide to learn how to easily automate supply chain risk reports with Chat GPT and news data.
How to Automate Mergers and Acquisitions Reports: A Guide for Developers
A quick guide for developers to automate mergers and acquisitions reports with Python and AI. Learn to fetch data, analyze content, and generate reports automatically.
