zackw / tbbscraper

Automated website scraping over Tor

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

# Automated website scraping (not actually using TBB)

This software collects webpages, using a headless browser (PhantomJS),
from many different network locations, via proxy servers.  It could
in principle use Tor for the proxy but right now it does not.

There is also some software for analyzing the contents of the
collected webpages.

The management cannot guarantee that this is of any use to anyone or
indeed that it works at all outside the context where it is used.

License labeling is pretty spotty, but the intent is to use the
Apache license for everything ( http://www.apache.org/licenses/LICENSE-2.0 )

About

Automated website scraping over Tor


Languages

Language:Python 63.5%Language:AngelScript 10.4%Language:C++ 8.8%Language:C 8.8%Language:PLpgSQL 5.2%Language:JavaScript 1.1%Language:TSQL 0.5%Language:Lua 0.4%Language:SQLPL 0.4%Language:Makefile 0.4%Language:Max 0.3%Language:R 0.2%Language:Shell 0.1%Language:Awk 0.0%