BrongoObenge / Crawler

https://github.com/Frenesius/CrawlerProject56

CrawlerProject56

Repository is inactive. It has been moved to a private repo.

Crawler computer specs

Dependencies

• Python 2.7
• Pip
• Scrapy 0.24.4
• Py2neo 1.6.4
• Neo4j 2.1.4
• pymongo 2.7.2
• TorCtl
• Tor

Installation

Comming soon ...

Usage(Outdated)

Install Tor and set it up on port 9050. Install Clone the repo using git clone https://github.com/Frenesius/CrawlerProject56.git Get in to the folder using cd Make Start.sh executable(run as root) chmod +x Start.sh

Make sure Neo4j database is up and running. User Neo4j-octo-love to create base database. Execute ./Start.sh

and have fun

Licence

see Licence.md