Simple web crawler in Python.
Use of MongoDB for saving URL-s (you can stop and continue crawling at any time)
You can pick whether you want spiders to crawl just your domain space, or you want to go crawling outside your domain space.
If you encounter any problem (or bug) feel free to make a pull request.
NOTICE: Code needs to be refactored, expect that to be done until 1/1/2017.