Labyrinth
Have you ever crawled a site? You probably have.
Now try to crawl this site. It should take some time :)
Installing
- Use rvm to use Ruby 2.1.0
- Run
gem install bundler - Run
bundlerPossibly from the rvm install dir - Run
rake:setup - Run
rake:migrate - Run
rackup
TODO
Add logging for requested urls- Use something not SQLite for logging This is real!
- Cache the wordlist to give the disc some peace