An awesome web crawler written in Go
- Opens URLs in parrallel
- Canonicalizes URLs
- Visits URLs only once
- Allows a parameter for the maximum number of pages to crawl
- Is very polite :) waits 300 milliseconds between HTTP GET requests
- Follows the rules from robots.txt
Petur Orri Ragnarsson peturor@gmail.com
Trausti Saemundsson trauzti@gmail.com
$ go get github.com/trauzti/crawler/gocrawler
$ go build
$ ./crawler <URL> <TOPIC> <QUERYWORDS> [N]
Where N is the maximum number of pages to crawl
Note that the querywords must be within quotation marks (see example.sh)