Obey robots.txt
eklem opened this issue · comments
Get robots.txt, and check before each URL is crawled.
Will be own server. Could be nice to have down the road, but not now.
eklem / browsercrawler
Crawling content from a site within the browser. A basis for i.e. a search solution for static sites.