Giters

juhanurmi / ahmia

Ahmia hidden service search engine

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Rewrite tools as Scrapy spiders

juhanurmi opened this issue · comments

commented

Scrapy: The crawler is developed using Scrapy spider framework [1]. This onion site crawler is called OnionBot[2].

At the moment we have many different test tools for hidden services [3]. It might be cleaner solution to rewrite these tools with Scrapy.

[1] http://scrapy.org/
[2] https://github.com/juhanurmi/ahmia/tree/master/onionbot
[3] https://github.com/juhanurmi/ahmia/tree/master/tools