Another crawler to search .onion links from the public Internet
juhanurmi opened this issue · comments
Use an another crawler to search .onion pages from the public Internet. Search new .onion domains from different online sources. Ask help from organizations that are crawling. This is an excellent case to test open source crawlers like Heritrix and Apache Nutch? Or use the search engines that exist.
2 workweeks
Heritrix and Apache Nutch are totally overkill for this. There are few good sites that list onion addresses. I will fetch new URLs daily from these sites.
Furthermore, with Tor2web integration I am downloading the visit history from each Tor2web nodes and this seems to be the best way to find new onions.