Hakrawler return urls of out of scope domains
brabbit10 opened this issue · comments
When using hakrawler on a site, it returns out of scoped domains, like Facebook, Google, Youtube, etc.
I saw in another issue that this was fixed but it seems this is still an issue.
for example, running hakrawler on "https://ynet.co.il" like this:
echo https://ynet.co.il | hakrawler -u -subs -insecure -d 2 -h "User-Agent: ${DEFAULT_UA}"
will return URLs from google.com, Instagram.com and others
Hey! The -subs option determines what gets crawled, not what URLs are returned. If there are URLs in a page that go to out-of-scope domains, hakrawler will still print them, but not navigate to them.