BotMap.io tries crawling other websites..
AdrienLemaire opened this issue · comments
Trying to generate a sitemap for my site, BotMap.io goes to a bunch of urls before switching to facebook, urls all starting with http://www.facebook.com/share.php?u=...
This is surprising, shouldn't you filter out urls unrelated to the domain of the site being crawled ?
I tried to disallow www.facebook.com, but didn't work.
Yes there is logic that checks against the target site URL. So this shouldn't be happening. I am looking into it.