How to debug when getting no output or errors?
skiffkiddie opened this issue · comments
Hello. First, I appreciate your script and hard work for this project that we can all use. I think its great.
I have been struggling to crawl a website. Read the Readme over and over. Tried all the flags to see if it would fix the lack of output/crawling.
What can I do to see whats actually going wrong? Not familiar with go specifically. Would a cookie help and which cookie would I want to input? Whats the best way to attain that cookie i could use (for any attempt, figured id ask)
Sorry to bother, know this might be a no-brainer for a more experienced programmer. If you could point me in the right direction or let me know if I'm missing something id be really grateful!
Thanks so much.
Running this on the most up to date Kali, installed with go (used the apt installed one before quickly figured out how to run the newer one from the go install so both are installed)
Hey it's likely that the URL you're crawling just doesn't have any links, or maybe the links are generated dynamically on the client-side, in which case hakrawler might not detect them.
Try doing a curl of the same URL and seeing what it returns.
If you provide me the url you're trying to call I can give you more info.
Hi Hakluke, I tested the hakrawler with links such as "https://www.discord.com", "https://www.tryhackme.com" and the output is empty. With gospider, there are links that can be extracted.
hakrawler can extract link from "https://www.google.com"
Can you check this on your side?
Plus, can you add some error message to handle this scenario?
In addition, in theory, even the link is dynamically generated, it is still included in the website's html so it can still be extracted, is that correct?
One more thing, can you add a flag for link to rather than just pipelining it? Cause in that way, the tool is more independent
Hi Hakluke, never mind the previous comment, those sample links are not existed, their correct versions do not contain "www" in the domain name.
However, Plus, can you add some error message to handle this scenario? (Such as the domain name may not exist or their maybe typo in your domain name)
And can you explain a bit more about the scenario you mentioned above "maybe the links are generated dynamically on the client-side, in which case hakrawler might not detect them."?
One more thing, can you add a flag for link to rather than just pipelining it? Cause in that way, the tool is more independent
Thank you very much for your time