Add option to display full redirect chain (http codes 301 and 302) as separate lines

[Quinn Comendant]

Currently, Scout displays only the final http code for URLs that direct. It would be useful to know when a page redirects, and to where.. I would like to suggest if the URL they redirect to is a positive match, add it as a separate line. For example, the URL https://www.strangecode.com/wm response is code 301, and redirects to https://webmail.strangecode.com/ which has response code 200. When a page redirects successfully, you can display it separately, like this:

…
[301] https://www.strangecode.com/wm
[200] https://webmail.strangecode.com/
…

This might result in a chain of interesting redirects; in the following example, it might be useful to learn about the existence of the host oauth.example.com:

…
[301] https://example.com/admin
[302] https://oauth.example.com/interesting/url
[200] https://destination.example.com/
…

I would even enable this by default, but you can hide it under an option such as --expand-redirects, -e. :)

[Liam Galvin]

Good spot, thank you. I think you're right and enabling by default is the right way to go. I'm planning on adding a few more "by default" options, such as spidering found pages for links, and scanning robots.txt for further spidering. I think opting the user in for everything and including opt-out flags e.g. --no-spider would be the nicest solution.

[Liam Galvin]

This now works by default, though if you omit 301 from the positive status codes and then a request hits a 301 -> 200, the 200 will be reported. If you include both codes, you will see both reported. Hope that makes sense!