7-of-9 / google-site-spider

Automatically exported from code.google.com/p/google-site-spider

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

SSL redirect page listed as unable to load

GoogleCodeExporter opened this issue · comments

One of my pages links to http://wiki.github.com/ook/isbg but the spider lists 
it as unable to load.

The URL is fine although retrieving it does a 301 permanent redirect to an 
https version of the URL.  Arguably the extension should also list permanent 
redirects as such so that the source can be changed to point to the new 
permanent URL.

Original issue reported on code.google.com by rogerbinns on 18 May 2011 at 7:37

This isn't an SSL issue - also happens with other redirects such as 
http://juniper.net/ (which does a 302 temporary redirect).

Original comment by rogerbinns on 18 May 2011 at 7:42

Confirmed.  It seems to be trying (and failing) to inject the spidering code on 
the redirect.  No attempt is made to inject the spidering code into the 
subsequent page.  I'll look into trying to detect this situation.

Original comment by neil.fra...@gmail.com on 12 Jun 2011 at 6:23