Crawl stops when 1st page is a redirection
flmommens opened this issue · comments
Using the same code as issue 352, with the starting page being https://www.future-processing.com/blog that redirects to https://www.future-processing.com/blog/ (trailing slash) the crawler stops after crawling that page.
With https://www.future-processing.com/blog/ as the starting page, the crawl goes on following the links.
You are creating the crawler without client configuration options. Crawler's Default client options has FOLLOW_REDIRECTS value as false. You can check it here
Line 86 in 1f954a5
Pass your own client options to the Crawler::create method with follow to true and you should have no problem.