Crawler simply stops
mrtnmueller opened this issue · comments
I have an issue where the crawler stops without giving any hint about why. Seems to be non-deterministic, always at another point during the crawl process.
I have some middlewares (retry, proxy, user agent header) and a redis queue but I don't think they create the problem.
The real problem seems to be that I add URLs dynamically in the observer using $crawler->addToCrawlQueue in the crawled function.
Do I need to call something else to make the crawler crawl all URLs on queue?