Cannot create a sitemap for www.forthea.com, my app just freezes
Chamomile11 opened this issue · comments
Do you want to request a feature or report a bug?
BUG
What is the current behavior?
I am building a small console Node app. I successfully managed to create sitemaps for several thousands websites using this package. However, when I try to use it for www.forthea.com my app just hangs forever (even when I use it inside a promise with a timeout timer to force it to stop and even despite its own built-in timeout value of 30000 ms). My app just freezes but does not crash and there is no any errors.
If the current behavior is a bug, please provide the steps to reproduce.
const sitemapGenerator = require('sitemap-generator');
const generator = sitemapGenerator('http://www.forthea.com', {
stripQuerystring: false
});
generator.start(); // Loops forever, no errors, no sitemap, even when used with a timer.
What is the expected behavior?
The package must be capable of building a sitemap for www.forthea.com (or at least generate some error).
Will have a look at it tomorrow.
Will have a look at it tomorrow.
Alright, thank you Lars.
Seems like the package blocks the main event loop altogether when parsing www.forthea.com so that even timers set via setTimeout stop working. I fixed my problem by running my sitemap generation code inside a separate child process and killing it after a predetermined amount of time, as I originally needed. This way it works alright and I can kill the child process even if it hangs.
Interesting. Would still be good to know why the main event loop is blocked for this website especially.
that is weird, I've never experienced this infinite loop error. Are you setting the crawler to respect robots.txt? @Chamomile11