zero sites found for https://www.newyorker.com/sitemap.xml

[palash kulshreshtha]

sitemapper is not able to find any site in https://www.newyorker.com/sitemap.xml . not able to figure out why.

[Sean Thomas Burke]

That doesn’t look like a proper .xml file, it’s called sitemap.xml but it looks like just a list of links.

[palash kulshreshtha]

It shows xml format in my browser pfa screenshot. Please let me know if I am missing anything

[palash kulshreshtha]

What i think is happening is that, since there are a lot of child sitemaps, sitemapper is trying to parallely fetch all the sitemaps which leads to no request getting complete. digging further into it. Trying to figure out how to reduce concurrency of parallel requests.

[palash kulshreshtha]

yes, thats the case. It is not able to fetch because there is no limit to the parallel requests that could be made at a time. And hence all request fails.

@seantomburke any ideas on how to curb infinite parallel requests.

[Sean Thomas Burke]

Oh it must return no sites because it’s reaching the timeout limit, you could try increasing the timeout limit first. If that doesn’t work, then there will need to be some sort of processing queue introduced.

On Mon, Aug 5, 2019 at 10:20 AM palash kulshreshtha < ***@***.***> wrote: yes, thats the case. It is not able to fetch because there is no limit to the parallel requests that could be made at a time. And hence all request fails. @seantomburke <https://github.com/seantomburke> any ideas on how to curb infinite parallel requests. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34?email_source=notifications&email_token=AAHLVMXNOS4556B36RASJN3QDBOMJA5CNFSM4IJFVWY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3SPQCQ#issuecomment-518322186>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHLVMRHILYXV7IIQ3XKTW3QDBOMJANCNFSM4IJFVWYQ> .

-- Sean Thomas Burke [e-mail] http://www.seantburke.com [phone]

[palash kulshreshtha]

let me check with increasing timeout. But eventually need to check the uncontrolled hits. I am trying to find a way for it. Will raise a PR if i find a way

[palash kulshreshtha]

i dont think increasing the timeout will solve the problem . will need to work on concurrency.

[Sean Thomas Burke]

The new yorker blocks the request due to the user agent.
Try adding this to your options:

const sitemapper = new Sitemapper({
  url: 'https://www.newyorker.com/sitemap.xml',
  timeout: 15000,
  headers: {
    'User-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36',
  }
});