Support for Basic Authentication with multiple sitemap files

[Artem Kolotilkin]

I need to parse a sitemap on a site protected by a basic authentication. If there is just one sitemap file and I pass basic auth parameters in the url like http://{user}:{pass}@example.com, it works great. But if sitemap.xml is just an index of sitemap files, then those files will not be parsed and an empty result set is returned.

It would be great to have a basic auth parameter in Sitemapper class that would be applied to all sitemap files.

[Matthew Holloway]

Hi @artemkolotilkin @seantomburke I added Basic Auth with the requestHeaders option.

const username = "username";
const password = "password123";

const usernamePasswordBuffer = Buffer.from(
  `${username}:${password}`,
  "utf-8"
);
const base64UsernamePassword = usernamePasswordBuffer.toString("base64");
const requestHeaders = {
  Authorization: `Basic ${base64UsernamePassword}`,
};
const sitemap = new Sitemapper({ requestHeaders });

It would be great if this was a built in feature.

[Sean Thomas Burke]

@artemkolotilkin @holloway I think I'm in favor of @holloway's implementation and keeping that as an object passed into the requestHeaders.

We can add it to the README.MD as a way to help users implement it. Integrating username and password options that passes a plain text username and password strings feels less secure.

[Hiski Valli]

This doesn't seem to work correctly. I have an index file for index files, and I get 401 error for all subindex files.