selectors config file

[Espen Klem]

Split out the selectors config to a .json-file so it's less scary to make it match your content. Should be required in the main.js file

[Espen Klem]

Skip the require-step and just fetch the JSON file with the fetch API. Format of setup.json can be like this:

{
  "urls": [
    "https://example.com/1",
    "https://example.com/2",
    "https://example.com/3"
  ],
  "selectors": {
    "title": "$('title').text()",
    "body": "$('body').text()"
  }
}

[Espen Klem]

Not a config file, but the crawler should take it as input, so a setup/config step

[Espen Klem]

https://stackoverflow.com/questions/12797118/how-can-i-declare-optional-function-parameters-in-javascript