Can this tool actually crawl/spider, or just scrape pages?

[Bogdan Stanciu]

Hey,

I may not fully understand these terms, but can this tool actually crawl/spider all the pages under a domain, or does it just scrape a specific url?

When I say crawl/spider, I am thinking of something like the ScreamingFrog Spider tool, where you can provide an url, and it will find all (most) other pages on that site.

Please feel free to close this issue, but I feel the tool description needs to be a bit more clear.

Thanks

[Paul Kretschel]

I agree, node-scraper would be a more fitting name for this tool. Or is there an easy configuration to make it behave like an actual crawler?

[Mike Chen]

Yes, "scraper" should be much better. But never mind, you may implement a spider by yourself based on this.

[Raquel Smith]

But never mind, you may implement a spider by yourself based on this.

how do you do this?

[Mike Chen]

1. Figure out the home page or entrance URL which is good to start;
2. Send request to the URL(s);
3. Parse the page content that you get from the response to get all the URLs you care about, which may be the same domain as the previous one;
4. Save the page content to a file or Db whatever you want;
5. Repeat from step 2 to end.

[Mike Chen]

solved in #420