glwbr / crawler

One of the crawlers ever created.

Basic Usage

1. Clone the repot to your machine
2. Make sure that you have nodejs installed
3. On the project folder run npm install
4. Call node pointing to main.js and pass a URL to it

Example

node main.js https://www.example.com/

Limitations

1. It'll take a long time to run on websites like stackoverflow and at somepoint it'll get a 429 - Too many requests exception but it works :).
2. It doesn't handle links of external domains
3. Probably many others...