trondeh80 / site-crawler

Small script to crawl a website

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

site-crawler

Crawls a site for 404 errors. This script will also limit the crawl to a subsection of a site. ie: vg.no/nyheter > would make the crawler only search pages with a url that starts with vg.no/nyheter

Run

Node

Requires modern NodeJS to run as it has no transpiling step.

About

Small script to crawl a website


Languages

Language:JavaScript 100.0%