kwsp / crawl

A fast crawler that detects broken links on your website and builds a graph representation of the network. Based on libcurl and libxml2.

crawl

A fast web crawler based on libcurl and libxml2 that can spawn hundreds of concurrent connections.

Features:

• Find broken links given a starting url
• Build a graph of the network topology and output to the GraphViz dot format

Developing

mkdir build && cd build
cmake ..
make