tawilkinson / rust_web_crawler

A simple web crawler in rust

Rust Web Crawler

A very simple web crawler in rust.

Usage

cargo run <url to crawl>

The crawler will then check the robots.txt and crawl the site if allowed.