arghyadipchak / craww

Gemini (protocol) crawler written in Rust

Craww

Gemini Crawler written in Rust. Information Retrieval project of Arghyadip and Gurdit @CMI

Getting Started

For Docker (Recommended)

1. Install docker and docker-compose-plugin
2. Clone the repository

git clone https://github.com/arghyadipchak/craww

3. Create a config.toml file (example config given)
4. Build and Run

docker compose up

For Non-Docker

1. Install Rust
2. Clone the repository

git clone https://github.com/arghyadipchak/craww

3. Build Craww

cargo build --release

4. Create a config.toml file (example config below)
5. Run Craww

./target/release/craww

OR You can run Craww directly with

cargo run

Configuration

Example config file (config.toml)

root = "gemini.circumlunar.space" #Root Seed
timeout = 5                       #Connection Timeout(in secs)
database = "store.db"             #Sqlite file

[cache]                           #Bloom Filter config
expected_web_pages = 100000
false_positive_rate = 0.01