spider-rs

spider

Web crawler and scraper for Rust

spider-py

Spider ported to Python

headless-browser

Headless browser - reliable high performance setup for cloud providers like Fargate, CloudRun, K8, and more.

spider-nodejs

Spider ported to Node.js

html2md

HTML to markdown for Rust

ua_generator

Pre-compiled random real User-Agents. Updated weekly for windows, mac, linux, and android

chromey

Concurrent high-level API to control Chrome over the DevTools Protocol for Rust

web-crawling-guides

How to guides on web-crawling or scraping

spider-clients

Python, Javascript, and Rust libraries for the Spider Cloud API.

readability

The readability library for Rust

spider_fingerprint

The fingerprinting library for Rust

jsdom

JSDOM for Rust [incomplete]

spider-cloud-live-code-viewer

Visual web crawler

case_insensitive_string

A case-insensitive string for Rust

async_job

Simple async cron jobs for Rust

proxier

Auto proxy whitelisting for 3rd parties

spider_transformations

HTML transformation library for Rust

auto-encoder

Auto encoding library bytes to strings

spider_network_blocker

Network blocking lists on ads, scripts, and analytics for Spider

spider_scraper

html5ever web scraper used for some of spider-rs CSS transformations

bad_websites

websites that should be blocked via firewall

firewall

Firewall for Rust

homebrew-spider-cloud-cli

spider-wasm

The spider project ported to wasm [wip]