nlevitt

monie

man-in-the-middle http/https proxy library in rust

libhdfs3-deb

Steps to produce a .deb that can be installed dpkg for libhdfs3 (for use with python)

warc

Read and write WARC files in Go

brozzler

brozzler - distributed browser-based web crawler

umbra

A queue-controlled browser automation tool for improving web crawl quality

warcio-rs

warcprox-rs

cdxj-indexer

CDXJ Indexing of WARC/ARCs

hdfs

A native go client for HDFS

heritrix3

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

hyper

An HTTP library for Rust

logaggrd

simple udp log aggregator daemon

lru-cache

A cache that holds a limited number of key-value pairs

outbackcdx

A Wayback RemoteResourceIndex server using RocksDB

pretty-env-logger

A pretty, easy-to-use logger for Rust.

psutil

A cross-platform process and system utilities module for Python

psutilz

utilities built on the psutil library

pywb

rocksdb

A library that provides an embeddable, persistent key-value store for fast storage.

snakebite

A pure python HDFS client

sshrc

bring your .bashrc, .vimrc, etc. with you when you ssh

swift-nio

Event-driven network application framework for high performance protocol servers & clients, non-blocking.

swift-nio-examples

examples of how to use swift-nio

tempfile

Temporary file library for rust

tokio

A runtime for writing reliable, asynchronous, and slim applications with the Rust programming language.

trough

Trough: Big data, small databases.

urlcanon

url canonicalization library for python and java

warcio

Streaming WARC/ARC library for fast web archive IO

warcprox

WARC writing MITM HTTP/S proxy

websocket-client

websocket client for python