DataHenHQ

till

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

useragent

DataHen useragent tool is a Golang package and standalone tool that generates a random combination of millions of user-agents strings. Currently used in production at DataHen to crawl/scrape through billions of pages.

datahen-ruby

Datahen Client for Ruby

datahen-python

DataHen Python Library

henqa

HenQA is a standalone tool for validating massive amounts of data using the JSON schema.

license

license package signs and verifies responses based on public and private key and timestamp

afero

A FileSystem Abstraction System for Go

cookie_store

An implementation of RFC6265

dh_easy-qa

QA library that runs on Fetch

go-envparse

Minimal environment variable parser for Go

gojspromise

henqa_shared

HenQA shared components

proxy_benchmark

Proxy benchmark script

request_tester

ujson

ujson package does marshalling like json but without escaping html

datahen-api-doc

DataHen API Documentation

datahen-python-helper

dh_easy-core

Datahen Easy Core Toolkit

dh_easy-router

Datahen Router Core Toolkit

docker-pgbouncer

Minimal PgBouncer image that is easy to configure

documentations

gid

gid package is a golang package that is used to generate globally unique IDs (GID) for web pages (HTTP requests). Useful for troubleshooting web scrapers, and reusing web page caches.

hudsucker

Intercepting HTTP/S proxy

imgix-rails

A Rails gem for integrating imgix into Rails projects

lightning-fs

A lean and fast 'fs' for the browser

reqwest-actix-stream

A Stream to link between Reqwest and Actix-web two systems.

reqwest_cookie_store

test-scraper

Test Scraper

useragentr

a Rust library that generates a random combination of millions of user-agents strings.

website-crawler

Crawls a web site