Scrapinghub

Frontera backend to guide a crawl using PageRank, HITS or other ranking algorithms based on the link structure of the web graph, even when making big crawls (one billion pages).

Language:CBSD-3-Clause54 118 19

exporters

Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations

Language:PythonBSD-3-Clause40 101 20

scrapinghub-entrypoint-scrapy

Scrapy entrypoint for Scrapinghub job runner

Language:PythonBSD-3-Clause25 25 14

andi

Library for annotation-based dependency injection

Language:PythonBSD-3-Clause20 9 3

shublang

Pluggable DSL that uses pipes to perform a series of linear transformations to extract data

Language:PythonBSD-3-Clause15 80 37

shub-workflow

Language:PythonBSD-3-Clause13 18 8

autologin

A project to attempt to automatically login to a website given a single seed

Language:PythonApache-2.09 50

tutorials

Language:Python8 1190

hcf-backend

Crawl Frontier HCF backend

Language:PythonBSD-3-Clause7 84 9

scrapy-monkeylearn

A Scrapy pipeline to categorize items using MonkeyLearn

Language:Python7 70

varanus

A command line spider monitoring tool

Language:Python7 9 4

Formasaurus

Formasaurus tells you the type of an HTML form and its fields using machine learning

Language:HTML500

luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Language:PythonApache-2.04 750