innoplexus

flink

Apache Flink

luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

pyspider

A Powerful Spider(Web Crawler) System in Python.

ceph

Ceph is a distributed object, block, and file storage platform

WebFundamentals

Former git repo for WebFundamentals on developers.google.com

arangodb

🥑 ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

beef

The Browser Exploitation Framework Project

GloVe

Software in C and data files for the popular GloVe model for distributed word representations, a.k.a. word vectors or embeddings

tabula

Tabula is a tool for liberating data tables trapped inside PDF files

titan

Distributed Graph Database

openlibrary

One webpage for every book ever published!

oboe.js

A streaming approach to JSON. Oboe.js speeds up web applications by providing parsed objects before the response completes.

flasgger

Easy OpenAPI specs and Swagger UI for your Flask API

heritrix3

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

conceptnet5

Code for building ConceptNet from raw data.

facebook-sdk

Python SDK for Facebook's Graph API

gitinspector

:bar_chart: The statistical analysis tool for git repositories

abot

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

dr-elephant

Dr. Elephant is a job and flow-level performance monitoring and tuning tool for Apache Hadoop and Apache Spark

rappor

RAPPOR: Privacy-Preserving Reporting Algorithms

wayback

IA's public Wayback Machine (moved from SourceForge)

brozzler

brozzler - distributed browser-based web crawler

linkedin-scraper

Scrapes the public profile of the linkedin page

ebot

Ebot, an Opensource Web Crawler built on top of a nosql database (apache couchdb, riak), AMQP database (rabbitmq), webmachine and mochiweb. Ebot is written in Erlang and it is a very scalable, distribuited and highly configurable web cawler. See wiki pages for more details

OpenNlp

Open source NLP tools (sentence splitter, tokenizer, chunker, coref, NER, parse trees, etc.) in C#

commoncrawl-crawler

The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)

cdx-index-client

A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/

TextMaps

analyze_ocr

Parse OCR result files for pagenos, tables of contents, etc.

webarchive-indexing

Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.