adbar

trafilatura

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

German-NLP

Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

htmldate

Fast and robust date extraction from web pages, with Python or on the command-line

courlan

Clean, filter and sample URLs to optimize data collection – includes spam, content type and language filters

py3langid

Faster, modernized fork of the language identification tool langid.py

geokelone

integrates spatial and textual data processing tools into a modular software package which features preprocessing, geocoding, disambiguation and visualization

german-reddit

Extraction of a German Reddit Corpus

awesome-crawler

A collection of awesome web crawler,spider in different languages

awesome-web-scraper

A collection of awesome web scaper, crawler.

flux-toolchain

Filtering and Language-identification for URL Crawling Seeds (FLUCS) a.k.a. FLUX-Toolchain

tweets-tools

Diverse tools used with Twitter data

coronakorpus

Material zum Aufbau eines deutschsprachigen COVID-19-Webkorpus / Building a corpus in German dedicated to coronavirus

jlcl-style

Experiments to modernize the LaTeX class of the JLCL

toponyms

Old prototype for toponym extraction in historical texts written in German

trafilatura_gui

vardial-experiments

Experiments conducted on the occasion of the VarDial shared tasks

adbar

archiveis

A simple Python wrapper for the archive.is capturing service

btw21

Visualization of the most frequent words in the German federal election in 2021

cChardet

universal character encoding detector

datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

dateparser

python parser for human readable dates

dwdsmor

SFST/SMOR/DWDS-based German Morphology

jparser

A readability parser which can extract title, content, images from html pages

jusText

Heuristic based boilerplate removal tool

python-readability

fast python port of arc90's readability tool, updated to match latest readability.js!

shoten

valency-oriented-chunker

A one-pass FSA valency-oriented chunker for German (proof of concept)

wee-benchmarking-tool