ParaCrawl

extractor

corset

Corset is a web-based data selection portal that helps you getting relevant data from massive amounts of parallel data.

keops

Tool for manual evaluation of parallel sentences.

DataCollection

Data collection, alignment and TAUS repository

cirrus-scripts

Scripts for running bitextor/paracrawl/europat jobs on cirrus.ac.uk

human-evaluations

Results of the human evaluation

synthesis

Data synthesis by contextualizing glossary translations

embedding

Mine parallel corpora with embeddings

europat-scripts

Scripts for obtaining patent data

tmxutil

Tools to generate & filter Europat tmx files.

opus-train

Automate download and training with OPUS corpora

b64filter

Program for operating on one document per Base 64 encoded line files

Domain_Adaptation

InDomain detection is a tool designed to extract in-domain data from a large collections of data.

giashard

Sharding program for Paracrawl

giawarc

Processing utilities for Internet Archive

corpus-issues

Open here any Paracrawl corpus related issue

go-warc

A golang library to work with WARC files from the common crawl

multilingual-ted

targeted-crawling

url_language_analysis