hplt-project / OpusCleaner

OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.

https://pypi.org/project/opuscleaner/

hplt-project/OpusCleaner Issues

Refactor filters as transfomers & scorers
Updated 8 months ago
Feature request: Do not rerun the whole processing pipeline when adding filters in the web frontend
Closed 8 months ago1
Configure the diff view to select diffs between different steps
Updated 8 months ago
Tooltip that says which filter did it
Updated 8 months ago
OpusCleaner get stuck on ASGI error when adding filters while sample is still loading
Closed 8 months ago1
add version somewhere in the frontend
Closed 8 months ago
How does the num_mismatch filter tokenize?
Closed 8 months ago2
num_mismatch fails on CCMatrix
Updated 9 months ago7
opuscleaner-clean hangs on CCMatrix
Closed 9 months ago6
Show overlap scores when downloading datasets
Updated 9 months ago
Add license
Updated 9 months ago
FileNotFoundError: [WinError 2] The system cannot find the file specified
Closed 10 months ago3
My browser is going slow
Updated 10 months ago1
Update UI
Updated 10 months ago
Download all of them!
Closed 10 months ago
Color coding download states
Closed 10 months ago
How to use opuscleaner-clean with stdin?
Closed 10 months ago1
LASER threshold broken?
Closed 10 months ago1
opuscleaner-clean returns status code 0 if one of the processes fail
Closed 10 months ago
shuffle data
Updated 10 months ago2
bicleaner-ai?
Updated 10 months ago
Why do I select cleanliness when I'm not looking at the data?
Closed 10 months ago1
A simple length ratio filter shouldn't require manually updating pyhash submodules
Closed 10 months ago2
Add notes
Updated 10 months ago
Any plans to share the filter configs?
Updated a year ago5
Support a hierarchy of filters
Updated a year ago1
Setting languages for individual rules is redundant
Closed a year ago2
How do I say this dataset is so garbage it shouldn't be used?
Updated a year ago2
langid vs fasttext_filter
Updated a year ago1
What does the pie "show dataset statistics" do?
Updated a year ago1
Clicking "Datalor" logo from "import dataset page" results in empty folder even though some have been downloaded
Updated a year ago
Downloads keep running after interrupt
Closed a year ago2
No way to retry failed download
Closed a year ago
Modify URL to reflect language selection
Closed a year ago
num_mismatch filter does not recognize equivalent numbers
Closed a year ago
Error when filters is an empty array
Closed a year ago2
Add support for monolingual datasets
Closed a year ago
Move filters to their own packages
Updated a year ago
CLI to deal with data categories
Updated a year ago2
Interface sorts language columns on full name, backend on language code
Closed a year ago1
Boolean filter properties are always `true`
Closed a year ago2
OpusCleaner leaves `\r` in output?
Closed a year ago4
Missing minor installation instructions in README
Closed a year ago
Undo/redo
Updated a year ago
Store download urls in opuscleaner configuration files
Updated a year ago
Add instructions for running it without installing it
Updated a year ago1
Add tests
Updated a year ago
Corpus analytics
Updated a year ago
Special quote filter for CCMatrix and CCAligned
Updated a year ago
[Discussion] Filter that checks for numerical sequences?
Closed a year ago3