bitextor / bifixer

Tool to fix bitexts and tag near-duplicates for removal

bitextor/bifixer Issues

Output file encoding should be set to UTF-8
Closed a year ago2
Monofixer adds empty extra lines when there are multiple columns
Closed 2 years ago1
Long sentences are not being removed apparently
Closed 2 years ago2
Tests don't pass
Closed 2 years ago5
'charmap' codec can't decode input_test_2.txt
Closed 2 years ago9
Detokenization introduce new error to the data and ignore_detokenization is not working
Closed 2 years ago3
Bifixer Indexerror: list index out of range
Closed 3 years ago4
Bifixer doesn't work with new ftfy >=6.0
Closed 3 years ago2
	 introduces tabs in tsv output
Closed 3 years ago1
Obtain a clean output
Closed 4 years ago2
Bifixer doesn't see input file
Closed 5 years ago2