google-research/deduplicate-text-datasets Issues
Distributed running
Updateddoes this tool can process Chinese?
Updated 1Question: Upper Bound
Closed 1customized dataset deduplication
Closed 1where the data is?
Closed 1Incomplete Sentences
Closed 1Error when running the code
Closed 15RAM crash when use collect method
Updated 2Simple test
ClosedOff-by-1 error in `collect`?
Updatedone bug when I use
Closed 2Should newline char be removed
Closed 1false positives
Closed 1Error on self deduplication
Closed 10Why not use Simhash?
Closed 3