huggingface/datatrove Issues
Fastwarc reader
Updatedhow to turn log/traceback color off?
Updated 3Log progress
UpdatedPeriodical logging of stats
Updated 2OpensearchWriter
Updated 2PineconeWriter
UpdatedSupport int32 in substring dedup
Updated 4[Feature] Packing
Closed 1Local fasttext model
Closed 3Memory overhead in multiprocessing
Updated 1Bug in url filter
Closed 12Unreadable log
Closed 1About local dataset
Closed 3Timeout warning/error
Closed 2Need help for url filter
Closed 2Potential issues in substring dedup
Updated 10Job depends in local executor
Closed 3Load local tokenizer
Closed 10LocalExecutor Speedup
Closed 2Slurm in containers
Closed 2Spark support
Updated 1Multi-node parallelism
Closed 5In-file parallelism
UpdatedWriter adapter
Closed 4