NVIDIA / NeMo-Curator

Scalable data pre processing and curation toolkit for LLMs

NVIDIA/NeMo-Curator Issues

Retire text_bytes_aware_shuffle and directly use shuffle
Updated 11 days ago1
Incorrect Shuffle results with dask-cuda 24.06 & above
Closed 11 days ago3
removing fuzzy duplicates bug in single node tutorial
Closed 11 days ago1
tutorials/peft-curation-with-sdg Failing with Runtime Exceptions.
Updated 11 days ago5
Grammar and punctuation nits in Jupyter Notebooks
Updated 12 days ago3
Re-add `test_uneven_common_crawl_range` PyTest
Updated 12 days ago
Default int_to_str_id error in jaccard shuffle
Closed 12 days ago1
Make `max_text_bytes_per_part` configurable
Updated 15 days ago
Update SemDeDup README
Updated 16 days ago1
Semantic Deduplication Docs not in Index
Closed 18 days ago1
Explore Dask jobque's slurm runner for multi node slurm setups.
Updated 25 days ago1
KeyError in Map Buckets
Closed 25 days ago2
handling inconsistent metadata
Closed 25 days ago2
[FEA] Assert that on connecting to cluster you have at least > 1 workers
Updated a month ago
DocumentDataset read errors when other files are present in directory
Updated a month ago
[FEA] Align the character pruning vs sequence length based pruning for our models.
Updated a month ago
Running into OOM with add id
Updated a month ago2
[BUG] Semdedup Embedding Restart not working cleanly
Updated a month ago
[FEA] Create intermediate representation (IR) from code source
Updated a month ago
[FEA] Add license detector for code repositories
Updated a month ago
Add CPU Dask DataFrame support for `DistributedDataClassifier`
Updated a month ago3
`DocumentDataset` bug for reading relative file paths
Updated a month ago
single_gpu_tutorial.ipynb fails to run on GPU
Updated a month ago2
Given JSON input is too large when using "cudf" backend
Closed a month ago3
Improved Semantic Deduplication Docs
Updated a month ago
Pandas and cuDF DataFrames in `DocumentDataset`
Updated a month ago
Add HuggingFace support for Quality Classifier
Closed a month ago1
Error starting dask cuda worker with nemo framework container
Closed a month ago3
[META] Update python version to include python 3.11
Updated a month ago1
Update single_gpu_tutorial.ipynb to use a recent snapshot
Updated 2 months ago
[FEA] Improve performance for score_document (NumberOfLinesOfCodeFilter) and _line_statistics (PerExtensionFilter)
Closed 2 months ago
[FEA] Allow specify fields when reading files with DocumentDataset.
Updated 2 months ago
Reading compressed json files
Closed 2 months ago2
Remove duplicated assigments in GeneralCommentToCodeFilter
Closed 2 months ago
Remove text field requirement from Download and Extract
Updated 2 months ago
Add pytests for semantic dedup
Updated 3 months ago
Broken Tutorial: single_gpu_tutorial.ipynb
Closed 3 months ago3
[FEA] Support an installation path using RAPIDS nightly packages & source installs of Curator
Updated 3 months ago
ValueError: More than one filename found in partition
Closed 3 months ago1
Import fails on cpu
Closed 3 months ago
[FEA] Add nemo_curator.__version__ and also display it in command line when invoked --help
Updated 3 months ago
FastTextQualityFilter model file release
Updated 3 months ago1
shuffle datasets
Updated 3 months ago3
README.md broken links
Closed 3 months ago
Remove Numpy<2.0 pin
Updated 3 months ago
exact_deduplication.py out_of_memory
Closed 3 months ago6
Update tutorial & Slurm documentation
Updated 3 months ago
fuzzy dedup in cpu
Updated 3 months ago
Update download documentation to include client creation
Updated 3 months ago3
[DOC] Incorrect reference to domain_classifier_example.py
Closed 4 months ago