togethercomputer / RedPajama-Data

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

togethercomputer/RedPajama-Data Issues

Exact dedup details
Updated 2 months ago
regarding to deduplication
Updated 2 months ago5
Other language data
Updated 2 months ago4
Thresholds for all quality signals
Updated 2 months ago2
what does the prefix "rps_" mean?
Closed 3 months ago2
slow transfer speeds from URL sources
Updated 3 months ago5
Difference between RedPajama-Data-1T, RedPajama-Data-V2, RedPajama-Data-V1
Updated 4 months ago1
Inconsistent IDs lead to distributed computing woes.
Updated 4 months ago1
Spanish artifact building error
Updated 4 months ago2
Step 2) "Invalid option: ---input_base_uri"
Updated 4 months ago1
Potential Language Contamination Inquiry
Updated 4 months ago1
About the final result
Updated 5 months ago2
What purpose cutoff.csv used in the cc_net pipeline?
Updated 5 months ago2
What is the output of `run_lsh.py`?
Closed 6 months ago8
Running the pipeline on cloud or a big data platform
Updated 5 months ago1
Running full pipeline on a small part of CC
Updated 6 months ago
Unavailable Parameters
Updated 6 months ago
Invalid uri: ParseResult(...) must be of the form s3://<bucket>/<key> or file://<path>
Closed 6 months ago
what's the specific meaning of dsir?
Updated 6 months ago4
Recommended way to load wget-downloaded data using HF datasets API?
Updated 6 months ago1
Is there a specific meaning of the snapshot id?
Closed 6 months ago2
possibly missing shard from host
Closed 6 months ago2
Impossible unpack tail data... took time to download, but impossible to unpack dataset without quality signals with broken link.
Closed 7 months ago1
Are shards randomly created?
Closed 7 months ago1
Low Data Downloading Speed
Closed 7 months ago1
Train a new wikiref model
Closed 7 months ago1
Deduplicated version of RedPajama-v2
Closed 7 months ago4
Request: Enable artifact prep on local data
Closed 8 months ago1
Token counts
Updated 8 months ago2
regarding to quality classifier
Updated 8 months ago2
where should I go to get the file about "domain_to_category_id.json"?
Closed 8 months ago
quality_signals, minhash and duplicates missing for tail
Closed 8 months ago1
How is the SHA1 digest computed?
Closed 9 months ago2
Invalid argument when running cc_net
Updated 9 months ago2
Executing V2 issues
Updated 9 months ago6
Issue on book datasets download
Updated 9 months ago2
Failed building wheel for cc-net
Closed 9 months ago2
ArXiv cleaning issue
Closed 9 months ago1
cc-net failure on slurm cluster
Closed 9 months ago
cc_net processing local wet file
Closed 9 months ago1
New Features
Updated 9 months ago
What does "default" do in `load_dataset('togethercomputer/RedPajama-Data-1T', "default")`?
Updated a year ago3
Specifying arxiv dates
Updated a year ago1
Q: Why does RePajama exist? what problem are you solving?
Updated a year ago1
I got an issue when I use fasttext doing arxiv cleaning.
Updated a year ago1
Unlock open science for dataset generation
Updated a year ago
How do i prepare the data for Visualisation?
Updated a year ago2
Understanding the quality filter
Updated a year ago5
Fine tuning RedPajama Model
Updated a year ago1
We then run the same cc-net pipeline on warc_wikipedia.warc, which produces warc_wikipedia.warc.wet
Updated a year ago1