sangmichaelxie / doremi

Pytorch implementation of DoReMi, a method for optimizing the data mixture weights in language modeling datasets

https://arxiv.org/abs/2305.10429

sangmichaelxie/doremi Issues

Request for Redpajama Dataset Weights
Updated 3 months ago
AssertionError：assert q.dtype in [torch.float16, torch.bfloat16]
Closed 4 months ago2
Question about the initialization of the perdomain_scores
Closed 4 months ago1
Question about 8B model architecture
Closed 4 months ago1
ModuleNotFoundError: No module named 'flash_attn.models.falcon'
Updated 5 months ago11
Cuda version problem
Updated 5 months ago2
program stuck (when ”Loading cached shuffled indices for dataset at ...“)
Updated 5 months ago3
Question about model initialization
Updated 6 months ago
Cannot reproduce the results shown in Github repo with the 120M reference model on A800 (8*80G).
Updated 10 months ago17
Questions about the loss used for optimizing the proxy model
Updated 10 months ago3
List of pinned requirements / Dockerfile?
Closed 10 months ago2
Speed decrease during training
Updated 10 months ago1
Questions about directly applying the weights from paper or the repo to train main model
Updated 10 months ago2
Edge Case Discussion
Closed 10 months ago1
Question about optimized weights in the paper
Closed a year ago4
Training time for baseline model and proxy model
Closed a year ago1
question about only updating the domain weights on process 0
Closed a year ago4
How many rounds do we need to converge domain weights on The Pile?
Updated a year ago1
easy HF dataset doremi?
Updated a year ago2
How do you get the model to be good at code if it downsamples code?
Updated a year ago1
Should reference model initialize weights uniformly?
Closed a year ago3
loss computation wrong?
Closed a year ago2
Question about Flash-attention version.
Closed a year ago1
Domain weights are mostly near one-hot
Closed a year ago3
question about domain weights initialization value in paper fingure 8
Closed a year ago1
Multi-nodes support
Updated a year ago1
about loss
Closed a year ago1
step 1 baseline_280M loss large
Closed a year ago5
Adding a license
Closed a year ago1