jquesnelle / yarn

YaRN: Efficient Context Window Extension of Large Language Models

jquesnelle/yarn Issues

Have you tried Rope_base = 500k in YaRN?
Updated 3 months ago
OOM error of distributed training on 80GB GPUs with Mistral-7b
Updated 3 months ago3
An OOM error occurred while computing the perplexity of 128k Proofpoint documents with a maximum token count set to 128k.
Updated 4 months ago1
OOM on two 80GB GPUs
Updated 5 months ago6
Question related to _yarn_linear_ramp_mask
Updated 5 months ago1
For models with other architectures, such as Qwen family, how to find the best `\alpha`, `\beta` and `\sqrt{1/t}` parameters?
Updated 5 months ago
Can we run the replication of the results，8 * 80 A100
Updated 5 months ago1
the detail of code implements.
Updated 5 months ago
cannot load safetensor: Trying to set a tensor of shape torch.Size([0]) in "weight" (which has shape torch.Size([32000, 4096]))
Closed a year ago4
Training takes a long time
Closed a year ago3
Question about the 128K context dataset
Closed 7 months ago1
What the recommended GPU setup for fine-tuning ?
Closed a year ago8
Sliding window perplexity with truncated documents
Closed a year ago5
Unexpected larger perplexity on PG19
Updated 7 months ago1
deepspeed config crashed for `auto` and OOM
Closed a year ago3
Mistral-train error on deepspeed config
Updated 7 months ago1
Running Error
Updated 7 months ago2
cannot connect to hugging face
Updated 8 months ago
Why the updated cache is initialized with seqlen=256?
Updated 8 months ago
Trying to set a tensor of shape torch.Size([257, 1024]) in "weight" (which has shape torch.Size([1226, 1024])), this look incorrect
Updated 8 months ago
How should I proceed with conducting an evaluation for lm-evaluation-harness?
Updated 8 months ago
Questions about DynamicNTK
Updated 8 months ago
Phi 2
Updated 9 months ago
Could this repository be used for sft based on YaRN?
Updated 9 months ago
Compute Requirements
Closed a year ago3
Testing yarn on practical tasks.
Closed a year ago3
Yarn gets worse results than NTK-aware-scaling policy, under non-fine-tuned scenarios
Updated a year ago8
Question about Yarn environment configuration (v2)
Closed a year ago4
RoPE scaling config confusing
Updated a year ago
Inquiry Regarding Evaluation Metrics in Your Paper
Updated a year ago2
context length and dataset size
Updated a year ago
Hardware equipments and training time?
Updated a year ago
What is the purpose of `finetuned` parameter in `LlamaDynamicYaRNScaledRotaryEmbedding`?
Updated a year ago
Questions about max-position-embeddings
Updated a year ago
How do i increases the context of already fined tuned or base model of llama2 ?
Updated a year ago
Can it be debug with deepspeed + trainer
Closed a year ago1
Runtime error
Updated a year ago
Training system configuration
Closed a year ago1
dataset preprocessing script
Updated a year ago1
It is the highest at the lowest dimension and the lowest at the highest dimension.
Updated a year ago
inv_freq seems not calculated right
Closed a year ago9
A potential bug in scaled_rope/LlamaDynamicScaledRotaryEmbedding.py
Updated a year ago
Are 7B and 13B Models fine-tuned?
Updated a year ago3
Should the training incremental? From 64k to 128k having the output of first training passed as input to the next?
Updated a year ago5
Discussion: how to apply this experiment to the llama2 70B model?
Updated a year ago6
v2版本的论文什么时候提交到arxiv上？
Updated a year ago2
Please, writting LISENSE
Closed a year ago2
OOM when doing text generation
Updated a year ago3
OSError: [Errno 28] No space left on device
Closed a year ago1
Confirmation of License
Closed a year ago2