dvlab-research / LongLoRA

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

http://arxiv.org/abs/2309.12307

dvlab-research/LongLoRA Issues

Something wrong with the torch version
Updated a month ago
norm层不是没有参数矩阵吗
Updated 2 months ago
I am unable to reproduce the results from the paper for llama-7B-32k-longlora ppl.
Updated 2 months ago1
模型完全没法正常输出
Closed 2 months ago1
embedding 为什么要resize成32001？
Updated 2 months ago
What's the trainset is used to obtain “Model with contextg extension via improved LoRA fine-tuning” (LoRA+)？
Updated 3 months ago
How did make questions and answers for long context(LongAlpaca)?
Updated 5 months ago
When I set `per_device_train_batch_size=2`, the S2-Attn would not shift as expected
Updated 5 months ago2
HF models missing rope scaling in the config
Updated 5 months ago
Machine don't install Flash Attention
Updated 5 months ago
global_step文件
Updated 5 months ago
Regarding the results in Table 8 and Table 14
Updated 6 months ago
About the different datasets and corresponding models
Updated 6 months ago
The proof-pile/test-sample-ids is not the exact ids for the proof-pile-testsample.bin
Closed 6 months ago
Memory usage "too small" for 7B Llama-2
Updated 6 months ago
training a LLM w/ shifted sparse attention from the scratch?
Updated 6 months ago
merge_lora_weights_and_save_hf_model.py Error while deserializing header: HeaderTooLarge
Updated 6 months ago
Distributed inference issue
Updated 6 months ago
LongLoRA + Flash Attention 2 causing illigal memory access
Updated 6 months ago7
Is it possible to increase the context length of phi-2 using LongLora? If yes, what changes need to be done to support it?
Updated 6 months ago1
论文中的evaluate结果，推理时用的attention是shifted sparse attention？还是full attention？
Updated 6 months ago
the value of loss is too unstable when supervised-finetune the 7b-100k-ft model
Updated 6 months ago1
streaming llm problem
Updated 6 months ago
How can I use the Llama-2-7b-longlora-100k-ft model correctly
Updated 6 months ago
bug report : RuntimeError: probability tensor contains either inf, nan or element < 0
Updated 6 months ago
Is LongLoRA can be mixed with YaRN ?
Updated 7 months ago
推理时候显存分配
Updated 7 months ago1
Adapting to new models
Updated 7 months ago2
如何在LoRA训练中加入embed和norm层的训练？
Updated 7 months ago
Lora+deepspeed zero3 无法保存lora权重问题
Closed 7 months ago6
What llama attn replacement to use for SFT-based inference?
Updated 7 months ago
To save model in HF format after supervised-fine-tune-qlora
Updated 7 months ago7
在没有报错的情况下，LongAlpaca-7B只对文本的第一段文字进行了响应
Updated 7 months ago
Configs in inference.py necessary for context length expansion in model serving?
Updated 7 months ago
训练的时候使用的什么外推方式
Updated 7 months ago
支持qwen、baichuan等中文模型微调吗
Updated 7 months ago
torch.cuda.OutOfMemoryError: CUDA out of memory
Closed 8 months ago3
inference OOM
Updated 8 months ago
推理 group整除问题
Closed 8 months ago1
Is LongAlpaca model fine-tuned from llama-2 or the Alpaca model?
Updated 8 months ago
Can LongLoRA be used for incremental pre-training?
Updated 8 months ago
the current text generation call will exceed the model's predefined maximum length (4096)
Updated 8 months ago4
中文领域进展
Closed 8 months ago1
32k inference result is garbled
Updated 8 months ago8
Question about inference use Llama-2-7b-longlora-8k-ft output nothing
Closed 8 months ago4
微调数据
Closed 8 months ago
Inquiry Regarding the Tokenize Function
Closed 8 months ago3
Qustions about dynamic NTK interpolation fine-tuning and non-linear interpolation methods
Updated 8 months ago1
扩充词表后，不改变其他代码和参数，预训练过程中能否对新添加的词元进行训练
Closed 8 months ago
How did you design questions and answers in the LongQA dataset?
Closed 8 months ago1