LoRA results in 4-6% lower performance compared to full fine-tuning

Question

LoRA results in 4-6% lower performance compared to full fine-tuning

digvijayingle016 opened this issue a year ago · comments

I am working on fine-tuning LLMs (6B to 40B parameters) using the LoRA framework on an instruction tuning dataset comprising of instructions corresponding to ~20 tasks (a mix of factual as well as open-ended tasks). The input to the model consists of a conversation snippet between two individuals along with a task-specific prompt. The results I am observing do not align with the performance improvements reported in the paper. Specifically, the paper reports that fine-tuning using LoRA generally results in performance at par with or better than full fine-tuning of the model, however, throughout our experiments I observe a performance lower than full fine-tuning by an absolute margin of ~4-6% in terms of RougeL score.

Sharing some of the training details below:

[Framework versions]
Python: 3.8
PyTorch: 1.13.1
Transformers: 4.27.4
PEFT: 0.3.0

[Infrastructure]
8 X A100 40 GB GPUs

[Hyper-parameter Range]
Learning rate: 5e-5 to 3e-3
Learning rate scheduler: [Constant, Linear]
Epochs: [1, 2]
Batch size: [2, 4, 8]
Weight decay: 0.0
Precision: bf16

Specifically, I tried fine-tuning of google/flan-t5-xxl model in following two scenarios:

Scenario 1
Full fine-tuning with constant learning rate = 5e-5, batch size = 8, epochs = 1
Scenario 2
Fine-tuning using LoRA with constant learning rate = 1e-3, batch size = 8, epochs = 1 and LoraConfig as follows:
LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05, bias='none', task_type="SEQ_2_SEQ_LM")

Observation: Scenario 2 resulted in 4% lower RougeL as compared to scenario 1. I have also tried tuning the hyper-parameters in Scenario 2 as per the range specified above, however, the best I could get is to a gap of ~4% RougeL.

Thank you very much for your time and consideration. Looking forward to any relevant insights here.

Younes Belkada · Answer 1 · Fri Jun 23 2023 19:01:20 GMT+0800 (China Standard Time)

Thanks for brining up this discussion, what I would try first is probably increasing the r argument to 16 or 32 to add more trainable parameters to the LoRA modules
Also note that in the QLoRA paper: https://arxiv.org/abs/2305.14314 the authors suggest to adapt all linear layers including FFN layers and not the attention layers only (default)

digvijayingle016 · Answer 2 · Mon Jun 26 2023 13:46:25 GMT+0800 (China Standard Time)

@younesbelkada - Thank you for the suggestion above. I did try experimenting with higher rank (16, 32), however the performance does not seem to change drastically. This is in alignment with the observations in the QLoRA paper stating that that the projection dimension does not impact the performance.

Although, based on the above suggestion, I also tried adapting all other linear layers (including FFN layers). The performance improved by 1%, however, there is still a gap of ~3% compared to full fine-tuning.

Younes Belkada · Answer 3 · Mon Jun 26 2023 15:56:26 GMT+0800 (China Standard Time)

I see thanks for the experiments, can you also double check that the LoRA weights are set on the encoder as well?
Also what is the task you are trying to fine-tune on, and do you quantize the base model in 8bit?

digvijayingle016 · Answer 4 · Mon Jun 26 2023 17:56:35 GMT+0800 (China Standard Time)

Yes, the LoRA weights are set on both the encoder and decoder as well. Here is the list of modules for first block in encoder and decoder

Encoder

base_model.model.encoder.block.0.layer.0.SelfAttention.q
base_model.model.encoder.block.0.layer.0.SelfAttention.q.lora_dropout
base_model.model.encoder.block.0.layer.0.SelfAttention.q.lora_dropout.default
base_model.model.encoder.block.0.layer.0.SelfAttention.q.lora_A
base_model.model.encoder.block.0.layer.0.SelfAttention.q.lora_A.default
base_model.model.encoder.block.0.layer.0.SelfAttention.q.lora_B
base_model.model.encoder.block.0.layer.0.SelfAttention.q.lora_B.default
base_model.model.encoder.block.0.layer.0.SelfAttention.q.lora_embedding_A
base_model.model.encoder.block.0.layer.0.SelfAttention.q.lora_embedding_B
base_model.model.encoder.block.0.layer.0.SelfAttention.k
base_model.model.encoder.block.0.layer.0.SelfAttention.k.lora_dropout
base_model.model.encoder.block.0.layer.0.SelfAttention.k.lora_dropout.default
base_model.model.encoder.block.0.layer.0.SelfAttention.k.lora_A
base_model.model.encoder.block.0.layer.0.SelfAttention.k.lora_A.default
base_model.model.encoder.block.0.layer.0.SelfAttention.k.lora_B
base_model.model.encoder.block.0.layer.0.SelfAttention.k.lora_B.default
base_model.model.encoder.block.0.layer.0.SelfAttention.k.lora_embedding_A
base_model.model.encoder.block.0.layer.0.SelfAttention.k.lora_embedding_B
base_model.model.encoder.block.0.layer.0.SelfAttention.v
base_model.model.encoder.block.0.layer.0.SelfAttention.v.lora_dropout
base_model.model.encoder.block.0.layer.0.SelfAttention.v.lora_dropout.default
base_model.model.encoder.block.0.layer.0.SelfAttention.v.lora_A
base_model.model.encoder.block.0.layer.0.SelfAttention.v.lora_A.default
base_model.model.encoder.block.0.layer.0.SelfAttention.v.lora_B
base_model.model.encoder.block.0.layer.0.SelfAttention.v.lora_B.default
base_model.model.encoder.block.0.layer.0.SelfAttention.v.lora_embedding_A
base_model.model.encoder.block.0.layer.0.SelfAttention.v.lora_embedding_B
base_model.model.encoder.block.0.layer.0.SelfAttention.o
base_model.model.encoder.block.0.layer.0.SelfAttention.o.lora_dropout
base_model.model.encoder.block.0.layer.0.SelfAttention.o.lora_dropout.default
base_model.model.encoder.block.0.layer.0.SelfAttention.o.lora_A
base_model.model.encoder.block.0.layer.0.SelfAttention.o.lora_A.default
base_model.model.encoder.block.0.layer.0.SelfAttention.o.lora_B
base_model.model.encoder.block.0.layer.0.SelfAttention.o.lora_B.default
base_model.model.encoder.block.0.layer.0.SelfAttention.o.lora_embedding_A
base_model.model.encoder.block.0.layer.0.SelfAttention.o.lora_embedding_B
base_model.model.encoder.block.0.layer.0.SelfAttention.relative_attention_bias
base_model.model.encoder.block.0.layer.0.layer_norm
base_model.model.encoder.block.0.layer.0.dropout
base_model.model.encoder.block.0.layer.1
base_model.model.encoder.block.0.layer.1.DenseReluDense
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_0
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_0.lora_dropout
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_0.lora_dropout.default
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_0.lora_A
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_0.lora_A.default
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_0.lora_B
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_0.lora_B.default
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_0.lora_embedding_A
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_0.lora_embedding_B
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_1
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_1.lora_dropout
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_1.lora_dropout.default
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_1.lora_A
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_1.lora_A.default
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_1.lora_B
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_1.lora_B.default
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_1.lora_embedding_A
base_model.model.encoder.block.0.layer.1.DenseReluDense.wi_1.lora_embedding_B
base_model.model.encoder.block.0.layer.1.DenseReluDense.wo
base_model.model.encoder.block.0.layer.1.DenseReluDense.wo.lora_dropout
base_model.model.encoder.block.0.layer.1.DenseReluDense.wo.lora_dropout.default
base_model.model.encoder.block.0.layer.1.DenseReluDense.wo.lora_A
base_model.model.encoder.block.0.layer.1.DenseReluDense.wo.lora_A.default
base_model.model.encoder.block.0.layer.1.DenseReluDense.wo.lora_B
base_model.model.encoder.block.0.layer.1.DenseReluDense.wo.lora_B.default
base_model.model.encoder.block.0.layer.1.DenseReluDense.wo.lora_embedding_A
base_model.model.encoder.block.0.layer.1.DenseReluDense.wo.lora_embedding_B
base_model.model.encoder.block.0.layer.1.DenseReluDense.dropout
base_model.model.encoder.block.0.layer.1.DenseReluDense.act
base_model.model.encoder.block.0.layer.1.layer_norm
base_model.model.encoder.block.0.layer.1.dropout

Decoder

base_model.model.decoder.block.0.layer.0.SelfAttention.q
base_model.model.decoder.block.0.layer.0.SelfAttention.q.lora_dropout
base_model.model.decoder.block.0.layer.0.SelfAttention.q.lora_dropout.default
base_model.model.decoder.block.0.layer.0.SelfAttention.q.lora_A
base_model.model.decoder.block.0.layer.0.SelfAttention.q.lora_A.default
base_model.model.decoder.block.0.layer.0.SelfAttention.q.lora_B
base_model.model.decoder.block.0.layer.0.SelfAttention.q.lora_B.default
base_model.model.decoder.block.0.layer.0.SelfAttention.q.lora_embedding_A
base_model.model.decoder.block.0.layer.0.SelfAttention.q.lora_embedding_B
base_model.model.decoder.block.0.layer.0.SelfAttention.k
base_model.model.decoder.block.0.layer.0.SelfAttention.k.lora_dropout
base_model.model.decoder.block.0.layer.0.SelfAttention.k.lora_dropout.default
base_model.model.decoder.block.0.layer.0.SelfAttention.k.lora_A
base_model.model.decoder.block.0.layer.0.SelfAttention.k.lora_A.default
base_model.model.decoder.block.0.layer.0.SelfAttention.k.lora_B
base_model.model.decoder.block.0.layer.0.SelfAttention.k.lora_B.default
base_model.model.decoder.block.0.layer.0.SelfAttention.k.lora_embedding_A
base_model.model.decoder.block.0.layer.0.SelfAttention.k.lora_embedding_B
base_model.model.decoder.block.0.layer.0.SelfAttention.v
base_model.model.decoder.block.0.layer.0.SelfAttention.v.lora_dropout
base_model.model.decoder.block.0.layer.0.SelfAttention.v.lora_dropout.default
base_model.model.decoder.block.0.layer.0.SelfAttention.v.lora_A
base_model.model.decoder.block.0.layer.0.SelfAttention.v.lora_A.default
base_model.model.decoder.block.0.layer.0.SelfAttention.v.lora_B
base_model.model.decoder.block.0.layer.0.SelfAttention.v.lora_B.default
base_model.model.decoder.block.0.layer.0.SelfAttention.v.lora_embedding_A
base_model.model.decoder.block.0.layer.0.SelfAttention.v.lora_embedding_B
base_model.model.decoder.block.0.layer.0.SelfAttention.o
base_model.model.decoder.block.0.layer.0.SelfAttention.o.lora_dropout
base_model.model.decoder.block.0.layer.0.SelfAttention.o.lora_dropout.default
base_model.model.decoder.block.0.layer.0.SelfAttention.o.lora_A
base_model.model.decoder.block.0.layer.0.SelfAttention.o.lora_A.default
base_model.model.decoder.block.0.layer.0.SelfAttention.o.lora_B
base_model.model.decoder.block.0.layer.0.SelfAttention.o.lora_B.default
base_model.model.decoder.block.0.layer.0.SelfAttention.o.lora_embedding_A
base_model.model.decoder.block.0.layer.0.SelfAttention.o.lora_embedding_B
base_model.model.decoder.block.0.layer.0.SelfAttention.relative_attention_bias
base_model.model.decoder.block.0.layer.0.layer_norm
base_model.model.decoder.block.0.layer.0.dropout
base_model.model.decoder.block.0.layer.1
base_model.model.decoder.block.0.layer.1.EncDecAttention
base_model.model.decoder.block.0.layer.1.EncDecAttention.q
base_model.model.decoder.block.0.layer.1.EncDecAttention.q.lora_dropout
base_model.model.decoder.block.0.layer.1.EncDecAttention.q.lora_dropout.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.q.lora_A
base_model.model.decoder.block.0.layer.1.EncDecAttention.q.lora_A.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.q.lora_B
base_model.model.decoder.block.0.layer.1.EncDecAttention.q.lora_B.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.q.lora_embedding_A
base_model.model.decoder.block.0.layer.1.EncDecAttention.q.lora_embedding_B
base_model.model.decoder.block.0.layer.1.EncDecAttention.k
base_model.model.decoder.block.0.layer.1.EncDecAttention.k.lora_dropout
base_model.model.decoder.block.0.layer.1.EncDecAttention.k.lora_dropout.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.k.lora_A
base_model.model.decoder.block.0.layer.1.EncDecAttention.k.lora_A.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.k.lora_B
base_model.model.decoder.block.0.layer.1.EncDecAttention.k.lora_B.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.k.lora_embedding_A
base_model.model.decoder.block.0.layer.1.EncDecAttention.k.lora_embedding_B
base_model.model.decoder.block.0.layer.1.EncDecAttention.v
base_model.model.decoder.block.0.layer.1.EncDecAttention.v.lora_dropout
base_model.model.decoder.block.0.layer.1.EncDecAttention.v.lora_dropout.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.v.lora_A
base_model.model.decoder.block.0.layer.1.EncDecAttention.v.lora_A.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.v.lora_B
base_model.model.decoder.block.0.layer.1.EncDecAttention.v.lora_B.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.v.lora_embedding_A
base_model.model.decoder.block.0.layer.1.EncDecAttention.v.lora_embedding_B
base_model.model.decoder.block.0.layer.1.EncDecAttention.o
base_model.model.decoder.block.0.layer.1.EncDecAttention.o.lora_dropout
base_model.model.decoder.block.0.layer.1.EncDecAttention.o.lora_dropout.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.o.lora_A
base_model.model.decoder.block.0.layer.1.EncDecAttention.o.lora_A.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.o.lora_B
base_model.model.decoder.block.0.layer.1.EncDecAttention.o.lora_B.default
base_model.model.decoder.block.0.layer.1.EncDecAttention.o.lora_embedding_A
base_model.model.decoder.block.0.layer.1.EncDecAttention.o.lora_embedding_B
base_model.model.decoder.block.0.layer.1.layer_norm
base_model.model.decoder.block.0.layer.1.dropout
base_model.model.decoder.block.0.layer.2
base_model.model.decoder.block.0.layer.2.DenseReluDense
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_0
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_0.lora_dropout
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_0.lora_dropout.default
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_0.lora_A
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_0.lora_A.default
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_0.lora_B
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_0.lora_B.default
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_0.lora_embedding_A
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_0.lora_embedding_B
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_1
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_1.lora_dropout
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_1.lora_dropout.default
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_1.lora_A
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_1.lora_A.default
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_1.lora_B
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_1.lora_B.default
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_1.lora_embedding_A
base_model.model.decoder.block.0.layer.2.DenseReluDense.wi_1.lora_embedding_B
base_model.model.decoder.block.0.layer.2.DenseReluDense.wo
base_model.model.decoder.block.0.layer.2.DenseReluDense.wo.lora_dropout
base_model.model.decoder.block.0.layer.2.DenseReluDense.wo.lora_dropout.default
base_model.model.decoder.block.0.layer.2.DenseReluDense.wo.lora_A
base_model.model.decoder.block.0.layer.2.DenseReluDense.wo.lora_A.default
base_model.model.decoder.block.0.layer.2.DenseReluDense.wo.lora_B
base_model.model.decoder.block.0.layer.2.DenseReluDense.wo.lora_B.default
base_model.model.decoder.block.0.layer.2.DenseReluDense.wo.lora_embedding_A
base_model.model.decoder.block.0.layer.2.DenseReluDense.wo.lora_embedding_B
base_model.model.decoder.block.0.layer.2.DenseReluDense.dropout
base_model.model.decoder.block.0.layer.2.DenseReluDense.act
base_model.model.decoder.block.0.layer.2.layer_norm
base_model.model.decoder.block.0.layer.2.dropout

Also, I am not quantizing the base model in 8bit, instead using bf16 precision for fine-tuning.

Additionally, the training dataset consists of instructions for ~20 tasks (a mixture of summarization, classification, paraphrasing, etc.)

I am wondering if the setup is sensitive to hyperparameters. Specifically,

Is there a thumb-rule that one can use to determine what range of hyper-parameters generally work best for a given class of models?
Any specific stopping criteria that one should use while fine-tuning using LoRA? If full fine-tuning leads to performance X with N epochs, is LoRA also expected to result in similar performance in N epochs. Or should one be training longer while using LoRA?

Sourab Mangrulkar · Answer 5 · Tue Jun 27 2023 21:22:38 GMT+0800 (China Standard Time)

Hello @digvijayingle016, have you tried also making biases as trainable via bias config param ('all' or 'lora_only'). Also, try training longer with LoRA. In LoRA paper, they train for 10-30 epochs on GLUE with Roberta-large while in RoBerta paper, the max epochs for GLUE dataset was 10. So, indeed training longer seems important with PEFT methods.

github-actions · Answer 6 · Sun Jul 23 2023 23:03:07 GMT+0800 (China Standard Time)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Younes Belkada · Answer 7 · Mon Jul 24 2023 20:12:18 GMT+0800 (China Standard Time)

Closing the issue, feel free to-reopen if you have more questions