Add qlora to our current codebase
samsja opened this issue · comments
Context
we want to add qlora (lora + 4 bits int quant) to our codebase.
The goal is to reduce memory usage and the cost of finetuning without degrading quality.
Steps
- activate 4bits in Peft. Should be as easy as turning on a flag
- run the modal on one GPU for one epoch and look at memory consumption compared to 8 bits training
- run a full training (3 epochs) with the same parameters as 8bits training and compare result to see if we don't degrade quality
you can get inspired by this PR: tloen/alpaca-lora#487