ryan-air / Alpaca-350M-Fine-Tuned

In this project, I have provided code and a Colaboratory notebook that facilitates the fine-tuning process of an Alpaca 350M parameter model originally developed at Stanford University. The model was adapted using LoRA to run with fewer computational resources and training parameters and used HuggingFace's PEFT library.

Alpaca-350M-Fine-Tuned

Professional work-related project

In this project, I have provided code and a Colaboratory notebook that facilitates the fine-tuning process of an Alpaca 350M parameter model originally developed at Stanford University. The particular model that is being fine-tuned has around 350 million parameters, which is one of the smaller Alpaca models (smaller than my previous fine-tuned model).

The model uses low-rank adaptation LoRA to run with fewer computational resources and training parameters. We use bitsandbytes to set up and run in an 8-bit format so it can be used on colaboratory. Furthermore, the PEFT library from HuggingFace was used for fine-tuning the model.

Hyper Parameters:

1. MICRO_BATCH_SIZE = 4 (4 works with a smaller GPU)
2. BATCH_SIZE = 32
3. GRADIENT_ACCUMULATION_STEPS = BATCH_SIZE // MICRO_BATCH_SIZE
4. EPOCHS = 2 (Stanford's Alpaca uses 3)
5. LEARNING_RATE = 2e-5 (Stanford's Alpaca uses 2e-5)
6. CUTOFF_LEN = 256 (Stanford's Alpaca uses 512, but 256 accounts for 96% of the data and runs far quicker)
7. LORA_R = 4
8. LORA_ALPHA = 16
9. LORA_DROPOUT = 0.05

Credit for Original Model: Qiyuan Ge

Fine-Tuned Model: RyanAir/Alpaca-350M-Fine-Tuned (HuggingFace)