πTraining LLaMA with MMEngine!
LLaMA.MMEngine is an experimental repository that leverages the MMEngine training engine, originally designed for computer vision tasks, to train and fine-tune language models. The primary goal of this project is to explore the compatibility of MMEngine with language models, learn about fine-tuning techniques, and engage with the open-source community for knowledge sharing and collaboration.
- Support for loading LLaMA models with parameter sizes ranging from 7B to 65B
- Instruct tuning support
- low-rank adaptation (LoRA) fine-tuning support
- int8 quantization support
- improve the generate script
- support show validation loss
-
Install PyTorch
Following this guide https://pytorch.org/get-started/locally/
-
Setup this repo
Clone the repo
git clone https://github.com/RangiLyu/llama.mmengine cd llama.mmengine
Install dependencies
pip install -r requirements.txt
Run setup.py
python setup.py develop
Please Download the model weights from the official LLaMA repo.
The checkpoints folder should be like this:
checkpoints/llama
βββ 7B
β βββ checklist.chk
β βββ consolidated.00.pth
β βββ params.json
βββ 13B
β ...
βββ tokenizer_checklist.chk
βββ tokenizer.model
Convert the weights (Thanks for the script from Lit-LLaMA):
python scripts/convert_checkpoint.py \
--output_dir checkpoints/mm-llama \
--ckpt_dir checkpoints/llama \
--tokenizer_path checkpoints/llama/tokenizer.model \
--model_size 7B
python tools/train.py configs/llama-7B_finetune_3e.py
python tools/generate.py configs/llama-7B_finetune_3e.py work_dirs/llama-7B_finetune_3e/epoch_3.pth
I greatly appreciate your interest in contributing to LLaMA.MMEngine! Please note that this project is maintained as a personal side project, which means that available time for development and support is limited. With that in mind, I kindly encourage members of the community to get involved and actively contribute by submitting pull requests!
- @Lightning-AI for Lit-LLaMA οΈ
- @tloen for Alpaca-LoRA
- Stanford Alpaca
- LLaMA