This repository is implementation of Homework 3 for CSIE5431 Applied Deep Learning course in 2023 Fall semester at National Taiwan University.
![instruction_tuning](https://private-user-images.githubusercontent.com/55970911/286495045-68bba37f-8695-4335-b296-960f9e753cd9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE0ODg3NzUsIm5iZiI6MTcyMTQ4ODQ3NSwicGF0aCI6Ii81NTk3MDkxMS8yODY0OTUwNDUtNjhiYmEzN2YtODY5NS00MzM1LWIyOTYtOTYwZjllNzUzY2Q5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzIwVDE1MTQzNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTA0NGEzMjY2NDE1YzBiYTRjMjY5ZTUyZTRjMWQwYTUzYzlhMGQ0YmI0YTAzY2M4NmM3MDNhZjFlNWFkYmFlZDQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.VUVjFHJCQdfSudUDW2G0o03xPLwtxDFmGrnpiCvhrYg)
![lora](https://private-user-images.githubusercontent.com/55970911/286495685-6d7cb5db-63db-4767-8897-8f140dfc2ebe.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE0ODg3NzUsIm5iZiI6MTcyMTQ4ODQ3NSwicGF0aCI6Ii81NTk3MDkxMS8yODY0OTU2ODUtNmQ3Y2I1ZGItNjNkYi00NzY3LTg4OTctOGYxNDBkZmMyZWJlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzIwVDE1MTQzNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM5ZjdmYmIyNzA2MGU1NDRlOTI3NmM1YTQxOTNkNTRmODAzNTgzNTlkZGNmYzc0MGFmNzJhMTRlZmU0NmVmNzQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.o-Kl7SWukimgdIxo3EO1SyOxdJgNgwfhU7OFFPiOG04)
To set the environment, you can run this command:
pip install -r configs/requirements.txt
To download the LoRA checkpoint, you can run the command:
bash ./download.sh
The dataset comprises keys labeled id, instruction, and output. An example of the data structure is displayed as follows:
{
"id": "2fb7d211-978f-41c8-a3ab-e51d9df06280",
"instruction": "翻譯成文言文:\n於是,廢帝讓瀋慶之的堂侄、直將軍瀋攸之賜瀋慶之毒藥,命瀋慶之自殺。",
"output": "帝乃使慶之從父兄子直閣將軍攸之賜慶之藥。"
}
Note: If you wish to download the Taiwan-LLaMa pretrained weights and dataset, please uncomment the lines in the download.sh
script.
To reproduce our best result, you can run the command:
bash ./run.sh <pretrain model folder> <lora model folder> <input data path> <output file path>
For example:
bash ./run.sh \
pretrain/Taiwan-LLM-7B-v2.0-chat \
adapter_checkpoint \
data/public_test.json \
public_prediction.json
To fine-tune the Taiwan-LLaMa model, you can run the command:
python train.py --base_model_path <pretrain model folder> \
--train_data_path <train data path> \
--valid_data_path <valid data path> \
--train_num <number of used training data> \
--epoch <number of epochs> \
--batch_size <number of training batch size> \
--accum_grad_step <number of accumulated gradient batch size> \
--lr <learning rate> \
--lr_scheduler <learning rate scheduler> \
--warm_up_step <number of warm up step>
--lora_rank <rank of LoRA>
To inference the Taiwan-LLaMa model, you can run the command:
python infer.py --method <support method: lora-fine-tune, zero-shot, and few-shot> \
--base_model_path <pretrain model folder> \
--peft_path <lora model folder> \
--test_data_path <test data path> \
--output_path <output file oath>
To demo the conversation with Taiwan-LLaMa model, you can run the command:
python demo.py --method <support method: lora-fine-tune, zero-shot, and few-shot>
--base_model_path <pretrain model folder> \
--peft_path <lora model folder> \
--test_data_path <test data path>
We implemented the code on an environment running Ubuntu 22.04.1, utilizing a 12th Generation Intel(R) Core(TM) i7-12700 CPU, along with a single NVIDIA GeForce RTX 4090 GPU equipped with 24 GB of dedicated memory.
We thank the Taiwan-LLaMa repository: https://github.com/MiuLab/Taiwan-LLaMa
@misc{
title = {Taiwan-LLaMa Instruction Tuning},
author = {Jia-Wei Liao},
url = {https://github.com/jwliao1209/Taiwan-LLaMa-Instruction-Tuning.git},
year = {2023}
}