This repository is implementation of Homework 3 for CSIE5431 Applied Deep Learning course in 2023 Fall semester at National Taiwan University.
![instruction_tuning](https://private-user-images.githubusercontent.com/55970911/286495045-68bba37f-8695-4335-b296-960f9e753cd9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTg4Mzg5MjEsIm5iZiI6MTcxODgzODYyMSwicGF0aCI6Ii81NTk3MDkxMS8yODY0OTUwNDUtNjhiYmEzN2YtODY5NS00MzM1LWIyOTYtOTYwZjllNzUzY2Q5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjE5VDIzMTAyMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTcwMDEzNzUwNDdiMzI2NmIyY2E1MzVkMmY4NTQ1NTkyMTFjM2NmNGMwZjM3Yzk3YzQyN2I4ZTc0MTRjNTY2MWYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.6y_YM2QBtHAYPaCGZv6Zy3PqomS7UmADGAqBgiaduig)
![lora](https://private-user-images.githubusercontent.com/55970911/286495685-6d7cb5db-63db-4767-8897-8f140dfc2ebe.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTg4Mzg5MjEsIm5iZiI6MTcxODgzODYyMSwicGF0aCI6Ii81NTk3MDkxMS8yODY0OTU2ODUtNmQ3Y2I1ZGItNjNkYi00NzY3LTg4OTctOGYxNDBkZmMyZWJlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjE5VDIzMTAyMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTMzZWY0ZmNhYjNmNWM4N2FjYmVmOTFhMzBhNDQzMmNmNWIyMzNmODVkZDE4ZjRiZDFlYjNhZGU5MDNiMzg4NjImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.DyqAbexA8rPD9O3NfZM-d4Ao20Wvob-2L_Huy6cPVn4)
To set the environment, you can run this command:
pip install -r configs/requirements.txt
To download the LoRA checkpoint, you can run the command:
bash ./download.sh
The dataset comprises keys labeled id, instruction, and output. An example of the data structure is displayed as follows:
{
"id": "2fb7d211-978f-41c8-a3ab-e51d9df06280",
"instruction": "翻譯成文言文:\n於是,廢帝讓瀋慶之的堂侄、直將軍瀋攸之賜瀋慶之毒藥,命瀋慶之自殺。",
"output": "帝乃使慶之從父兄子直閣將軍攸之賜慶之藥。"
}
Note: If you wish to download the Taiwan-LLaMa pretrained weights and dataset, please uncomment the lines in the download.sh
script.
To reproduce our best result, you can run the command:
bash ./run.sh <pretrain model folder> <lora model folder> <input data path> <output file path>
For example:
bash ./run.sh \
pretrain/Taiwan-LLM-7B-v2.0-chat \
adapter_checkpoint \
data/public_test.json \
public_prediction.json
To fine-tune the Taiwan-LLaMa model, you can run the command:
python train.py --base_model_path <pretrain model folder> \
--train_data_path <train data path> \
--valid_data_path <valid data path> \
--train_num <number of used training data> \
--epoch <number of epochs> \
--batch_size <number of training batch size> \
--accum_grad_step <number of accumulated gradient batch size> \
--lr <learning rate> \
--lr_scheduler <learning rate scheduler> \
--warm_up_step <number of warm up step>
--lora_rank <rank of LoRA>
To inference the Taiwan-LLaMa model, you can run the command:
python infer.py --method <support method: lora-fine-tune, zero-shot, and few-shot> \
--base_model_path <pretrain model folder> \
--peft_path <lora model folder> \
--test_data_path <test data path> \
--output_path <output file oath>
To demo the conversation with Taiwan-LLaMa model, you can run the command:
python demo.py --method <support method: lora-fine-tune, zero-shot, and few-shot>
--base_model_path <pretrain model folder> \
--peft_path <lora model folder> \
--test_data_path <test data path>
We implemented the code on an environment running Ubuntu 22.04.1, utilizing a 12th Generation Intel(R) Core(TM) i7-12700 CPU, along with a single NVIDIA GeForce RTX 4090 GPU equipped with 24 GB of dedicated memory.
We thank the Taiwan-LLaMa repository: https://github.com/MiuLab/Taiwan-LLaMa
@misc{
title = {Taiwan-LLaMa Instruction Tuning},
author = {Jia-Wei Liao},
url = {https://github.com/jwliao1209/Taiwan-LLaMa-Instruction-Tuning.git},
year = {2023}
}