ayulockin / lit-gpt

Hackable implementation of state-of-the-art open-source LLMs based on nanoGPT. Supports flash attention, 4-bit and 8-bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

⚡ Lit-GPT

 

⚡ Lit-GPT

Hackable implementation of state-of-the-art open-source large language models released under the Apache 2.0 license.

Supports the following popular model checkpoints:

Model and usage	Reference
Meta AI Llama 2	Touvron et al. 2023
Stability AI FreeWilly2	Stability AI 2023
TII UAE Falcon	TII 2023
OpenLM Research OpenLLaMA	Geng & Liu 2023
LMSYS Vicuna	Li et al. 2023
Together RedPajama-INCITE	Together 2023
EleutherAI Pythia	Biderman et al. 2023
StabilityAI StableLM	Stability AI 2023

This implementation extends on Lit-LLaMA and nanoGPT, and it's powered by Lightning Fabric ⚡.

 

---

🏆 NeurIPS 2023 Large Language Model Efficiency Challenge: 1 LLM + 1 GPU + 1 Day

The Lit-GPT repository is the official starter kit for the NeurIPS 2023 LLM Efficiency Challenge, which is a competition focused on finetuning an existing non-instruction tuned LLM for 24 hours on a single GPU. The competition has two tracks, one for the A100 and another for the 4090 GPUs.

If you are interested in participating, you can learn more about the NeurIPS LLM Efficiency Challenge on the official website here. Also see the Lit-GPT NeurIPS Challenge Quickstart Guide for helpful tips.

The submission deadline is Oct 15th, 2023.

---

 

Lit-GPT design principles

This repository follows the main principle of openness through clarity.

Lit-GPT is:

• Simple: Single-file implementation without boilerplate.
• Correct: Numerically equivalent to the original model.
• Optimized: Runs fast on consumer hardware or at scale.
• Open-source: No strings attached.

Avoiding code duplication is not a goal. Readability and hackability are.

 

Get involved!

Join our Discord to build high-performance, truly open-source models for the common benefit of the community.

 

Setup

Clone the repo

git clone https://github.com/Lightning-AI/lit-gpt
cd lit-gpt

Lit-GPT currently relies on flash attention from PyTorch nightly. Until PyTorch 2.1 is released you'll need to install nightly manually. Luckily this is straightforward, as shown below.

 

On CUDA

pip install --index-url https://download.pytorch.org/whl/nightly/cu118 --pre 'torch>=2.1.0dev'

On CPU (incl Macs)

pip install --index-url https://download.pytorch.org/whl/nightly/cpu --pre 'torch>=2.1.0dev'

(Optional) install Flash Attention 2

MAX_JOBS=4 pip install 'flash-attn>=2.0.0.post1' --no-build-isolation

All good, now install the dependencies plus some optional ones:

pip install -r requirements.txt tokenizers sentencepiece

You are all set! 🎉

 

Use the model

To generate text predictions, you need to download the model weights. If you don't have them, check out our guide.

Run inference:

python generate/base.py --prompt "Hello, my name is"

This will run the 3B pre-trained model and require ~7 GB of GPU memory using the bfloat16 datatype.

Full guide for generating samples from the model.

You can also chat with the model interactively:

python chat/base.py

 

Run large models on smaller consumer devices

We support 4-bit quantization (as in QLoRA), LLM.int8, and GPTQ.int4 inference by following this guide.

 

Finetune the model

We provide a simple training scripts (finetune/adapter.py, finetune/adapter_v2.py, and finetune/lora.py) that instruction-tunes a pretrained model on the Alpaca dataset.

1. Download the data and generate an instruction tuning dataset:

python scripts/prepare_alpaca.py

2. Run the finetuning script

For example, you can either use

Adapter (Zhang et al. 2023):

python finetune/adapter.py

or Adapter v2 (Gao et al. 2023):

python finetune/adapter_v2.py

or LoRA (Hu et al. 2021):

python finetune/lora.py

(Please see the tutorials/finetune_adapter for details on the differences between the two adapter methods.)

The finetuning requires at least one GPU with ~12 GB memory (RTX 3060).

It is expected that you have downloaded the pretrained weights as described above. More details about each finetuning method and how you can apply it to your own data can be found in our technical how-to guides.

 

Finetuning How-To Guides

These technical tutorials illustrate how to run the finetuning code.

• Finetune with Adapters
• Finetune with LoRA or QLoRA

 

Understanding Finetuning -- Conceptual Tutorials

Looking for conceptual tutorials and explanations? We have some additional articles below:

• Understanding Parameter-Efficient Finetuning of Large Language Models: From Prefix Tuning to LLaMA-Adapters

• Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA)

 

Pre-training

Porting from Lit-LLaMA in progress 👷

 

Get involved!

We are on a quest towards fully open source AI.

Join us and start contributing, especially on the following areas:

• Pre-training
• Fine-tuning
• Quantization
• Sparsification

We welcome all individual contributors, regardless of their level of experience or hardware. Your contributions are valuable, and we are excited to see what you can accomplish in this collaborative and supportive environment.

Unsure about contributing? Check out our How to Contribute to Lit-GPT and Lit-LLaMA guide.

Don't forget to join our Discord!

 

Acknowledgements

• @karpathy for nanoGPT
• @EleutherAI for GPT-NeoX
• @TimDettmers for bitsandbytes
• @IST-DASLab for GPTQ
• @Microsoft for LoRA
• @tridao for Flash Attention 2

 

License

Lit-GPT is released under the Apache 2.0 license.