Giters

Lightning-AI / lit-llama

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Looking for LLaMA 2?

carmocca opened this issue · comments

commented

Meta AI has since released LLaMA 2. Additionally, new Apache 2.0 licensed weights are being released as part of the Open LLaMA project.

To run LLaMA 2 weights, Open LLaMA weights, or Vicuna weights (among other LLaMA-like checkpoints), check out the Lit-GPT repository.

commented

Hi @carmocca,
I wonder if it supports Full Parameter Supervised Training for LLaMA2 on customized dataset?
Thank you.

commented

Yes, full finetuning is supported via finetune/full.py script given a Llama 2 model provided via the --checkpoint_dir in Lit-GPT.

You can also use a custom dataset given that you prepare it in the right format. You can see the prepare_*.py scripts here for guidance.

commented

Is this repo still intended to be supported given since it seems like the lit-gpt repo supports more and newer models.