Giters

Lightning-AI / lit-llama

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Using llama3 through lit lama

fireyanci opened this issue · comments

commented

If I want to use llama3 through lit lama, how can I modify it? I found that the model structure of llama3 has changed

commented

While this repository is only focused on the first Llama model to keep the code as simple and readable as possible, we have the LitGPT repository (which is an extension of Lit-Llama) that also now supports Llama 3 as of Friday, thanks to @awaelchli

Hopefully this addressed your needs. And please let us know in case you bump into any issues.

commented

While this repository is only focused on the first Llama model to keep the code as simple and readable as possible, we have the LitGPT repository (which is an extension of Lit-Llama) that also now supports Llama 3 as of Friday, thanks to @awaelchli

Hopefully this addressed your needs. And please let us know in case you bump into any issues.

Thank you for your answer

commented

While this repository is only focused on the first Llama model to keep the code as simple and readable as possible, we have the LitGPT repository (which is an extension of Lit-Llama) that also now supports Llama 3 as of Friday, thanks to @awaelchli

Hopefully this addressed your needs. And please let us know in case you bump into any issues.

Due to meta-ai lama-3-8b not using sentencePiece, but litgpt/tokenizer. py still uses sentencePiece. When I use the
litgpt chat -- checkpoint_dir E: \ checkpoint \ meta llama \ Meta-Llama-3-8B Instrument
RuntimeError: Internal: could not part ModelProto from E: \ checkpoint \ meta llama \ Meta Llama-3-8B Instrument \ tokenizer. model
Is there a new version of tokenizer. py that is compatible with llama-3 in Litgpt

commented

I just saw your comment also in Lightning-AI/litgpt#1333. Let's continue the discussion there.

commented

@fireyanci A few notes here:

  • LitLlama is no longer being developed. The replacement is LitGPT. If you want to use Lama3 without major effort, please use LitGPT directly.
  • It is possible to make changes in LitLlama to support Llama3, but you'll have to update several files. If this is very important for you, I suggest start by copying over model.py, config.py and tokenizer.py and then work from there. But again, at this point you could use LitGPT directly.
  • In LitGPT, the tokenizer supports loading both sentenciepiece and huggingface tokenizers (including LLama3). Follow the download tutorial to get the LLama3 weights from HF.
  • Your messages indicate you are using a model called "Meta-Llama-3-8B Instrument". This does not exist. meta-llama/Meta-Llama-3-8B-Instruct is the correct name. Please try with this again.