Lightning-AI / lit-llama

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

Where is tokenizer.model? tokenizer path

andreamigliorati opened this issue 3 months ago · comments

andreamigliorati commented 3 months ago

Hi, I'm confused about where to find the tokenizer:

--tokenizer_path checkpoints/lit-llama/tokenizer.model

Referring here to the readme:

Where can I download it?

daxian-lh commented 2 months ago

Did you solve the problem?