Giters

Lightning-AI / lit-llama

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Ban some tokens

AnnaKholkina opened this issue · comments

commented

Hello. I would like the model to not use some tokens (such as \n). When training a model, can I remove unnecessary tokens from the tokenizer and how to do this? And how will the removal of most tokens affect the quality of training? (let's say I want to train a model to speak only one language).

Thanks for your answers!