Warning about tokenizer
xushilin1 opened this issue · comments
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be pro
perly handled. We recommend you to read the related pull request available at huggingface/transformers#24565