QwenLM / Qwen2

Qwen2 is the large language model series developed by Qwen team, Alibaba Cloud.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

What is the difference between id_to_token() and decode() in tokenizer?

yileld opened this issue · comments

6CDB801D-5CC9-481c-90B3-2035F015004E
If I use tokenizer.id_to_token(), the text is very strange, but if I use tokenizer.decode() to decode ids, the result is right. Are these two method different?