warning upon loading Tokenizer?
math-yyj opened this issue · comments
I've encountered the following warnings when trying fast-mode,
I've tried loading BlipTokenizer, but why turned out to be the "BlipTokenizer" warning?
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'BertTokenizer'.
The class this function is called from is 'BlipTokenizer'.
Besides, my demo run on an offline env,
where I can not download pretrained weights online
Is there any suggestions? Many thanks!
This warning is expected to be normal. We use the tokenizer config from here. Compared to the Bert tokenizer, Blip tokenizer only introduces two additional special tokens. Therefore, the code initially loads the Bert tokenizer and then appends two special tokens, which triggers the warning during tokenizer loading. For further details, refer to this file.
As for the pre-trained weights, you can download them to your local machine first, and then modify the paths in the config.
problem solved, thank you very much.