Tokenizer does not have a padding token
tomcobley opened this issue · comments
Hi, thanks for sharing your model!
I am trying to use it to generate embeddings of batches of sequences of text of different lengths (Gene Ontology annotations). However, when I try to do this using huggingface
, I get the following error at the tokenization stage.
Code:
tokenizer = GPT2Tokenizer.from_pretrained("stanford-crfm/pubmed_gpt_tokenizer")
inputs = tokenizer(sequences, padding=True, return_tensors="pt")
Error:
Asking to pad but the tokenizer does not have a padding token. Please select a token to use as
`pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via
`tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
How should I resolve this?
Thanks!
It seems this was a more general question which has an explanation here (the choice of pad_token
doesn't actually matter since the attention mask causes padded indices to be ignored).
Adding the line
tokenizer.pad_token = tokenizer.eos_token
removed the error and solved the problem :).