Index not in list error when evaluating models zero-shot

[Rohan Taori]

Hi,

I would like to evaluate a few different BERT models zero-shot on MNLI, SNLI, QNLI, and RTE. As such, I am running this following command, for example:

TYPE=prompt TASK=MNLI BS=2 LR=1e-5 SEED=42 K=16 MODEL=roberta-large bash run_experiment.sh "--no_train"

Unfortunately, this gives me a somewhat cryptic error message:

Traceback (most recent call last):
  File "run.py", line 632, in <module>
    main()
  File "run.py", line 465, in main
    FewShotDataset(data_args, tokenizer=tokenizer, mode="test", use_demo=("demo" in model_args.few_shot_type))
  File "/sailhome/rtaori/zero-shot-generalization/LM-BFF/src/dataset.py", line 456, in __init__
    self.features.append(self.convert_fn(
  File "/sailhome/rtaori/zero-shot-generalization/LM-BFF/src/dataset.py", line 575, in convert_fn
    inputs = tokenize_multipart_input(
  File "/sailhome/rtaori/zero-shot-generalization/LM-BFF/src/dataset.py", line 243, in tokenize_multipart_input
    mask_pos = [input_ids.index(tokenizer.mask_token_id)]
ValueError: 50264 is not in list

A similar error occurs for SNLI, QNLI, and RTE.

Do you have any advice on how to deal with this? Any help would be much appreciated!

Thanks in advance,
Rohan

[Tianyu Gao]

Hi Rohan, sorry for the trouble! This is caused by a bug in the training script (it did not load task-specific parameters before). Now this is fixed. Thanks for letting us know!

Tianyu

[Rohan Taori]

Great! Thank you so much @gaotianyu1350 !