Negative CoLA MCC - Prompt-Based Finetuning error?

[yash_b]

Hi, currently I am getting negative values for CoLA prompt-tuning tests.

python run.py
--task_name CoLA
--overwrite_cache
--data_dir data/k-shot/CoLA/16-13
--do_train
--do_eval \
--do_predict
--evaluate_during_training
--model_name_or_path roberta-base
--few_shot_type prompt
--num_k 16
--max_steps 1000
--eval_steps 100
--per_device_train_batch_size 2
--learning_rate 1e-5
--num_train_epochs 0
--output_dir result/BERT-LARGE-13-CoLA
--seed 13
--template "cls**sent_0_This_ismask.sep+"
--mapping "{'0':'incorrect','1':'correct'}"
--num_sample 16 \

My result was: -0.01883200893

I am just a little bit concerned if I am training the model correctly.

Any help would be greatly appreciated. Thank you for your assistance so far.

[Tianyu Gao]

Hi,

It is possible since you use the roberta-base and in the paper we use roberta-large. We noticed that for CoLA the prompt-based fine-tuning performed pretty bad in general. If you refer to Table 3 from our paper, you can see that for CoLA standard fine-tuning actually performs the best.

[yash_b]

Hi.

Thank you for your time.

Best,
Yash.