Giters

thunlp / PTR

Prompt Tuning with Rules

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

The dev set is init by the test set?

CheaSim opened this issue · comments

commented

PTR/code_script/run_prompt.py

Line 149 in 7cce6f1

name = "test",

I found that the val_dataset is actually the test dataset.

commented

If you want to use the dev set to select the best checkpoint, you can set name="val".

commented

In this code, the same code can work for both dev and test set.

commented

@THUCSTHanxu13 May I ask the score reported in the paper is obtained by selecting from the dev set or test set?

commented

For few-shot settings, the scores reported in the paper is obtained by selecting from the dev set. For full-set settings, the scores reported in the paper is obtained by the performance of the last checkpoint on the dev set.