Table results from the last model or the best model?

Question

Table results from the last model or the best model?

fanq15 opened this issue 2 years ago · comments

Hi, thanks for your great work!
I wonder the model performance in your paper tables.
Is it from the last trained model? Or from the model with the best testing performance?

Qi Fan commented 2 years ago

Thanks!

Yuanhan Zhang · Answer 1 · Tue Jun 28 2022 08:35:34 GMT+0800 (China Standard Time)

Best model

Zheda (Marco) Mai · Answer 2 · Fri Apr 07 2023 10:36:37 GMT+0800 (China Standard Time)

Thanks for your answer. When you said the best model, does it mean the model with the best testing performance? Shouldn't we use the validation set to find the best model? Thanks in advance and look forward to hearing back from you.

Yuanhan Zhang · Answer 3 · Fri Apr 07 2023 11:23:23 GMT+0800 (China Standard Time)

Thanks for your answer. When you said the best model, does it mean the model with the best testing performance? Shouldn't we use the validation set to find the best model? Thanks in advance and look forward to hearing back from you.

Greetings,

My apologies for any confusion caused earlier.

The NOAH approach comprises two stages. (1) find the optimal prompt structure. (2) the model, which is implied by the optimal prompt structure. is retrained using this optimal prompt structure.

In the first stage, the optimal prompt structure is determined based on the validation set. Subsequently, during the second stage, the model's performance is evaluated and reported using the test set.

The reported result for the retrained model typically does not correspond to the final training step, but rather reflects the highest performance attained on the test set.

In summary, your assertion is correct, and we indeed use the validation set to identify the most effective model.