Evaluate on PACS

[GA-17]

Hi @KaiyangZhou ,
Thanks for sharing the code. I have the following questions:

1. How many times did you repeat on PACS?
2. Would you mind sharing the standard deviation of your PACS performances?
3. How did you select the model to report your results?

Thanks!

[Kaiyang]

How many times did you repeat on PACS?

The provided script runs the code 5 times

In the paper, I think I have run more than 5 times to have a smaller std

Would you mind sharing the standard deviation of your PACS performances?

Please see the paper

How did you select the model to report your results?

simply use the last-step checkpoint

another way is to use the median performance among the last, e.g., 10 steps, or use their average

[GA-17]

Thanks for your reply.
For the last question, I think the common way is to use the checkpoint which has the highest accuracy on validation dataset. And it usually has a relatively high variance. What do you think about it?

[Kaiyang]

due to domain shift, a higher performance on the same-domain val set doesn't necessarily give a better result on unseen domains (e.g. overfitting), but it's not uncommon to follow the standard cross-val way for selecting checkpoints

as I mentioned, you could also simply use the last, converged checkpoint which might lead to a smaller variance

I'd assume there is no further issue regarding the code so I'm closing it now