How to choose which epoch to use?

[Yingda Xia]

Hi authors,

Great work! I really appreciate you can release code of your Neurips paper.

One question is that how do you select which epoch to use (for next step training or report final results). I trained a baseline for Cifar10 and find that the results varies from epoch to epoch. Do you have any specific criterions?

Thanks!

[Charles Corbière]

Hi YingdaXia,

Regarding the original classifier model, best epoch is selected on validation set accuracy. MCP, MCDropout and TrustScore results are all computed using this same baseline.

From that model, we attach ConfidNet layers and train them. We select best epoch for ConfidNet based on AUPR-Error on validation dataset and report results for that epoch.

More details about the implementation are available also on the NeurIPS paper's supplemental:
https://papers.nips.cc/paper/8556-addressing-failure-prediction-by-learning-model-confidence

Best,
Charles

[Yingda Xia]

Thank you so much for your explanations!

Best,
Yingda