Why in-distribution classification results are so bad?
algoteam5 opened this issue · comments
I tested your model. The model could detect ood images quite well, but when it comes to classifying in-distribution classes. The model produced very bad results with wrong class label. So why in-distribution classification results are so bad?
Hello. Can you be more specific your test scenario? Is it CIFAR10 and 100 as we tested? In table2, we did report in-distribution test error and it does not harm the performance much.
If you use it in other dataset, you might want to tune the bounds min and mout so that they do not shift the energy of samples produced by the pre-trained models too much. Otherwise, the model will strive to adapt the bounds. For example, if the pre-trained model of in-dist samples is -0.5 and out-of-dist samples -0.3, you can set the min to -1 and OOD samples to -0. But if you set them to be min=-20 and mout=-19, the model will try to adapt to this new bounds for all the samples it trains on and this will harm your classificaiton performance.
@wetliu Thank you very much for your answers. I am using my custom dataset. What are the rules of thumb for setting m_in and m_out? In my custom dataset I set m_in=-27, m_out=-5, is that the reason why in-distribution classification results are so bad?
You could first use the in-distribution test data to compute the E_in, and use the TinyImages80 (auxiliary dataset for fine-tuning) to compute E_out. It is expected E_in should be smaller than E_out. You can then set m_in slightly smaller than E_in and m_out slightly bigger than E_out. The key is that we hope this loss term does not chime in very often to degrade the in-distribution classification performance.