About the unstable performance

Question

About the unstable performance

tjulyz opened this issue 5 years ago · comments

Hi,
Thanks for your kind share. When I try to run the train step several times to get several models, I found that the performance are unstable. Sometimes, the AP3D for hardset may degrade by 3%. I have got results (~51.1, ~34.5, ~30.1 for easy, moderate, hard). Could you please help me?
Thanks a lot!

Peiliang · Answer 1 · Mon Sep 02 2019 11:48:59 GMT+0800 (China Standard Time)

Hi, it is the best model you trained or the worst model?

lyz · Answer 2 · Mon Sep 02 2019 12:26:55 GMT+0800 (China Standard Time)

Hi, it is the worst model maybe. But I am not sure. I have got about 27％ AP for hard set on another computer. So is the performance reported by average or by model selection？Thanks a lot!

Peiliang · Answer 3 · Mon Sep 02 2019 14:25:58 GMT+0800 (China Standard Time)

Usually, ~1% variance should be normal due to the random property. But I didn't see such a different AP. Please make sure using the exact parameters as we provide.

lyz · Answer 4 · Mon Sep 02 2019 14:36:40 GMT+0800 (China Standard Time)

Thanks. I train the model with your provided parameters. For the easy set，the variance is much larger than other sets.

lyz · Answer 5 · Wed Sep 04 2019 00:54:28 GMT+0800 (China Standard Time)

Hi, I have another question about the test set performance. Did you directly test the model which is trained on training set(3000+images) and reported the test set(7500+images) performance? Have you merged the training set and validation set to retrain the model when evaluated on the test set?
Thanks a lot!