Question about overfitting

Question

Question about overfitting

MrChenFeng opened this issue 4 years ago · comments

Hi,

Thanks so much for sharing your code and work!
I wonder have you tried asym noise at a low ratio? I tried some different noise mode such as mixing asym and sym together, sometimes the network seems overfit quickly in the initial epochs of warmup. Do you have any suggestions about modifying the loss and regularization tricks in this condition? Actually, I'm curious and confused about the relation between noise mode and loss distribution. Any suggestions will be highly appreciated!

Best,
Chen

Junnan Li · Answer 1 · Fri Oct 30 2020 20:28:24 GMT+0800 (China Standard Time)

Hi,
Have you tried to activate the confidence penalty that is used in asym noise? Usually asym noise is easier to overfit because the noise has structure.

Chen Feng · Answer 2 · Fri Oct 30 2020 20:59:08 GMT+0800 (China Standard Time)

Hi,
Actually, I added a weights hyperparameter for the confidence-regularization term. Seems it resulted in the loss distribution moved right-side as the weight get bigger but still one-peak distribution.
Sadly, it didn't work.

Junnan Li · Answer 3 · Fri Oct 30 2020 21:28:04 GMT+0800 (China Standard Time)

Can I know what kind of noise distribution do you use? You may want to also try different warm-up epochs and see which epoch results in more separation in the loss distribution. Moreover, a larger learning rate may also help.

Chen Feng · Answer 4 · Sun Nov 01 2020 13:03:31 GMT+0800 (China Standard Time)

I would say the noise mode I tried tend to be noisier. Such as one real class has may be blended with noisy samples from two or three more other classes.

Junnan Li · Answer 5 · Sun Nov 01 2020 20:26:47 GMT+0800 (China Standard Time)

It might be possible that there is too much noise? From my experience, the model needs to be able to learn something during warmup in order to start noise cleaning.