rho for Adaptive Sharpness Aware Minimization (ASAM)

[jungminkwon]

Hi.
This is Jungmin Kwon, one of the authors of Adaptive Sharpness Aware Minimization (ASAM).
We really appreciate your great implementation!
I have performed cifar10 tests with your code and we found that ASAM with rho=2.0 shows the best accuracy among [0.5, 1.0, 2.0, 5.0].
The test error rates obtained from the grid search are as follows:

rho	Test error rate
0.5	2.75 %
1.0	2.69 %
2.0	2.55 %
5.0	2.90 %

In our implementation without bias (or beta for BatchNorm) normalization (https://github.com/SamsungLabs/ASAM), ASAM with rho=0.5 shows the best accuracy (2.37 % for WRN16-8), so we performed all the cifar10 tests with rho=0.5.
If you don't mind, could you update the table of test error rate with the result of rho=2.0 (2.55 %)?
Thank you.

[David Samuel]

Hi,

thank you very much for working on ASAM, it's a very clever improvement of the original SAM. I've updated the results, thanks for running the grid search! :)

[jungminkwon]

Thanks for the update! :)