BatchNorm Momentum Setting in Pytorch
yuhuan-wu opened this issue · comments
Hello kazuto! An awesome job for re-implenting deeplab-v2 in Pytorch!
I find a BN typo in this repo, although such typo will not affect the performance, as people will not train BN layers as default.
In Pytorch , the momentum parmameter of nn.BatchNorm2d class is 0.1.
In fact, this momentum param is actually 1 - true_momentum, and the true momentum is 0.9 as default. It can refer to the documentation of Pytorch 0.4.1 torch.nn.BatchNorm2d.
However, in your code, libs/models/resnet.py,
self.add_module("bn", _BATCH_NORM(out_ch, eps=1e-5, momentum=0.999))
I know you wanna set the true momentum to be 0.999 to avoid the unstabitily of the mean value/std caused by small batchsize. I think it should be set as a right value: 0.001 and it will work well :).
Thank you for pointing it out! I will fix all the related lines.