Not training in DIGITS

Question

Not training in DIGITS

achaiah opened this issue 8 years ago · comments

Hi, I was wondering if you tried running the models in NVIDIA DIGITS? I imported one of your models (resnet 36) - it runs but does not train. Any ideas?

Thanks!

Jay Mahadeokar · Answer 1 · Sun Nov 20 2016 10:58:57 GMT+0800 (China Standard Time)

I haven't tried it. What do you mean by does not train? Can you be more specific?

achaiah · Answer 2 · Tue Nov 22 2016 00:02:55 GMT+0800 (China Standard Time)

It literally doesn't train (i.e. zero progress over 200 epochs). I found a fix here: BVLC/caffe#3919 There's a difference in BatchNorm between bvlc and NVIDIA that is confusing.

mrgloom · Answer 3 · Sat Nov 26 2016 16:40:31 GMT+0800 (China Standard Time)

@achaiah Can you be more specific about differences in BN layer?
Seems realted thread is here ? NVIDIA/DIGITS#629

achaiah · Answer 4 · Sun Nov 27 2016 13:42:29 GMT+0800 (China Standard Time)

The thread you pointed to is correct as well... same issue. The CUDNN BN implementation has changed and that has broken existing nets that use BN.