cudnn deterministic not guaranteed when seed already set in imagenet

[叶志晟]

Context

• Pytorch version: not related
• Operating System and version: not related

Your Environment

• Is it a CPU or GPU environment?: GPU
• Which example are you using: imagenet/main.py https://github.com/pytorch/examples/blob/32f0c1e1073913e75fa75b848bccdbf011b2a569/imagenet/main.py

Expected Behavior

The whole training will be deterministic when the seed is set, according to the annotations here:

examples/imagenet/main.py

Lines 91 to 95 in 32f0c1e

 warnings.warn('You have chosen to seed training. ' 
               'This will turn on the CUDNN deterministic setting, ' 
               'which can slow down your training considerably! ' 
               'You may see unexpected behavior when restarting ' 
               'from checkpoints.') 

Current Behavior

The cudnn.benchmark = true here will indeed introduce nondeterminism, according to PyTorch docs

examples/imagenet/main.py

Line 207 in 32f0c1e

 cudnn.benchmark = True 

I will send a pr soon.