RuntimeError: CUDA error: an illegal memory access was encountered

[hmm0852]

Traceback (most recent call last):
File "/media/mm/62f65e0e-b396-44ac-b1aa-b2fb260e70d1/mm/experiment/DE-Resnet18/main.py", line 251, in
main(args, logger)
File "/media/mm/62f65e0e-b396-44ac-b1aa-b2fb260e70d1/mm/experiment/DE-Resnet18/main.py", line 171, in main
train(model, train_loader, optimizer, criterion, epoch, args, logger)
File "/media/mm/62f65e0e-b396-44ac-b1aa-b2fb260e70d1/mm/experiment/DE-Resnet18/main.py", line 61, in train
loss.backward()
File "/home/mm/anaconda3/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/mm/anaconda3/lib/python3.7/site-packages/torch/autograd/init.py", line 132, in backward
allow_unreachable=True) # allow_unreachable flag
File "/home/mm/anaconda3/lib/python3.7/site-packages/torch/autograd/function.py", line 89, in apply
return self._forward_cls.backward(self, *args) # type: ignore
File "/media/mm/62f65e0e-b396-44ac-b1aa-b2fb260e70d1/mm/experiment/DE-Resnet18/softpool.py", line 45, in backward
saved[-1][torch.isnan(saved[-1])] = 0

My environment info is:
GPU: Titan
CUDA: 10.1.243
cuDNN: 7.6.5
PyTorch: 1.7.1

[Alex Stergiou]

Hi @hmm0852 ,

You can comment out saved[-1][torch.isnan(saved[-1])] = 0 as it is mainly used for underflows and may throw an illegal memory access for some drivers.

Best,
Alex

[Alex Stergiou]

Will be closing this due to inactivity. Feel free to open a new issue in case of a different problem.

Best,
Alex