RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:343

[BrainPotter]

when I ran python test.py tracking --exp_id kitti_fulltrain --dataset kitti_tracking --dataset_version test --track_thresh 0.4 --load_model ../models/kitti_fulltrain_model_last.pth --resume --is_recurrent --gru_filter_size 3 --num_gru_layers 1 --visibility --visibility_thresh_eval 0.2 --stream_test --flip_test --trainval
I got the following problem
Traceback (most recent call last):
File "test.py", line 301, in
prefetch_test(opt)
File "test.py", line 114, in prefetch_test
ret = detector.run(pre_processed_images)
File "/home/dehazing/permatrack/src/lib/detector.py", line 149, in run
images, self.pre_images, self.pre_hms, pre_inds, return_time=True, original_batch=pre_processed_images)
File "/home/dehazing/permatrack/src/lib/detector.py", line 419, in process
output, self.h = self.model.step(batch_list, self.h)
File "/home/dehazing/permatrack/src/lib/model/networks/base_model.py", line 117, in step
feats = self.imgpre2feats(x, None, torch.zeros(1))
File "/home/dehazing/permatrack/src/lib/model/networks/dla.py", line 678, in imgpre2feats
y = self.do_tensor_pass(x, pre_img, pre_hm)
File "/home/dehazing/permatrack/src/lib/model/networks/dla.py", line 633, in do_tensor_pass
x = self.dla_up(x)
File "/home/dehazing/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/dehazing/permatrack/src/lib/model/networks/dla.py", line 574, in forward
ida(layers, len(layers) -i - 2, len(layers))
File "/home/dehazing/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/dehazing/permatrack/src/lib/model/networks/dla.py", line 545, in forward
layers[i] = upsample(project(layers[i]))
File "/home/dehazing/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/dehazing/permatrack/src/lib/model/networks/dla.py", line 518, in forward
x = self.conv(x)
File "/home/dehazing/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/dehazing/permatrack/src/lib/model/networks/DCNv2/dcn_v2.py", line 128, in forward
self.deformable_groups)
File "/home/dehazing/permatrack/src/lib/model/networks/DCNv2/dcn_v2.py", line 31, in forward
ctx.deformable_groups)
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:343
My OS is ubuntu16.04 CUDA10.1 pytorch1.4.0 Nvidia Quadro P6000
Please help me! Thank you very much!