CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)

[BigDong]

Hi,
I tried to run SOLOv1 on the slurm, by using
GPUS=8 GPUS_PER_TASK=8 ./tools/slurm_train.sh XX XX config/solo/solo_r50_1x.py

while validating, it got a CUDA error
CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)

This seems got an error at torch.mm, but I don't know how to fix this bug.
When I rerun the code, it won’t have this bug anymore.

I run the code under Pytorch 1.4. CUDA 9.0
Meanwhile, I rewrite your code under MMDetection v2.0+, with Pytorch 1.5, CUDA 9.0. And sometimes, I also get this bug.

Do you know how to fix this bug? or have any Ideas?

Yudong