microsoft / DynamicHead

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

cuda11.1 train fail?

vison20080808 opened this issue · comments

Traceback (most recent call last):
File "train_net.py", line 224, in
args=(args,),
File "/sda/program/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/engine/launch.py", line 82, in launch
main_func(*args)
File "train_net.py", line 210, in main
return trainer.train()
File "/sda/program/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/engine/defaults.py", line 484, in train
super().train(self.start_iter, self.max_iter)
File "/sda/program/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/engine/train_loop.py", line 149, in train
self.run_step()
File "/sda/program/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/engine/defaults.py", line 494, in run_step
self._trainer.run_step()
File "/sda/program/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/engine/train_loop.py", line 273, in run_step
loss_dict = self.model(data)
File "/sda/program/anaconda3/envs/detectron2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/sda/okay/ai/project/ocrquesseg-svr/extra/atss.py", line 204, in forward
losses = self.losses(anchors, pred_logits, gt_labels, pred_anchor_deltas, gt_boxes, pred_centers)
File "/sda/okay/ai/project/ocrquesseg-svr/extra/atss.py", line 236, in losses
cls_loss = self.classification_loss_func(box_cls_flatten, labels_flatten.int()) / num_pos_avg_per_gpu
File "/sda/program/anaconda3/envs/detectron2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/sda/okay/ai/project/ocrquesseg-svr/extra/sigmoid_focal_loss.py", line 48, in forward
loss = sigmoid_focal_loss_cuda(logits, targets, self.gamma, self.alpha)
File "/sda/okay/ai/project/ocrquesseg-svr/extra/sigmoid_focal_loss.py", line 20, in forward
logits, targets, num_classes, gamma, alpha
RuntimeError: cuda runtime error (209) : no kernel image is available for execution on the device at /sda/okay/ai/project/ocrquesseg-svr/dyhead/csrc/cuda/SigmoidFocalLoss_cuda.cu:139