error while training
mohamedgamal7 opened this issue · comments
Traceback (most recent call last):
File "/content/pseudo-LiDAR_e2e/PointRCNN/tools/train_rcnn_depth.py", line 336, in
lr_scheduler_each_iter=(cfg.TRAIN.OPTIMIZER == 'adam_onecycle')
File "/content/pseudo-LiDAR_e2e/PointRCNN/tools/../tools/train_utils/train_utils.py", line 212, in train
loss, tb_dict, disp_dict = self._train_it(batch)
File "/content/pseudo-LiDAR_e2e/PointRCNN/tools/../tools/train_utils/train_utils.py", line 143, in _train_it
loss.backward()
File "/usr/local/lib/python3.7/dist-packages/torch/tensor.py", line 102, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/local/lib/python3.7/dist-packages/torch/autograd/init.py", line 90, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
Hi, I guess this sort of problems mostly come from the difference of environment, like pytorch and cuda version, please refer to the readme as well as the original pointrcnn repo for assistance.