Giters

xuchen-ethz / fast-snarf

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Multi-gpu failed

yf1019 opened this issue · comments

commented

Hi, thanks to your great work.

I just trying to make this module into my own projects. While I found when I tried to use multi-gpu for training. The code will report errors as:

precompute_cuda.precompute(self.lbs_voxel_final, tfs, voxel_d, voxel_J, self.offset, self.scale)

RuntimeError: CUDA error: an illegal memory access was encountered

And I just check the source code "train.py" in this repo and I found it also reports an error when I try to use gpu > 2:

RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries. If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).

Anyone meets this error before?

commented

Meeting the same error: "RuntimeError: CUDA error: an illegal memory access was encountered" when training on multi-gpus, have you solved this problem? @yf1019