RuntimeError: scan failed to synchronize: an illegal memory access was encountered

[Xiaobing Han]

Hi @loicland

After running a number of epochs, an illegal memory access was encountered. Have you ever met this bug before? Thanks.

Traceback (most recent call last):
File "learning/main.py", line 394, in
main()
File "learning/main.py", line 290, in main
acc, loss, oacc, avg_iou = train()
File "learning/main.py", line 187, in train
outputs = model.ecc(embeddings)
File "/export/home/hanxiaobing/anaconda3/envs/SPG/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/graphnet.py", line 97, in forward
input = module(input)
File "/export/home/hanxiaobing/anaconda3/envs/SPG/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/modules.py", line 54, in forward
input = ecc.GraphConvFunction(nc, nc, idxn, idxe, degs, degs_gpu, self._edge_mem_limit)(hx, weights)
File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/ecc/GraphConvModule.py", line 67, in forward
cuda_kernels.conv_aggregate_fw(output.narrow(0,startd,numd), products.view(-1,self._out_channels), self._degs_gpu.narrow(0,startd,numd))
File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/ecc/cuda_kernels.py", line 120, in conv_aggregate_fw
csdegs = torch.cumsum(degs,0)
RuntimeError: scan failed to synchronize: an illegal memory access was encountered

[Xiaobing Han]

Hi @loicland

After running a number of epochs, an illegal memory access was encountered. Have you ever met this bug before? Thanks.

Traceback (most recent call last): File "learning/main.py", line 394, in main() File "learning/main.py", line 290, in main acc, loss, oacc, avg_iou = train() File "learning/main.py", line 187, in train outputs = model.ecc(embeddings) File "/export/home/hanxiaobing/anaconda3/envs/SPG/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in call result = self.forward(*input, **kwargs) File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/graphnet.py", line 97, in forward input = module(input) File "/export/home/hanxiaobing/anaconda3/envs/SPG/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in call result = self.forward(*input, **kwargs) File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/modules.py", line 54, in forward input = ecc.GraphConvFunction(nc, nc, idxn, idxe, degs, degs_gpu, self._edge_mem_limit)(hx, weights) File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/ecc/GraphConvModule.py", line 67, in forward cuda_kernels.conv_aggregate_fw(output.narrow(0,startd,numd), products.view(-1,self._out_channels), self._degs_gpu.narrow(0,startd,numd)) File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/ecc/cuda_kernels.py", line 120, in conv_aggregate_fw csdegs = torch.cumsum(degs,0) RuntimeError: scan failed to synchronize: an illegal memory access was encountered

Hi @loicland @nicolas-chaulet @mys007 @bermanmaxim
Have you ever met this bug before? Thanks a lot.

[Loic Landrieu]

Hi!

We are releasing a new version of SuperPoint Graph called SuperPoint Transformer (SPT).

https://github.com/drprojects/superpoint_transformer

It is better in any way:

✨ SPT in numbers ✨
📊 SOTA results: 76.0 mIoU S3DIS 6-Fold, 63.5 mIoU on KITTI-360 Val, 79.6 mIoU on DALES
🦋 212k parameters only!
⚡ Trains on S3DIS in 3h on 1 GPU
⚡ Preprocessing is x7 faster than SPG!
🚀 Easy install (no more boost!)

If you are interested in lightweight, high-performance 3D deep learning, you should check it out. In the meantime, we will finally retire SPG and stop maintaining this repo.