Error occured while reproducing result
wanyunfeiAlex opened this issue · comments
Hello, thanks for your great work. I am reproducing this code but some errors spurted out. It complained while training:
../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:365: operator(): block: [0,0,0], thread: [7,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds"
failed.
../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:365: operator(): block: [0,0,0], thread: [9,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds"
failed.
../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:365: operator(): block: [0,0,0], thread: [11,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds"
failed.
I used the openLane dataset, and tried training with configs "anchor3dlane_effb3.py" & "anchor3dlane_iter.py", neither trained some iters and end up failing, throwing the errors mentioned above.
Noted that the openLane dataset seems not coincide with this code at first(which was also complained about in another open issue: #5). So I reconstructed "data_lists/traning.txt" and "data_lists/validation.txt" w.r.t the file lists in cache_dense to circumvent the file-missing-error, and I ended up here.
I assume that this index-exceeding error is the result of different versions of the openLane datasets and there might be some hard-coded index that causes this error?
This error is caused by abnormal lane categories, e.g., more than 21 or less than 0. You can check your generated pickle files in the training set and exclude the abnormal ones. PS As far as I know, training/segment-1926967104529174124_5214_780_5234_780_with_camera_labels/150914294030533900.pkl contains abnormal label in both versions of openlane.