Cannot reproduce nuScenes results in paper
weixmath opened this issue · comments
Hello! Great work!
I have been trying to reproduce the results on Semantic KITTI and nuScenes datasets. The model can reach promising performance on SemanticKITTI dataset, but only get 77.0% mIoU on nuScenes dataset, which has a large gap from 78.4% reported in the paper.
I suspect some key nuScenes configurations are missing, e.g. class weight. It would be greatly appreciated if you could provide the configuration files for reproducing the nuScenes results in the paper.
Please let me know if any other information is needed. Thanks!
Hi, the configuration files are correct. FYI, the results on nuScenes often fluctuate much, especially on the validation set. The fluctuation within 1.0% mIoU is acceptable. Besides, may I know whether you have used the Testing-Time Augmentations (TTA)?
I conduct experiments on nuScenes again, the mIoU result on val set is 77.2. If TTA is used, the result is 78.4. These results are still lower than the results (78.4 and 79.5) in the paper. I think this gap is unacceptable since the RPVNet could achieve 77.6 mIoU as reported in the paper. As SphereFormer achieves excellent results on SemanticKITTI which are easy to reproduce, I wonder if this result gap could be caused by minor bugs in the nuScenes implementation?
I am not sure whether the results of RPVNet use TTA or not. But I can make sure that the current codebase can reproduce a result around 78.0% mIoU without using TTA. That is to say, it is not likely that there are some bugs in the codebase.
Besides, the high variance in the results of nuScenes may be caused by many things (including the dataset itself, the training environment like cuda version, pytorch version, spconv version, or even GPU types), and it is common to see that fluctuation.
Hope that can help you.
Hello! Great work!
I have been trying to reproduce the results on Semantic KITTI and nuScenes datasets. The model can reach promising performance on SemanticKITTI dataset, but only get 77.0% mIoU on nuScenes dataset, which has a large gap from 78.4% reported in the paper.
I suspect some key nuScenes configurations are missing, e.g. class weight. It would be greatly appreciated if you could provide the configuration files for reproducing the nuScenes results in the paper.
Please let me know if any other information is needed. Thanks!
Hello! Great work!
I have been trying to reproduce the results on Semantic KITTI and nuScenes datasets. The model can reach promising performance on SemanticKITTI dataset, but only get 77.0% mIoU on nuScenes dataset, which has a large gap from 78.4% reported in the paper.
I suspect some key nuScenes configurations are missing, e.g. class weight. It would be greatly appreciated if you could provide the configuration files for reproducing the nuScenes results in the paper.
Please let me know if any other information is needed. Thanks!
May I ask what is the reproduced result on the semantickitti dataset and what is the hyperparameter configuration? I have tried many times and can only reach around 67%. I look forward to your reply, Thanks!