I have a question about the performance of PTv3 on nuScenes

[WeijianZhang]

Hi, thanks for your great work!

I ran the PTv3 on nuScenes val set with the default config semseg-pt-v3m1-0-base.py, but only got 78.33 mIoU (lower than 80.4 reported in the paper). Could you please give me some advice?

Here's my environment:
4 RTX 4090 GPUs
PyTorch == 1.12
CUDA==11.3
flash-attn==2.5.6

Here's my result:

[Xiaoyang Wu]

Hi, running three times and choosing the highest might be a good choice. Model performances on point cloud datasets are always not so stable. An example log is available here (https://huggingface.co/Pointcept/PointTransformerV3/blob/main/nuscenes-semseg-pt-v3m1-0-base/train.log)

[WeijianZhang]

Thanks for your quick reply! I have found the reason.

I got the 78.33 mIoU from the val evaluation stage:

And the performance is equal to your 78.31 mIoU:

Your reported 80.32 mIoU is from the test evaluation stage:

Could you kindly explain the difference between these two types of evaluations? ( Maybe it's a stupid question 😓

[Xiaoyang Wu]

Hi, It is not a stupid question and I think Readme here (https://github.com/Pointcept/Pointcept?tab=readme-ov-file#testing) might be helpful.

[WeijianZhang]

Thanks for your helpful reply!