About using PTv3 to inference our LiDAR point clouds

[BotaoSun]

Hi, I encountered one problem when I tried to use the pre-trained model on Nuscenes of PTv3 to infer our own LiDAR point clouds. The category 'bicycles' is hard to categorize. 'bicycles' is often categorized as 'static manmade‘. Could you please provide some suggestions on the config when using this model?

[Xiaoyang Wu]

Would fine-tuning the model (or joint training with PPT) with your personal dataset be better?

[BotaoSun]

Would fine-tuning the model (or joint training with PPT) with your personal dataset be better?

Thanks for your reply. However, we don't have the labels for LiDAR point clouds. We want to generate the labels using pre-trained models. The labels are not required to be perfect, but in nearly every frames, PTv3 seems to ignore the category 'bicycle'. However, it performs well on category 'vehicle' or scenario objects.

Could you provide some suggestions or comment on this situation?

[Bryan Chang]

@BotaoSun @Gofinge I am also interested in running the pre-trained model on Nuscenes of PTv3 to infer my own LiDAR point to do semantic segmentation. Would you mind sharing the process of doing this? Here's what I have in mind:

1. Convert my dataset such that it has following format
   -train
   -test
   --scence0
   ---coord.npy
   ---strength.npy
   --scence1
   ---coord.npy
   ---trength.npy
   -valid
   Leaving train and valid folder empty for now
2. Modify the pointcept.datasets.default.py VALID_ASSETS by leaving only coord, strength, segment and rename the file to custom_default.py
3. Download the Nuscene PTv3 Exp record (where do I put this folder?)
4. Run the test script like this: sh scripts/test.sh -p python -d custom_default -n nuscenes-semseg-pt-v3m1-0-base -w model_best

Then I should be able to get the segment.npy file? Please let me know if I am missing anything in this process.

[BotaoSun]

@BotaoSun @Gofinge I am also interested in running the pre-trained model on Nuscenes of PTv3 to infer my own LiDAR point to do semantic segmentation. Would you mind sharing the process of doing this? Here's what I have in mind:

1. Convert my dataset such that it has following format
   -train
   -test
   --scence0
   ---coord.npy
   ---strength.npy
   --scence1
   ---coord.npy
   ---trength.npy
   -valid
   Leaving train and valid folder empty for now
2. Modify the pointcept.datasets.default.py VALID_ASSETS by leaving only coord, strength, segment and rename the file to custom_default.py
3. Download the Nuscene PTv3 Exp record (where do I put this folder?)
4. Run the test script like this: sh scripts/test.sh -p python -d custom_default -n nuscenes-semseg-pt-v3m1-0-base -w model_best

Then I should be able to get the segment.npy file? Please let me know if I am missing anything in this process.

Hi, I think transforming your data to the same format as Nuscenes is a good idea.
If you are confused with the folder path, maybe you could directly run test.py and add the path like this

python tools/test.py --config-file ${CONFIG_PATH} --num-gpus ${NUM_GPU} --options save_path=${SAVE_PATH} weight=${CHECKPOINT_PATH}

[Bryan Chang]

Thanks for the responds! Now I am trying to use the pre-trained nuscene model and test it on semantic kitti but I am getting very low accuracy. Is this normal or am I doing something wrong? So the first picture is the actual label and the second is the predicted label. For example, if I select the car in the middle of the picture I am getting 252: "moving-car" which is correct but in the predicted label I am getting 44: "parking", both corresponds for the SemanticKITTI labeling. I understand the labeling objects for nuscenes and semantic kitti might be different but both has car labeled. So I would assume it's inferencing the car correctly. What I did was modifying the config file's test data for the exp record of the NuScenesDataset to SemanticKITTIDataset test=dict( type='SemanticKITTIDataset', split='val', oh and a side note, I did not enable flash attention will that be a problem?