Title: LiDAR-Camera Panoptic Segmentation via Geometry-Consistent and Semantic-Aware Alignment (ICCV 2023)
Download: ICCV Formal Version is here
Download: Arxiv preprint paper is here
Authors: Zhiwei Zhang, Zhizhong Zhang, Qian Yu, Ran Yi, Yuan Xie*, Lizhuang Ma*
$\dagger$ Equal Contribution *Corresponding author
- [2023/08/24] Demo release.
- [2023/08/04] Arxiv preprint released. π
- [2023/07/15] Accepted to ICCV 2023! π₯π₯π₯
3D panoptic segmentation is a challenging perception task that requires both semantic segmentation and instance segmentation. In this task, we notice that images could provide rich texture, color, and discriminative information, which can complement LiDAR data for evident performance improvement, but their fusion remains a challenging problem. To this end, we propose LCPS, the first LiDAR-Camera Panoptic Segmentation network. In our approach, we conduct LiDAR-Camera fusion in three stages: 1) an Asynchronous Compensation Pixel Alignment (ACPA) module that calibrates the coordinate misalignment caused by asynchronous problems between sensors; 2) a Semantic-Aware Region Alignment (SARA) module that extends the oneto-one point-pixel mapping to one-to-many semantic relations; 3) a Point-to-Voxel feature Propagation (PVP) module that integrates both geometric and semantic fusion information for the entire point cloud. Our fusion strategy improves about 6.9% PQ performance over the LiDAR-only baseline on NuScenes dataset. Extensive quantitative and qualitative experiments further demonstrate the effectiveness of our novel framework.
Code Structure (Full Projects will be updated soon):
Current:
LCPS/
βββ docs # Detailed Documentations
β βββ prepare_nusc.md # Prepare NuScenes Dataset
β βββ prepare_kitti.md # Prepare SemanticKITTI Dataset
β βββ env_install.md # Prepare Environment
β βββ train_test_repro.md # Start training, validation, testing and reproducing
β βββ other_toolkits.md # Get some visualization pictures & statistic results
βββ requirements.txt # Package Install
βββ README.md # Quick Start
- Step-1 Preparing NuScenes
- Step-2 Preparing SemanticKITTI
- Step-3 Environment Installation
- Step-4 Training, Testing and Reproducing
- Step-5 Other Toolkits (Will be Updated Soon)
- Step-6 (Optional) Further Issues? Find QA
Full visualizations and quick explainations can be seen here.
All assets and code are under the Apache 2.0 license unless specified otherwise.
If you find this project helpful, please consider citing the following paper:
@InProceedings{Zhang_2023_ICCV,
author = {Zhang, Zhiwei and Zhang, Zhizhong and Yu, Qian and Yi, Ran and Xie, Yuan and Ma, Lizhuang},
title = {LiDAR-Camera Panoptic Segmentation via Geometry-Consistent and Semantic-Aware Alignment},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2023},
pages = {3662-3671}
}
Many thanks to the following awesome open-source projects, which provide helpful guidance for usοΌ
Closely Relevant Project:
Other Tools:
-
Visualization Tools Del:
-
QR-Code Generation: https://www.hlcode.cn