Giters

zhangguanghui1 / lcps

Official Implementation for `LiDAR-Camera Panoptic Segmentation via Geometry-Consistent and Semantic-Aware Alignment` (ICCV 2023)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

LCPS: First LiDAR-Camera Panoptic Segmentation Framework for 3D Perception

Description

Title: LiDAR-Camera Panoptic Segmentation via Geometry-Consistent and Semantic-Aware Alignment (ICCV 2023)

Download: ICCV Formal Version is here

Download: Arxiv preprint paper is here

Authors: Zhiwei Zhang, Zhizhong Zhang, Qian Yu, Ran Yi, Yuan Xie*, Lizhuang Ma*

$\dagger$ Equal Contribution *Corresponding author

Demo

The Visualization video is Captured on NuScenes Dataset, compressed and converted to gif for efficient playting. Left: Semantic Segmentation, Right: Instance Segmentation. Original MP4 video can be downloaded at Dropbox or Aliyun (Code: q5h7).

News

  • [2023/08/24] Demo release.
  • [2023/08/04] Arxiv preprint released. πŸ“Ž
  • [2023/07/15] Accepted to ICCV 2023! πŸ”₯πŸ”₯πŸ”₯

Introduction

3D panoptic segmentation is a challenging perception task that requires both semantic segmentation and instance segmentation. In this task, we notice that images could provide rich texture, color, and discriminative information, which can complement LiDAR data for evident performance improvement, but their fusion remains a challenging problem. To this end, we propose LCPS, the first LiDAR-Camera Panoptic Segmentation network. In our approach, we conduct LiDAR-Camera fusion in three stages: 1) an Asynchronous Compensation Pixel Alignment (ACPA) module that calibrates the coordinate misalignment caused by asynchronous problems between sensors; 2) a Semantic-Aware Region Alignment (SARA) module that extends the oneto-one point-pixel mapping to one-to-many semantic relations; 3) a Point-to-Voxel feature Propagation (PVP) module that integrates both geometric and semantic fusion information for the entire point cloud. Our fusion strategy improves about 6.9% PQ performance over the LiDAR-only baseline on NuScenes dataset. Extensive quantitative and qualitative experiments further demonstrate the effectiveness of our novel framework.

Getting Started

Code Structure (Full Projects will be updated soon):

Current:
LCPS/
β”œβ”€β”€ docs                    # Detailed Documentations
β”‚   β”œβ”€β”€ prepare_nusc.md		# Prepare NuScenes Dataset
β”‚   β”œβ”€β”€ prepare_kitti.md	# Prepare SemanticKITTI Dataset
β”‚   β”œβ”€β”€ env_install.md		# Prepare Environment
β”‚   β”œβ”€β”€ train_test_repro.md	# Start training, validation, testing and reproducing
β”‚   └── other_toolkits.md	# Get some visualization pictures & statistic results
β”œβ”€β”€ requirements.txt		# Package Install
└── README.md 				# Quick Start

More Visualizations

Full visualizations and quick explainations can be seen here.

License

All assets and code are under the Apache 2.0 license unless specified otherwise.

Citation

If you find this project helpful, please consider citing the following paper:

@InProceedings{Zhang_2023_ICCV,
    author    = {Zhang, Zhiwei and Zhang, Zhizhong and Yu, Qian and Yi, Ran and Xie, Yuan and Ma, Lizhuang},
    title     = {LiDAR-Camera Panoptic Segmentation via Geometry-Consistent and Semantic-Aware Alignment},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {3662-3671}
}

Acknowledgement Awesome

Many thanks to the following awesome open-source projects, which provide helpful guidance for us！

Closely Relevant Project:

Other Tools:

About

Official Implementation for `LiDAR-Camera Panoptic Segmentation via Geometry-Consistent and Semantic-Aware Alignment` (ICCV 2023)

License:Apache License 2.0

Languages

Language:Python 99.7%Language:Shell 0.3%