MetaBEV: Solving Sensor Failures for BEV Detection and Map Segmentation
Chongjian GE*,
Junsong Chen*,
Enze Xie+,
Zhongdao Wang,
Lanqing Hong,
Huchuan Lu,
Zhenguo Li,
Ping Luo+
（* denotes equal contribution, + denotes corresponding authors）
Project Page | arXiv | youtube demo
Updates
- (20/04/2023) MetaBEV is released on arxiv.
Abstract
Perception systems in modern autonomous driving vehicles typically take inputs from complementary multi-modal sensors, e.g., LiDAR and cameras. However, in real-world applications, sensor corruptions and failures lead to inferior performances, thus compromising autonomous safety.
In this paper, we propose a robust framework, called MetaBEV, to address extreme real-world environments, involving overall six sensor corruptions and two extreme sensor-missing situations.
Experiments show MetaBEV outperforms prior arts by a large margin on both full and corrupted modalities. For instance, when the LiDAR signal is missing, MetaBEV improves 35.5% detection NDS and 17.7% segmentation mIoU upon the vanilla BEVFusion model; and when the camera signal is absent, MetaBEV still achieves 69.2% NDS and 53.7% mIoU, which is even higher than previous works that perform on full-modalities. Moreover, MetaBEV performs fairly against previous methods in both canonical perception and multi-task learning settings, refreshing state-of-the-art nuScenes BEV map segmentation with 70.4% mIoU.
Results
Our model achieves the following performance on :
1-Single Complementary Modalities.
- Detection on nuScenes val set with LiDAR and Camera.
| Methods | Modality | Multi-Task | mAP(val) | NDS(val) |
|---|---|---|---|---|
| MetaBEV-Transfusion | Camera | x | 49.4 | 49.7 |
| MetaBEV-Centerhead | Camera | x | 55.5 | 60.4 |
| MetaBEV-Transfusion | LiDAR | x | 62.5 | 68.6 |
| MetaBEV-Centerhead | LiDAR | x | 64.2 | 69.3 |
| MetaBEV-Transfusion | Camera+LiDAR | x | 68 | 71.5 |
| MetaBEV-Transfusion | Camera+LiDAR | √ | 65.4 | 69.8 |
- Segmentation on nuScenes val set with LiDAR and Camera.
| Methods | Modality | Drivable | Ped.Cross | Walkway | Stop Line | Carpark | Divider | Mean |
|---|---|---|---|---|---|---|---|---|
| MetaBEV | Camera | 83.3 | 56.7 | 61.4 | 50.8 | 55.5 | 48 | 59.3 |
| MetaBEV | LiDAR | 87.9 | 63.4 | 71.6 | 55 | 55.1 | 55.7 | 64.8 |
| MetaBEV | Camera+LiDAR | 89.6 | 68.4 | 74.8 | 63.3 | 64.4 | 61.8 | 70.4 |
| MetaBEV | Camera+LiDAR | 88.5 | 64.9 | 71.8 | 56.7 | 61.1 | 58.2 | 66.9 |
2-Missing Modalities.
| Methods | Camera+LiDAR | Missing Camera | Missing LiDAR | ||||||
| mAP | NDS | mIoU | mAP | NDS | mIoU | mAP | NDS | mIoU | |
| MetaBEV | 68.0 | 71.5 | 70.4 | 63.6 | 69.2 | 53.7 | 39.0 | 42.6 | 54.4 |
3-Corrupted Modalities.
Acknowledgements
The project is based on mmdetection3d, BEVFusion, robust benchmark. Thanks for their awesome works.
License
This project is under the MIT license. See LICENSE for details.
Citation
If you find MetaBEV useful or relevant in your research please consider citing our paper:
@article{ge2023metabev,
title={MetaBEV: Solving Sensor Failures for BEV Detection and Map Segmentation},
author={Ge, Chongjian and Chen, Junsong and Xie, Enze and Wang, Zhongdao and Hong, Lanqing and Lu, Huchuan and Li, Zhenguo and Luo, Ping},
journal={arXiv preprint arXiv:2304.09801},
year={2023}
}