happinesslz / LION

Official repository of ”LION: Linear Group RNN for 3D Object Detection in Point Clouds“

LION: Linear Group RNN for 3D Object Detection in Point Clouds

Zhe Liu ^1,* , Jinghua Hou ^1,* , Xinyu Wang ^1,* , Xiaoqing Ye ³, Jingdong Wang ³, Hengshuang Zhao ², Xiang Bai ^1,✉
1 Huazhong University of Science and Technology, ² The University of Hong Kong, ³ Baidu Inc.
* Equal contribution, ✉ Corresponding author.

Project Page | arXiv 2024

🔥 Highlights

• Strong performance. LION achieves state-of-the-art performance on Waymo, nuScenes, Argoverse V2, and ONCE datasets. 💪

• Strong generalization. LION can support almost all linear RNN operators including Mamba, RWKV, RetNet, xLSTM, and TTT. Anyone is welcome to verify more linear RNN operators. 😀

• More friendly. LION can train all models on less 24G GPU memory~(i.e., RTX 3090, RTX4090, V100 and A100 are enough to train our LION) 😀

News

• 2024.07.25: LION paper released. 🔥
• 2024.07.02: Our new works OPEN and SEED have been accepted by ECCV 2024. 🎉

Results

• Waymo Val Set

Model	mAP/mAPH_L1	mAP/mAPH_L2	Vec_L1	Vec_L2	Ped_L1	Ped_L2	Cyc_L1	Cyc_L2	Config
LION-RetNet	80.9/78.8	74.6/72.7	79.0/78.5	70.6/70.2	84.6/80.0	77.2/72.8	79.0/78.0	76.1/75.1	config
LION-RWKV	81.0/79.0	74.7/72.8	79.7/79.3	71.3/71.0	84.6/80.0	77.1/72.7	78.7/77.7	75.8/74.8	config
LION-Mamba	81.4/79.4	75.1/73.2	79.5/79.1	71.1/70.7	84.9/80.4	77.5/73.2	79.7/78.7	76.7/75.8	config
LION-Mamba-L	82.1/80.1	75.9/74.0	80.3/79.9	72.0/71.6	85.8/81.4	78.5/74.3	80.1/79.0	77.2/76.2	config

• NuScenes

Model	Split	NDS	mAP	Config	Download
LION-RetNet	Val	71.9	67.3	config
LION-RWKV	Val	71.7	66.8	config
LION-Mamba	Val	72.1	68.0	config
LION-Mamba	Test	73.9	69.8

• Argoverse V2 Val Set

Model	mAP	Config	Download
LION-RetNet	40.7	config
LION-RWKV	41.1	config
LION-Mamba	41.5	config

• ONCE Val Set

Model	Vehicle	Pedestrian	Cyclist	mAP	Config	Download
LION-RetNet	78.1	52.4	68.3	66.3	config
LION-RWKV	78.3	50.6	68.4	65.8	config
LION-Mamba	78.2	53.2	68.5	66.6	config

Quick Validation

• We provide some examples of LION models on KITTI dataset for quick validation of any Linear RNN operators.
• Here, we provide the results of moderate difficulty for LION with RetNet, RWKV, Mamba, xLSTM, and TTT.
• Anyone is welcome to verify more linear RNN operators. 😀

Model	Car	Pedestrian	Cyclist	Config	Download
LION-TTT	78.0	58.6	69.6	config
LION-xLSTM	77.9	59.3	67.4	config
LION-RetNet	77.9	60.2	69.6	config
LION-Mamba	78.3	60.2	68.6	config
LION-RWKV	78.3	62.2	71.2	config

Installation

Please refer to INSTALL.md for the installation of LION codebase.

Getting Started

We provide all training&evaluation scripts for training our LION, please refer to tools/

• Train all models of LION on nuScenes

bash run_train_lion_for_nus.sh

• Train all models of LION on Waymo

bash run_train_lion_for_waymo.sh

• Train all models of LION on Argoverse V2

bash run_train_lion_for_argov2.sh

• Train all models of LION on ONCE

bash run_train_lion_for_once.sh

• Train all models of LION on KITTI

bash run_train_lion_for_kitti.sh

For more details about LION, please refer to GETTING_STARTED.md to learn more usage about LION.

TODO

• Release the paper.
• Release the code of LION on the Waymo.
• Release the code of LION on the nuScenes.
• Release the code of LION on the Argoverse V2.
• Release the code of LION on the ONCE.
• Release the code of LION on the KITTI.
• Release all checkpoints of LION.
• Support more linear RNNs.

Citation

@article{liu2024lion,
  title={LION: Linear Group RNN for 3D Object Detection in Point Clouds},
  author={Liu, Zhe and Hou, Jinghua and Wang, Xinyu and Ye, Xiaoqing and Wang, Jingdong and Zhao, Hengshuang and Bai, Xiang},
  year={2024},
  eprint={2407.18232},
  archivePrefix={arXiv},
}

Acknowledgements

We thank these great works and open-source repositories: OpenPCDet, DSVT, FlatFormer, HEDNet, Mamba, RWKV, Vision-RWKV, RMT, xLSTM, TTT, and flash-linear-attention.