zxczrx123 / e3d

e3d: Efficient Methods for 3D Deep Learning

We open source e3d: Efficient Methods for 3D Deep Learning, a repository containing our recent advances in efficient 3D point cloud understanding.

News

[2020-08] Please check out our ECCV 2020 tutorial on AutoML for Efficient 3D Deep Learning, which summarizes the methods released in this codebase.

[2020-07] Our paper Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution is accepted to ECCV 2020.

[2020-03] Our work PVCNN is deployed on MIT Driverless racing cars, please check of this video.

[2019-12] We give the spotlight talk of PVCNN at NeurIPS 2019.

Content

• Installation
• PVCNN (NeurIPS 2019, spotlight)
• SPVNAS (ECCV 2020)

Installation

Please run:

git clone https://github.com/mit-han-lab/e3d --recurse-submodules

to clone this code base. If you forget to add the —recursive-submodules flag when cloning the codebase, please run:

git submodule update --init

after you run:

git clone https://github.com/mit-han-lab/e3d

To use all the codebases presented in this repository, please following the instructions in each folder.

PVCNN

@inproceedings{liu2019pvcnn,
  title={Point-Voxel CNN for Efficient 3D Deep Learning},
  author={Liu, Zhijian and Tang, Haotian and Lin, Yujun and Han, Song},
  booktitle={Advances in Neural Information Processing Systems},
  year={2019}
}

[Paper] [NeurIPS 2019 spotlight talk] [Deploy on MIT Driverless] [NVIDIA Jetson Community Project Spotlight] [Playlist] [Website]

Overview

In PVCNN, we present a new efficient 3D deep learning module, Point-Voxel Convolution (PVConv) as is illustrated below.

PVConv takes advantage of the regularity of volumetric representation and small footprint of point cloud representation, achieving significantly faster inference speed and much lower memory footprint comparing with both point cloud-based and voxel-based 3D deep learning methods.

Here is a demo comparing PVCNN and PointNet in 3D shape part segmentation on NVIDIA Jetson Nano:

Evaluation

To test the PVCNN models, please run cd pvcnn first and download our pretrained models as is indicated in the README file. Then, please run this code template

python train.py [config-file] --devices [gpu-ids] --evaluate --configs.evaluate.best_checkpoint_path [path to the model checkpoint]

to do the evaluation. If you want to do inference on S3DIS with GPU 0,1, you can run:

python train.py configs/s3dis/pvcnn/area5.py --devices 0,1 --evaluate --configs.evaluate.best_checkpoint_path s3dis.pvcnn.area5.c1.pth.tar

SPVNAS

@inproceedings{
  tang2020searching,
  title = {Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution},
  author = {Tang, Haotian* and Liu, Zhijian* and Zhao, Shengyu and Lin, Yujun and Lin, Ji and Wang, Hanrui and Han, Song},
  booktitle = {European Conference on Computer Vision},
  year = {2020}
}

[Paper] [ECCV 10-min Talk] [MIT News] [State-of-the-Art on SemanticKITTI Leaderboard]

Overview

We present SPVNAS, the first AutoML method for efficient 3D scene understanding. In this work, we first adapt Point-Voxel Convolution to large-scale outdoor LiDAR scans by introducing Sparse Point-Voxel Convolution (SPVConv):

We then apply 3D Neural Architecture Search (3D-NAS) to automatically search for the best architectures built from SPVConv under efficiency constraints.

Here is a demo comparing SPVNAS and MinkowskiNet, SPVNAS reaches 9.1 FPS on NVIDIA GTX1080Ti, which is 3 times faster than MinkowskiNet.

Evaluation

Please run cd spvnas and run the following code template to evaluate pretrained SPVNAS/SPVCNN/MinkowskiNet models:

torchpack dist-run -np [num_of_gpus] python evaluate.py configs/semantic_kitti/default.yaml --name [num_of_net]

For example, if you want to run inference with the model SemanticKITTI_val_SPVNAS@65GMACs, you may run:

torchpack dist-run -np 1 python evaluate.py configs/semantic_kitti/default.yaml --name SemanticKITTI_val_SPVNAS@65GMACs