ispc-lab / NeuralPCI

✨ [CVPR 2023] NeuralPCI: Spatio-temporal Neural Field for 3D Point Cloud Multi-frame Non-linear Interpolation

NeuralPCI

NeuralPCI: Spatio-temporal Neural Field for 3D Point Cloud Multi-frame Non-linear Interpolation
Zehan Zheng*, Danni Wu*, Ruisi Lu, Fan Lu, Guang Chen†, Changjun Jiang.
(* Equal contribution, † Corresponding author)
CVPR 2023

Paper (arXiv) | Paper (CVPR) | Project Page | Video (6min) | Talk (15min, Chinese) | Slides | Poster

Indoor Scenario	Outdoor Scenario
demo_indoor.mp4	demo_outdoor.mp4

Table of Contents

1. Changelog
2. Get Started
3. Introduction
4. Dataset
5. Benchmark
6. Visualization
7. Citation
8. License

Changelog

2023-5-21: ✨ We release the code of NeuralPCI along with the NL-Drive dataset.
2023-3-27: We post the preprint paper on arXiv and release the project page.
2023-2-28: This paper is accepted by CVPR 2023 🎉🎉.

Introduction

This repository is the PyTorch implementation for NeuralPCI.

NeuralPCI is an end-to-end 4D spatio-temporal Neural field for 3D Point Cloud Interpolation, which implicitly integrates multi-frame information to handle nonlinear large motions for both indoor and outdoor scenarios.

The 4D neural field is constructed by encoding the spatio-temporal coordinates of the multi-frame input point clouds via a coordinate-based multi-layer perceptron network. For each point cloud frame of the input, the interpolation time is set to the corresponding timestamps of four input frames for NeuralPCI to generate the corresponding point cloud. And then the neural field is optimized on runtime in a self-supervised manner without relying on ground truth. In the inference stage after optimization, NeuralPCI receives a reference point cloud and an arbitrary interpolation frame moment as input to generate the point cloud of the associated spatio-temporal location.

Get Started

Installation

Please follow instructions to setup the environment.

git clone https://github.com/ispc-lab/NeuralPCI.git
cd ./NeuralPCI/

conda create -n npci python=3.9
conda activate npci

conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch -y

conda install -c open3d-admin open3d==0.9.0 -y
conda install -c conda-forge -c fvcore -c iopath fvcore iopath -y
conda install -c bottler nvidiacub -y
conda install pytorch3d -c pytorch3d -y

conda install pyg -c pyg -y
conda install -c conda-forge shapely -y

# compile cuda CD&EMD
cd ./utils/CD/chamfer3D/
python setup.py install

cd ../../EMD/
python setup.py install
cp build/lib.linux-x86_64-cpython-39/emd_cuda.cpython-39-x86_64-linux-gnu.so .

# compile point cloud utils
cd ../point_cloud_query/
python setup.py install

cd ../pointnet2/
python setup.py install

Run NeuralPCI

Refer to Dataset for data downloading first.
Create a soft link to the dataset folder for DHB and NL-Drive.

ln -s /PATH/TO/DHB ./data/DHB-dataset
ln -s /PATH/TO/NL_Drive ./data/NL_Drive

Make sure you are in the root directory of this repo.

For DHB dataset, run

bash run_DHB.sh

For NL-Drive dataset, run

bash run_NL_Drive.sh

Dataset

NL-Drive

Download link: [OneDrive] [Google Drive (testset only)]

NL-Drive dataset is a challenging multi-frame interpolation dataset for autonomous driving scenarios. Based on the principle of hard-sample selection and the diversity of scenarios, NL-Drive dataset contains point cloud sequences with large nonlinear movements from three public large-scale autonomous driving datasets: KITTI, Argoverse and Nuscenes.

DHB

Download link: [Google Drive]

Dynamic Human Bodies dataset (DHB), containing 10 point cloud sequences from the MITAMA dataset and 4 from the 8IVFB dataset. The sequences in DHB record 3D human motions with large and non-rigid deformation in real world. The overall dataset contains more than 3000 point cloud frames. And each frame has 1024 points.

Benchmark

• DHB Dataset Results (Chamfer Distance)

Method	Longdress	Loot	Red&Black	Soldier	Squat	Swing	Overall ↓
IDEA-Net	0.89	0.86	0.94	1.63	0.62	1.24	1.02
PointINet	0.98	0.85	0.87	0.97	0.90	1.45	0.96
NSFP	1.04	0.81	0.97	0.68	1.14	3.09	1.22
PV-RAFT	1.03	0.82	0.94	0.91	0.57	1.42	0.92
NeuralPCI (Ours)	0.70	0.61	0.67	0.59	0.03	0.53	0.54

• DHB Dataset Results (Earth Mover's Distance)

Method	Longdress	Loot	Red&Black	Soldier	Squat	Swing	Overall ↓
IDEA-Net	6.01	8.62	10.34	30.07	6.68	6.93	12.03
PointINet	10.87	12.10	10.68	12.39	13.99	14.81	12.25
NSFP	7.45	7.13	8.14	5.25	7.97	11.39	7.81
PV-RAFT	6.88	5.99	7.03	5.31	2.81	10.54	6.14
NeuralPCI (Ours)	4.36	4.76	4.79	4.63	0.02	2.22	3.68

• NL-Drive Dataset Results (Chamfer Distance)

Method	Frame-1	Frame-2	Frame-3	Average ↓
NSFP	0.94	1.75	2.55	1.75
PV-RAFT	1.36	1.92	1.63	1.64
PointINet	0.93	1.24	1.01	1.06
NeuralPCI (Ours)	0.72	0.94	0.74	0.80

• NL-Drive Dataset Results (Earth Mover's Distance)

Method	Frame-1	Frame-2	Frame-3	Average ↓
NSFP	95.18	132.30	168.91	132.13
PV-RAFT	104.57	146.87	169.82	140.42
PointINet	97.48	110.22	95.65	101.12
NeuralPCI (Ours)	89.03	113.45	88.61	97.03

Visualization

Citation

If you find our code or paper useful, please cite

@inproceedings{zheng2023neuralpci,
  title     = {NeuralPCI: Spatio-temporal Neural Field for 3D Point Cloud Multi-frame Non-linear Interpolation},
  author    = {Zheng, Zehan and Wu, Danni and Lu, Ruisi and Lu, Fan and Chen, Guang and Jiang, Changjun},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2023}
  }

License

All code within this repository is under Apache License 2.0.