Silhouette-Guided-3D

PyTorch implementation of our WACV 2020 paper: "Silhouette Guided Point Cloud Reconstruction beyond Occlusion"

Our short introduction video

Network architecture:

Requirements

Python 3
PyTorch >= 0.4.0
numpy, scipy, pickle, skimage, sklearn, random, re
torchvision
Matlab (for FSSR based post process)

Installation

Install mve by following the instructions. This is for FSSR based point cloud refinement.
Under ./matlab folder, install gptoolbox for Matlab based Poisson-Disc Sampling
[Optional] Install Pix3D evaluation toolkit under the current folder. Note that this requires Tensorflow.
[Optional] Install PCN evaluation toolkit under the current folder. Note that this requires Tensorflow. PCN toolkit is for object-centered evaluation.
[Optional] Install Mask-RCNN benchmark under the current folder. This is for getting visible silhouette for completion in Pix3D (We've included pre-processed results as below).

Download Data and Pre-trained Model

Download pre-trained models and put them under the ./model/ folder.
Download pre-processed DYCE dataset and put them under the ./data/ folder.
Download pre-processed Pix3D dataset and put them under the ./data/ folder. This includes pre-computed complete silhouette and ground truth point clouds rotated w.r.t. camera position. We've excluded examples with incorrect Mask-RCNN detections.
Download ShapeNet dataset and put them under the ./data/ folder.
Download pre-processed LSUN dataset and put them under the ./data/ folder
Download pre-computed result. and put them under the ./result/ folder. This includes point clouds prediction on ShapeNet and Pix3D after FSSR refinement.
[Optional] Download ShapeNet rendered images, and put them under ./data/ShapeNet/ folder. This is for comparing to object-centered point cloud reconstruction approach.
[Optional] Download ShapeNet object-centered point cloud ground truth, , and put them under ./data/ShapeNet/ folder. This is for comparing to object-centered point cloud reconstruction approach.

Training

Point cloud reconstruction
```
python train.py
python test.py
```
This will save network predictions for the downstream FSSR post-refinement step.
Silhouette completion First train on DYCE dataset:
```
python train_sc.py
```
Then finetune on Pix3D dataset, using 5-fold cross validation ( you will need to run it 5 times by changing the fold number in L32-35 ):
```
python train_sc_ft.py
python test_sc_pix3d.py
```
Silhouette guidede point cloud reconstruction
```
python train_occ.py
python test_rec_pix3d.py
```
Then perform FSSR post-refinement step as describe below

Surface Based Point Clouds Refinement

Start matlab
```
cd matlab
./matlab
```
pre-compute FSSR params (per-pixel normal and scale), change folder name based on your saved network predictions path
```
FssrPostRefine
```
FSSR Here we provide the sample batch-process code (need to go back to the main folder):
```
cd ..
python fssr_batch_process.py
```
smoothing
```
cd matlab
preComputeFssrParam
cd ..
```
This produces the refined point clouds for evaluation.

Evaluation

Pix3D
- ICP-based fitting since Pix3D ground truth is object-centered (you can skip this step since we've included pre-computered ground truth and predictions). Code is derived from 3D-LMNET, which also includes the ground truth point cloud.
```
cd pcn
python metrics_pix3d.py
cd ..
```
- You need to use TensorFlow 3.0+ to run the evaluation:
```
cd pix3d/eval/
python eval_pix3d.py
cd ../../
```
ShapeNet
- You need to use TensorFlow 3.0+ to run the evaluation:
```
cd pcn
python eval_shapenet.py
cd ..
```
- To compare with object-centered point cloud prediction appraoch (3D-LMNET), you need to perform ICP-based fitting first:
```
cd pcn
python metrics_shapenet.py
cd ../pix3d/eval/
python eval_shapenet_object_centered.py
cd ../../
```
DYCE (silhouette completion)
- This is PyTorch based
```
python test_sc_DYCE.py
```

Citation

Please cite our paper for any purpose of usage.

@inproceedings{zou2020silhouette,
  title={Silhouette Guided Point Cloud Reconstruction beyond Occlusion},
  author={Zou, Chuhang and Hoiem, Derek},
  booktitle={The IEEE Winter Conference on Applications of Computer Vision},
  pages={41--50},
  year={2020}
}

zouchuhang / Silhouette-Guided-3D