devernay/photometric-mesh-optim

Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction

Chen-Hsuan Lin, Oliver Wang, Bryan C. Russell, Eli Shechtman, Vladimir G. Kim, Matthew Fisher, and Simon Lucey
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

Project page: https://chenhsuanlin.bitbucket.io/photometric-mesh-optim
Paper: https://chenhsuanlin.bitbucket.io/photometric-mesh-optim/paper.pdf
arXiv preprint: https://arxiv.org/abs/1903.08642

We provide PyTorch code for the following experiments:

ShapeNet+SUN360 sequences
(coming soon!) Real-world videos

Prerequisites

This code is developed with Python3 (python3). PyTorch 1.0+ is required.
(If you wish to run using PyTorch 0.4, please switch to the pytorch-0.4 branch.)

First, create a Python virtual environment by running

virtualenv -p python3 PMO

Before installing dependencies and/or running the code, activate the virtual environment by running

source PMO/bin/activate

The dependencies can be installed by running (within the virtual environment)

pip3 install --upgrade -r requirements.txt

Dataset

Test sequences

The test sequences are composited from the leave-out set of ShapeNet and SUN360. To download the dataset (64GB), run the script file download_sequences.sh under the directory data.
After downloading, run tar -zxf sequences.tar.gz under the directory data. The files will be extracted to a directory sequences.

ShapeNet renderings

We render images from ShapeNet with higher quality (with resolution of 224x224). To download the dataset (33GB), run the script file download_rendering.sh under the directory data.
After downloading, run tar -xf rendering.tar. The files will be extracted to a directory rendering.

ShapeNet point clouds

Please follow the instructions in the AtlasNet repository to download the ground-truth point clouds. The directory customShapeNet should be placed under the directory data.

SUN360 cropped backgrounds

The cropped background images from SUN360 (92GB) can be downloaded by running the script file download_background.sh under the directory data.
After downloading, run tar -xf background.tar. The files will be extracted to a directory background.

Pretrained models

The pretrained models (626MB each) can be downloaded by running the command

wget https://cmu.box.com/shared/static/oryysitkhn2eldgb90qkr3lh7j469sj1.npz # airplane
wget https://cmu.box.com/shared/static/jgif23ytibtektwwcji8wiv0jbubzs08.npz # car
wget https://cmu.box.com/shared/static/zakir5pi9xma4l3d5c2g74i8r0lggp36.npz # chair

Compiling the CUDA libraries

The meshrender library can be compiled by running python3 setup.py install under the directory meshrender.
The chamfer library can be compiled by running python3 setup.py install under the directory chamfer. The source code is taken from the AtlasNet repository.
When compiling CUDA code, you may need to modify CUDA_PATH accordingly.

Running the code

To try a demo of the photometric mesh optimization, download our pretrained model for cars. Then run (setting the model variable to the downloaded checkpoint)

model=pretrained/02958343_atl25.npz
python3 main.py --load=${model} --code=5e-2 --scale=2e-2 --lr-pmo=3e-3 --noise=0.1 --video

This will create the following output files:

the optimized object mesh (saved into the directory optimized_mesh),
the input video sequence with the overlayed 3D mesh (saved to video), and
(coming soon!) a 3D mesh model (in .obj format) with textures estimated from the input RGB sequence.

The flags --log-tb and --log-vis toggles visualization of the optimization process. More optional arguments can be found by running python3 main.py --help.

Pretraining

To pretrain AtlasNet with our new dataset (high-resolution ShapeNet rendering + SUN360 cropped backgrounds), run the following command (taking the airplane category for example)

cat=02691156
python3 main_pretrain.py --category=${cat} --name=${cat}_pretrain \
	--imagenet-enc --pretrained-dec=pretrained/ae_atlasnet_25.pth

By default, we initialize the encoder with an ImageNet-pretrained ResNet-18 and the decoder with the pretrained AtlasNet (Please refer to the AtlasNet repository for downloading their pretrained models).
More optional arguments can be found by running python3 main_pretrain.py --help.

Visualizing the results

We've included code to visualize the training over TensorBoard(X). To execute, run

tensorboard --logdir=summary/GROUP --port=6006

where GROUP is specified in the pretraining arguments.
For pretraining, we provide three types of data visualization:

SCALARS: training and test loss curves over epochs
IMAGES: sample input images

If you find our code useful for your research, please cite

@inproceedings{lin2019photometric,
  title={Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction},
  author={Lin, Chen-Hsuan and Wang, Oliver and Russell, Bryan C and Shechtman, Eli and Kim, Vladimir G and Fisher, Matthew and Lucey, Simon},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2019}
}

Please contact me (chlin@cmu.edu) if you have any questions!

devernay / photometric-mesh-optim