PyTorch implementation for LayoutNet v2 in the paper:
https://arxiv.org/pdf/1910.04099.pdf
- New: Our MatterportLayout Annotation dataset is released!
Original Torch implementation for LayoutNet is here.
You may also be interested in the source code of the methods for comparison in the paper: DuLa-Net and HorizonNet
- Extend to general Manhattan layout (on our newly labeled MatterportLayout dataset)
- Use ResNet encoder instead of SegNet encoder
- Training details and implementation details
- Gradient ascent based post optimization, revised from sunset1995's PyTorch implementation
- Add random stretching data augmentation
- Python 3
- PyTorch >= 0.4.0
- numpy, scipy, pickle, skimage, sklearn, random, cv2, shapely
- torchvision
- Matlab (for depth rendering)
- Download pre-trained models and put them under the ./model/ folder.
- Download pre-processed PanoContext and Stanford 2D-3D dataset in .pkl form and put them under the ./data/ folder.
- Download gt from original LayoutNet and the processed one by sunset1995 and put them under the ./data/ folder
- (Optional) Download original LayoutNet's .t7 file and put them under the ./data/ folder
- Download our newly labeled MatterportLayout dataset and put them under the ./data/ folder.
- Download pre-computed depth maps from our models trained on MatterportLayout dataset and put them under the current folder.
- We've provided sample code to transform original LayoutNet's .t7 file to .pkl file for PyTorch
python t72pkl.py
- On PanoContext (note that we use Stanford 2D-3D as additional data in this script):
python train_PC.py
- On Stanford 2D-3D (note that we use PanoContext as additional data in this script):
python train_stanford.py
- On MatterportLayout
python train_matterport.py
-
On PanoContext (Corner error, pixel error and 3D IoU)
python test_PC.py
-
On Stanford 2D-3D (Corner error, pixel error and 3D IoU)
python test_stanford.py
-
On Matterport3D (3D IoU, 2D IoU on under top-down view, RMSE for depth and delta_1 for depth)
python test_matterport.py
For depth related evaluation, we need to render depth map from predicted corner position on equirectangualr view (you can skip this step as we've provided pre-computed depth maps from our approach)
First, uncomment L313-L314 in test_matterport.py, and comment out lines related to evaluation for depth. Run test_matterport.py and save intermediate corner predictions to folder ./result_gen. Then open matlab:
cd matlab cor2depth cd ..
Rendered depth maps will be saved to folder ./result_gen_depth/. Then comment out L313-L314 in test_matterport.py, uncomment lines related to evaluation for depth, and run test_matterport.py again
Please cite our paper for any purpose of usage.
@article{zou20193d,
title={3D Manhattan Room Layout Reconstruction from a Single 360 Image},
author={Zou, Chuhang and Su, Jheng-Wei and Peng, Chi-Han and Colburn, Alex and Shan, Qi and Wonka, Peter and Chu, Hung-Kuo and Hoiem, Derek},
journal={arXiv preprint arXiv:1910.04099},
year={2019}
}