VCL3D / SphericalViewSynthesis

Code accompanying the paper "Spherical View Synthesis for Self-Supervised 360 Depth Estimation", 3DV 2019

Spherical View Synthesis for Self-Supervised 360^o Depth Estimation

---

Data

An updated dataset is now available which fixes a critical issue with 3D60, the lighting bias introduced by the light source placed at the origin. More information can be found at the Pano3D project page.

The 360^o stereo data used to train the self-supervised models are available here and are part of a larger dataset [1, 2] that contains rendered color images, depth and normal maps for each viewpoint in a trinocular setup.

---

Train

Training code to reproduce our experiments is available in this repository:

A set of training scripts are available for each different variant:

• train_ud.py for vertical stereo (UD) training
• train_lr.py for horizontal stereo (LR) training
• train_tc.py for trinocular stereo (TC) training, using the photo_ratio argument to train the different TC variants.
• train_sv.py for supervised (SV) training

The PyTorch implementation of the differentiable depth-image-based forward rendering (splatting), presented in [3] and originally implemented in TensorFlow, is also available.

Test

Our evaluation script test.py also includes the adaptation of the metrics calculation to spherical data that includes spherical weighting and spiral sampling.

Pre-trained Models

Our PyTorch pre-trained models (corresponding to those reported in the paper) are available at our releases and contain these model variants:

• UD @ epoch 16
• TC8 @ epoch 16
• TC6 @ epoch 28
• TC4 @ epoch 17
• TC2 @ epoch 20
• LR @ epoch 18
• SV @ epoch 24

---

Citation

If you use this code and/or data, please cite the following:

@inproceedings{zioulis2019spherical,
  author       = "Zioulis, Nikolaos and Karakottas, Antonis and Zarpalas, Dimitris and Alvarez, Federic and Daras, Petros",
  title        = "Spherical View Synthesis for Self-Supervised $360^o$ Depth Estimation",
  booktitle    = "International Conference on 3D Vision (3DV)",
  month        = "September",
  year         = "2019"
}

References

[1] Zioulis, N.*, Karakottas, A.*, Zarpalas, D., and Daras, P. (2018). Omnidepth: Dense depth estimation for indoors spherical panoramas. In Proceedings of the European Conference on Computer Vision (ECCV).

[2] Karakottas, A., Zioulis, N., Samaras, S., Ataloglou, D., Gkitsas, V., Zarpalas, D., and Daras, P. (2019). 360^o Surface Regression with a Hyper-sphere Loss. In Proceedings of the International Conference on 3D Vision (3DV).

[3] Tulsiani, S., Tucker, R., and Snavely, N. (2018). Layer-structured 3d scene inference via view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV).