Jerrypiglet / rui-indoorinv-data

Script for generating customized data for indoor scene inverse rendering (e.g. FIPT, FVP, Li22, MonoSDF)

FIPT-data: data preparation for FIPT

Project Page | Paper | Data download for FIPT

Overview

This repo (branch: fipt) contains the code for generating customized data for FIPT, from scratch.

The repo is also useful for loading/generating data for other indoor inverse rendering pipelines/datasets, by adding load_{DATASET}Scene3D.py and lib/class_{DATASET}Scene3D.py for loading other data formats, and customized formats to lib/class_exporter.py for export from existing datasets to those new formats.

Currently supported datasets include:

1. indoor_synthetic

   • See README_indoor_synthetic.md for details.
   • Based on Mitsuba XML files by Benedikt Bitterli.
   • Scripts support: sampling poses, rendering numerous modalities, visualization and export to FIPT/Monosdf/FVP/Li22.

2. real

   • See README_real.md for details.
   • Captured for FIPT.
   • Scripts support: visualization and export to FIPT/Monosdf/FVP/Li22.

See ## Related Works for a brief overview of aforementioned methods.

Installation

Please refer to README_env.md for instructions for installing the environment.

Related Works

• FIPT

  • Wu and Zhu et al. 2023, FIPT: Factorized Inverse Path Tracing
  • Optimization-based multi-view inverse rendering method.

• Monosdf

  • Yu et al. NeurIPS 2022, MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
  • NeRF-like methods for multi-view scene reconstruction using SDF (signed-distance function) representation.
  • Used in FIPT for acquiring scene geometry (meshes).

• IPT

  • Azinović et al. CVPR 2019, Inverse Path Tracing for Joint Material and Lighting Estimation
  • Optimization-based multi-view inverse rendering method.
  • Used as baseline in FIPT.

• MILO

  • Yu et al. TPAMI 2023, MILO: Multi-bounce Inverse Rendering for Indoor Scene with Light-emitting Objects
  • Optimization-based multi-view inverse rendering method.
  • Used as baseline in FIPT.

• FVP

  • Philip et al. TOG 2021, Free-viewpoint Indoor Neural Relighting from Multi-view Stereo
  • Takes multiple images and aggregate multiview irradiance and albedo information to a pre-trained network to synthesize a relit image.
  • Used as baseline in FIPT.
  • Our code | Original code

• Li22

  • Li et al. ECCV 2022, Physically-Based Editing of Indoor Scene Lighting from a Single Image
  • Learning-based single image inverse rendering and relighting.
  • Used as baseline in FIPT.
  • Our code | Original code

• NeILF

  • Yao et. al. ECCV 2022, NeILF: Neural Incident Light Field for Material and Lighting Estimation
  • NeRF-like methods for multi-view inverse rendering by estimating neural representations of surface lighting and BRDF.
  • Used as baseline in FIPT.
  • Our code | Original code

See Related Works section by the end of FIPT website for overview of most recent works.

TODO

• Add code links for re-implemented baseline methods: FVP, NeILF, Li22

Citation

If you find our work is useful, please consider cite:

@misc{fipt2023,
      title={Factorized Inverse Path Tracing for Efficient and Accurate Material-Lighting Estimation}, 
      author={Liwen Wu and Rui Zhu and Mustafa B. Yaldiz and Yinhao Zhu and Hong Cai and Janarbek Matai and Fatih Porikli and Tzu-Mao Li and Manmohan Chandraker and Ravi Ramamoorthi},
      year={2023},
      eprint={2304.05669},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}