kyotovision-public / fmbrdf

Fresnel Microfacet BRDF: Unification of Polari-Radiometric Surface-Body Reflection, CVPR2023

This repository provides an inplementation of our paper Fresnel Microfacet BRDF: Unification of Polari-Radiometric Surface-Body Reflection in CVPR 2023. If you use our code and data, please cite our paper.

Please note that this is research software and may contain bugs or other issues – please use it at your own risk. If you experience major problems with it, you may contact us, but please note that we do not have the resources to deal with all issues.

@InProceedings{Ichikawa_2023_CVPR,
    author    = {Ichikawa, Tomoki and Fukao, Yoshiki and Nobuhara, Shohei and Nishino, Ko},
    title     = {Fresnel Microfacet BRDF: Unification of Polari-Radiometric Surface-Body Reflection},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {16489-16497}
}

Prerequisites

We tested our code with Python 3.8.10 on Ubuntu 20.04 LTS. Our code depends on the following modules.

• matplotlib
• numba
• numpy
• opencv_contrib_python
• scipy
• torch

Please browse requirements.txt generated by pipreqs to find the specific versions we used in our test environment, or you can simply do

$ python3 -m pip install -r requirements.txt

in your (virtual) environment.

Alternatively, you can use fmbrdf.def to build your singularity container by

$ singularity build --fakeroot ./fmbrdf.sif ./fmbrdf.def
INFO:    Starting build...
Getting image source signatures
(snip)
INFO:    Creating SIF file...
INFO:    Build complete: ./fmbrdf.sif

The file size of the generated container file (fmbrdf.sif in this example) will be around 7.4GB.

Data

If you would like to implement our code with our data, please download it from here (3.2GB) and put it in data/.

Following data is included in a directory for each object:

• data/{object}/calibration/
  • IntrinsicL.npy : Intrinsic camera parameters
  • L.txt : List of light source directions (The index is the same as that of an image)
  • distortionL.npy : Distortion coefficients
  • vpL_img.npy : Viewing direction for each pixel
  • intensities.npy (only for horse_shape) : The ratio of intensities of light sources between different images. Due to a large change in light source height, the intensities of light sources are weakened slightly for some images.
• data/{object}/image/
  • img : Images before undistortion (16-bit Bayer pattern polarimetric image)
  • img_ud : Undistorted images (16-bit Bayer pattern polarimetric image)
• data/{object}/normal/
  • normal_GT.npy : Ground truth of surface normal

Directories for objects with rough mesogeometry (sponge_sandpaper and texture_sand) are a little different:

• data/{object}/calib/
  • IntrinsicL.npy : Intrinsic camera parameter
  • light.npz : A light direction for a certain image and the index of the image
  • distortionL.npy : Distortion coefficients
  • vpL_img.npy : Viewing direction for each pixel
• data/{object}/image/
  • img : Images before undistortion (16-bit Bayer pattern polarimetric image)
  • img_ud : Undistorted images (16-bit Bayer pattern polarimetric image)

BRDF Model

As mentioned in the paper, we approximate body reflection and Smith's shadowing/masking function with multilayer perceptrons (MLPs). The MLP parameters are put in BRDFmodel/NN_model/. body_reflection.pth and shadowing.pth are MLP parameters for the body reflection and Smith's shadowing/masking function, respectively. Each MLP is defined in FMBRDF.py as Net_BodyReflection and Net_Shadowing.

Usage

Please implement the following shell scripts.

Fitting to Images with DoLP

$ sh DoLP_fitting.sh

This script implements

• python DoLP_fitting.py <object>: Fitting our model to multiple polarimetric images with DoLPs
• python DoLP_fitting_rendering.py <object>: Output fitting results as DoLP images and plots

The output result is in data/{object}/fitting-DoLP/:

• DoLP/: Rendered DoLP images with obtained parameters
• DoLP_plot/: Rendered DoLP values plotted as a function of the angle between the global surface normal and the viewing direction
• DoLP_error.txt: The RMSEs of rendered DoLP images for each image and all images.

Fitting to Images with Intensity

$ sh intensity_fitting.sh

This script implements

• python intensity_fitting.py <object>: Fitting our model to multiple polarimetric images with intensities
• python intensity_fitting_rendering.py <object>: Output fitting results as intensity images and plots

The output results are in data/{object}/fitting-intensity/:

• intensity/: Rendered intensity images with obtained parameters
• intensity_plot/: Rendered intensity values plotted as a function of the angle between the global surface normal and the lighting source direction
• intensity_error.txt: The RMSEs of rendered intensity images for each image and all images.

Fitting to Images of A Plane with Rough Mesogeometry with Intensity

$ sh mesogeometry.sh

This script implements

• python save_intensities_plane.py <object> <rotation angle of camera> <settings of ChArUco borad> <IDs of ArUco markers at corners> -s <offset of start index> -e <offset of end index>: Save the average intensity of the plane sample and compute the camera pose for each image
• python mesogeometry_fitting.py <object>: Implement fitting to the intensity of the plane sample and show the fitting result

The output result is data/{object}/FMBRDF_intensity.pdf.

BRDF Estimation

$ sh estimate_BRDF.sh

This script implements

• python estimate_BRDF.py <object> <index of image>: Estimate FMBRDF parameters from a single polarimetric image
• python estimate_BRDF_rendering.py <object> <index of image>: Render an image under another lighting condition with estimated BRDF

The output result is data/{object}/estimation_BRDF/

• img_ud_{index}_FMBRDF.png: The rendered image.
• intensity_error.txt: The RMSE of rendered radiance image devided by the mean of observed intensity

Fitting to MERL BRDF Dataset

Please download MERL BRDF database and put brdfs directory in MERL_BRDFDatabase/.

$ sh MERL_fitting.sh

This script implements

• python MERL_fitting.py: Implement fitting to all materials of MERL BRDF dataset
• python MERL_error.py: Compute fitting error of all materials
• python MERL_result.py: Show the fitting error as a graph

The output result is in code/MERL_fitting_result/MERL_error_RGB.pdf.

Shape Reconstruction

$ sh shape_reconstruction.sh

This script implements

• python shape_reconstruction.py <object>: Reconstruct the surface normal from multiple polarimetric normal images
• python shape_reconstruction_error.py <object>: Compute the angular error of the reconstructed surface normal

The output result is in data/{object}/shape_reconstruction/:

• normal_FMBRDF.png: Reconstructed surface normal with our model
• normal_errorFMBRDF.png: The error map of surface normal reconstructed with our model
• error.txt: The mean/median/standard deviation of the angular errors.