- An attempt to reproduce the data generation part of this paper.
- The github repository for the aforementioned paper can be found here.
- As mentioned in the paper the optimization makes use Basel Face Model(BFM).
- The mean shape and albedo parameters along with the principal components and standard deviations for identity and albedo can be found in BFM.
- The prinicipal components for the expression can be found in the CoarseData dataset which can be downloaded from the author's repository.
- The standard deivation for expression can be found in
std_exp.txt - The inner mouth vertices of the BFM model has been discarded based on 3DDFA. The indices of the vertices ignored are stored in
model_info.mat - 300-W Dataset is used for the optimization process. The dataset also provides facial landmarks.
- There are three implementations of the optimization algorithm.
Optimization based Inverse Rendering-PyTorch.ipynb/.py- Pytorch implementation. The rasterization process alone uses numpy as the pytorch version of it is slow. Since torch tensor is converted to numpy, it gets detached from the graph and therefore torch.backward() cannot be used for calculating gradients and jacobain. To use the pytorch implementation convert all the numpy parts to torch tensor.Optimization based Inverse Rendering-Numpy.ipynb/.py- Numpy Implementation. Although the rendering and rasterization process is fast the autograd part of the code used to calculate the jacobian for Gauss-Newton method is extremely slow! This is because numpy runs on CPU.Optimization based Inverse Rendering-MXNet.ipynb/.py- MXNet Implementation. To overcome the hurdle in the numpy implementation, MXNet provides GPU accelerated numpy but doesn't have a jacobian function as in numpy. The code is not complete since the jacobian computation proves to be tricky.
- Guo, Yudong, et al. "CNN-based real-time dense face reconstruction with inverse-rendered photo-realistic face images." IEEE transactions on pattern analysis and machine intelligence 41.6 (2018): 1294-1307.
- Rendering and rasterization code based on this implemention
- Jacobian matrix computation function in pytorch is based on this implementation and in numpy is based on this implementation