AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild":

Published in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020)

AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs:

Published in IEEE Transaction on Pattern Analysis and Machine Intelligence (TPAMI 2021)

Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Baris Gecer,
Abhijeet Ghosh, Stefanos Zafeiriou
Imperial College London

Overview

AvatarMe++ is a new method for reconstructing renderable photorealistic 3D faces from a single ‘in-the-wild”. Extending on AvatarMe we define a fast facial photorealistic differentiable rendering methodology with accurate facial skin diffuse and specular reflection, self-occlusion and subsurface scattering approximation. With this, we train a network that disentangles the facial diffuse and specular BRDF components from a shape and texture with baked illumination, reconstructed with a state-of-the-art 3DMM fitting method.

Method

Summary of the AvatarMe++ method: We acquire a 3DMM fitting and upscale its texture using a state-of-the-art super resolution network, trained on synthetic data rendered in the texture’s domain. A image-translation network is then used to transform the upscaled texture and normals to reflectance maps, namely the Diffuse Albedo, Specular Albedo, Diffuse Normals and Specular Normals.

We create a rendering module, with subsurface-scattering and self-occlusion approximation. During training, it is used to create synthetic data pairs, by rendering the captured data in the target’s environment and random ones, which are used for photorealistic rendering loss. The complete high resolution (up to 6k×4k) BRDF maps can be used for photorealistic rendering, while the specular normals can be used to enhance the original geometry with fine details.

Results

Our high resolution mesh and reflectance textures (Diffuse Albedo, Specular Albedo, Diffuse Normals, Specular Normals) can be used with most open-source or commercial renderers to render the reconstructed subjects in various illumination environments.

Thanks to the photorealistic differentiable renderer and the stochastic rendering loss, in contrast to AvatarMe, AvatarMe++ can be generalize for most examples with frontal facing illumination. As shown above, AvatarMe++ can be effectively used in reconstructions with (Chen et al., 2019), (Gecer et al., 2021) and datasets such as Facescape, Superface and subjects captured with a 3dMD device.

Citation

If you find this work useful in your research, please cite our papers AvatarMe, AvatarMe++ using:

@InProceedings{lattas2020avatarme,
  author = {Lattas, Alexandros and Moschoglou, Stylianos and Gecer, Baris and Ploumpis, Stylianos and Triantafyllou, Vasileios and Ghosh, Abhijeet and Zafeiriou, Stefanos},
  title = {AvatarMe: Realistically Renderable 3D Facial Reconstruction "In-the-Wild"},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2020}
}

@article{lattas2021avatarme++,
  title={AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs},
  author={Lattas, Alexandros and Moschoglou, Stylianos and Ploumpis, Stylianos and Gecer, Baris and Ghosh, Abhijeet and Zafeiriou, Stefanos P},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2021},
  publisher={IEEE}
}

Contact

For any enquiries please contact the first author at the email listed on the above papers.

ehipasika / avatarme