steliosploumpis/Universal_Head_3DMM

Towards a complete 3D morphable model of the human head

Stylianos Ploumpis^1,2, Evangelos Ververas^1,2, Eimear O'Sullivan^1,2, Stylianos Moschoglou^1,2, Haoyang Wang¹, Nick Pears³, William A. P. Smith³, Baris Gecer^1,2, & Stefanos Zafeiriou^1,2
¹Imperial College London, UK
²FaceSoft.io
³University of York, UK
^{(3D Graphic visuals by Vasilis Triantafyllou)}

[CVPR 2019] Extension in [TPAMI 2020]

Abstract

Three-dimensional Morphable Models (3DMMs) are powerful statistical tools for representing the 3D shapes and textures of an object class. Here we present the most complete 3DMM of the human head to date that includes face, cranium, ears, eyes, teeth and tongue. To achieve this, we propose two methods for combining existing 3DMMs of different overlapping head parts: i. use a regressor to complete missing parts of one model using the other, ii. use the Gaussian Process framework to blend covariance matrices from multiple models. Thus we build a new combined face-and-head shape model that blends the variability and facial detail of an existing face model (the LSFM) with the full head modelling capability of an existing head model (the LYHM). Then we construct and fuse a highly-detailed ear model to extend the variation of the ear shape. Eye and eye region models are incorporated into the head model, along with basic models of the teeth, tongue and inner mouth cavity. The new model achieves state-of-the-art performance. We use our model to reconstruct full head representations from single, unconstrained images allowing us to parameterize craniofacial shape and texture, along with the ear shape, eye gaze and eye color.

Approach

Detailed overview of our regression modeling pipeline. 1) The left part illustrates the matrix formulation from the original LYHM head model; 2) the central part demonstrates how we utilize the MeIn3D database to produce highly-detailed head shapes; 3) the final part on the right depicts the registration framework along with the per-vertex template weights and the statistical modeling.

A graphical representation of the non-rigid registration of all mean meshes along with our head template St and the calculation of the local covariance matrix Ki,j based on the locations of the ith and jth points.

Embeded Ear and Eye 3DMM

Head Reconstruction Results from single images

Citation

If you find this work is useful for your research, please cite our papers:

@inproceedings{ploumpis2019combining,
  title={Combining 3D Morphable Models: A Large scale Face-and-Head Model},
  author={Ploumpis, Stylianos and Wang, Haoyang and Pears, Nick and Smith, William AP and Zafeiriou, Stefanos},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={10934--10943},
  year={2019}
}

@article{ploumpis2020towards,
  title={Towards a complete 3D morphable model of the human head},
  author={Ploumpis, Stylianos and Ververas, Evangelos and O'Sullivan, Eimear and Moschoglou, Stylianos and Wang, Haoyang and Pears, Nick     and Smith, William and Gecer, Baris and Zafeiriou, Stefanos P},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2020},
  publisher={IEEE}
}

Public release models

The head models have been freely available for research and education purposes. To obtain access to the models, you need to complete and sign the user agreement form (can be found in this repository, user_agreement.pdf). This agreement should be completed by a full-time academic staff member (not a student) or a craniofacial clinician. The form should be signed, and emailed to Stylianos Ploumpis (s.ploumpis@gmail.com), Nick Pears (nick.pears@york.ac.uk). We will verify your request and contact you on how to download the model package. Note that the agreement requires that:

The models are used for non-commercial research and education purposes only.
You agree not copy, sell, trade, or exploit the model for any commercial purposes.
In any published research using the models, you cite the following papers:

Combining 3D Morphable Models: A Large scale Face-and-Head Model, S.Ploumpis, H.Wang, N.Pears, W. AP Smith, S. Zafeiriou, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (CVPR) 2019

Towards a complete 3D morphable model of the human head, S.Ploumpis, E.Ververas, E.Sullivan, S.Moschoglou, H.Wang, N.Pears, W.Smith and S.Zafeiriou, Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2020

steliosploumpis / Universal_Head_3DMM