ChickenRushKR / AUFART

AUFART: Action Unit-Based 3D Face Reconstruction Using Transformers

Using Action Unit in single & monocular 3D Face reconstruction

This is the official Pytorch implementation of AUFART.

AUFART reconstructs 3D head models sensitive to AU activations from a single monocular in-the-wild face image.
Please refer to the paper preprint for more details.

Main contributions:

• AU-feature based reconstruction: We propose a Transformer-based 3D face reconstruction framework that leverages the features of AUs in the frame-based 3D face reconstruction process, explicitly consider-ing their correlations.
• Employing professional AU recognition model: We integrate a state-of-the-art AU feature extraction module for effective AU feature extraction from in-the-wild images.
• Effective architecture: We propose a Transformer encoder-based model that can utilize both image features and AU features.
• Novel loss functions: We design AU-based loss fuctions for training our proposed 3D face reconstruction framework

Related works:

• for the accurate AU prediction: ME-GraphAU
• for the better skin estimation: TRUST