neverwinHao / Awesome-Face-Restoration

A comprehensive summary of deep face restoration methods.

Survey on Deep Face Restoration: From Non-blind to Blind and Beyond

A comprehensive statistic on methods related to deep face restoration.
[Paper]   [Supplementary Material]

| | | | |

| | | | |

💥 Note: More visual comparisons can be found in the Paper and Supplementary Material.

---

📋 Citation

@article{li2023survey,
  title={Survey on Deep Face Restoration: From Non-blind to Blind and Beyond},
  author={Li, Wenjie and Wang, Mei and Zhang, Kai and Li, Juncheng and Li, Xiaoming and Zhang, Yuhang and Gao, Guangwei and Deng, Weihong and Lin, Chia-Wen},
  journal={arXiv preprint arXiv:2309.15490},
  year={2023}
}

Table of contents

• Non-Blind Tasks
• Blind Tasks
• Joint Tasks
  • Joint Face Completion
  • Joint Face Frontalization
  • Joint Face Alignment
  • Joint Face Recogntion
  • Joint Face Illumination Compensation
  • Joint Face Fairness
  • Joint 3D Face Reconstruction
• Perfermance
  • Non-Bind Face Super-Resoution
  • Blind Face Restoration
  • Blind Face Super-Resoution
  • Joint Face Restoration
• Benchmarks
  • Populur Datasets
  • Evaluation Metrics

Non-Blind Tasks

Pub	Paper	Technology
AAAI2015	Learning Face Hallucination in the Wild	CNN-based
ECCV2016	Deep cascaded bi-network for face hallucination	CNN-based
ECCV2016	Ultra-resolving face images by discriminative generative networks	GAN-based
ICCV2017	Wavelet-SRNet: A Wavelet-based CNN for Multi-scale Face Super Resolution	Wavelet transform
CVPR2017	Attention-Aware Face Hallucination via Deep Reinforcement Learning	Attention-based
CVPR2017	Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders	GAN-based
IJCAI2017	Learning to Hallucinate Face Images via Component Generation and Enhancement	Prior-based
ICASSP2017	Face Hallucination Based on Key Parts Enhancement	CNN-based
CVPR2018	FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors	Prior-based
CVPRW2018	Attribute Augmented Convolutional Neural Network for Face Hallucination	GAN-based
ECCV2018	Super-Identity Convolutional Neural Network for Face Hallucination	Prior-based
ECCV2018	Face Super-resolution Guided by Facial Component Heatmaps	Prior-based
IJCAI2018	Deep CNN Denoiser and Multi-layer Neighbor Component Embedding for Face Hallucination	CNN-based
ICASSP2018	FACE HALLUCINATION BASED ON KEY PARTS ENHANCEMENT	Prior-based
CVPRW2019	Exemplar Guided Face Image Super-Resolution without Facial Landmarks	Prior/GAN-based
BMVC2019	Progressive Face Super-Resolution via Attention to Facial Landmark	Prior/GAN-based
AAAI2019	Residual Attribute Attention Network for Face Image Super-Resolution	Attention-based
TMM2019	ATMFN: Adaptive-threshold-based Multi-model Fusion Network for Compressed Face Hallucination	CNN/GAN/RNN-based
IJCV2019	Joint Face Hallucination and Deblurring via Structure Generation and Detail Enhancement	Prior-based
PR2019	Face hallucination from low quality images using definition-scalable inference	Frequency-based
TIP2020	Learning Spatial Attention for Face Super-Resolution	Attention-based
TIP2020	Deblurring Face Images using Uncertainty Guided Multi-Stream Semantic Networks	Prior-based
TMM2020	Learning Face Image Super-Resolution through Facial Semantic Attribute Transformation and Self-Attentive Structure Enhancement	Prior-based
TNNLS2020	Dual-Path Deep Fusion Network for Face Image Hallucination	CNN-based
ECCV2020	Face Super-Resolution Guided by 3D Facial Priors	Prior-based
CVPR2020	Deep Face Super-Resolution with Iterative Collaboration between Attentive Recovery and Landmark Estimation	Prior-based
CVPR2020	MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks	GAN-based
AAAI2020	Facial Attribute Capsules for Noise Face Super Resolution	Prior-based
MM2020	PCA-SRGAN: Incremental Orthogonal Projection Discrimination for Face Super-resolution	GAN-based
ICASSP2020	PARSING MAP GUIDED MULTI-SCALE ATTENTION NETWORK FOR FACE HALLUCINATION	Prior-based
WACV2020	Component Attention Guided Face Super-Resolution Network: CAGFace	Prior-based
FG2020	ATFaceGAN: Single Face Image Restoration and Recognition from Atmospheric Turbulence	GAN-based
TBIOM2021	Semantic-Driven Face Hallucination Based on Residual Network	Prior-based
TBBLS2021	E-ComSupResNet: Enhanced Face Super-Resolution Through Compact Network	CNN-based
TCYB2021	Sequential Gating Ensemble Network for Noise Robust Multiscale Face Restoration	CNN-based
TIP2021	Features Guided Face Super-Resolution via Hybrid Model of Deep Learning and Random Forests	Prior-based
TPAMI2021	Face restoration via plug-and-play 3d facial priors	Prior-based
CVPR2021	GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolutio	GAN/Prior-based
MM2021	Face Hallucination via Split-Attention in Split-Attention Network	Attention-based
ICME2021	HEATMAP-AWARE PYRAMID FACE HALLUCINATION	Prior-based
ICME2021	SAAN: SEMANTIC ATTENTION ADAPTATION NETWORK FOR FACE SUPER-RESOLUTION	Prior-based
ICME2021	ORGAN-BRANCHED CNN FOR ROBUST FACE SUPER-RESOLUTION	GAN/Prior-based
ICME2021	FACE SUPER-RESOLUTION THROUGH DUAL-IDENTITY CONSTRAINT	Prior-based
ICIP2021	Progressive Face Super-Resolution with Non-Parametric Facial Prior Enhancement	Prior-based
TCSVT2022	Progressive Face Super-Resolution with Non-Parametric Facial Prior Enhancement	Attention-based
TCSVT2022	Propagating Facial Prior Knowledge for Multitask Learning in Face Super-Resolution	Prior-based
TNNLS2022	Rethinking Prior-Guided Face Super-Resolution: A New Paradigm With Facial Component Prior	Prior-based
TIP2022	Attention-Driven Graph Neural Network for Deep Face Super-Resolution	Attention-based
TPAMI2022	Image Super-Resolution via Iterative Refinement	Diffusion-based
Nercom2022	SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models	Diffusion-based
ICME2022	RCNet: Recurrent Collaboration Network Guided by Facial Priors for Face Super-Resolution	Prior-based
IJCB2022	Face Super-Resolution with Progressive Embedding of Multi-scale Face Priors	Prior-based
ICME2022	RCNet: Recurrent Collaboration Network Guided by Facial Priors for Face Super-Resolution	Prior-based
Arxiv2022	Bi-Noising Diffusion: Towards Conditional Diffusion Models with Generative Restoration Priors	Diffusion-based
Arxiv2022	Stochastic Attribute Modeling for Face Super-Resolution	Prior-based
Arxiv2022	HAZE-Net: High-Frequency Attentive Super-Resolved Gaze Estimation in Low-Resolution Face Images	Frequency-based
TMM2023	An Efficient Latent Style Guided Transformer-CNN Framework for Face Super-Resolution	Attention-based
TMM2023	SCTANet: A Spatial Attention-Guided CNN-Transformer Aggregation Network for Deep Face Image Super-Resolution	Attention-based
TMM2023	Exploiting Multi-scale Parallel Self-attention and Local Variation via Dual-branch Transformer-CNN Structure for Face Super-resolution	Attention-based
TBIOM2023	Super-Resolving Face Image by Facial Parsing Information	Prior-based
TIP2023	CTCNet: A CNN-Transformer Cooperation Network for Face Image Super-Resolution	Attention-based
TIM2023	Deep HyFeat Based Attention in Attention Model for Face Super-Resolution	GAN/Attention-based
PR2023	A Composite Network Model for Face Super-Resolution with Multi-Order Head Attention Facial Priors	Attention/Prior-based
CVPR2023	Spatial-Frequency Mutual Learning for Face Super-Resolution	Attention-based
CVPR2023	Implicit Diffusion Models for Continuous Super-Resolution	Diffusion-based
IJCAI2023	Learning Attention from Attention: Effcient Self-Refnement Transformer for Face Super-Resolution	Attention-based

Blind Tasks

Pub	Paper	Technology
CVPRW2017	Deep Face Deblurring	CNN-based
ECCV2018	Learning to Super-Resolve Blurry Face and Text Images	GAN-based
ECCV2018	Learning Warped Guidance for Blind Face Restoration	CNN-based
ECCV2018	Attribute-guided face generation using conditional cyclegan	CycleGAN-based
ECCV2018	To learn image super-resolution, use a GAN to learn how to do image degradation first	CycleGAN-based
CVPR2018	Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans	Prior/GAN-based
CVPR2018	Exploiting Semantics for Face Image Deblurring	Prior-based
CVPRW2018	Learning Face Deblurring Fast and Wide	Prior-based
CVPRW2018	Learning Face Deblurring Fast and Wide	CNN-based
ICCV2019	DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better	GAN-based
ICIP2019	GUIDED CYCLEGAN VIA SEMI-DUAL OPTIMAL TRANSPORT FOR PHOTO-REALISTIC FACE SUPER-RESOLUTION	CycleGAN-based
IJCV2020	Exploiting Semantics for Face Image Deblurring	Prior-based
IJCV2020	Identity-preserving Face Recovery from Stylized Portraits	Prior-based
AAAI2020	Learning to Deblur Face Images via Sketch Synthesis	GAN-based
ECCV2020	Blind Face Restoration via Deep Multi-scale Component Dictionaries	GAN/Prior-based
CVPR2020	Enhanced Blind Face Restoration with Multi-Exemplar Images and Adaptive Spatial Feature Fusion	GAN/Prior-based
CVPR2020	Image Processing Using Multi-Code GAN Prior	GAN/Prior-based
MM2020	HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment	GAN-based
Arxiv2020	DeepDeblur: Fast one-step blurry face images restoration	GAN-based
CVPR2021	Progressive Semantic-Aware Style Transformation for Blind Face Restoration	GAN-based
CVPR2021	Progressive Semantic-Aware Style Transformation for Blind Face Restoration	GAN-based
CVPR2021	Progressive Semantic-Aware Style Transformation for Blind Face Restoration	GAN/Prior-based
CVPR2021	Towards Real-World Blind Face Restoration with Generative Facial Prior	GAN/Prior-based
CVPR2021	GAN Prior Embedded Network for Blind Face Restoration in the Wild	GAN/Prior-based
Arxiv2021	Universal Face Restoration With Memorized Modulation	GAN-based
Arxiv2021	Implicit Subspace Prior Learning for Dual-Blind Face Restoration	GAN-based
CVPR2022	RestoreFormer: High-Quality Blind Face Restoration from Undegraded Key-Value Pairs	GAN/Attention/Prior-based
CVPR2022	Blind Face Restoration via Integrating Face Shape and Generative Priors	GAN/Prior-based
CVPR2022	Rethinking Deep Face Restoration	GAN/Prior-based
CVPR2022	GCFSR: a Generative and Controllable Face Super Resolution Method Without Facial and GAN Priors	GAN-based
CVPRW2022	Unpaired Face Restoration via Learnable Cross-Quality Shift	CycleGAN-based
AAAI2022	Panini-Net: GAN Prior Based Degradation-Aware Feature Interpolation for Face Restoration	GAN/Prior-based
ECCV2022	VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder	GAN/Prior-based
ECCV2022	From Face to Natural Image: Learning Real Degradation for Blind Image Super-Resolution	GAN-based
Nips2022	Towards Robust Blind Face Restoration with Codebook Lookup Transformer	GAN/Prior-based
ICME2022	FEATURE-GUIDED BLIND FACE RESTORATION WITH GAN PRIOR	GAN/Prior-based
WACV2022	Deep Feature Prior Guided Face Deblurring	GAN/Prior-based
TCSVT2022	DEAR-GAN: Degradation-Aware Face Restoration with GAN Prior	GAN/Prior-based
TPAMI2022	GLEAN: Generative Latent Bank for Image Super-Resolution and Beyond	GAN/Prior-based
TPAMI2022	Learning Dual Memory Dictionaries for Blind Face Restoration	GAN/Prior-based
SIGGRAPH2022	Face Deblurring using Dual Camera Fusion on Mobile Phones	CNN-based
Arxiv2022	FaceFormer: Scale-aware Blind Face Restoration with Transformers	GAN/Attention-based
Arxiv2022	Multi-prior learning via neural architecture search for blind face restoration	GAN/Prior-based
Arxiv2022	Blind Face Restoration: Benchmark Datasets and a Baseline Model	GAN/Attention-based
Arxiv2022	Enhancing Quality of Pose-varied Face Restoration with Local Weak Feature Sensing and GAN Prior	GAN/Prior-based
Arxiv2022	Difface: Blind Face Restoration with Diffused Error Contraction	Diffusion-based
Arxiv2022	Face Deblurring Based on Separable Normalization and Adaptive Denormalization	Prior/GAN-based
CVPR2023	DR2: Diffusion-based Robust Degradation Remover for Blind Face Restoration	Diffusion-based
CVPRW2023	TFRGAN: Leveraging Text Information for Blind Face Restoration with Extreme Degradation	GAN/Prior-based
CVPRW2023	FRR-Net: A Real-Time Blind Face Restoration and Relighting Network	GAN/Prior-based
AAAI2023	GAN Prior Based Null-Space Learning for Consistent Super-resolution	GAN-based
IJCAI2023	Analyzing and Combating Attribute Bias for Face Restoration	GAN/Prior-based
ICML2023	ZERO-SHOT IMAGE RESTORATION USING DENOISING DIFFUSION NULL-SPACE MODEL	Diffusion-based
ICCV2023	Towards Authentic Face Restoration with Iterative Diffusion Models and Beyond	Diffusion-based
MM2023	DiffBFR: Bootstrapping Diffusion Model Towards Blind Face Restoration	Diffusion-based
Nips2023	PGDiff: Guiding Diffusion Models for Versatile Face Restoration via Partial Guidance	Diffusion-based
TIP2023	Semi-Cycled Generative Adversarial Networks for Real-World Face Super-Resolution>	CycleGAN-based
TPAMI2023	Old Photo Restoration via Deep Latent Space Translation	GAN/Prior-based
TPAMI2023	Multi-Modality Deep Restoration of Extremely Compressed Face Videos	GAN/Prior-based
TPAMI2023	RestoreFormer++: Towards Real-World Blind Face Restoration from Undegraded Key-Value Pairs	Prior/GAN-based
TCSVT2023	Blind Face Restoration for Under-Display Camera via Dictionary Guided Transformer	Prior/Attention-based
WACV2023	AT-DDPM: Restoring Faces degraded by Atmospheric Turbulence using Denoising Diffusion Probabilistic Models	Diffusion-based
Arxiv2023	ResDiff: Combining CNN and Diffusion Model for Image Super-Resolution	Diffusion-based
Arxiv2023	DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior	Diffusion-based
Arxiv2023	Exploiting Diffusion Prior for Real-World Image Super-Resolution	Diffusion-based
Arxiv2023	HDTR-Net: A Real-Time High-Definition Teeth Restoration Network for Arbitrary Talking Face Generation Methods	CNN-based
Arxiv2023	Towards Real-World Blind Face Restoration with Generative Diffusion Prior	Prior/Diffusion-based
ICLR2024	DUAL ASSOCIATED ENCODER FOR FACE RESTORATION	Prior/GAN-based

Joint Tasks

Joint Face Completion

Pub	Paper	Technology
ICME2019	Facial Image Inpainting Using Multi-level Generative Network	CNN-based
FG2020	IF-GAN: Generative Adversarial Network for Identity Preserving Facial Image Inpainting and Frontalization	GAN-based
TBOIM2020	FCSR-GAN: Joint Face Completion and Super-Resolution via Multi-Task Learning	GAN/Prior-based
TCSVT2020	Occluded Face Recognition in the Wild by Identity-Diversity Inpainting	GAN/Prior-based
ICIP2022	Swin-CasUNet: Cascaded U-Net with Swin Transformer for Masked Face Restoration	Attention-based
WACV2022	3DFaceFill: An Analysis-By-Synthesis Approach to Face Completion	Prior/GAN-based
TMM2022	Pluralistic Face Inpainting with Transformation of Attribute Information	GAN/Prior-based
TMM2023	FT-TDR: Frequency-Guided Transformer and Top-Down Refinement Network for Blind Face Inpainting	GAN/Prior-based
TMM2023	JDSR-GAN: Constructing an Efficient Joint Learning Network for Masked Face Super-Resolution	GAN/Prior-based
TCSVT2022	SwapInpaint: Identity-Specific Face Inpainting With Identity Swapping	GAN/Prior-based
JVCI2023	Joint Face Completion and Super-resolution using Multi-scale Feature Relation Learning	GAN/Prior-based
ICASSP2023	Fine-Grained Blind Face Inpainting with 3D Face Component Disentanglement	Prior-based
TIP2023	Deep Face Video Inpainting via UV Mapping	Attention/GAN/Prior-based
PR2023	Joint face completion and super-resolution using multi-scale feature relation learning	Attention/GAN/Prior-based

Joint Face Frontalization

Pub	Paper	Technology
TPAMI2019	Can We See More? Joint Frontalization and Hallucination of Unaligned Tiny Faces	GAN-based
FG2020	IF-GAN: Generative Adversarial Network for Identity Preserving Facial Image Inpainting and Frontalization	GAN-based
TCSVT2021	Joint Face Image Restoration and Frontalization for Recognition	GAN/Prior-based
TCSVT2021	Simultaneous Face Completion and Frontalization via Mask Guided Two-Stage GAN	GAN/Attention-based
TIP2021	Face Hallucination With Finishing Touches	GAN/Prior-based

Joint Face Alignment

Pub	Paper	Technology
AAAI2017	Face Hallucination with Tiny Unaligned Images by Transformative Discriminative Neural Networks	GAN-based
CVPR2017	Hallucinating Very Low-Resolution Unaligned and Noisy Face Images by Transformative Discriminative Autoencoders∗	GAN-based
CVPR2018	Semantic Face Hallucination: Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes	GAN-based
AAAI2020	Joint Super-Resolution and Alignment of Tiny Faces	Prior-based
IJCV2020	Hallucinating Unaligned Face Images by Multiscale Transformative Discriminative Networks	GAN-based

Joint Face Recogntion

Pub	Paper	Technology
ECCV2018	Super-Identity Convolutional Neural Network for Face Hallucination	Prior-based
ICIP2019	Low-Resolution Face Recognition Based on Identity-Preserved Face Hallucination	CNN-based
TIP2019	SiGAN: Siamese Generative Adversarial Network for Identity-Preserving Face Hallucination	GAN/Prior-based
TIP2019	Face Hallucination Using Cascaded Super-Resolution and Identity Priors	GAN/Prior-based
Nercom2019	Edge and Identity Preserving Network for Face Super-Resolution	GAN/Prior-based
Arxiv2019	Verification of Very Low-Resolution Faces Using An Identity-Preserving Deep Face Super-Resolution Network	GAN/Prior-based
IJCV2019	Wavelet Domain Generative Adversarial Network for Multi-scale Face Hallucination	GAN/Prior-based
TPAMI2020	Adversarial Cross-Spectral Face Completion for NIR-VIS Face Recognition	GAN-based

Joint Face Illumination Compensation

Pub	Paper	Technology
ICB2019	SeLENet: A Semi-Supervised Low Light Face Enhancement Method for Mobile Face Unlock	CNN-based
CVPR2020	Copy and Paste GAN: Face Hallucination from Shaded Thumbnails	CNN-based
CVPR2020	From Less to More: Spectral Splitting and Aggregation Network for Hyperspectral Face Super-Resolution	CNN-based
ICME2020	Learning To See Faces In The Dark	CNN-based
Arxiv2021	Network Architecture Search for Face Enhancement	CNN-based
TPAMI2021	Recursive Copy and Paste GAN: Face Hallucination From Shaded Thumbnails	GAN-based
TMM2023	Learning to Hallucinate Face in the Dark	CNN-based

Joint Face Fairness

Pub	Paper	Technology
ICIP2021	Identity and Attribute Preserving Thumbnail Upscaling	Prior-based
ICML2021	Fairness for image generation with uncertain sensitive attributes	CNN-based
TPAMI2022	EDFace-Celeb-1M: Benchmarking Face Hallucination With a Million-Scale Dataset	Dataset
Arxiv2022	Blind Face Restoration: Benchmark Datasets and a Baseline Model	Dataset

Joint Face 3D Face Reconstruction

Pub	Paper	Technology
FG202O	Face Denoising and 3D Reconstruction from A Single Depth Image	CNN-based
IJCB2021	3D Face Point Cloud Super-Resolution Network	CNN-based
Sensors2022	Incomplete Region Estimation and Restoration of 3D Point Cloud Human Face Datasets	CNN-based
CVPR2022	Learning to Restore 3D Face from In-the-Wild Degraded Images	Prior/GAN-based

Perfermance

Non-Blind Face Super-Resoution

Blind Face Restoration

Blind Face Super-Resoution

Joint Face Restoration

💥 Note: It can be found in the Supplementary Material.

Benchmarks

Populur Datasets

Dataset	Quantity	Introduction	Year
FFHQ	70,000	non-paired dataset for training	2018
CelebA	202,599	non-paired dataset for training	2015
CelebA-HQ	30,000	non-paired dataset for training	2020
LFW	13,233	non-paired dataset for training	2008
Multi-PIE	75,000	non-paired dataset for training	2010
Helen	2,330	non-paired dataset for testing	2012
CelebA-Test	3,000	non-paired dataset for testing	2021
CelebChild-Test	180	non-paired real-world dataset for testing	2021
CelebAdult-Test	180	non-paired real-world dataset for testing	2021
WebPhoto-Test	407	non-paired real-world dataset for testing	2021
LFW-Test	1,711	non-paired real-world dataset for testing	2021
Wider-Test	970	non-paired real-world dataset for testing	2022

💥 Note: More datasets can be found in the Paper.

Evaluation Metrics

🚩 Note: A detailed description of the evaluation metrics and how to use it can be found here.

📧 Contact

If you have any question, please email [e-mail]