Face Image Quality Assessment

15.05.2020 SER-FIQ (CVPR2020) was added.

18.05.2020 Bias in FIQ (IJCB2020) was added.

SER-FIQ: Unsupervised Estimation of Face Image Quality Based on Stochastic Embedding Robustness

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2020

Abstract
Key Points
Results
Installation
Bias in Face Quality Assessment
Citing
Acknowledgement
License

Abstract

Face image quality is an important factor to enable high-performance face recognition systems. Face quality assessment aims at estimating the suitability of a face image for recognition. Previous works proposed supervised solutions that require artificially or human labelled quality values. However, both labelling mechanisms are error-prone as they do not rely on a clear definition of quality and may not know the best characteristics for the utilized face recognition system. Avoiding the use of inaccurate quality labels, we proposed a novel concept to measure face quality based on an arbitrary face recognition model. By determining the embedding variations generated from random subnetworks of a face model, the robustness of a sample representation and thus, its quality is estimated. The experiments are conducted in a cross-database evaluation setting on three publicly available databases. We compare our proposed solution on two face embeddings against six state-of-the-art approaches from academia and industry. The results show that our unsupervised solution outperforms all other approaches in the majority of the investigated scenarios. In contrast to previous works, the proposed solution shows a stable performance over all scenarios. Utilizing the deployed face recognition model for our face quality assessment methodology avoids the training phase completely and further outperforms all baseline approaches by a large margin. Our solution can be easily integrated into current face recognition systems and can be modified to other tasks beyond face recognition.

Key Points

Quality assessment with SER-FIQ is most effective when the quality measure is based on the deployed face recognition network, meaning that the quality estimation and the recognition should be performed on the same network. This way the quality estimation captures the same decision patterns as the face recognition system.
To get accurate quality estimations, the underlying face recognition network for SER-FIQ should be trained with dropout. This is suggested since our solution utilizes the robustness against dropout variations as a quality indicator.
The provided code is only a demonstration on how SER-FIQ can be utilized. The main contribution of SER-FIQ is the novel concept of measuring face image quality.
If the last layer contains dropout, it is sufficient to repeat the stochastic forward passes only on this layer. This significantly reduces the computation time to a time span of a face template generation.

Results

Face image quality assessment results are shown below on LFW (left) and Adience (right). SER-FIQ (same model) is based on ArcFace and shown in red. The plots show the FNMR at $\Large 10^{-3}$ FMR as recommended by the best practice guidelines of the European Border Guard Agency Frontex. For more details and results, please take a look at the paper.

Installation

We recommend Anaconda to install the required packages. This can be done by creating an virtual environment via

conda env create -f environment.yml

or by manually installing the following packages.

conda create -n serfiq python=3.6.9
conda install cudatoolkit
conda install cudnn
conda install tensorflow=1.14.0
conda install mxnet
conda install mxnet-gpu
conda install tqdm
conda install -c conda-forge opencv
conda install -c anaconda scikit-learn
conda install -c conda-forge scikit-image
conda install keras=2.2.4

After the required packages have been installed, also download the Insightface codebase at the needed git point in the repository history to a location of your choice and extract the archive if necessary.

We will refer to this location as $Insightface in the following.

The path to the Insightface repository must be passed to the InsightFace class in face_image_quality.py. To avoid any problems, absolute paths can be used. Our InsightFace class automatically imports the required dependencies from the Insightface repository.

insightface = InsightFace(insightface_path = $Insightface) # Repository-path as parameter

Please be aware to change the location in our example code according to your setup.

A pre-trained Arcface model is also required. We recommend using the "LResNet100E-IR,ArcFace@ms1m-refine-v2" model. This can be downloaded from the Insightface Model-Zoo.

Extract the downloaded model-0000.params and model-symbol.json to the following location on your computer:

$Insightface/models/

After following these steps you can activate your environment (default: conda activate serfiq) and run the example code.

The implementation for SER-FIQ based on ArcFace can be found here: Implementation.
In the Paper, this is refered to SER-FIQ (same model) based on ArcFace.

Bias in Face Quality Assessment

The best face quality assessment performance is achieved when the quality assessment solutions build on the templates of the deployed face recognition system. In our work on (Face Quality Estimation and Its Correlation to Demographic and Non-Demographic Bias in Face Recognition), we showed that this lead to a bias transfer from the face recognition system to the quality assessment solution. On all investigated quality assessment approaches, we observed performance differences based on on demographics and non-demographics of the face images.

Citing

If you use this code, please cite the following papers.

@inproceedings{DBLP:conf/cvpr/TerhorstKDKK20,
  author    = {Philipp Terh{\"{o}}rst and
               Jan Niklas Kolf and
               Naser Damer and
               Florian Kirchbuchner and
               Arjan Kuijper},
  title     = {{SER-FIQ:} Unsupervised Estimation of Face Image Quality Based on
               Stochastic Embedding Robustness},
  booktitle = {2020 {IEEE/CVF} Conference on Computer Vision and Pattern Recognition,
               {CVPR} 2020, Seattle, WA, USA, June 13-19, 2020},
  pages     = {5650--5659},
  publisher = {{IEEE}},
  year      = {2020},
  url       = {https://doi.org/10.1109/CVPR42600.2020.00569},
  doi       = {10.1109/CVPR42600.2020.00569},
  timestamp = {Tue, 11 Aug 2020 16:59:49 +0200},
  biburl    = {https://dblp.org/rec/conf/cvpr/TerhorstKDKK20.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@article{DBLP:journals/corr/abs-2004-01019,
  author    = {Philipp Terh{\"{o}}rst and
               Jan Niklas Kolf and
               Naser Damer and
               Florian Kirchbuchner and
               Arjan Kuijper},
  title     = {Face Quality Estimation and Its Correlation to Demographic and Non-Demographic
               Bias in Face Recognition},
  journal   = {CoRR},
  volume    = {abs/2004.01019},
  year      = {2020},
  url       = {https://arxiv.org/abs/2004.01019},
  archivePrefix = {arXiv},
  eprint    = {2004.01019},
  timestamp = {Wed, 08 Apr 2020 17:08:25 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2004-01019.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

If you make use of our SER-FIQ implementation based on ArcFace, please additionally cite the original .

Acknowledgement

This research work has been funded by the German Federal Ministry of Education and Research and the Hessen State Ministry for Higher Education, Research and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE.

License

This project is licensed under the terms of the Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license. Copyright (c) 2020 Fraunhofer Institute for Computer Graphics Research IGD Darmstadt

ZhengHui-Z / FaceImageQuality