fake-face-detection
some collected paper and personal notes relevant to Fake Face Detetection
Challenge
Study
- [arXiv 2019] Deep Learning for Deepfakes Creation and Detection
- [ACM SIGSAC 2019] Poster: Towards Robust Open-World Detection of Deepfakes
- [arXiv 2020] DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection
- [arXiv 2019] Zooming into Face Forensics: A Pixel-level Analysis
- [arXiv 2020] DeepFake Detection: Current Challenges and Next Steps
- [arXiv 2020] DeepFakes Evolution: Analysis of Facial Regionsand Fake Detection Performance
- [arXiv 2020] The Creation and Detection of Deepfakes: A Survey
- [arXiv 2020] Preliminary Forensics Analysis of DeepFake Images
I. Dataset
-
[Google] DeepFakeDetection Dataset
- homepage
- over 363 original sequences from 28 paid actors in 16 different scenes
- over 3000 manipulated videos using Deep-Fakes.
- homepage
-
DeepFake Forensics (Celeb-DF) Dataset
- paper: [arXiv 2019] Celeb-DF: A New Dataset for DeepFake Forensics
- real and DeepFake synthesized videos having similar visual quality on par with those circulated online
- 408 original videos collected from YouTube with subjects of different ages, ethic groups and genders, and 795 DeepFake videos synthesized from these real videos.
- paper: [arXiv 2019] Celeb-DF: A New Dataset for DeepFake Forensics
-
[Facebook] Deepfake Detection Challenge (DFDC) Dataset
-
paper : [arXiv 2019] The Deepfake Detection Challenge (DFDC) Preview Dataset
-
- paper: [arXiv 2020] DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection
- represents the largest face forgery detection dataset by far, with 60;000 videos constituted by a total of 17.6 million frames, including 50,000 original collected videos and 10,000 manipulated videos
- fake videos are generated by a newly proposed end-to-end face swapping framework
- code
- paper: [arXiv 2020] DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection
-
- paper: [Expert Systems With Applications 2019] Face image manipulation detection based on a convolutional neural network
- 8,950 facial images with unconstrained conditions such as pose, background cluttered, illumination change
- 1,500 images labeled as “fake” and 7,450 images labeled as “normal”.
- paper: [Expert Systems With Applications 2019] Face image manipulation detection based on a convolutional neural network
-
- paper: [CVPRW 2017] Two-Stream Neural Networks for Tampered Face Detection
- generated by using one iOS application called SwapMe and an open source face swap application called FaceSwap
- contains 705 fake faces and 1,400 normal faces
- paper: [CVPRW 2017] Two-Stream Neural Networks for Tampered Face Detection
-
Deep Fakes Dataset
- [to be released]
- paper: [arXiv 2019] FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals
- more ''in the wild" portrait videos
- totaling up to 142 videos, 32 minutes, and 30 GBs
-
Fake Faces in the Wild (FFW) Dataset
- paper: [BIOSIG 2018] Fake Face Detection Methods: Can They Be Generalized?
- more than 53,000 images (from 150 videos)
- paper: [BIOSIG 2018] Fake Face Detection Methods: Can They Be Generalized?
-
Swapped Face Detection Dataset
- [to be released]
- paper: [arXiv 2019] Swapped Face Detection using Deep Learning and Subjective Assessment
- A public dataset comprising 86 celebrities using 420,053 images.
- This dataset is created using still images, different from other datasets created using video frames that may contain highly correlated images.
II. Current Work
(1) Special Artifact-Based
- [CVPRW 2019] Protecting World Leaders Against Deep Fakes
- note;
- capture the distinct facial expression and movements of a specific person use Action Unit (AU)
- [CVPRW 2019] Exposing DeepFake Videos By Detecting FaceWarping Artifacts
- [WIFS 2018] In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking
- [ICASSP 2019] EXPOSING DEEP FAKES USING INCONSISTENT HEAD POSES
- [arXiv 2019] FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals
- note;
- biological signals hidden in portrait videos can be used as an implicit descriptor of authenticity, because they are neither spatially nor temporally preserved in fake content.
- [WACVW 2019] Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations
- [ICCVW 2019] Deepfake Video Detection through Optical Flow Based CNN
- we propose the adoption of optical flow fields to exploit possible inter-frame dissimilarities.
- [IMVOP 2018] Detection of Deepfake Video Manipulation
- To contribute to a solution, photo response non uniformity (PRNU) analysis is tested for its effectiveness at detecting Deepfake video manipulation
- [arXiv 2019] Face X-ray for More General Face Forgery Detection
- note
- We observe that most existing face manipulation methods share a common step: blending the altered face into an existing background image.
- The face X-ray of an input face image is a greyscale image that reveals whether the input image can be decomposed into the blending of two images from different sources.
(2) CNN-Based
- [ICCV 2019] FaceForensics++: Learning to Detect Manipulated Facial Images
- [ISITC 2018] Forensics Face Detection From GANs Using Convolutional Neural Network
- note;
- VGGFace + 2-way FN
- [ICASSP 2019] Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos
- [arXiv 2019] Swapped Face Detection using Deep Learning and Subjective Assessment
- ResNet18 pretrained on ImageNet
- [arXiv 2020] Video Face Manipulation Detection Through Ensemble of CNNs
- code
- ensemble of CNNs & attention layers & siamese training
- DFDC challenge performance: the final solution proposed by our team was an ensemble of the 4 proposed models, which led us to top3% on the leaderboard computed against the public test set.
- [arXiv 2020] Fake Face Detection via Adaptive Residuals Extraction Network
- Novel residual extractor for residual feature extraction
(3) CNN+RNN
- [AVSS 2018] Deepfake Video Detection Using Recurrent Neural Networks
- note;
- CNN (InceptionV3) + LSTM
- [CVPR 2019] Recurrent Convolutional Strategies for Face Manipulation Detection in Videos
- note;
- CNN (DenseNet) + bidirectional RNN
- [arxiv 2020] Deepfakes Detection with Automatic Face Weighting
(4) Two Stream
- [CVPRW 2017] Two-Stream Neural Networks for Tampered Face Detection
- note;
- Face Classification stream(GoogLeNet) + Patch Triplet stream(Steganalysis feature)
- [TIFS 2019] Attention-Based Two-Stream Convolutional Networks for Face Spoofing Detection
- note;
- RGB stream(contain texture details) + MSR stream(illumination invariant) & Attention-based fusion
- [arXiv 2019] Complement Face Forensic Detection and Localization with Facial Landmarks
- face landmark + RGB
(5) Auto-encoder
- [arXiv 2018] ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection
- note;
- input image -> [Encoder] -> Forensic Embedding -> [Decoder] -> reconstructed image
- [BTAS 2019] Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos
- [arXiv 2019] Towards Generalizable Forgery Detection with Locality-aware AutoEncoder
- note
- To bridge generalization gap, in this paper we propose Locality-aware AutoEcoder (LAE), which combines fine-grained representation learning and enforcing locality in a unified frame-work.
- A key characteristic of LAE is the augmented local interpretability, which could be regularized using extra pixel wise forgery masks, in order to learn intrinsic and meaningful forgery representations.
(6) Frequency Domain
- [arXiv 2019] Unmasking DeepFakes with simple Features
- [arXiv 2020] Manipulated Face Detector: Joint Spatial and Frequency Domain Attention Network
(7) General image manipulation
- [CVPR 2019] ManTraNet: Manipulation Tracing Network For Detection And Localization of Image Forgeries With Anomalous Features
- [arXiv 2019] Hybrid LSTM and Encoder-Decoder Architecture for Detection of Image Forgeries
- note
- This paper proposes a high-confidence manipulation localization architecture which utilizes resampling features, Long-Short Term Memory (LSTM) cells, and encoder-decoder network to segment out manipulated regions from non-manipulated ones
- [CVPR 2018] Learning Rich Features for Image Manipulation Detection
- [arXiv 2019] Full-Image Full-Resolution End-to-End-Trainable CNN Framework for Image Forgery Detection
(8) Novel Network or Module
- [WIFS 2018] MesoNet: a Compact Facial Video Forgery Detection Network
- [Expert Systems With Applications 2019] Face image manipulation detection based on a convolutional neural network
- note;
- a customized convolutional neural network model for Manipulated Face (MANFA) & A hybrid framework (HF-MANFA) that uses Adaptive Boosting (AdaBoost) and eXtreme Gradient Boosting (XGBoost) to deal with the imbalanced dataset challenge
- [arXiv 2019] On the Detection of Digital Face Manipulation
- note
- proposed a novel attention-based layer to improve classification performance and produce an attention map indicating the manipulated facial regions.
- [arXiv 2019] Exploiting Human Social Cognition for the Detection of Fake and Fraudulent Faces via Memory Networks
- note
- We propose a Hierarchical Memory Network (HMN) architecture, which is able to successfully detect faked faces by utilizing knowledge stored in neural memories as well as visual cues to reason about the perceived face and anticipate its future semantic embeddings.
(9) GAN-fake face detection
- [ICLR 2020 submission] Real or Fake: An Empirical Study and Improved Model for Fake Face Detection
- propose to introduce “Gram Block” into the CNN architecture and propose a novel architecture coined as Gram-Net as shown. The “Gram Block” captures the global image texture feature by calculating the Gram matrix in different semantic level
- [WIFS 2019] AutoGAN : Detecting and Simulating Artifacts in GAN Fake Images
- AutoGAN: which can simulate the artifacts produced by the common pipeline shared by several popular GAN models
- [CVPR 2020] Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions
- common up-sampling methods,i.e. known as up-convolution or transposed convolution, are causing the inability of such models to reproduce spectral distributions of natural training data correctly
- [CVPR 2020] CNN-generated images are surprisingly easy to spot... for now
- with careful pre- and post-processing and data augmentation, a standard image classifier trained on only one specific CNN generator is able to generalize surprisingly well to unseen architectures, datasets, and training methods
- [arXiv 2020] DeepFake Detection by Analyzing Convolutional Traces
(10) Domain Adaption
- [arXiv 2020] One-Shot Domain Adaptation For Face Generation
(11) Metric Learning
- [arXiv 2020] Detecting Deepfakes with Metric Learning