laimis-dev / LRNet

Landmark Recurrent Network: An efficient and robust framework for Deepfakes detection

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

LRNet

Landmark Recurrent Network: An efficient and robust framwork for Deepfakes detection

The implementation for paper: Improving the Efficiency and Robustness for Deepfakes Detection through Precise Geometric Features (CVPR2021).

[Paper Link]

Overview

LRNet is a light-weight and effective Deepfakes detection framework. It analyzes the sequence of geometric features in facial videos to capture the temporal artifacts (such as unnatural expressions) and make predictions.

It takes three steps to discriminate a forged face video:

  1. Extract the 2-D facial landmark sequence from the video. It can be conveniently realized by some open-source toolkit like Dlib, OpenFace, MediaPipe, etc.
  2. De-noise the sequence by our carefully-devised calibration module. It utilizes optical flow and Kalman filter techniques to alleviate the noise caused by different landmark detectors.
  3. Embed the landmark sequence into two feature sequences and input them into two RNN. The output of two RNN are merged to obtain the final prediction.
LRNet.png

Description

To go through the whole procedure of LRNet's forgery detection (from an input video sample to its corresponding predicted label), you can enter the ./demo/ for more details.

For easily training and evaluating the LRNet, you can enter the ./training/ for more details.

If you would like to use your own landmark detector to generate the dataset, you can enter the ./calibrator/ for more details.

Citation

If our work helps your research, welcome for the citation:

@inproceedings{sun2021improving,
  title={Improving the Efficiency and Robustness of Deepfakes Detection through Precise Geometric Features},
  author={Sun, Zekun and Han, Yujie and Hua, Zeyu and Ruan, Na and Jia, Weijia},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={3609--3618},
  year={2021}
}

TODO & Update List

  • Upload the demo.
    • Organize the codes into the sub-folder. (2022/3/16)
  • Update the calibration module.
    • Update the pyramidal LK with numpy and openCV.
    • Implement the pyramidal LK with openCV API.
    • Provide API for easier use.
    • Provide utils for calibrating the dataset (under OpenFace workflow).
  • Optimize the model.
    • Update the training codes.
    • For PyTorch version, optimize the dropout layer.
    • Greatly refactor the training and evaluation codes for PyTorch version. (2022/3/10)
    • Update the Jupyter Notebook version training/evaluation codes. (2022/3/20, LATEST)
    • Update the weights for the whole FF++ dataset. (2022/4/28, LATEST)
    • Update AUC evaluation codes. (Both .py and .ipynb) (2022/5/2, LATEST)
    • Model structure exploration
  • Gradually release the datasets
    • FF++
      • c23
      • raw
      • c40

Log

2021/5/18
  • Update the demo's model weights. Now we provide demo's weights trained on FF++ (./demo/modelweights/ff) and Deeperforensics-1.0 (./demo/modelweights/deeper). We use deeper by default, because the given examples are from this dataset. You can change to ff in classify.py L107--L113.
  • Provide faster LK tracker. We now use openCV optical api to achieve LK track. It would be faster and more stable.
  • Update GPU support. We restrict Tensorflow not to use up the memory of GPU when it is supported. Although CPU is enough for inference. The GPU memory occupation may be < 1G.
2021/11/13
  • Update the training and testing codes.
  • Release FF++(c23) landmark datasets.
  • Update the plug-and-use landmark calibrator API.
2022/3/10
  • Provide utils for calibrating the dataset. Currently it is devised for OpenFace work flow. We consider add more workflow supports in the future.
  • Greatly refactor the training and evaluation codes for PyTorch version. They can help perform more flexible training and evaluating strategies, which is convenient for further research.
2022/3/16
  • Organize the demo codes. Now the demo codes are gathered in the ./demo/ folder , making the project's home page clean and tidy.
  • Discussion: the selection of landmark detector in demo. We also try several landmark detectors and find that Dlib is relatively good solution. Although it's accuracy and stability are not the SOTA, it helps alleviate preparing burdensome dependencies (especially for deep-learning-based models). Besides, it's inference speed is fast and suitable for reproducing the LRNet's framework. You could also replace it with other advanced landmark detectors.

About

Landmark Recurrent Network: An efficient and robust framework for Deepfakes detection

License:MIT License


Languages

Language:Python 55.6%Language:Jupyter Notebook 44.4%