Video Face Manipulation Detection Through Ensemble of CNNs
This is the official repository of Video Face Manipulation Detection Through Ensemble of CNNs, submitted to ICPR2020 and currently available on arXiv.
We participated as the ISPL team in the Kaggle Deepfake Detection Challenge. With this implementation, we reached the 41st position over 2116 teams (top 2%) on the private leaderboard.
This repository is currently under maintenance, if you are experiencing any problems, please open an issue.
Getting started
Prerequisites
- Install conda
- Create the
icpr2020
environment with environment.yml
$ conda env create -f environment.yml
$ conda activate icpr2020
Quick run
If you just want to test the pre-trained models against your own videos or images:
The whole pipeline
You need to preprocess the datasets in order to index all the samples and extract faces. Just run the script make_dataset.sh
$ ./scripts/make_dataset.sh
Please note that we use only 32 frames per video. You can easily tweak this parameter in extract_faces.py
Train
In train_all.sh you can find a comprehensive list of all the commands to train the models presented in the paper. Please refer to the comments in the script for hints on their usage.
Training a single model
If you want to train some models without lunching the script:
- for the non-siamese architectures (e.g. EfficientNetB4, EfficientNetB4Att), you can simply specify the model in train_binclass.py with the --net parameter;
- for the siamese architectures (e.g. EfficientNetB4ST, EfficientNetB4AttST), you have to:
- train the architecture as a feature extractor first, using the train_triplet.py script and being careful of specifying its name with the --net parameter without the ST suffix. For instance, for training the EfficientNetB4ST you will have to first run
python train_triplet.py --net EfficientNetB4 --otherparams
; - finetune the model using train_binclass.py, being careful this time to specify the architecture's name with the ST suffix and to insert as --init argument the path to the weights of the feature extractor trained at the previous step. You will end up running something like
python train_binclass.py --net EfficientNetB4ST --init path/to/EfficientNetB4/weights/trained/with/train_triplet/weights.pth --otherparams
- train the architecture as a feature extractor first, using the train_triplet.py script and being careful of specifying its name with the --net parameter without the ST suffix. For instance, for training the EfficientNetB4ST you will have to first run
Test
In test_all.sh you can find a comprehensive list of all the commands for testing the models presented in the paper.
Pretrained weights
We also provide pretrained weights for all the architectures presented in the paper.
Please refer to this Dropbox link.
Each directory is named $NETWORK_$DATASET
where $NETWORK
is the architecture name and $DATASET
is the training dataset.
In each directory, you can find bestval.pth
which are the best network weights according to the validation set.
Additionally, you can find notebooks for results computations in the notebook folder.
Datasets
References
Credits
Image and Sound Processing Lab - Politecnico di Milano
- Nicolò Bonettini (nicolo.bonettini@polimi.it)
- Edoardo Daniele Cannas (edoardodaniele.cannas@polimi.it)
- Sara Mandelli (sara.mandelli@polimi.it)
- Luca Bondi (luca.bondi@polimi.it)
- Paolo Bestagini (paolo.bestagini@polimi.it)