somuchtome / Faceswap

Face swap via diffusion models [Lora+IP-Adapter+Controlnet+text embedding optimization]

Face Swap via Diffusion Model

PyTorch implementation of Face Swap via Diffusion Model. Technical Report:https://arxiv.org/abs/2403.01108

Environment setup

cd Faceswap
conda env create -f environment.yaml 
conda activate swap
cd ldm
pip install -e ".[torch]"

Note

the diffusers have been modified and we define new pipe "StableDiffusionControlNetFaceInpaintPipeline" to support IP-adapter and our text embedding optimization which is different from huggingface-released.

Download Pretrained Weights

The weights required for FaceParser can be downloaded from link. The weights required for ArcFace can be downloaded from link.

mkdir checkpoints
mv arcface_checkpoint.tar checkpoints/ 
mv FaceParser.pth checkpoints/ 

Directories structure

The dataset and checkpoints should be placed in the following structures below

Faceswap
├── checkpoints
    ├── arcface_checkpoint.tar
    ├── FaceParser.pth
├── data
    └── src
        ├── 001.png
        └── ...
    └── targ
        ├── 001.png
        └── ...

Face swap

Place source and target images in data/src, and data/targ. Then run the following.

sh swap.sh

Face restoration postprocess

# git clone this repository
git clone https://github.com/sczhou/CodeFormer.git
cd CodeFormer
python basicsr/setup.py develop

Quick Inference

Download Pre-trained Models:

Download the facelib and dlib pretrained models from [Releases | Google Drive | OneDrive] to the weights/facelib folder. You can manually download the pretrained models OR download by running the following command:

python scripts/download_pretrained_models.py facelib
python scripts/download_pretrained_models.py dlib (only for dlib face detector)

Download the CodeFormer pretrained models from [Releases | Google Drive | OneDrive] to the weights/CodeFormer folder. You can manually download the pretrained models OR download by running the following command:

python scripts/download_pretrained_models.py CodeFormer

🧑🏻 Face Restoration (cropped and aligned face)

# For cropped and aligned faces (512x512)
python inference_codeformer.py -w 0.5 --has_aligned --input_path [image folder]|[image path]

Acknowledgments

This code borrows heavily from DiffFace, Diffusers, Codeformer and Lora.