Davidyao99 / image_morphing

image_morphing

git submodule update --init -> initialize submodule SPIGA

This is a final project for CS445:Computational Photography at UIUC. Given input images of faces, automatic morphing is performed from one face to another.

Morphing using Manual Key Points Selection

Morphing using Custom Facial KeypointDetection

Morphing using SPIGMA

Methodology

1. Given two images (I1 and I2), facial keypoints are automatically identified and detected.
2. Find average of corresponding points and get triangular mesh using Delaunay Triangulation
3. To create frame t, interpolate corresponding points
4. For each triangle, affine project to corresponding triangles in I1 and I2.
5. Use affine projection to map pixel values from I1 and I2 to frame.
6. Set final pixel value to be weighted average of pixel in I1 and I2.
7. Frame t is done! Repeat for frame t+1

We implemented our own facial keypoint detector using training data from kaggle. This works well for in-distribution faces, but has difficulties in scaling to all faces due to training data size limitations. We also provide the option to leverage existing SOTA in facial keypoint detection SPIGA for better in-the-wild performance.

Setup

conda env create --name morph --file=environment.yml

Notebooks

custom_vis.ipynb -> Visualize custom trained face keypoint ResNet18 model detector_vis.ipynb -> visualize SPIGA face keypoint detection model image_morph_vis.ipynb -> visualize image morph with user input keypoint selection feature keypoint_detector_train.ipynb -> Training notebook for face keypoint detection

Main

python3 engine.py $custom -> takes in directory of images and morphs them into a video sequentially. Set argument $custom as 1 or 0 to use custom or SPIGA respectively.

Training Data

https://www.kaggle.com/datasets/prashantarorat/facial-key-point-data

References

For training -> https://github.com/nalbert9/Facial-Keypoint-Detection