Lifespan Age Transformation Synthesis
Project Page | Paper | Data
Roy Or-El1 ,
Soumyadip Sengupta1,
Ohad Fried2,
Eli Shechtman3,
Ira Kemelmacher-Shlizerman1
1University of Washington, 2Stanford University, 3Adobe Research
Updates
- August 19th, 2020: Added alternate download urls for pretrained models to bypass Google Drive "quota exceeded" error.
Overview
Lifespan Age Transformation Synthesis is a GAN based method designed to simulate the continuous aging process from a single input image.
This code is the official PyTorch implementation of the paper:
Lifespan Age Transformation Synthesis
Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, Ira Kemelmacher-Shlizerman
ECCV 2020
https://arxiv.org/pdf/2003.09764.pdf
Ethics & Bias statement
Intended use:
- This algorithm is designed to hallucinate the aging process and produce an approximation of a person's appearance throughout his/her/their lifespan.
- The main use cases of this method are for art and entertainment purposes (CGI effects, Camera filters, etc.). This method might also be useful for more critical applications, e.g. approximating the appearance of missing people. However, we would like to stress that as a non perfect data-driven method, results might be inaccurate and biased. The output of our method should be critically analyzed by a trained professional, and not be treated as an absolute ground truth.
- The results of this method should not be used as grounds for detention/arrest of a person or as any other form of legal evidence under any circumstances.
We have devoted considerable efforts in our algorithm design to preserve the identity of the person in the input image, and to minimize the influence of the inherent dataset biases on the results. These measures include:
- Designing the identity encoder architecture to preserve the local structures of the input image.
- Including training losses that were designed to maintain the person's identity.
- Latent Identity loss: encourages identity features that are consistent across ages.
- Cycle loss: drives the network to reproduce the original image from any aged output.
- Self-reconstruction loss: makes the network learn to reconstruct the input when the target age class is the same as the source age class.
- The FFHQ dataset contains gender imbalance within age classes. To prevent introducing these biases in the output, e.g. producing male facial features for females or vice versa, we have trained two separate models, one for males and one for females. The decision of which model to apply is left for the user. We acknowledge that this design choice restricts our algorithm from simulating the aging process of people whose gender is non-binary. Further work is required to make sure future algorithms will be able to simulate aging for the entire gender spectrum.
Despite these measures, the network might still introduce other biases that we did not consider when designing the algorithm. If you spot any bias in the results, please reach out to help future research!
Pre-Requisits
You must have a GPU with CUDA support in order to run the code.
This code requires PyTorch and torchvision to be installed, please go to PyTorch.org for installation info.
We tested our code on PyTorch 1.4.0 and torchvision 0.5.0, but the code should run on any PyTorch version above 1.0.0, and any torchvision version above 0.4.0.
The following python packages should also be installed:
- opencv-python
- visdom
- dominate
- numpy
- scipy
- pillow
- unidecode
- requests
- tqdm
If any of these packages are not installed on your computer, you can install them using the supplied requirements.txt
file:
pip install -r requirements.txt
Quick Demo
You can try running the method on your own images!!!
You can either run the demo localy or explore it in Colab
Running locally:
- Download the pre-trained models.
python download_models.py
- Create a txt file with the paths to all images you want to try (for example, see
males_image_list.txt
orfemales_image_list.txt
) - Run the model:
- Open,
./run_scripts/in_the_wild.sh
(Linux) or./run_scripts/in_the_wild.bat
(windows). - Select which model to use in the
--name
flag (males_model
orfemales_model
). - Enter the path to the txt file you created in step 2 after the
--image_path_file
flag. - Run the script.
- Open,
- The outputs can be seen in
results/males_model/test_latest/traversal/
orresults/females_model/test_latest/traversal/
(according to the selected model).
Please refer to Using your own images for guidelines on what images are good to use.
If you get a CUDA out of memory error, slightly increase the --interp_step
parameter until it fits your GPU. This parameter controls the number of interpolated frames between every 2 anchor classes. Increasing it will reduce the length of the output video.
Using your own images
For best results, use images according to the following guidelines:
- The image should contain a single face.
- Image was taken from a digital camera (phone cameras are fine). Old images from film cameras would produce low quality results.
- Pure RGB images only. No black & white, grayscale, sepia, or filtered images (e.g. Instagram filters).
- Person's head should directly face the camera. Looking sideways/downwards/upwards degrades the results.
- The person's face should not be occluded (or partially occluded) by any item.
- Both eyes should be open and visible. (Eyeglasses are ok, but might produce artifacts. No sunglasses)
Training/Testing on FFHQ-Aging
-
Download the FFHQ-Aging dataset. Go to the FFHQ-Aging dataset repo and follow the instructions to download the data.
-
Prune & organize the raw FFHQ-Aging dataset into age classes:
cd datasets
python create_dataset.py --folder <path to raw FFHQ-Aging directory> --labels_file <path to raw FFHQ-Aging labels csv file> [--train_split] [num of training images (default=69000)]
- Download pretrained models (Optional)
python download_models.py
Training:
- Open a visdom port to view loss plots and intermediate results. Run
visdom
and monitor results at http://localhost:8097. If you run the code on a remote server openhttp://hostname:8097
instead. - Open
run_scripts/train.sh
(Linux) orrun_scripts/train.bat
(windows) and set:- The GPUs you intend to use
--gpu_ids
as well as theCUDA_VISIBLE_DEVICES
environment variable.
NOTE: the scripts are currently set to use 4 GPUs - The dataset relative path
--dataroot
- The model name
--name
- The batch size
--batchSize
according to your GPU's maximum RAM capacity and the number of GPU's available.
- The GPUs you intend to use
- Train the model: Run
./run_scripts/train.sh
(Linux) or./run_scripts/train.bat
(windows)
Testing:
- Open
run_scripts/test.sh
(Linux) orrun_scripts/test.bat
(windows) and set:- The dataset relative path
--dataroot
- The model name
--name
- Which checkpoint to load the model from
--which_epoch
. This can be either an epoch number e.g.400
or the latest saved modellatest
.
- The dataset relative path
- Test the model: Run
./run_scripts/test.sh
(Linux) or./run_scripts/test.bat
(windows) - The outputs can be seen in
results/<model name>/test_<model_checkpoint>/index.html
Generate Video
- Prepare a
.txt
file with a list of image paths to generate videos for. See examples inmales_image_list.txt
andfemales_image_list.txt
- Open
run_scripts/traversal.sh
(Linux) orrun_scripts/traversal.bat
(windows) and set:- The dataset relative path
--dataroot
- The model name
--name
- Which checkpoint to load the model from
--which_epoch
. This can be either an epoch number e.g.400
or the latest saved modellatest
. - The relative path to the image list
--image_path_file
- The dataset relative path
- Run
./run_scripts/traversal.sh
(Linux) or./run_scripts/traversal.bat
(windows) - The output videos will be saved to
results/<model name>/test_<model_checkpoint>/traversal/
Generate Full Progression
This will generate an image of progressions to all anchor classes
- Prepare a
.txt
file with a list of image paths to generate videos for. See examples inmales_image_list.txt
andfemales_image_list.txt
- Open
run_scripts/deploy.sh
(Linux) orrun_scripts/deploy.bat
(windows) and set:- The dataset relative path
--dataroot
- The model name
--name
- Which checkpoint to load the model from
--which_epoch
. This can be either an epoch number e.g.400
or the latest saved modellatest
. - The relative path to the image list
--image_path_file
- The dataset relative path
- Run
./run_scripts/deploy.sh
(Linux) or./run_scripts/deploy.bat
(windows) - The output images will be saved to
results/<model name>/test_<model_checkpoint>/deploy/
Citation
If you use this code for your research, please cite our paper.
@inproceedings{orel2020lifespan,
title={Lifespan Age Transformation Synthesis},
author={Or-El, Roy
and Sengupta, Soumyadip
and Fried, Ohad
and Shechtman, Eli
and Kemelmacher-Shlizerman, Ira},
booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
year={2020}
}
Acknowledgments
This code is inspired by pix2pix-HD and style-based-gan-pytorch.