owenzlz / latent-composition

Latent Composition

Project Page | Paper

Using latent space regression to analyze and leverage compositionality in GANs.
International Conference on Learning Representations, 2021
Lucy Chai, Jonas Wulff, Phillip Isola

Prerequisites

• Linux
• Python 3
• NVIDIA GPU + CUDA CuDNN

Table of Contents:

1. Colab - run it in your browser without installing anything locally
   
2. Setup - download pretrained models and resources
3. Pretrained Models - quickstart with pretrained models
   
4. Notebooks - jupyter notebooks for interactive composition
   
5. Training - pipeline for training encoders
   

Colab

1. Interactive Masking Demo: Demonstrates using a masked encoder to investigate image priors in GANs.
2. Interactive Composition Demo: Interface to compose multiple images using masked encoder.
3. Finetune and Edit: For real images, finetune the encoder towards a specific image for better reconstruction. Further composition can be done in real time.

Setup

• Clone this repo:

git clone https://github.com/chail/latent-composition.git

• Install dependencies:
  • we provide a Conda environment.yml file listing the dependencies. You can create a Conda environment with the dependencies using:

conda env create -f environment.yml

• Download resources:
  • we provide a script for downloading associated resources and pretrained models. Fetch these by running:

bash resources/download_resources.sh

Quickstart with pretrained models

Pretrained models can be downloaded from here or here. Using the above download script will automatically download the pretrained models, as well as other resources for automatic collaging.

See the following code snippet for a basic example. An notebook format is provided in notebooks/quickstart.ipynb

from networks import networks
from utils import masking

nets = networks.define_nets('proggan', 'celebahq')
# proggan: celebahq, livingroom, church
# stylegan: ffhq, church, car, horse

with torch.no_grad():
    im = nets.seed2image(1,seed=10)
    hints, mask = masking.mask_upsample(im)
    rec = nets.invert(hints, mask=mask)

Notebooks

Important: First, setup symlinks required for notebooks: bash notebooks/setup_notebooks.sh, and add the conda environment to jupyter kernels: python -m ipykernel install --user --name latent-composition.

We provide a few interactive examples:

1. notebooks/quickstart.ipynb: basic usage example
2. notebooks/interactive-masking.ipynb: investigate GAN priors from incomplete images
3. notebooks/interactive-composition.ipynb: compose multiple images
4. notebooks/finetune-and-edit.ipynb: finetune the model on a real image, and then compose in real-time

Training

Coming soon!

TODOs

• Add experiment scripts
• Add training code
• Website: Webpage of random samples
• Move pretrained models hosting from gdrive to CSAIL, package models individually for faster download

Acknowledgements

We thank the authors of these repositories:

• Gan Seeing for GAN and visualization utilities
• StyleGAN 2 Pytorch for pytorch implementation of StyleGAN 2 and pretrained models
• Pixel2Style2Pixel for identity loss implementation
• Pytorch FID for FID implementation

Citation

If you use this code for your research, please cite our paper:

@inproceedings{chai2021latent,
  title={Using latent space regression to analyze and leverage compositionality in GANs},
  author={Chai, Lucy and Wulff, Jonas and Isola, Phillip},
  booktitle={International Conference on Learning Representations},
  year={2021}
 }