DeepLandscape: Adversarial Modeling of Landscape Videos
Implementation of DeepLandscape: Adversarial Modeling of Landscape Video in PyTorch
Project Page | Video Explanation | Paper | Teaser [1min]
Official repository for the paper E. Logacheva, R. Suvorov, O. Khomenko, A. Mashikhin, and V. Lempitsky. "DeepLandscape: Adversarial Modeling of Landscape Videos" In 2020 European Conference on Computer Vision (ECCV).
GIF sample
Requirements
pip3 install -r requirements.txt
Usage
Download everything from here and put in the results directory.
Homographies
Use homographies/manual_homographies to reproduce the paper; use homographies/manual_homographies_x2.5 if you want the speed to match the speed of real videos in test data; use homographies/selected_homographies to get the best visual results.
Animate Images
Move images you like to animate to results/test_images. Then run
PYTHONPATH=`pwd`:$PYTHONPATH runfiles/encode_and_animate_test_all256_with_style.sh <homography_dir>
Results will be saved in results/encode_and_animate_results.
Generate Synthetic Videos
To use the 256x256 generator run
./generate.py config/train/256.yaml --homography_dir <homography_dir>
Results will be saved in results/generated
Calculate Metrics
TBD
Super-Resoultion
Train
Train the Generator
You should prepare an lmdb dataset:
./prepare_data.py <data type (images or videos)> <input data path> --out <output lmdb directory> --n_worker <number of workers>
To train the 256x256 generator:
./train.py configs/train/256.yaml --restart -i <path to image data> -v <path to video data>
Train the Encoder
You should prepare a dataset:
PYTHONPATH=`pwd`:$PYTHONPATH runfiles/gen_encoder_train_data_256.sh
To train the 256x256 encoder:
PYTHONPATH=`pwd`:$PYTHONPATH runfiles/train_encoder_256.sh
Download Test Data
vid_dl/main.py <output directory>
Acknowledgment
This repository is based on Kim Seonghyeon's Implementation A Style-Based Generator Architecture for Generative Adversarial Networks in PyTorch
The Superresolution part is based on https://github.com/xinntao/BasicSR
Mean optical flow calculation is taken from https://github.com/avinashpaliwal/Super-SloMo
Segmentation is taken form https://github.com/CSAILVision/semantic-segmentation-pytorch
Metrics
- LPIPS metric https://github.com/richzhang/PerceptualSimilarity
- SSIM https://github.com/Po-Hsun-Su/pytorch-ssim
- FID https://github.com/mseitzer/pytorch-fid
Citation
If you found our work useful, please don't forget to cite
@inproceedings{Logacheva_2020_ECCV,
author = {Logacheva, Elizaveta and
Suvorov, Roman and
Khomenko, Oleg and
Mashikhin, Anton and
Lempitsky, Victor
},
title = {DeepLandscape: Adversarial Modeling of Landscape Videos},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
month = {August},
year = {2020},
}