pacifinapacific / Super-SloMo

PyTorch implementation of Super SloMo by Jiang et al.

Super-SloMo

PyTorch implementation of "Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation" by Jiang H., Sun D., Jampani V., Yang M., Learned-Miller E. and Kautz J. [Project] [Paper]

Results

Results on UCF101 dataset using the evaluation script provided by paper's author. The get_results_bug_fixed.sh script was used. It uses motions masks when calculating PSNR, SSIM and IE.

Method	PSNR	SSIM	IE
DVF	29.37	0.861	16.37
SepConv - L_1	30.18	0.875	15.54
SepConv - L_F	30.03	0.869	15.78
SuperSloMo_Adobe240fps	29.80	0.870	15.68
pretrained mine	29.77	0.874	15.58
SuperSloMo	30.22	0.880	15.18

Prerequisites

This codebase was developed and tested with pytorch 0.4.1 and CUDA 9.2. Install:

• PyTorch

For GPU, run

conda install pytorch=0.4.1 cuda92 -c pytorch
pip install torchvision

For CPU, run

conda install pytorch-cpu=0.4.1 -c pytorch
pip install torchvision

• TensorboardX for training visualization
• tensorflow for tensorboard
• matplotlib for training graph in notebook.
• tqdm for progress bar in video_to_slomo.py
• numpy

Training

Preparing training data

In order to train the model using the provided code, the data needs to be formatted in a certain manner. The create_dataset.py script uses ffmpeg to extract frames from videos.

Adobe240fps

For adobe240fps, download the dataset, unzip it and then run the following command

python data\create_dataset.py --ffmpeg_dir path\to\folder\containing\ffmpeg --videos_folder path\to\adobe240fps\videoFolder --dataset_folder path\to\dataset --dataset adobe240fps

Custom

For custom dataset, run the following command

python data\create_dataset.py --ffmpeg_dir path\to\folder\containing\ffmpeg --videos_folder path\to\adobe240fps\videoFolder --dataset_folder path\to\dataset

The default train-test split is 90-10. You can change that using command line argument --train_test_split.

Run the following commmand for help / more info

python data\create_dataset.py --h

Training

In the train.ipynb, set the parameters (dataset path, checkpoint directory, etc.) and run all the cells.

or to train from terminal, run:

python train.py --dataset_root path\to\dataset --checkpoint_dir path\to\save\checkpoints

Run the following commmand for help / more options like continue from checkpoint, progress frequency etc.

python train.py --h

Tensorboard

To get visualization of the training, you can run tensorboard from the project directory using the command:

tensorboard --logdir log --port 6007

and then go to https://localhost:6007.

Evaluation

Pretrained model

You can download the pretrained model trained on adobe240fps dataset here.

Video Converter

You can convert any video to a slomo or high fps video (or both) using video_to_slomo.py. Use the command

python video_to_slomo.py --ffmpeg path\to\folder\containing\ffmpeg --video path\to\video.mp4 --sf N --checkpoint path\to\checkpoint.ckpt --fps M --output path\to\output.mkv

If you want to convert a video from 30fps to 90fps set fps to 90 and sf to 3 (to get 3x frames than the original video).

Run the following commmand for help / more info

python video_to_slomo.py --h

You can also use eval.py if you do not want to use ffmpeg. You will instead need to install opencv-python using pip for video IO. A sample usage would be:

python eval.py data/input.mp4 --checkpoint=data/SuperSloMo.ckpt --output=data/output.mp4 --scale=4

Use python eval.py --help for more details

More info TBA

References:

Parts of the code is based on TheFairBear/Super-SlowMo