360SR
A PyTorch implementation of Super-resolution of Omnidirectional Images Using Adversarial Learning
Abstract
An omnidirectional image (ODI) enables viewers to look in every direction from a fixed point through a head-mounted display providing an immersive experience compared to that of a standard image. Designing immersive virtual reality systems with ODIs is challenging as they require high resolution content. In this paper, we study super-resolution for ODIs and propose an improved generative adversarial network based model which is optimized to handle the artifacts obtained in the spherical observational space. Specifically, we propose to use a fast PatchGAN discriminator, as it needs fewer parameters and improves the super-resolution at a fine scale. We also explore the generative models with adversarial learning by introducing a spherical-content specific loss function, called 360-SS. To train and test the performance of our proposed model we prepare a dataset of 4500 ODIs. Our results demonstrate the efficacy of the proposed method and identify new challenges in ODI super-resolution for future investigations.
Requirements
- Anaconda
- PyTorch
conda install pytorch torchvision -c pytorch
- tqdm
conda install tqdm
- opencv
conda install opencv
Datasets
The train, val, and test datasets are sampled from SUN360. The dataset has 4500 ODIs. We used the equirectangular projection format for training and evaluations. 3500 ODIs were randomly selected from the total set for training purposes, and 500 ODIs were used for validation and test sets.
Download the datasets and then extract it into data
directory.
Testing pre-trained models
You need to download pre-trained models with this link and extract in into models
directory.
Usage
Train
python train.py
optional arguments:
--crop_size training images crop size [default value is 88]
--upscale_factor super resolution upscale factor [default value is 8](choices:[2, 4, 8])
--num_epochs train epoch number [default value is 100]
Test
python test.py
optional arguments:
--upscale_factor super resolution upscale factor [default value is 8](choices:[2, 4, 8])
--test_mode [default is 'GPU'] and choices:['GPU', 'CPU']
Results
Citation
| Paper accepted in IEEE 21st International Workshop on Multimedia Signal Processing (MMSP 2019) |
Please cite our paper in your publications if it helps your research:
@inproceedings{sr360,
author={C. {Ozcinar} and A. {Rana} and A. {Smolic}},
booktitle={IEEE 21st International Workshop on Multimedia Signal Processing (MMSP 2019)},
title={Super-resolution of Omnidirectional Images Using Adversarial Learning},
year={2019},
month={Sep}
}
Authors
| Cagri Ozcinar | Aakanksha Rana | Aljosa Smolic |
Acknowledgement
This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under the Grant Number 15/RP/2776. We gratefully acknowledge the support of NVIDIA Corporation for the donation of GPUs used in this work.
Contact
If you have any question, send an e-mail at ozcinarc@scss.tcd.ie or ranaa@scss.tcd.ie