Pytorch implementation of GQN, Generative Query Network proposed in the paper Neural scene representation and rendering, Eslami et al., Science.
The following figures are side-by-side comparisons of prediction and ground-truth. The left one is the prediction from GQN after 400k iterations, and the right image is the ground-truth.
After cloning the repository, move into the directory you've just cloned and run following code to configure virtual environment.
# setup conda virtual environment 'torch-gqn'
conda env create --file environment.yaml
The official dataset is distributed here.
Visit the page, follow the instructions to download the dataset of your preference. It's recommended to download the dataset under data directory because there's a python script for converting serialized tfrecord files into pickle files which are used in this project.
Note: The code is tested on 'rooms_ring_camera' dataset only. For other examples like shepard_metzler_*_parts, you may wish to take a look at Implementation from iShohei220 or Implementation from Jesper Wohlert
In the project root directory, execute the following code fragment:
cd data
python convert_to_torch.py rooms_ring_camera
For me, converting 1 million data took almost three days. Also, I set the code to terminate after converting 1 million training data and 200k test data due to limited disk capacity, you should modify the code depending on your environment.
Again in the project root directory, execute the following code fragment:
Warning! Training takes a lot of time, so you may want to use tmux or other terminal multiplexers of your preference so that the training keeps running even when you close the terminal.
python train_gqn.py
You can adjust parameters associated with training (+ hyperparameters) via argument parser on the top of the train_gqn.py file.