Deep Image Reconstruction

Data and demo codes for Shen, Horikawa, Majima, & Kamitani (2017). Deep image reconstruction from human brain activity. bioRxiv.

Requirements

Python 2.7
icnn
Numpy
Scipy
Pillow (PIL)
Caffe with up-convolutional layer
- https://github.com/dosovits/caffe-fr-chairs (Branch: deepsim)
- Both CPU and GPU installation are OK

Usage

Preparation

Download data files from figshare (see data/README.md).
Download Caffe networks (see net/README.md).

CNN feature decoding from brain activity

You can skip the feature decoding from brain activity since we provide the decoded CNN features used in the original paper (see data/README.md).

We used the same methodology in our previous study for the feature decoding (Horikawa & Kamitani, 2017, Generic decoding of seen and imagined objects using hierarchical visual features, Nat Commun.). Demo programs for Matlab and Python are available at https://github.com/KamitaniLab/GenericObjectDecoding.

Image reconstruction from decoded CNN features

We provide seven scripts that reproduce main figures in the original paper.

1_reconstruct_natural_image.py
- Reconstructing natural images from the CNN features decoded from the brain with deep generator network (DGN); reproducing results in Figure 2(a)
2_reconstruct_natural_image_without_DGN.py
- Reconstructing natural images from CNN features decoded from the brain without deep generator network (DGN); reproducing results in Figure 2(b)
3_reconstruct_natural_image_different_combinations_of_CNN_layers.py
- Reconstructing natural images from CNN features decoded from the brain with different combinations of CNN layers; reproducing results in Figure 2(d)
4_reconstruct_shape_image.py
- Reconstructing colored artificial shapes from CNN features decoded from the brain; reproducing results in Figure 3(a)
5_reconstruct_shape_image_different_ROI.py
- Reconstructing colored artificial shapes from CNN features decoded from multiple visual areas in the brain; reproducing results in Figure 3(c)
6_reconstruct_alphabet_image.py
- Reconstructing alphabetical letters shapes from CNN features decoded from the brain; reproducing results in Figure 3(e)
7_reconstruct_imagined_image.py
- Reconstructing imagined image from CNN features decoded from the brain; reproducing results in Figure 4

Data

fMRI data: Deep Image Reconstruction@OpenNeuro
Decoded CNN features: Deep Image Reconstruction@figshare

Notes

Enable back-propagation in the CNNs

In the demo code, we use pre-trained VGG19 model (http://www.robots.ox.ac.uk/~vgg/software/very_deep/caffe/VGG_ILSVRC_19_layers.caffemodel) and pre-trained deep generator network (DGN; https://lmb.informatik.uni-freiburg.de/resources/binaries/arxiv2016_alexnet_inversion_with_gans/release_deepsim_v0.zip; Dosovitskiy & Brox, 2016, Generating Images with Perceptual Similarity Metrics based on Deep Networks. arXiv.). In order to make back-propagation work, one line should be added to the prototxt files (the file describes the configuration of the CNN model):

force_backward: true.

CNN features before or after ReLU

In our study, we define CNN features of conv layers or fc layers as the output immediately after the convolutional or fully-connected computation, before applying the Rectified-Linear-Unit (ReLU). However, as default setting, ReLU operation is an in-place computation, which will override the CNN features we need. In order to use the CNN features before the ReLU operation, we need to modify the prototxt file. Taking the VGG19 prototxt file as an example:

In the original prototxt file, ReLU is in-place computation:

layers {
  bottom: "conv1_1"
  top: "conv1_1"
  name: "relu1_1"
  type: RELU
}

Now, we modify it as:

layers {
  bottom: "conv1_1"
  top: "relu1_1"
  name: "relu1_1"
  type: RELU
}

legutierr / DeepImageReconstruction