shatu / n2nmn

Learning to Reason: End-to-End Module Networks for Visual Question Answering

Learning to Reason: End-to-End Module Networks for Visual Question Answering

This repository contains the code for the following paper:

• R. Hu, J. Andreas, M. Rohrbach, T. Darrell, K. Saenko, Learning to Reason: End-to-End Module Networks for Visual Question Answering. in arXiv preprint arXiv:1704.05526, 2017. (PDF)

@article{hu2017learning,
  title={Learning to Reason: End-to-End Module Networks for Visual Question Answering},
  author={Hu, Ronghang and Andreas, Jacob and Rohrbach, Marcus and Darrell, Trevor and Saenko, Kate},
  journal={arXiv preprint arXiv:1704.05526},
  year={2017}
}

Project Page: http://ronghanghu.com/n2nmn

Installation

1. Install Python 3 (Anaconda recommended: https://www.continuum.io/downloads).
2. Install TensorFlow v1.0.0 (Note: newer or older versions of TensorFlow may fail to work due to incompatibility with TensorFlow Fold):
   pip install tensorflow-gpu==1.0.0
3. Install TensorFlow Fold (which is needed to run dynamic graph):
   pip install https://storage.googleapis.com/tensorflow_fold/tensorflow_fold-0.0.1-py3-none-linux_x86_64.whl
4. Download this repository or clone with Git, and then enter the root directory of the repository:
   git clone https://github.com/ronghanghu/n2nmn.git && cd n2nmn

Train and evaluate on the CLEVR dataset

Download and preprocess the data

1. Download the CLEVR dataset from http://cs.stanford.edu/people/jcjohns/clevr/, and symbol link it to exp_clevr/clevr-dataset. After this step, the file structure should look like

exp_clevr/clevr-dataset/
  images/
    train/
      CLEVR_train_000000.png
      ...
    val/
    test/
  questions/
    CLEVR_train_questions.json
    CLEVR_val_questions.json
    CLEVR_test_questions.json
  ...

2. Extract visual features from the images and store them on the disk. In our experiments, we keep the original 480 x 320 image size in CLEVR, and use the pool5 layer output of shape the (1, 10, 15, 512) from VGG-16 network (feature stored as numpy array in HxWxC format). Then, construct the "expert layout" from ground-truth functional programs, and build image collections (imdb) for clevr. These procedures can be down as follows.

./exp_clevr/tfmodel/vgg_net/download_vgg_net.sh  # VGG-16 converted to TF

cd ./exp_clevr/data/
python extract_visual_features_vgg_pool5.py  # feature extraction
python get_ground_truth_layout.py  # construct expert policy
python build_clevr_imdb.py  # build image collections
cd ../../

The saved features will take up approximately 29GB disk space (for all images in CLEVR train, val and test).

Training

1. Train with ground-truth layout (cloning expert + policy search after cloning)

   • Step a (cloning expert):
     python exp_clevr/train_clevr_gt_layout.py
   • Step b (policy search after cloning):
     python exp_clevr/train_clevr_rl_gt_layout.py

2. Train without ground-truth layout (policy search from scratch)
   python exp_clevr/train_clevr_scratch.py

Note: by default, the above scripts use GPU 0. To train on a different GPU, set the --gpu_id flag. During training, the script will write TensorBoard events to exp_clevr/tb/ and save the snapshots under exp_clevr/tfmodel/.

Test

1. Evaluate clevr_gt_layout (cloning expert):
   python exp_clevr/eval_clevr.py --exp_name clevr_gt_layout --snapshot_name 00600000 --test_split val

2. Evaluate clevr_rl_gt_layout (policy search after cloning):
   python exp_clevr/eval_clevr.py --exp_name clevr_rl_gt_layout --snapshot_name 00300000 --test_split val

3. Evaluate clevr_scratch (policy search from scratch):
   python exp_clevr/eval_clevr.py --exp_name train_clevr_scratch --snapshot_name 00100000 --test_split val

Note:

• The above evaluation scripts will print out the accuracy (only for val split) and also save it under exp_clevr/results/. It will also save a prediction output file under exp_clevr/eval_outputs/.
• By default, the above scripts use GPU 0, and evaluate on the validation split of CLEVR. To evaluate on a different GPU, set the --gpu_id flag.
• To evaluate on the test split, use --test_split tst instead. As there is no ground-truth answers for test split in the downloaded CLEVR data, the evaluation script above will print out zero accuracy on the test split. You may email the prediction outputs in exp_clevr/eval_outputs/ to the CLEVR dataset authors for the test split accuracy.

Train and evaluate on the VQA dataset

Download and preprocess the data

1. Download the VQA dataset annotations from http://www.visualqa.org/download.html, and symbol link it to exp_vqa/vqa-dataset. After this step, the file structure should look like

exp_vqa/vqa-dataset/
  Questions/
    OpenEnded_mscoco_train2014_questions.json
    OpenEnded_mscoco_val2014_questions.json
    OpenEnded_mscoco_test-dev2015_questions.json
    OpenEnded_mscoco_test2015_questions.json
  Annotations/
    mscoco_train2014_annotations.json
    mscoco_val2014_annotations.json

2. Download the COCO images from http://mscoco.org/, extract features from the images, and store them under exp_vqa/data/resnet_res5c/. In our experiments, we resize all the COCO images to 448 x 448, and use the res5c layer output of shape (1, 14, 14, 2048) from the ResNet-152 network pretrained on ImageNET classification (feature stored as numpy array in HxWxC format).

The saved features will take up approximately 307GB disk space (for all images in COCO train2014, val2014 and test2015). After feature extraction, the file structure for the features should look like

exp_vqa/data/resnet_res5c/
  train2014/
    COCO_train2014_000000000009.npy
    ...
  val2014/
    COCO_val2014_000000000042.npy
    ...
  test2015/
    COCO_test2015_000000000001.npy
    ...

where each of the *.npy file contains COCO image feature extracted from the res5c layer of the ResNet-152 network, which is a numpy array of shape (1, 14, 14, 2048) and float32 type, stored in HxWxC format.

3. Build image collections (imdb) for VQA:

cd ./exp_vqa/data/
python build_vqa_imdb.py
cd ../../

Note: this repository already contains the parsing results from Stanford Parser for the VQA questions under exp_vqa/data/parse/new_parse, with the converted ground-truth (expert) layouts under exp_vqa/data/gt_layout_*_new_parse.npy.

Training

1. Train with ground-truth layout (cloning expert)
   python exp_vqa/train_vqa_gt_layout.py

Note: by default, the above scripts use GPU 0, and train on the union of train2014 and val2014 splits. To train on a different GPU, set the --gpu_id flag. During training, the script will write TensorBoard events to exp_vqa/tb/ and save the snapshots under exp_vqa/tfmodel/.

Test

1. Evaluate on test-dev2015:
   python exp_vqa/eval_vqa.py --exp_name vqa_gt_layout --snapshot_name 00040000 --test_split test-dev2015

2. Evaluate on test2015:
   python exp_vqa/eval_vqa.py --exp_name vqa_gt_layout --snapshot_name 00040000 --test_split test2015

Note: the above evaluation scripts will not print out the accuracy, but will write the prediction outputs to exp_vqa/eval_outputs/, which can be uploaded to the evaluation sever (http://www.visualqa.org/roe.html) for evaluation.

Train and evaluate on the SHAPES dataset

A copy of the SHAPES dataset is contained in this repository under exp_shapes/shapes_dataset. The ground-truth module layouts (expert layouts) we use in our experiments are also provided under exp_shapes/data/*_symbols.json. The script to obtain the expert layouts from the annotations is in exp_shapes/data/get_ground_truth_layout.ipynb.

Training

1. Train with ground-truth layout (behavioral cloning from expert):
   python exp_shapes/train_shapes_gt_layout.py

2. Train without ground-truth layout (policy search from scratch):
   python exp_shapes/train_shapes_scratch.py

Note: by default, the above scripts use GPU 0. To train on a different GPU, set the --gpu_id flag. During training, the script will write TensorBoard events to exp_shapes/tb/ and save the snapshots under exp_shapes/tfmodel/.

Test

1. Evaluate shapes_gt_layout (behavioral cloning from expert):
   python exp_shapes/eval_shapes.py --exp_name shapes_gt_layout --snapshot_name 00040000 --test_split test

2. Evaluate shapes_scratch (policy search from scratch):
   python exp_shapes/eval_shapes.py --exp_name shapes_scratch --snapshot_name 00400000 --test_split test

Note: the above evaluation scripts will print out the accuracy and also save it under exp_shapes/results/. By default, the above scripts use GPU 0, and evaluate on the test split of SHAPES. To evaluate on a different GPU, set the --gpu_id flag. To evaluate on the validation split, use --test_split val instead.