This repository contains the code for the Visual Bidirectional Kernelized Network for Visual Question Answering, presented at the VQA Challenge at CVPR'16. With this module, you can replicate our experiments and easily deploy new models. VIBIKNet is built upon the Keras (version 1.2) framework and tested for the Theano backend.
VIBIKNet requires the following libraries:
- Our version of Keras v1.0.4 or newer
- Multimodal Keras Wrapper v0.1 or newer
- VQA challenge evaluation scripts
Additionally, if you want to run the tutorials and the visualization module, you'll need:
If you want to extract KCNN features you will need (see the following README for more info):
- Matlab 2014a or newer or Octave
- Caffe. Download into ~/code/caffe folder or change path in extractCNNfeatures.m
- EdgeBoxes object detection. Download into /repository_root/edges folder.
- Piotr Dollar toolbox. Download into /repository_root/piotr_toolbox folder.
- Inria's Yael library for Matlab. Download into /repository_root/yael folder and compile for you system. See yael's release notes for more information.
If you want to use pretrained word embeddings, you can either train them by yourself using Glove or Word2Vec, or download pretrained word embeddings (recommended):
- For extracting KCNN features, see the following README.
- For training a new model, follow the train README.
- For visualizing the results, follow the visualize_results notebook.
See CVPR poster here.
These answers have been automatically generated by VIBIKNet:
If you use this project, please, cite the following publication:
Bolaños, M., Peris, Á., Casacuberta, F., & Radeva, P.
VIBIKNet: Visual Bidirectional Kernelized Network for Visual Question Answering
Iberian Conference on Pattern Recognition and Image Analysis, IbPRIA '17 (In press).
Liu Z.
Kernelized Deep Convolutional Neural Network for Describing Complex Images.
arXiv preprint arXiv:1509.04581. 2015 Sep 15.
Peris Á, Bolaños M, Radeva P, Casacuberta F.
Video Description using Bidirectional Recurrent Neural Networks.
arXiv preprint arXiv:1604.03390. 2016 Apr 12.
Malinowski M, Rohrbach M, Fritz M.
Ask your neurons: A neural-based approach to answering questions about images.
In Proceedings of the IEEE International Conference on Computer Vision 2015 (pp. 1-9).
Joint collaboration between the Computer Vision at the University of Barcelona (CVUB) group at Universitat de Barcelona-CVC and the PRHLT Research Center at Universitat Politècnica de València.
Marc Bolaños (web page): marc.bolanos@ub.edu
Álvaro Peris: lvapeab@prhlt.upv.es