This is the source code of our IEEE TCYB 2020 paper "Visual-textual Hybrid Sequence Matching for Joint Reasoning". Please cite the following paper if you use the resources.
Xin Huang, Yuxin Peng and Zhang Wen, "Visual-textual Hybrid Sequence Matching for Joint Reasoning", IEEE Transactions on Cybernetics (TCYB), 2020.
python 2.7
tensorflow 1.5
You can download the [SNLI] dataset and [Flickr30K] images used in our paper.
[SNLI] dataset (https://drive.google.com/file/d/1CxjKsaM6YgZPRKmJhNn7WcIC3gISehcS/view?usp=sharing)
[Flickr30K] images (http://shannon.cs.illinois.edu/DenotationGraph/data/index.html)
All the data files should be unzipped and saved in directory data/ (You need to create the directory data under directory VHSM).
$ sh run.sh
$ sh test.sh You can edit the configuration variables in the training and testing scripts to your own setting. After running these scripts, you can train the model and obtain the testing results.