swstarlab / GLACNet

GLAC Net: GLocal Attention Cascading Network for the Visual Storytelling Challenge

This repository corresponds to CogMap in the ALTA project.

GLAC Net: GLocal Attention Cascading Network

This repository is the implementation of GLAC Net: GLocal Attention Cascading Network for the Visual Storytelling Challenge 2018 as a team SnuBiVtt. Our model got the highest score in the human evaluation of the challenge.

Dependencies

Python 2.7
Pytorch >= 0.4.0

Prerequisites

1. Clone the repository

git clone https://github.com/AcousticRicky/GLACNet.git
cd GLACNet

2. Download requirements

pip install -r requirements.txt

3. Download sentence tokenizer

python
>>> import nltk
>>> nltk.download('punkt')
>>> exit()

Preprocessing

1. Download the dataset

VIST homepage

2. Resize images and build vocabulary

All the images should be resized to 256x256.

python resize.py --image_dir [train_image_dir] --output_dir [output_train_dir]
python resize.py --image_dir [val_image_dir] --output_dir [output_val_dir]
python resize.py --image_dir [test_image_dir] --output_dir [output_test_dir]
python build_vocab.py

Training & Validation

python train.py

Evaluation

1. Download the evaluation tool (METEOR score) for the VIST Challenge

git clone https://github.com/windx0303/VIST-Challenge-NAACL-2018 ../VIST-Challenge-NAACL-2018

2. Install Java

sudo apt-get install default-jdk

3. Run eval.py script

python eval.py --model_num [my_model_num]

The result.json file will be found in the root directory.

Pretrained model

We provide the pretrained model. Please download the link and move to <GLACNet root>/models/.

Citation

@article{Kim2018GLAC,
title={GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation},
author={Taehyeong Kim and Min-Oh Heo and Seonil Son and Kyoung-Wha Park and Byoung-Tak Zhang},
journal={CoRR},
year={2018},
volume={abs/1805.10973}
}

License

MIT License
This repository refer to pytorch tutorial by yunjey.