This repo contains the code of using SCN for video captioning, based on the CVPR 2017 paper “Semantic Compositional Networks for Visual Captioning”.
To keep things simple, SCN for image captioning is provided in another separate repo.
This code is written in python. To use it you will need:
- Python 2.7 (do not use Python 3.0)
- Theano 0.7 (you can also use the most recent version)
- A recent version of NumPy and SciPy
We provide the code on how to train SCN for video captioning on the Youtube2Text dataset. The SCN used in this experiment is a slightly different version of the original SCN. That is, we feed the video features to each step of the SCN-LSTM, instead of only the first step.
-
In order to start, please first download the C3D, ResNet features and tag features for the Youtube2Text dataset we used in the experiments. Put the
youtube2textfolder inside the./datafolder. -
We also provide our pre-trained model on Youtube2Text. Put the
pretrained_modelfolder into the current directory. -
In order to evaluate the model, please download the standard coco-caption evaluation code. Copy the folder
pycocoevalcapinto the current directory. -
Now, everything is ready.
- Run
SCN_training.pyto start training.
THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python SCN_training.py
-
Based on our pre-trained model, run
SCN_decode.pyto generate captions on the Youtube2Text test set. The generated captions are also provided, namedyoutube2text_scn_test.txt. -
Now, run
SCN_evaluation.pyto evaluate the model. The code will output
CIDEr: 0.777, Bleu-4: 0.511, Bleu-3: 0.606, Bleu-2: 0.697, Bleu-1: 0.810, ROUGE_L: 0.706, METEOR: 0.335.
Please cite our CVPR paper in your publications if it helps your research:
@inproceedings{SCN_CVPR2017,
Author = {Gan, Zhe and Gan, Chuang and He, Xiaodong and Pu, Yunchen and Tran, Kenneth and Gao, Jianfeng and Carin, Lawrence and Deng, Li},
Title = {Semantic Compositional Networks for Visual Captioning},
booktitle={CVPR},
Year = {2017}
}