Duel-Level Collaborative Transformer for Image Captioning
This repository contains the reference code for the paper Duel-Level Collaborative Transformer for Image Captioning.
Experiment setup
please refer to m2 transformer
Data preparation
- Annotation. Download the annotation file annotation.zip. Extarct and put it in the project root directory.
- Feature. You can download our ResNeXt-101 feature (hdf5 file) here. Access code: 7fhb.
- evaluation. Download the evaluation tools here. Access code: 7fhb. Extarct and put it in the project root directory.
There are five kinds of keys in our .hdf5 file. They are
['%d_features' % image_id]
: region features (N_regions, feature_dim)['%d_boxes' % image_id]
: bounding box of region features (N_regions, 4)['%d_size' % image_id]
: size of original image (for normalizing bounding box), (2,)['%d_grids' % image_id]
: grid features (N_grids, feature_dim)['%d_mask' % image_id]
: geometric alignment graph, (N_regions, N_grids)
We extract feature with the code in grid-feats-vqa.
The first three keys can be obtained when extracting region features with extract_region_feature.py. The forth key can be obtained when extracting grid features with code in grid-feats-vqa. The last key can be obtained with align.ipynb
Training
python train.py --exp_name dlct --batch_size 50 --head 8 --features_path ./data/coco_all_align.hdf5 --annotation annotation --workers 4 --rl_batch_size 100 --image_field ImageAllFieldWithMask --model DLCT --rl_at 17 --seed 118
Evaluation
python eval.py --annotation annotation --workers 4 --features_path ./data/coco_all_align.hdf5 --model_path path_of_model_to_eval --model DLCT --image_field ImageAllFieldWithMask --grid_embed --box_embed --dump_json gen_res.json --beam_size 5
Important args:
--features_path
path to hdf5 file--model_path
--dump_json
dump generated captions to
References
[1] M2
[2] grid-feats-vqa
[3] butd
Acknowledgements
Thanks the original m2 and amazing work of grid-feats-vqa.