Introduction
This repository includes the code for paper Context-aware zero-shot recognition.
The code is highly based on Detectron.pytorch. Thanks for the effort from original author.
Requirement
- Pytorch 1.0
- Python 2 and python 3
- torchtext
- nltk
- pycocotools
Preparation
Download dataset
bash data/vg/download.sh
Convert from bansal train test split.
(In python 2)
python lib/datasets/vg/convert_from_bansal.py
Build necessary libs
cd lib
bash make.sh
cd ..
Download pretrained imagenet-model
python tools/download_imagenet_weights.py
To reproduce numbers in the paper
Download pretrained model from link.
run scripts in scripts/reproduce
Training
Train without unseen category annotations
bash scripts/train.sh
In this case, --cfg defines the general training test configuration, like roi align size, backbone bone etc.
--auto_resume enables it to resume from the latest snapshot, this has to be used with --id together.
--set overrides the items in the configuration files. Here we manually set NUM_CLASSES BASE_LR and MODEL.TAGGING.
--id allow us to identify where we save our model much easier.
iter_reason_new.yaml is borrowed from xinlei's iterative reasoning paper.
Train with word embedding as last layer
bash scripts/train_we.sh
This append MODEL.WORD_EMBEDDING_REGU True in '--set'.
Normally, the last fc layer of fast rcnn head is 2048x601, where here this layer is replaced by two linear layer 2048x300 and 300x601. The weight of 300x601 layer is the word embedding of all the classes.
Train with relation inference model.
bash train_rel.sh (bash train_we_rel.sh)
Test in region classification setting
Test with WE
bash scripts/test/we_infer.sh
bash scripts/test/we_noinfer.sh
Test with conse
bash scripts/test/conse_infer.sh
bash scripts/test/conse_noinfer.sh
Test with gcn
Download zsl-pth-gcn to ./externals.
(In python 2)
python lib/zsl-gcn/convert_gcn.py --weight_folder ./Outputs/rel_ft/ft_gt_relt_geo_sc
bash scripts/test/gcn_infer.sh
bash scripts/test/gcn_noinfer.sh
Test with sync
(In python 2)
python lib/zsl-gcn/convert_sync.py --weight_folder ./Outputs/iter_reason_new/irn_vg_gt_sc_step --dataset vgbansal --to_weight Outputs/rel_ft/ft_gt_relt_geo_sc_step/ckpt/model_final.pth
Test in detection setting:
Get edgebox proposals
cd ./external/edges
get_proposals.m
python to_raw_proposals.py
cd ../../data/vg/bansal/
python generate_proposals.py
or download from link, and put it under ./data/vg/bansal
Test
bash scripts/test/detection_gcn.sh
bash scripts/test/detection_sync.sh