The code for CLOSURE paper
CLOSURE: Assessing Systematic Generalization of CLEVR Models
by Dzmitry Bahdanau, Harm de Vries, Timothy J. O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, Aaron Courville
This repository contains the original code that was used to obtain the reported results. See the NOTICE_AND_LICENSE file for licensing information.
Setup
The recommended way is to use Conda.
cd closure_code
conda env create
conda activate closure
pip install -r requirements.txt -e .
export NMN=`pwd`
Data
Can be dowloaded from here.
Preprocessing
Apply the same preprocessing that is done for the original CLEVR data (see here).
After preprocessing, you should have <test>_features.h5 and <test>_questions.h5 files for each part of CLOSURE on which you want to test models.
Training on CLEVR
To train the GT-Vector-NMN model from the paper, run this command:
bash <closure_code>/scripts/train/ee_film_clevr.sh --data_dir <data> --val_part val --checkpoint_path model.pt
Here, <data> should contain preprocessed CLEVR data in files train_features.h5, train_questions.h5, val_features.h5 and val_questions.h5.
You can monitor the model's online performance on CLOSURE tests as well. To do this, preprocess the data as discussed above and use extra --val-part arguments.
scripts/train also contains the training scripts for Tensor-NMN, MAC, FiLM and the program generator (PG).
Evaluation
bash <closure_code>/run_model.py --execution_engine model.pt --data_dir <data> --part <closure_test> --output_h5 output.h5