This repository contains an implementation source code for Distance-Aware Explanation based Learning (XBL-D) paper. It contains scripts to extract COCO images, add confounding regions to them and generate a new dataset, tune a classifier, and refine a classifier using XBL-D.
To install all requirements:
pip install -r requirements.txt
For copyright issues, the decoy version of a subset of the MS-COCO dataset can not be shared here, and for research purposes it needs to be recreated using a Python script which is included in this repository. The decoy FashionMNSIT and decoy CIFAR-10 datasets can be downloaded from the following links:
- Decoy FashionMNIST dataset: this dataset is created and shared as part of the paper Schramowski et al. [2020] 'Making deep neural networks right for the right scientific reasons by interacting with their explanations.' It can be downloaded from the following link: https://codeocean.com/capsule/7818629/tree/v1/data/fashion/decoy-fashion.npz
- Decoy CIFAR-10 dataset: this dataset is shared anonymously. It can be downloaded from the following link: https://osf.io/w5f7y/?view_only=abb7f5f55bfc48fb8c891838f699c0d3
Download this repository and change the working directory to XBL-D, and follow the instructions below:
cd XBL-D
Download the 'instances_train2014.json' from this link and place it in the same directory as the scripts in this repository. It is found inside the '2014 Train/Val annotations.'
To extract training and testing images and object annotations for the Train and Zebra categories from the MS COCO database using the 'instances_train2014.json' file:
python coco_data_extraction.py
This will create a directory named 'MSCOCO' with two sub-directories named 'images' and 'annotations' and populate them with images and object annotations of the Train and Zebra categories.
To add confounding regions to the extracted images and generate masks of the confounders:
python coco_decoy_generate.py
This will create two sub-directories inside 'MSCOCO' named 'confounded' and 'confounded_mask' and populate them with confounded versions of the images and mask annotations of the confounders.
To search for the optimal hyperparameters: (we consider the hyperparameters: number and size of convolutional layers, number of pooling layers, number and size of fully connected layers, and learning rate for each datasets separately.)
python tuning_classifier.py
This will create a directory named 'models_coco' that contains tuned models. In addition, it will select the best performing model architecture and train it using classification loss. At then end, the trained model will be saved as 'coco.h5'
To start training a model using XBL-D:
python xbl-d_train.py
This will load the pretrained coco.h5 from above and refine it using XBL-D.