LaViSE Reproduction
This is an attempt to reproduce the results and findings of LaViSE: Explaining Deep Convolutional Neural Networks via Latent Visual-Semantic Filter Attention.
Requirements
Environment
The environment necessary to run the code can be created with the environment_lisa.yml file when running on lisa cluster and the environment_windesk when running locally on windows.
Data
To install the data, you can run the install_data.sh script. Alternatively, you can install them manually by following the instructions on Visual Genome Python Driver and COCO API WRAPPER. The wrappers and api are already installed in the environment.
To format the Visual Genome object, run the notebook.
Running the model
There are two job scripts available to run the code. The train_explainer.job file trains with a resnet18 model with Visual Genome as reference dataset. The infer_filter.job file contains several experiments that test different parameters, most of which have been commented out.
Alternatively, you can manually run
python train_explainer.py --refer <reference_dataset> --epochs <number_of_epochs> --anno_rate <annotation> --name <run_name>
on your desktop to train the explainer, and
python infer_filter.py --refer <reference_dataset> --anno_rate <annotation> --name <run_name>
to visualize the filters. Note that both parts should have matching hyperparameters.
Original README of LaViSE page
This is the official repository for paper "Explaining Deep Convolutional Neural Networks via Unsupervised Visual-Semantic Filter Attention" to appear in CVPR 2022.
Authors: Yu Yang, Seungbae Kim, Jungseock Joo
Datasets
Common Objects in Context (COCO)
- Please follow the instructions in the COCO API README and here to download and setup the COCO data.
Visual Genome (VG)
- Please follow the instructions in the README of the python wrapper for the Visual Genome API and here.
GloVe
- We load the pretrained GloVe word embeddings directly from the torchtext library.
Social Media Photographs of US Politicians (PoP)
- The list of entities used to discover new concepts is provided in
data/entities.txt
.
Getting started
Requirements
Required packages can be found in requirements.txt
.
Usage
Train an explainer with
python train_explainer.py
Explain a target filter of any model with
python infer_filter.py
More features will be added soon! 🍻
Citation
@inproceedings{yang2022explaining,
author = {Yang, Yu and Kim, Seungbae and Joo, Jungseock},
title = {Explaining Deep Convolutional Neural Networks via Unsupervised Visual-Semantic Filter Attention},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2022},
}