Interpretable Discovery in Large Image Datasets

Kiri Wagstaff, Jake Lee

in WHI 2018

This repository contains supplemental scripts and data used in the experiments presented in the paper.

Step-by-step instructions for running experiments

Download the imageset - It is recommended that the image filename include the class information. The images can be in class-related subfolders; if not, it can be placed in a single subfolder (ex. /data/1/*.jpg)
Preprocess the imageset - We recommend scaling/center-cropping your images to 227x227 first. We used imagemagick: $ mogrify -path imageset/# -format jpg -resize "227x227^" -gravity center -crop 227x227+0+0 +repage imageset/#/*.jpg
Download/install DEMUD - Available from https://github.com/wkiri/DEMUD
Extract features - Extract features from the images by using DEMUD/scripts/cnn_feat_extraction/feat_csv.py. The extracted features will be saved as a CSV, with the first column being the image name.

Note: You will need to install Caffe and to specify the trained Caffe model from which the features will be extracted. We used Caffe's pre-trained network called bvlc_reference_caffenet.

Run DEMUD on features - Run DEMUD by adding the path to the feature CSV in demud.config at floatdatafile and running python demud.py -v --init-item=svd --k=50 It will output recon-cnn-k=50..., the reconstructed features, resid-cnn-k50..., the residual features, select-cnn-k=50..., the features of the selected image, and selections-k50.csv, a list of selected image names (in corresponding order with the other outputs).
Download D&B 2015 - Available from https://lmb.informatik.uni-freiburg.de/resources/software.php
Visualize the output features - Using src/demud_inversion_2015.py provided in this repo (see below for instructions)
Plot discovery rates (optional) - Using src/plot/plot_exp.py provided in this repo
Organize your visualizations - Using src/gen_html.py provided in this repo

Scripts

src/gen_html.py collects the results from DEMUD and Dosovitskiy and Brox inversions into a single HTML file for convenient viewing. It requires the following paths to be edited into the appropriate variables:

outputname - The path of the html file to be generated
imageset_dir - The directory containing the full imageset, but not in subdirectories
dosov_dir - The directory containing the inversions generated by Dosovitskiy and Brox. It assumes the following subdirectories:
- dosov_dir/select/ - Inversions of the features in the select-cnn-*.csv output from DEMUD -v
- dosov_dir/recon/ - Inversions of the features in the recon-cnn-*.csv output from DEMUD -v
- dosov_dir/resid/ - Inversions of the features in the resid-cnn-*.csv output from DEMUD -v
demud_path - The selections-kn.csv output file from DEMUD -v
perfplot_path - The path to the discovery performance plot, if it was generated.

src/plot/plot_exp.py and src/plot/util_plot.py were used to generate the plots shown in the paper. plot_exp.py requires the directory containing DEMUD result folders. Note that, since these scripts were used specifically for our set of experiments, they may not be easy to use for other experiments.

src/demud_inversion_2015.py is a modified version of demo.py from Dosovitskiy and Brox, 2015. It reads each of the output CSVs from DEMUD -v/--cnn (selected features, reconstructed features, and residual features), and inverts them for visualization. The required pretrained model definitions are available here

src/build_imageset/ext_from_train.py builds the balanced ImageNet dataset used in the experiments from the ILSVRC 2012 training set. The classes to be pulled are specified in build_imageset/random_classes.txt. The files are renamed to match the pattern classID_imageNO.jpg and placed in their respective subfolders; for example, an image may be /output/n02009912/n02009912_0.jpg.

Data

data/sun-class/ includes an additional "sun" class to the Mars image dataset available on Zenodo. These 21 additional images were included for the experiments presented in the paper.

wkiri / whi2018-interp-img-disc