Human alignment of neural network representations

This is the official GitHub repository to the ICLR 2023 paper "Human alignment of neural network representations". The GitHub repository should allow you to reproduce the results of most, if not all, of the experiments we present in our paper.

Citation

@inproceedings{muttenthaler2023,
  author    = {Muttenthaler, Lukas and Dippel, Jonas and Linhardt, Lorenz and Vandermeulen,
  Robert A and Kornblith, Simon},
  title     = {Human alignment of neural network representations},
  booktitle = {11th International Conference on Learning Representations, {ICLR} 2023,
               Kigali, Rwanda, Mai 01-05, 2023},
  publisher = {OpenReview.net},
  year      = {2023}
}

Relevant resources used in this project

Environment setup and dependencies

We recommend to create a virtual environment (e.g., human_alignment), including all dependencies, via conda

$ conda env create --prefix /path/to/conda/envs/human_alignment --file envs/environment.yml
$ conda activate human_alignment
$ pip install git+https://github.com/openai/CLIP.git

Alternatively, dependencies can be installed via pip,

$ conda create --name human_alignment python=3.9
$ conda activate human_alignment
$ pip install --upgrade pip
$ pip install -r requirements.txt
$ pip install git+https://github.com/openai/CLIP.git

Repository structure

root
├── envs
├── └── environment.yml
├── data
├── ├── __init__.py
├── ├── cifar.py
├── └── things.py
├── utils
├── ├── __init__.py
├── ├── analyses/*.py
├── ├── evaluation/*.py
├── └── probing/*.py
├── models
├── ├── __init__.py
├── ├── custom_mode.py
├── └── utils.py
├── .gitignore
├── README.md
├── main_embedding_sim_eval.py
├── main_embedding_triplet_eval.py
├── main_model_comparison.py
├── main_model_sim_eval.py
├── main_model_triplet_eval.py
├── main_probing.py
├── requirements.txt
├── search_temp_scaling.py
├── show_triplets.py
└── visualize_embeddings.py

Usage

Run evaluation script on things triplet odd-one-out task with some pretrained model.

$ python main_model_triplet_eval.py --data_root /path/to/data/name \ 
--dataset name \
--model_names resnet101 vgg11 clip_ViT-B/32 clip_RN50 vit_b_16 \
--module logits \
--overall_source thingsvision \
--sources torchvision torchvision custom custom torchvision  \
--model_dict_path /path/to/model_dict.json \
--batch_size 128 \
--distance cosine \
--out_path /path/to/results \
--device cpu \
--verbose \
--rnd_seed 42

Run evaluation script on multi-arrangement similarity judgements with some pretrained model.

$ python main_model_sim_eval.py --data_root /path/to/data/name \ 
--dataset name \
--model_names resnet101 vgg11 clip_ViT-B/32 clip_RN50 vit_b_16 \
--module logits \
--overall_source thingsvision \
--sources torchvision torchvision custom custom torchvision  \
--model_dict_path /path/to/model_dict.json \
--batch_size 118 
--out_path /path/to/results \
--device cpu \
--verbose \
--rnd_seed 42 \

Plot Results

For each dataset, it is necessary to create a folder under resources/results (it is also possible to choose another root path).

resources/results
├── free-arrangement
│   ├── set1
│   │   ├── transform
│   │   └── zero-shot
│   └── set2
│       ├── transform
│       └── zero-shot
├── multi-arrangement
│   ├── transform
│   │   ├── google
│   │   ├── imagenet
│   │   ├── loss
│   │   ├── thingsvision
│   │   ├── vit_best
│   │   └── vit_same
│   └── zero-shot
│       ├── google
│       ├── imagenet
│       ├── loss
│       ├── thingsvision
│       ├── vit_best
│       └── vit_same
└── things
    ├── transform
    │   └── probing_results.pkl
    └── zero-shot
        └── results.pkl

Run the parse_results.py script to generate zero-shot.csv and transform.csv for each dataset. After creating the csv files, we can run plot_results.py which by default creates all plots for all datasets. This can be potentially limited with the --dataset and --typeflag.

KogniJannis / human_alignment