Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classification

Prerequisites

To use the repository, we provide a conda environment.

conda update conda
conda env create -f environment.yaml
conda activate Benchmark_TTA

Features

This repository allows to study a wide range of different datasets, models, settings, and methods. A quick overview is given below:

Datasets
- cifar10_c CIFAR10-C
- cifar100_c CIFAR100-C
- imagenet_c ImageNet-C
- domainnet126 DomainNet (cleaned)
- officehome Office-Home
- The dataset directory structure is as follows: |-- datasets
  
  |-- cifar-10
  
  |-- cifar-100
  
  |-- ImageNet
  
  |-- train
  
  |-- val
  
  |-- ImageNet-C
  
  |-- CIFAR-10-C
  
  |-- CIFAR-100-C
  
  |-- DomainNet
  
  |-- clipart
  
  |-- painting
  
  |-- real
  
  |-- sketch
  
  | -- clipart126_test.txt
  
  ......
  
  |-- office-home
  
  |-- Art
  
  |-- Clipart
  
  |-- Product
  
  |-- Real_World
You can download the .txt file for DomainNet in ./dataset/DomainNet, generate .txt file for office-home following SHOT
Models
- For adapting to ImageNet variations, ResNet-50 models available in Torchvision can be used and ViT available in timm · PyPI.
- For the corruption benchmarks, pre-trained models from RobustBench can be used.
- For the DomainNet-126 benchmark, there is a pre-trained model for each domain.
- The checkpoint of pretrained models is in directory ckpt
Methods
- The repository currently supports the following methods: source, PredBN, PredBN+, TENT, MEMO, EATA, CoTTA, AdaContrast, LAME, SHOT, NRC, PLUE, T3A, SAR
Modular Design
- Adding new methods should be rather simple, thanks to the modular design.

To run one of the following benchmarks, the corresponding datasets need to be downloaded.

Next, specify the root folder for all datasets _C.DATA_DIR = "./data" in the file conf.py.

The best parameters for each method and dataset are save in ./best_cfgs

download the ckpt of pretrained models and data load sequences from here and put it in ./ckpt

The entry file for SHOT, NRC, PLUE to run is SFDA-eva.sh

To evaluate this methods, modify the DATASET and METHOD in SFDA-eva.sh

and then

bash SFDA-eva.sh