TOAST: Top-Down Attention Steering
This is the official codebase of TOAST, from the following paper:
TOAST: Transfer Learning via Attention Steering
Baifeng Shi, Siyu Gai, Trevor Darrell, and Xin Wang
UC Berkeley, Microsoft Research
Motivation
We find previous transfer learning methods (fine-tuning, LoRA, prompt tuning, etc.) often fail to focus on the features relevant to the downstream tasks (see figure above). We show that refocusing the attention to task-relevant features can improve downstream performances.
What is TOAST?
TOAST is a transfer learning algorithm which transfers a large pre-trained model to a downstream task by refocusing the model's attention to task-relevant features. Specifically, TOAST freezes the pre-trained backbone and tunes a top-down attention module to refocus the attention (see figure below).
This repo contains
visual_classification: TOAST for visual classification (including transfer learning on FGVC and VTAB)language_generation: TOAST for language generation (including transfer learning on Alpaca)
Links
This codebase is largely built upon
- "Top-Down Visual Attention from Analysis by Synthesis"
- "Visual Prompt Tuning"
- "Stanford Alpaca: An Instruction-following LLaMA Model"
- "FastChat"
Citation
If you found this code helpful, please consider citing our work:
@article{shi2023toast,
title={TOAST: Transfer Learning via Attention Steering},
author={Shi, Baifeng and Gai, Siyu and Darrell, Trevor and Wang, Xin},
journal={arXiv preprint arXiv:2305.15542},
year={2023}
}