Official pytorch implementation of "Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation" (EMNLP2023 Findings).
CBBT is a black-box tuning method for few-shot adaptation of vision-language models
Please follow the steps below to build your environment.
# Create a conda environment (Omit if you already have a suitable environment)
conda create -n dassl python=3.8
conda activate dassl
conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge # torch (version >= 1.7.1)
# Clone this repo
git clone https://github.com/guozix/cbbt.git
cd cbbt
# install Dassl
cd Dassl.pytorch-master/
# Install dependencies
pip install -r requirements.txt
# Install this library (no need to re-build if the source code is modified)
python setup.py develop
cd ..
# Install CLIP dependencies
pip install -r requirements.txt
# Finished
Follow DATASETS.md to prepare the datasets.
Chage the variable DATA
in the training scripts main.sh
to your dataset directory.
Train TaI-DPT on the datasets:
bash scripts/coop/main.sh eurosat rn50 end 1 16 False eurosat_rn50_1ctx 1 True True False 256 150 6000
bash scripts/coop/main.sh fgvc_aircraft rn50 end 1 16 False fgvc_OURS_WO_rn50_1ctx_haug_fix 0 True True False 256 150 6000
bash scripts/coop/main.sh oxford_pets rn50 end 1 16 False oxford_pets_OURS_WO_rn50_1ctx_haug_fix 1 True True False 256 150 6000
bash scripts/coop/main.sh oxford_flowers rn50 end 1 16 False oxford_flowers_OURS_WO_rn50_1ctx_haug_fix 2 True True False 256 150 6000
...
If you make use of our work, please cite our paper.
@inproceedings{
guo2023blackbox,
title={Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation},
author={Zixian Guo and Yuxiang Wei and Ming Liu and Zhilong Ji and Jinfeng Bai and Yiwen Guo and Wangmeng Zuo},
booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
year={2023},
url={https://openreview.net/forum?id=gybvlVXT6z}
}
The code is based largely on the implementation of CoOp and Dassl. Thanks for their contributions!