This repo contains the codebase for our EMNLP 2022 paper CPL: Counterfactual Prompt Learning for Vision and Language Models
Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP. However, existing prompt tuning methods tend to learn spurious or entangled representations, which leads to poor generalization to unseen concepts. Towards non-spurious and efficient prompt learning from limited examples, this paper presents a novel Counterfactual Prompt Learning (CPL) method for vision and language models, which simultaneously employs counterfactual generation and contrastive learning in a joint optimization framework. Particularly, CPL constructs counterfactual by identifying minimal non-spurious feature change between semantically-similar positive and negative samples that causes concept change and learns more generalizable prompt representation from both factual and counterfactual examples via contrastive learning. Extensive experiments demonstrate that CPL can obtain superior few-shot performance on different vision and language tasks than previous prompt tuning methods on CLIP. On image classification, we achieve a 3.55% average relative improvement on unseen classes across seven datasets; on image-text retrieval and visual question answering, we gain up to 4.09% and 25.08% relative improvements across three few-shot scenarios on unseen test sets.
This code is built heavily on top of the toolbox Dassl.pytorch, so you need to install the dassl
environment first.
To install the toolbox, you need to follow the instructions described in the original codebase here to install dassl
as well as PyTorch.
The engine is updated to be adapted to this work. So you also need to clone Dassl.pytorch from this repo.
- Follow DATASETS.md and download the datasets used.
- Follow instructions in the dataset_processing repo to preprocess the datasets.
- The off-the-shelf prompts of VQAv2 that can be used for customized purpose are available at prompt
You'll need a working Python environment to run the code. The code is based on python 3.6
and pytorch 1.6
.
The recommended way to set up your environment is through the Anaconda Python distribution which provides the conda
package manager.
The required dependencies are specified in the file requirements.txt
.
Run the following command in the repository folder (where requirements.txt
is located) to create a separate environment and install all required dependencies in it:
conda create -n env_name python=3.6 # create new environment
source activate env_name
pip install -r requirements.txt
Before running any code you must activate the conda environment:
source activate env_name
or, if you're on Windows:
activate env_name
This will enable the environment for your current terminal session.
You can lauch corresponding experiments by passing the given arguments
cd CPL
python train.py
for baseline method; Or
cd CPL
python train_cf.py
for CPL method.
Avaliable arguments are:
Parameter | Description |
---|---|
--root ${DATA} \ | Your dataset location |
--seed ${SEED} \ | Random seeds |
--trainer ${TRAINER} \ | CoCoOpcf/CoCoOp/zsclip |
--dataset-config-file configs/datasets/${DATASET}.yaml \ | Dataset onfig location |
--config-file configs/trainers/${TRAINER}/${CFG}.yaml \ | Trainers config location |
--output-dir ${DIR} \ | output directory location |
DATASET.NUM_SHOTS ${SHOTS} \ | # shots for classification/ 0.5 1 3 5 for ITR and VQA |
DATASET.SUBSAMPLE_CLASSES fewshot | 'fewshot' for ITR and VQA/ number e.g. 4 8 16 for classifcation |
Examples are for baseline:
cd CPL
# TRAINER=ZeroshotCLIP
TRAINER=CoCoOp
python train.py \
--root /path/to/datasets \
--seed 1 \
--trainer ${TRAINER} \
--dataset-config-file configs/datasets/mscoco.yaml \
--config-file configs/trainers/${TRAINER}/vit_b16_c4_ep10_batch1_ctxv1.yaml \
--output-dir output \
DATASET.NUM_SHOTS 1 \
DATASET.SUBSAMPLE_CLASSES fewshot
for cpl:
cd CPL
python train_cf.py \
--root /path/to/datasets \
--seed 1 \
--trainer CoCoOpcf \
--dataset-config-file configs/datasets/mscoco.yaml \
--config-file configs/trainers/CoCoOpcf/vit_b16_c4_ep10_batch1_ctxv1.yaml \
--output-dir output \
DATASET.NUM_SHOTS 1 \
DATASET.SUBSAMPLE_CLASSES fewshot
Or you can use bash
cd scripts
sh cocoop/main_cpl.sh $dataset
for methods
Main/
├── Dassl.pytorch
│ ├── Dassl
│ │ ├── engine (modified trainer and dataset loader for CPL)
│ │ │ ├── ...
│ │ ├── evaluation (modified evaluator for ITR and VQA)
│ │ │ ├── ...
│ │ ├── metrics (mesaured metrics)
│ │ │ ├── ...
│ │ ├── ...
│ ├── ...
├── datasets (datatset processing files, added flickr8k, 30k, mscoco, and vqav2)
│ │ ├── flickr8k.py
│ │ ├── flickr30k.py
│ │ ├── mscoco.py
│ │ ├── vqav2.py
├── trainers
│ ├── cocoop.py (original CoCoOp)
│ ├── cocoopcf.py (CPL)
│ ├── zsclip.py (zeroshot CLIP)
├── prompt (T5 generated prompts for VQAv2)
├── train_cf.py (main training file for CPL)
├── train.py (main training file for CoCoOp)
├── ...
We thank the authors of CoOp and CoCoOp for releasing their code and engine. We thank the authors of CLIP Models are Few-shot Learners for their method of VQA prompt generation. This project is supported by the Google Ads Faculty Research Award.
Please consider citing the paper if you find it useful. Thks! :)
@inproceedings{he-2022-CPL,
title = "{CPL}: Counterfactual Prompt Learning for Vision and Language Models",
author = " Xuehai He and
Diji Yang and
Weixi Feng and
Tsu-Jui Fu and
Arjun Akula and
Varun Jampani and
Pradyumna Narayana and
Sugato Basu and
William Yang Wang and
Xin Eric Wang",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
year = "2022",
}