Unsolvable Problem Detection: Evaluating Trustworthiness of Vision Language Models

🤗 Dataset | 🏆 Leaderboard | 📖 arXiv | GitHub

⭐️ We are opening PRs for adding VLMs. Please do not hesitate to send them. We'll update 🏆 Leaderboard with your favorite VLMs! ⭐️

Atsuyuki Miyai¹ Jingkang Yang² Jingyang Zhang³ Yifei Ming⁴
Qing Yu^1,5 Go Irie⁶ Sharon Yixuan Li⁴ Hai Li³ Ziwei Liu²
Kiyoharu Aizawa¹

¹The University of Tokyo ²S-Lab, Nanyang Technological University ³Duke University ⁴University of Wisconsin-Madison
⁵LY Corporation ⁶Tokyo University of Science

News

2024.05: We have created our 🏆 Leaderboard with the support of the HuggingFace group.
2024.05: MM-UPD has been integrated into the LMMs-eval codebase.
2024.05: We have refactored the structure of the codebase to evaluate both UPD and Standard problems at once.
2024.03: The short version (4p.) of this paper has been accepted by ICLR 2024 R2-FM Workshop.

Introduction

This paper introduces a novel and significant challenge for Vision Language Models (VLMs), termed Unsolvable Problem Detection (UPD). UPD examines the VLM's ability to withhold answers when faced with unsolvable problems in the context of Visual Question Answering (VQA) tasks. UPD encompasses three distinct settings: Absent Answer Detection (AAD), Incompatible Answer Set Detection (IASD), and Incompatible Visual Question Detection (IVQD). To deeply investigate the UPD problem, extensive experiments indicate that most VLMs, including GPT-4V and LLaVA-Next-34B, struggle with our benchmarks to varying extents, highlighting significant room for the improvements. To address UPD, we explore both training-free and training-based solutions, offering new insights into their effectiveness and limitations. We hope our insights, together with future efforts within the proposed UPD settings, will enhance the broader understanding and development of more practical and reliable VLMs.

Requirements

Installation

We mainly follow the LLaVA's environment for the Installation.
For the implementations, we utlize Nvidia A100 GPUs with 80G. We utilize single GPU for the VLMs' inference and two GPUs for instruction tuning.

conda create -n upd_en python=3.10 -y
conda activate upd_en
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
pip install -e ".[train]"
pip install flash-attn --no-build-isolation --no-cache-dir

Data

MM-UPD Bench (Unnecessary)

We provide all benchmarks via huggingface 🤗. All benchmarks are automatically downloaded by running scripts. Please refer to our huggingface Dataset for more details.

If we encounter DatasetGenerationError, please delete the cache data of previous version.

Instruction tuning data (Optional)

We instruction tuning data (.json) via this url.
Please download, unzip and put it to ~/data.
As for the images for the instruction tuning data, we used the images for the official LLaVA's instruction tuning. Please download the images from constituting datasets:

COCO: train2017
GQA: images
OCR-VQA: download script, we save all files as .jpg
TextVQA: train_val_images
VisualGenome: part1, part2

Also, we provide the checkpoint for the instruction tuning via this url. Please download, unzip and put it to ~/checkpoints.

The overall file structure is as follows:

UPD
|-- checkpoints
    |-- llava-v1.6-vicuna-13b-task-lora
    |-- llava-v1.6-34b-task-lora
|-- data
    |-- inst_tuning
        |-- upd_tuning_data_20240303.json
        ├── coco
        │   └── train2017
        ├── gqa
        │   └── images
        ├── ocr_vqa
        │   └── images
        ├── textvqa
        │   └── train_images
        └── vg
            ├── VG_100K
            └── VG_100K_2

Note that the setting of these files is not necessary if you just want to inference VLMs.

API KEY

We need to create API keys for OpenAI and Gemini. With API keys, we can run the following commands:

export OPENAI_API_KEY='your-api-key-here'

export GEMINI_API_KEY='your-api-key-here'

About the setup of API keys, please refer to the OpenAI official page as for more detail.

Quick Start

1. Inference of VLMs

We put each script in ~/scripts/inference/<VLM name>/<UPD>. For example, to implement LLaVA-1.5 13B for the base setting, you can implement the following commands for each AAD and Standared senario:

base

bash scripts/inference/llava1.5/aad/base.sh

By implementing the above code, the result is automatically created under output/aad/answers_upload/llava1.5/base/mmaad_base/llava1.5-13b_<time_stamp>.xlsx

2. Evaluation

We put each evaluation script in ~/scripts/evaluation/<UPD>.

For example, to evaluate the performance of LLaVA-1.5 13B for the base setting, you can implement the following commands:

bash scripts/evaluation/aad/eval_base.sh <RESULT_PATH>

<RESULT_PATH> is output/aad/answers_upload/llava1.5/base/mmaad_base/llava1.5-13b_<time_stamp>.xlsx in this example.

By implementing the above code, the result is automatically created in each RESULT_PATH folder.

3. Instruction Tuning

We put each script in ~/scripts/inst_tuning. For example, to implement LLaVA-1.6 34B, you can implement the following commands:

bash scripts/inst_tuning/llava1.6_34b_lora_tuning.sh

As of March 2024, LLaVA1.6 has not yet released official LoRA tuning code. Therefore, please be aware that the our instruction tuning code may differ from the official LLaVA implementation.

Model Results

We provide a Google Sheet for the detailed results on each senario (Fig. 3, 4 ,5 ,6 in the paper). You can access the sheet here and can easily draw radar charts!

How to Add New VLMs

You can add your favorite VLMs in a very easy way!

Create vlms/<your_vlm>/<your_vlm>_vqa_updbench.py with reference to other files.
Create script files in scripts/inference/<your_vlm> with reference to other files.

After the performance check, let's send a PR!

Leaderboard Submission

We are opening your submission to our 🏆 Leaderboard. If you finish your eval with the eval scrip in scripts/evaluation and get a result file (_dual_detail_submission.json),
let's submit your result file into the leaderboard.

Acknowledgement

We adopt these codes to create this repository.

Visual Instruction Tuning, in NeurIPS, 2023.
OpenCompass
Otter

We thank the HF group for their kind assistance in the creation and promotion of our Leaderboard.

Contact

If you have questions, please open an issue mentioning @AtsuMiyai or send an email to miyai[at]hal.t.u-tokyo.ac.jp

If you are interested in this work, please refer to our other projects.

OpenOOD, in NeurIPS Dataset and Benchmark Track, 2022
OpenOOD v1.5, in NeurIPSW, 2023
LoCoOp, in NeurIPS, 2023.

Citaiton

If you find our work interesting or use our code/models, please consider citing:

@article{miyai2024upd,
  title={Unsolvable Problem Detection: Evaluating Trustworthiness of Vision Language Models},
  author={Miyai, Atsuyuki and Yang, Jingkang and Zhang, Jingyang and Ming, Yifei and Yu, Qing and Irie, Go and Li, Yixuan and Li, Hai and Liu, Ziwei and Aizawa, Kiyoharu},
  journal={arXiv preprint arXiv:2403.20331},
  year={2024}
}

AtsuMiyai / UPD