CLAIR: A (Surprisingly) simple semantic text metric leveraging large language models.

Project | Paper

Official implementation of the paper: "CLAIR: Evaluating Image Captions with Large Language Models".

The evaluation of machine-generated image captions poses an interesting yet persistent challenge. Effective evaluation measures must consider numerous dimensions of similarity, including semantic relevance, visual structure, object interactions, caption diversity, and specificity. Existing highly-engineered measures attempt to capture specific aspects, but fall short in providing a holistic score that aligns closely with human judgments. Here, we propose CLAIR, a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) to evaluate candidate captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Notably, on Flickr8K-Expert, CLAIR achieves relative correlation improvements over SPICE of 39.6% and over image-augmented methods such as RefCLIP-S of 18.3%. Moreover, CLAIR provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score.

Getting started

Setup

pip install git+https://github.com/DavidMChan/clair.git

Usage

import os
from clair import clair

os.environ['OPENAI_API_KEY'] = 'YOUR_API_KEY'

candidates = ['A photo of a cat sitting on a couch', 'A cat sitting on a couch']
references = ['A cat sitting on a couch', 'A cat is sitting on a couch', 'A cat is sitting on a sofa']

clair_score = clair(candidates, references, model='chat-gpt')

print(clair_score)

Models

CLAIR relies on the Grazier library to inteface with large language models. The following models are currently supported (see the linked repository for more information on configuring each model):

From OpenAI:

GPT-4 (Base, 32K) (Chat and Completion Engines)
GPT-3.5 (ChatGPT) (Chat and Completion Engines)
GPT-3 (Davinci (v2,v3), Ada, Babbage, Curie) (Completion Engine)

From Anthropic:

Claude 2 (Base) (Chat and Completion Engines)
Claude (Base, 100K) (Chat and Completion Engines)
Claude Instant (Base, 100K) (Chat and Completion Engines)

From Google/GCP:

PaLM (Chat and Completion Engines)

From Huggingface

GPT-2 (Base, Medium, Large, XL) (Completion Engine)
GPT-Neo (125M, 1.3B, 2.7B) (Completion Engine)
GPT-J (6B) (Completion Engine)
Falcon (7B, 40B, rw-1B, rw-7B) (Completion Engine)
Dolly (v1 - 6B, v2 - 3B, 7B, 12B) (Chat and Completion Engines)
MPT (Instruct - 7B, 30B) (Chat and Completion Engines)

From Facebook (via Huggingface)

Llama (7B, 13B, 30B, 65B) (Completion Engine)
Llama 2 (7B, 13B, 70B) (Completion Engine)
OPT (125M, 350M, 1.3B, 2.7B, 6.7B, 13B, 30B, 66B) (Completion Engine)

From Stanford (via Huggingface)

Alpaca (7B) (Chat and Completion Engines)

From Berkeley (via Huggingface)

Koala (7B, 13B_v1, 13B_v2) (Chat and Completion Engines)
Vicuna (7B, 13B) (Chat and Completion Engines)

From StabilityAI (via Huggingface)

StableLM (7B, 13B) (Chat and Completion Engines)

From AllenAI (via Huggingface)

Tulu (7B, 13B, 30B, 65B) (Chat and Completion Engines)
Open Instruct (ShareGPT) (7B, 13B, 30B, 65B) (Chat and Completion Engines)

From AI21

Jurassic 2 (Light, Mid, Ultra) (Completion Engines)

Citation

If you find this repository useful, please cite our paper:

@inproceedings{chan-etal-2023-clair-evaluating,
    title = "CLAIR: Evaluating Image Captions with Large Language Models",
    author = "Chan, David M and
        Petryk, Suzanne and
        Gonzalez, Joseph E and
        Darrell, Trevor and
        Canny, John",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore, Singapore",
    publisher = "Association for Computational Linguistics",
}

DavidMChan / clair