TingchenFu / InstructScore

First explanation metric (diagnostic report) for text generation evaluation

InstructScore (SEScore3)

An amazing explanation metric (diagnostic report) for text generation evaluation

Installation

We list all required dependencies in requirements.txt. You can create a conda environment and install all dependencies through following commands:

    conda create -n instructscore python=3.9
    conda activate instructscore
    pip install -r requirements.txt

Usage

There are two ways to use InstructScore.

Option 1: Have a fast try through Huggingface

We have uploaded our model to Huggingface, which can be found here. You can directly try InstructScore via several lines of code:

from InstructScore import InstructScore

refs = ["Normally the administration office downstairs would call me when there’s a delivery."]
outs = ["Usually when there is takeaway, the management office downstairs will call."]

scorer = InstructScore()
batch_explanations, batch_scores = scorer.score(refs, outs)

Option 2: Download weight from Google Drive

You can also download the checkpoint from this Google Drive link.