open-compass / Ada-LEval

Line 85 in a29c69e

if matched >= 1:

This line should be changed to '> -1 * nc'.

For example:

prediction : 'A11'
ground truth: 'A11'

In this case, find() will return 0, and it does not meet the condition: >=1.
However, it's still a correct answer.

find() return -1 rather than 0, if it truly cannot match something.

Hi, @Mooler0410,
Thanks a lot, it seems to be some typos in our released minimal example, we will create a PR to fix it ASAP.

Fixed in #3

Thank you for your quick and supportive response!

find() will return 0 if the prediction is the choice label itself, without any extra string (e.g. answer:)