hendrycks / test

Measuring Massive Multitask Language Understanding | ICLR 2021

Home Page:https://arxiv.org/abs/2009.03300

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Duplicate Answers in Validation Set

riedgar-ms opened this issue · comments

I have spotted (or more precisely, my schema validator has spotted) three questions where the choices feature a duplicate.

All are in the 'validation' set. Specifically:

In elementary_mathematics_val.csv

What is the value of |3 + 5| – |-4|?,12,-4,4,12,C

In high_school_mathematics_val.csv

"Sam has $\frac{5}{8}$ of a pound of chocolate. If he eats $\frac{1}{3}$ of the chocolate he has, how many pounds of chocolate does he eat?",\frac{5}{12},\frac{5}{24},\frac{3}{24},\frac{3}{24},B

In miscellaneous_val.csv

"How many balloons would be required to fill the Empire State Building, which is about 100 stories tall?","60,000,000","60,000,000","600,000,000","6,000,000,000",A

This one is a particular problem, since it is the correct answer which has been duplicated.