The third example problem is incorrect
melaniebeck opened this issue · comments
Melanie Beck commented
"With one person drinking 5, that brings the total drank to 5+9+8+3=25" is incorrect according to the problem setup and should instead be 5+9+8=22.
The accompanying paper estimates that fewer than 2% of problems have errors like the one above. Unless this example was explicitly cherry-picked to demonstrate an incorrect problem, it is either exceedingly unlucky that this problem made it into the showcased examples or the frequency of incorrect examples is likely higher.
Shuchang Zhou commented
Thanks for pointing out! Fortunately, they got it correct in the train.jsonl anyway.