salesforce / CodeGen

CodeGen is a family of open-source model for program synthesis. Trained on TPU-v4. Competitive with OpenAI Codex.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

MTPB benchmark question 1 is duplicate of 26

manleyroberts opened this issue · comments

It seems question 1 in the benchmark is actually a duplicate of 26.

In addition, the given description and name for question 1: "name": "Sandwich string", "description": "Append a string in the middle of another string." do not describe the task of question 1.

If this version of question 1 was used for paper evaluations, maybe it should not be altered now, but should the description be updated to reflect the actual task of question 1?