Giters

salesforce / CodeGen

CodeGen is a family of open-source model for program synthesis. Trained on TPU-v4. Competitive with OpenAI Codex.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

MTPB benchmark

giulio98 opened this issue · comments

commented

Hello,
I saw that you released the MTPB benchmark recently, do you plan to release also the script to evaluate CodeGen in multi-turn or single-turn using the MTPB dataset?

Thank you in advance.

commented

I'm also looking for a script of this kind. Thank you!

commented

Thank you for the interest! We will add the MTBP execution scripts soon.

commented

@giulio98 @yueatsprograms The MTBP sampling and execution scripts can be found here:

https://github.com/salesforce/CodeGen/tree/main/benchmark

commented

Thanks for uploading, how to run the script in single-turn evaluation?