Giters

mlcommons / inference

Reference implementations of MLPerf™ inference benchmarks

Home Page:https://mlcommons.org/en/groups/inference

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Performance requirement for GPT-J

sunghyuckhong opened this issue · comments

commented

Dear mlcommons,
I have a question regarding the performance requirement for GPT-J. According to inference_policies, the performance requirement is specified as "99% of FP32 and 99.9% of FP32 (rouge1=42.9865, rouge2=20.1235, rougeL=29.9881)". If the submission were to satisfy 99.9% of FP32, should the submission achieve equal to greater than 99% in all the three metrics? For example, is the submission with the metric "rouge1=42.95, rouge2=20.11, rougeL=29.93" considered invalid as it failed to meet the 99.9% performance requirement for rougeL despite scoring above 99.9% in rouge1 and rouge2?

Thanks in advance,

Sincerely,

Sung Hyuck Hong

commented

That's true except for the final parameter GEN_LEN for which 90% is fine. You can see the exact requirements here

commented

Thank you for your prompt response!