Are the evaluation results extracted automatically from research papers using an automated system, or are they submitted and curated by the end-users, such as researchers and developers?

[JIMMY ZHAO]

Dear PapersWithCode maintainers @alefnula @lambdaofgod @rstojnic @mkardas,

As a student researcher at Queen's University, I am interested in understanding how evaluation results are obtained and added to the PapersWithCode platform. Are the evaluation results extracted automatically from research papers using an automated system, or are they submitted and curated by the end-users, such as researchers and developers? For example, end-users can add new evaluation results to the HELM dataset:

I believe this information will help me better comprehend the reliability and scope of the evaluation results presented on the platform, as well as the potential opportunities for contributing to the platform as a researcher.

Thank you for your time and assistance in clarifying this matter.

Best regards,

Jimmy

[Viktor Kerkez]

HI Jimmy,

It's both. The results are extracted from the papers, but they are also curated and checked for potential errors that might have happened during extraction.

Users are also allowed to add results, but all user added results are curated and only verified results are published.

I hope that this helps :)