Giters

mlcommons / inference

Reference implementations of MLPerf™ inference benchmarks

Home Page:https://mlcommons.org/en/groups/inference

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

LlaMa2-70b run_accuracy.sh issue with consolidate_results.py

mrmhodak opened this issue · comments

commented

LlaMa2-70b has a run_accuracy.py script that as a last step that calls another script, consolidate_results.py. That last step does not work, it expects to read output pkl files, but those are not being created in previous steps.

The README marks this step as optional - so we should either remove if not needed, or fix.

@nvzhihanj , please help.

commented

I believe the consolidate_results.py is not needed if the pickle input file already has all the samples (24576). That script is a by-product of preprocessing that @nv-alicheng uses IIRC.

commented

consolidate_results.py is an optional step that was used to generate a pickle file for manual viewing / data analysis. It is not required to run the accuracy script. The pkl files that it consumes are generated here (https://github.com/mlcommons/inference/blob/master/language/llama2-70b/dataset.py#L90) during the accuracy inference.