mlcommons / inference

Reference implementations of MLPerf™ inference benchmarks

Home Page:https://mlcommons.org/en/groups/inference

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

host_processor_vcpu_count in <system_desc_id>.json

szutenberg opened this issue · comments

According to https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#57-system_desc_idjson-metadata it's required to provide field host_processor_vcpu_count.

I see that relevant checks are missing in https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py so the submitter get
submission_checker.py:3240 WARNING] <redacted>, field host_processor_vcpu_count is unknown.

My proposal is to remove host_processor_vcpu_count from the submission rules because this rule is not enforced anyway. In inference_results_v3.1 this field is present only in closed/Intel-HabanaLabs/systems/HLS-Gaudi2-PT.json.

If we provide only host_processor_vcpu_count instead of host_processor_core_count then submission checker fails:

submission_checker.py:3220 ERROR] <REDACTED>, field host_processor_core_count is missing
submission_checker.py:3240 WARNING] <REDACTED>, field host_processor_vcpu_count is unknown

A related issue: #1351