Giters

mlcommons / inference

Reference implementations of MLPerf™ inference benchmarks

Home Page:https://mlcommons.org/en/groups/inference

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

MLPerf-Inference Server: change metrics from scheduled to completed samples per second for all benchmarks

nvzhihanj opened this issue · comments

commented

As presented in https://docs.google.com/presentation/d/1Y_AKEJ6h1g5k3ntrL7nTazWw3xVDzJ_tjOGkLQ6VDMI/edit?usp=sharing the completed sample per second is a better representation of the throughput than scheduled QPS.
@pgmpablo157321 to help implement after the conclusion of v4.0

commented

Required code changes for this:

  • Change results.cc to report completed samples per second
  • Change submission checker to extract completed samples per second

Not necessary to change metrics in the compliance tests, they already use completed samples per second: test01, test04, test05