paperswithcode / sotabench-api

Easily benchmark Machine Learning models on selected tasks and datasets

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Run hash should give a warning for empty/all-zero inputs

rstojnic opened this issue · comments

In some cases the first batch might be too small, or simply contain no interesting predictions (eg empty or all zeros).

Run hash calculation should catch this, produce a warning and not try to retrieve the cached results.

This might be an issue for the frontends (sotabencheval, torchbench) and is task-specific.

I've solved for COCO in sotabencheval by doing the following:

  • Check whether all the bboxs in a list of dicts are empty
  • If they are all empty, then don't calculate a batch hash (wait until the first addition that has detections)