Implement mechanism to cache submission jobs
lewtun opened this issue · comments
Find a simple way to avoid re-computing a prior evaluation job. The simplest idea I have right now is to hash some combo like model-id__task__dataset-name__dataset-subset__dataset-split or the AutoTrain payload and add that hash to the prediction repo name. Then, when submitting evaluation jobs from the Space, do a quick check to see if any hashes exist and skip that evaluation.