huggingface / model-evaluator

Evaluate Transformers from the Hub 🔥

https://huggingface.co/spaces/autoevaluate/model-evaluator

huggingface/model-evaluator Issues

Extend support for the zero-shot text classification task to support encoder-decoder models
Updated a year ago
Evaluation job crashes with "KeyError: eval_info" when submitted
Closed 2 years ago2
Enable usage with private datasets / models
Updated 2 years ago
Filter models dropdown to those suitable for the task
Updated 2 years ago
Add FLOPS to reporting of results
Updated 2 years ago
Cap the number of zero-shot evaluations allowed simultaneously
Updated 2 years ago
Extend zero-shot classification language task to support a generalized likelihood task
Updated 2 years ago
Add semantic similarity task
Updated 2 years ago
Warn users if model-to-be-evaluated does not have a model card
Updated 2 years ago
Allow model evaluations to be re-run
Updated 2 years ago
Support concurrent submissions
Updated 2 years ago
CI against changes to Autotrain
Updated 2 years ago1
Extend list of compatible models to include Text2Text models for translation / summarisation
Updated 2 years ago
Add your redirection magic from dataset page
Closed 2 years ago1
Implement mechanism to cache submission jobs
Closed 2 years ago1
Add slice membership as column (default to entire split)
Updated 2 years ago