huggingface / evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

https://huggingface.co/docs/evaluate

huggingface/evaluate Issues

HuggingFace evaluate-cli create command results Error: "datasets[0]" must be a string
Updated 9 days ago2
evaluate consuming Memory and slow down the process
Closed a month ago
several breakages due to recent `datasets`
Updated a month ago9
Evaluate fails to acquire lock interrupting single-node multi-GPU training
Updated a month ago2
the difference of your bleu and sacrebleu
Updated a month ago
Multi-label with f1
Closed a month ago1
After fine-tuning Gemma and want to evaluate performance: AttributeError: module 'keras._tf_keras.keras' has no attribute '__internal__'
Updated 2 months ago
Add Precision@k and Recall@k metrics
Updated 2 months ago
Evaluation not working with `microsoft/deberta-v3-large`
Updated 2 months ago
need evaluator code reference for object detector & zero shot object dector task
Updated 2 months ago
Add geometric mean of per-token-Perplexities
Updated 2 months ago
Evaluate a quantize model
Updated 2 months ago2
MSE not accepting (n_samples, n_ouptuts) despite docs stating so
Updated 2 months ago
Try to import evaluate but get "Library not loaded: @rpath/libwebp.7.dylib" error
Closed 2 months ago2
`DatasetColumn` and `DatasetColumnPair` should have `__repr__()` methods defined
Updated 2 months ago
Token classification bootstrap crashing with custom dataset
Updated 3 months ago
Documentation claims that `label_mapping` is an accepted input to `TokenClassificationEvaluator.compute()`; it is not
Updated 3 months ago
Metric returned instead of the actual COMET scores
Closed 3 months ago7
Is there a plan to translate the documentation for this repository?
Updated 3 months ago
Is TextGenerationEvaluator incomplete?
Updated 3 months ago
How should i cite this repository?
Updated 3 months ago1
`perplexity` yields `IndexError: index out of range in self`
Updated 4 months ago1
Bug in BLEU calculation - BLEU of identical strings should be 1.0, not 0.0
Updated 4 months ago1
about evaluate/perplexity
Updated 4 months ago2
`.compute()` not compatible with `perplexity` for `text-generation`
Updated 4 months ago1
Cannot use evaluate.TextClassificationEvaluator for multi-label classification
Updated 4 months ago
Using huggingface evaluate for distributed evaluation
Updated 4 months ago
Getting error <lambda>() takes 0 positional arguments but 1 was given when trying to import evaluator
Updated 5 months ago
os.unlink(entry.name, dir_fd=topfd) TypeError: 'NoneType' object is not callable
Updated 5 months ago
DefaultCPUAllocator: can't allocate memory during perplexity.compute for Llama 2
Updated 5 months ago
`evaluate-cli create` fails with `subprocess.CalledProcessError`
Updated 5 months ago
Importing Error for evaluator module
Updated 5 months ago
Cannot list evaluation modules types
Updated 5 months ago
Can't calculate combined metric (WER + CER)
Updated 5 months ago2
Allow BLEURT optimisation parameters
Updated 5 months ago
add_batch taking very long time (SegFormer semantic segmentation).
Updated 5 months ago
perplexity.compute - Add args to pass model and tokenizer objects instead of `model_id`
Updated 6 months ago1
is it possible to evaluate a proprietary model without uploading it to HF
Updated 6 months ago7
Loading metrics that are shipped with the evaluate package is slow
Updated 6 months ago2
Perplexity Space Broken
Closed 6 months ago
Compatibility Issue with datasets library
Updated 6 months ago3
Move away from `Repository` to `HfApi` upload files
Updated 6 months ago
wrong rouge-Lsum results due to lack of parameter control
Updated 6 months ago
Wrong accuracy on the validation set!!!
Updated 7 months ago1
CER.py error in class CERConfig
Updated 7 months ago1
evaluate.load("wer") is returning Filenotfound error
Updated 7 months ago3
Feature request for metric result keys in `MetricInfo`
Updated 7 months ago
Matthew's Correlation Coefficient (MCC) always zero
Closed 7 months ago1
Concurrent Programs Sharing Evaluate Lib Would raise "ValueError: Error in finalize: another evaluation module instance is already using the local cache file."
Closed 8 months ago6
How to use `SubTask` with metrics that require valid `config_name`
Updated 8 months ago