huggingface / lighteval

LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.

huggingface/lighteval Issues

quantized model not loading
Updated a day ago6
Different results for gsm8k via lighteval compared to internal pipeline
Updated 2 days ago
ModuleNotFoundError: No module named 'lighteval'
Updated 7 days ago2
The helm|piqa task is generative but has generation_size=-1.
Updated 14 days ago1
Expose `stop_sequence` at command line
Updated 18 days ago
Zero scores on cnn-dm benchmark from HELM
Closed 18 days ago2
`LightevalTask.process_results()` is not aligned with `LightevalTask.get_request_type()`
Updated 22 days ago3
An apparent bug in drop's dealing with multi-span answer
Updated 22 days ago3
`apply_target_perplexity_metric` pops only the first response
Updated 24 days ago
Issue when saving results with fsspec==2023.12.1
Updated a month ago
Evaluate EncoderDecoderModels
Updated a month ago2
Error: `ModuleNotFoundError: No module named 'openai'`.
Updated a month ago5
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
Updated a month ago3
[New Task] Add AlpacaEval LC
Updated a month ago9
Expose a few model predictions / gold answers in the logs
Updated a month ago1
How to run 30b plus model with lighteval when accelerate launch failed? OOM
Closed a month ago4
Version of a task should be configurable.
Closed a month ago6
Add Sympy equivalence for MATH / GSM8K?
Updated a month ago1
DROP Evaluation with Llama3 (vs. lm-evaluation-harness)
Updated a month ago1
Performance compared to lm-evaluation-harness
Updated 2 months ago6
add transformers model to be used as judge
Updated 2 months ago2
Do an intro notebook on how to use `lighteval`
Updated 2 months ago2
`Could not initialize the JudgeOpenAI model` and `openi` import error
Updated 2 months ago1
[MATH] Too many values to unpack (expected 2)
Closed 2 months ago3
Feature: Checkpointing on task level.
Closed 2 months ago2
MMLU evaluation fails with Mistral
Updated 2 months ago6
Add Code-Centric Interface to LightEval for Enhanced Usability
Updated 2 months ago4
Add LLM as a judge as a metric
Closed 2 months ago1
`LatexTableWriter` created but never used.
Updated 2 months ago1
Homogeneize logging system
Updated 2 months ago
Add dtype management in inference endpoints
Closed 2 months ago1
Winogrande degraded results
Updated 2 months ago5
Add a nanotron model to the test suite
Updated 2 months ago
Use config files for the model parameters
Closed 2 months ago
Add a logger in the metric functions
Updated 2 months ago
Add MT-Bench
Closed 3 months ago
human eval run
Updated 3 months ago5
Problem with mutliple tasks from the same dataset
Closed 3 months ago1
Deploying evaluation for finetuned model as AWS SM pipeline step
Closed 3 months ago2
Add EQ Bench
Updated 3 months ago1
Add BBH subset back!
Closed 3 months ago
[BUG]: lighteval.utils import is_autogptq_available not working
Closed 3 months ago3
Push details to hub does not work
Closed 3 months ago1
[IFEVAL] Stopping criteria fails for models with ChatML special tokens
Closed 3 months ago1
With chat templates, instructions shouldn't be prepended to system prompt
Closed 3 months ago5
StarCoder2 3B SFT models give CUDA OOM on IFEval
Closed 3 months ago3
Add BBH
Closed 3 months ago
Collate items in GenerativeTaskDataset by similar EOS token
Updated 3 months ago
To remember for version upgrades
Updated 3 months ago
Add the license to all files headers
Closed 3 months ago