EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.

https://www.eleuther.ai

EleutherAI/lm-evaluation-harness Issues

running error
Updated 20 days ago3
BitsandBytesConfig in CLI?
Updated 20 days ago
What is likelihood-difference in CrowS-Pairs? Is a lower value better for this indicator?
Updated 21 days ago
Wrong (no) instruction for QQP evaluation in GLUE
Updated 21 days ago
Could lm_eval run on AMD gpu?
Updated 22 days ago
Speed up inference problems
Updated 22 days ago19
Different batch sizes give different results
Closed 22 days ago1
Minvera MATH
Updated 22 days ago1
How do I set a directory for HuggingFace weights to be downloaded to?
Closed 22 days ago1
Unnecessary installation: Nemo
Closed 22 days ago1
understanding context length behaviors
Updated 22 days ago
BUG: editable install failing
Updated 23 days ago1
Add better test coverage for models
Updated 23 days ago1
Clarification on API Endpoint: /v1/completions vs /v1/chat/completions
Updated 23 days ago2
After installing lm_eval, can't import and can't run the example lm_eval --model hf etc..
Closed 24 days ago3
after install， it is unknown
Updated 24 days ago3
When using `parallelize=True`, raise Runtime Error: expected all tensors to be on the same device
Updated 24 days ago11
Qwen 7B: RuntimeError: The expanded size of the tensor (32768) must match the existing size (32767) at non-singleton dimension 1.
Updated 25 days ago2
How can I evaluate on an output file?
Updated 25 days ago1
Different in MMLU Scores in lm-eval 0.4.2
Closed 25 days ago9
How to use token logits to get the loglikelihood result?
Updated 25 days ago
Image Dataset
Updated a month ago1
I want to add a custome model named mnn in lm-eval.models,How can I compute the output of the loglikelihood?
Updated a month ago
Issue with HF models not finding logits for log_softmax() during lm_eval.simple_evaluate
Closed a month ago3
(Question) Can we use in-code evaluation through the APIs, instead of using lm-eval in bash terminal?
Closed a month ago2
Add alternate (configurable) launcher / orchestration + sweep functionality
Updated a month ago
Allow registering custom LM implementations without requiring loading and modifying lm_eval code
Updated a month ago1
[Bug] `add_env_info` causes error when not running from base directory
Closed a month ago4
OpenAI Completions unexpected kwarg "until"
Closed a month ago1
Vllm Broken? TypeError: Can't instantiate abstract class VLLM with abstract method prefix_token_id
Closed a month ago
Package conflicts on minerva_math task for OLMo-1B
Updated a month ago3
Bug: `until` as a single `str`
Closed a month ago4
Vllm model is broken
Closed a month ago5
How to randomize fewshots?
Closed a month ago3
Make managing task variants / subversions easier
Updated a month ago
Regarding the question why hellaswag 0 shot has a higher score than 10 shot
Closed a month ago1
Negative perplexity values
Updated a month ago4
Very high perplexity values
Closed a month ago10
Repeated Question in Arabic MMLU
Updated a month ago1
Possibe Implementation for Cross Evaluation
Updated a month ago2
Sanity checking per-sample values from Advanced AI Risk Benchmark?
Closed a month ago3
XNLI weird result with gemma-2b
Updated a month ago1
Make Adding New MCQA Metrics Easier
Updated a month ago
Sanity checking the semantic meaning of "perplexity" in code
Updated a month ago1
(Question) How can I fully utilize the number of cores in my CPU?
Updated a month ago
0.4.1 release does not contain TaskManager
Closed a month ago1
What it the origin of Flan-COT prompt?
Closed a month ago2
How can i evaluate the local model
Closed a month ago5
I modified the structure of the llama, and I want to evaluate this model, how should I proceed?
Closed a month ago1
Expose Configuration Options for Perplexity calculations
Updated a month ago