llms-benchmarking

There are 0 repository under llms-benchmarking topic.

steel-dev / awesome-web-agents
🔥 A list of tools, frameworks, and resources for building AI web agents
ai llms llms-benchmarking ai-agents browser-automation
797
karthikv792 / LLMs-Planning
An extensible benchmark for evaluating large language models on planning
planning benchmark-suite llms llms-benchmarking llms-reasoning pddl llms-planning
Language:PDDL 405
bboylyg / BackdoorLLM
BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models
backdoor llms llms-benchmarking attack defense
Language:Python 213
JonathanChavezTamales / LLMStats
A comprehensive set of LLM benchmark scores and provider prices.
llm llm-agents llm-evaluation llmops llms-benchmarking
Language:JavaScript 178
ChemFoundationModels / ChemLLMBench
What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
benchmark chemistry large-language-models llm nlp ai4science llms-benchmarking
Language:Jupyter Notebook 159
lechmazur / nyt-connections
Benchmark that evaluates LLMs using 759 NYT Connections puzzles extended with extra trick words
benchmark evaluation llm llms-benchmarking puzzles reasoning testing claude gemini-pro gpt-5 grok4
Language:Python 139
lerogo / MMGenBench
Official repository of MMGenBench
llms-benchmarking mmgenbench mllm
Language:Python 120
parea-ai / parea-sdk-py
Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
llm llm-evaluation llm-tools llmops llms-benchmarking llm-eval llm-evaluation-framework llm-evaluation-toolkit prompt-engineering generative-ai good-first-issue metrics
Language:Python 78
lamalab-org / chembench
How good are LLMs at chemistry?
benchmark chemistry llm llms llms-benchmarking machine-learning materials-science safety
Language:Python 70
FSoft-AI4Code / XMainframe
Language Model for Mainframe Modernization
cobol code-summarization codellm llms-benchmarking mainframe migration
Language:Python 57
multinear / multinear
Develop reliable AI apps
evaluation llm llms llms-benchmarking reliability llm-eval llm-evaluation llm-evaluation-framework
Language:Python 44
lechmazur / generalization
Thematic Generalization Benchmark: measures how effectively various LLMs can infer a narrow or specific "theme" (category/rule) from a small set of examples and anti-examples, then detect which item truly fits that theme among a collection of misleading candidates.
benchmark evaluation generalization llm llms llms-benchmarking sonnet3-7 gpt-4-5
43
SuperBruceJia / Awesome-Mixture-of-Experts
Awesome Mixture of Experts (MoE): A Curated List of Mixture of Experts (MoE) and Mixture of Multimodal Experts (MoME)
large-language-models large-vision-language-models mixture-of-experts moe multimodal-learning mixtrure-of-multimodal-experts artificial-intelligence expert-network foundation-models gating-network large-language-model llms llms-benchmarking llms-reasoning mome sparse sparse-mixture-of-experts sparse-mixture-of-multimodal-experts sparse-moe load-balancing
39
RaptorMai / CompBench
[NeurIPS'25] MLLM-CompBench evaluates the comparative reasoning of MLLMs with 40K image pairs and questions across 8 dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. CompBench covers diverse visual domains, including animals, fashion, sports, and scenes
benchmark human-annotation large-language-models multimodal-deep-learning multimodal-large-language-models reasoning evaluation-llms foundation-models llms llms-benchmarking vision-and-language vision-language-model neurips-2024
Language:Jupyter Notebook 36
amazon-science / llm-code-preference
Training and Benchmarking LLMs for Code Preference.
code-generation llm-evaluation llm-training llms-benchmarking
Language:Python 35
epfl-dlab / cc_flows
The data and implementation for the experiments in the paper "Flows: Building Blocks of Reasoning and Collaborating AI".
agents ai aiflows competitive-coding competitive-programming competitive-programming-contests llms llms-benchmarking llms-reasoning
Language:Python 31
rajpurkarlab / craft-md
clinical-llm conversational-ai llms-benchmarking multiturn-conversations
Language:Python 30
declare-lab / resta
Restore safety in fine-tuned language models through task arithmetic
alignment alignment-algorithm llm llm-safety llm-safety-benchmark llms llms-benchmarking safety
Language:Python 28
Laoyu84 / 4onebench
A minimalist benchmarking tool designed to test the routine-generation capabilities of LLMs.
agents large-language-models llms-benchmarking
Language:Python 25
minnesotanlp / cobbler
Code and data for Koo et al's ACL 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"
bias evaluation llm nlp bias-detection llm-as-a-judge llm-as-evaluator llm-as-judge llm-evaluation llms llms-benchmarking
Language:Jupyter Notebook 21
gautierdag / plancraft
Plancraft is a minecraft environment and agent suite to test planning capabilities in LLMs
agentic-ai interactive-environments llm llms-benchmarking minecraft multimodal-large-language-models planning
Language:Python 17
logikon-ai / cot-eval
A framework for evaluating the effectiveness of chain-of-thought reasoning in language models.
chain-of-thought gen-ai leaderboard llm llms-benchmarking llms-reasoning
Language:Jupyter Notebook 16
Paulescu / text-embedding-evaluation
Join 15k builders to the Real-World ML Newsletter ⬇️⬇️⬇️
embeddings llms llms-benchmarking machine-learning
Language:Python 16
Kartik-3004 / facexbench
FaceXBench: Evaluating Multimodal LLMs on Face Understanding
age-estimation attributes crowd-counting deepfake-detection face face-antispoofing face-perception face-recognition face-segmentation facial-expression-recognition gender-classification headpose-estimation llms llms-benchmarking multimodal multimodal-llms race-detection
Language:Python 14
microsoft / private-benchmarking
A platform that enables users to perform private benchmarking of machine learning models. The platform facilitates the evaluation of models based on different trust levels between the model owners and the dataset owners.
benchmarking inference llms-benchmarking mpc private private-benchmarking secure ezpc large-language-models contamination platform confidential-computing trusted-execution-environment
Language:Python 10
nachoDRT / MERIT-Dataset
The MERIT Dataset is a fully synthetic, labeled dataset created for training and benchmarking LLMs on Visually Rich Document Understanding tasks. It is also designed to help detect biases and improve interpretability in LLMs, where we are actively working. This repository is actively maintained, and new features are continuously being added.
biases layoutlm layoutlmv2 layoutlmv3 layoutxlm llms-benchmarking synthetic-dataset synthetic-dataset-generation token-classification donut idefics2
Language:Python 10
PrincySinghal / Html-code-generation-from-LLM
Fine-Tuning and Evaluating a Falcon 7B Model for generating HTML code from input prompts.
code-generation fine-tuning-llm machine-learning llms llms-benchmarking
Language:Jupyter Notebook 10
cosmaadrian / romath
Official repository for "RoMath: A Mathematical Reasoning Benchmark in 🇷🇴 Romanian 🇷🇴"
dataset llms-benchmarking mathematics romanian
Language:Python 8
microsoft / MEGAVERSE
Official Codebase for MEGAVERSE: (published in ACL: NAACL 2024)
llms-benchmarking multilingual
Language:Python 8
whisk / leetgptsolver
Benchmarking LLMs (large language models) on leetcode algorithmic challenges
leetcode llms-benchmarking
Language:Go 6
desboisGIT / OpenPrompt
OpenPromptBank is an AI prompt library platform where users can explore, rank, and contribute AI prompts categorized by various topics. This platform features a searchable library, community-driven rankings, prompt performance benchmarks, and user profiles.
ai djnago llms-benchmarking prompt-engineering prompt-generator prompt-toolkit prompt-tuning react llms-prompting
Language:Python 5
dippatel1994 / Large-Language-Models-Evaluation-Benchmarks-Collection
This repository contains a list of benchmarks used by big orgs to evaluate their LLMs.
benchmarks large-language-models llm llms llms-benchmarking
4
lwachowiak / LLMs-for-Social-Robotics
Code and data for our IROS paper: "Are Large Language Models Aligned with People's Social Intuitions for Human–Robot Interactions?"
alignment hri llms llms-benchmarking social-robotics value-alignment vlm
Language:Jupyter Notebook 4
parea-ai / parea-sdk-ts
TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
llm llm-evaluation llm-evaluation-framework llm-evaluation-toolkit llm-tools llms llms-benchmarking llm-eval prompt-engineering
Language:TypeScript 4
stair-lab / melt
Multilingual Evaluation Toolkits
llm-evaluation-framework llms-benchmarking multilingual
Language:Python 4
matus-pikuliak / genderbench
GenderBench - Evaluation suite for gender biases in LLMs
fairness-ai gender-bias llms llms-benchmarking
Language:Python 3

llms-benchmarking

steel-dev / awesome-web-agents

karthikv792 / LLMs-Planning

bboylyg / BackdoorLLM

JonathanChavezTamales / LLMStats

ChemFoundationModels / ChemLLMBench

lechmazur / nyt-connections

lerogo / MMGenBench

parea-ai / parea-sdk-py

lamalab-org / chembench

FSoft-AI4Code / XMainframe

multinear / multinear

lechmazur / generalization

SuperBruceJia / Awesome-Mixture-of-Experts

RaptorMai / CompBench

amazon-science / llm-code-preference

epfl-dlab / cc_flows

rajpurkarlab / craft-md

declare-lab / resta

Laoyu84 / 4onebench

minnesotanlp / cobbler

gautierdag / plancraft

logikon-ai / cot-eval

Paulescu / text-embedding-evaluation

Kartik-3004 / facexbench

microsoft / private-benchmarking

nachoDRT / MERIT-Dataset

PrincySinghal / Html-code-generation-from-LLM

cosmaadrian / romath

microsoft / MEGAVERSE

whisk / leetgptsolver

desboisGIT / OpenPrompt

dippatel1994 / Large-Language-Models-Evaluation-Benchmarks-Collection

lwachowiak / LLMs-for-Social-Robotics

parea-ai / parea-sdk-ts

stair-lab / melt

matus-pikuliak / genderbench