llm-eval

There are 9 repositories under llm-eval topic.

promptfoo / promptfoo
Test your prompts, agents, and RAGs. AI Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.
llm prompt-engineering prompts llmops prompt-testing testing rag evaluation evaluation-framework llm-eval llm-evaluation llm-evaluation-framework ci ci-cd cicd pentesting red-teaming vulnerability-scanners
Language:TypeScript 8998
phoenix
Arize-ai / phoenix
AI Observability & Evaluation
llmops ai-monitoring ai-observability llm-eval aiengineering datasets agents llms prompt-engineering anthropic evals llm-evaluation openai langchain llamaindex smolagents
Language:Jupyter Notebook 7627
giskard-oss
Giskard-AI / giskard-oss
🐢 Open-Source Evaluation & Testing library for LLM Agents
agent-evaluation ai-red-team ai-security ai-testing fairness-ai llm llm-eval llm-evaluation llm-security llmops ml-testing ml-validation mlops rag-evaluation red-team-tools responsible-ai trustworthy-ai
Language:Python 4964
truera / trulens
Evaluation and Tracking for LLM Experiments and AI Agents
agent-evaluation agentops ai-agents ai-monitoring ai-observability evals explainable-ml llm-eval llm-evaluation llmops llms machine-learning neural-networks
Language:Python 2901
datachain
iterative / datachain
ETL, Analytics, Versioning for Unstructured Data
ai cv data-analytics data-wrangling embeddings llm llm-eval machine-learning mlops multimodal
Language:Python 2695
uptrain-ai / uptrain
UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
autoevaluation evaluation experimentation hallucination-detection jailbreak-detection llm-eval llm-prompting llm-test llmops machine-learning monitoring openai-evals prompt-engineering root-cause-analysis
Language:Python 2326
tuui
AI-QL / tuui
A desktop MCP client designed as a tool unitary utility integration, accelerating AI adoption through the Model Context Protocol (MCP) and enabling cross-vendor LLM API orchestration.
agent agentic-ai ai deepseek llm mcp openai-api qwen mcp-client mcp-host model-context-protocol ai-playground mcp-inspector llm-eval prompt testing anthropic claude dxt mcpb
Language:TypeScript 1094
athina-ai / athina-evals
Python SDK for running evaluations on LLM generated responses
evaluation evaluation-framework evaluation-metrics llm-eval llm-evaluation llm-evaluation-toolkit llm-ops llmops
Language:Python 292
Re-Align / just-eval
A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.
evaluation gpt4 llm llm-eval llm-evaluation llm-evaluation-toolkit
Language:Python 88
parea-ai / parea-sdk-py
Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
llm llm-evaluation llm-tools llmops llms-benchmarking llm-eval llm-evaluation-framework llm-evaluation-toolkit prompt-engineering generative-ai good-first-issue metrics
Language:Python 79
kuk / rulm-sbs2
Бенчмарк сравнивает русские аналоги ChatGPT: Saiga, YandexGPT, Gigachat
llm-eval russian-specific
Language:Jupyter Notebook 60
grigio / llm-eval-simple
llm-eval-simple is a simple LLM evaluation framework with intermediate actions and prompt pattern selection
llm llm-eval llm-evaluation-benchmark
Language:Python 52
circle-guard-bench
whitecircle-ai / circle-guard-bench
First-of-its-kind AI benchmark for evaluating the protection capabilities of large language model (LLM) guard systems (guardrails and safeguards)
ai benchmark large-language-model large-language-models llm llm-eval llm-evaluation guardrails safeguard benchmarking guardrail jailbreak llm-as-a-judge llm-jailbreaks llm-security
Language:Python 44
multinear / multinear
Develop reliable AI apps
evaluation llm llm-eval llm-evaluation llm-evaluation-framework llms llms-benchmarking reliability
Language:Python 43
ragrank
izam-mohammed / ragrank
🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it understands context, its tone, and more. This helps you see how good your LLM applications are.
evaluation language-model llm llm-eval llmops machine-learning prompt-engineering rag
Language:Python 42
alan-turing-institute / prompto
An open source library for asynchronous querying of LLM endpoints
hut23 large-language-models llm-eval llm-evaluation llms transformers deep-learning machine-learning natural-language-processing nlp python transformer
Language:Python 32
genia-dev / vibraniumdome
LLM Security Platform.
adversarial-attacks chatgpt large-language-model llm openai prompt-injection prompt-injection-tool security llm-agent llm-security llmops prompt-engineering prompts llm-framework llm-inference llm-serving llm-firewall llm-evaluation llm-eval
Language:Python 23
Supahands / llm-comparison-backend
This is an opensource project allowing you to compare two LLM's head to head with a given prompt, this section will be regarding the backend of this project, allowing for llm api's to be incorporated and used in the front-end
ai chatgpt llm llm-api llm-comparison llm-eval
Language:Python 20
honeyhiveai / realign
Realign is a testing and simulation framework for AI applications.
ai alignment evaluation llms prompt-engineering red-teaming simulation aiengineering llm-eval llm-evaluation llm-evaluation-framework llmops rag
Language:Python 17
amplifying-ai / ai-product-bench
dataset evals llm-eval
Language:HTML 11
ollama-multirun
attogram / ollama-multirun
Run a prompt against all, or some, of your models running on Ollama. Creates web pages with the output, performance statistics and model info. All in a single Bash shell script.
ai ai-evaluation-tools attogram-project bash-script llm-eval llm-evaluation llm-evaluation-metrics ollama ollama-app ollama-interface static-site-generator
Language:Shell 11
Networks-Learning / prediction-powered-ranking
Code for "Prediction-Powered Ranking of Large Language Models", NeurIPS 2024.
llm-eval llm-evaluation llm-evaluation-framework prediction-powered-inference rank-sets ranking-algorithm
Language:Jupyter Notebook 9
prompt-foundry / python-sdk
The prompt engineering, prompt management, and prompt evaluation tool for Python
llm llm-eval llm-evaluation open-ai prompt-engineering prompt-evaluation prompt-management python python3
Language:Python 8
IAAR-Shanghai / GuessArena
[ACL 2025] GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning
chatgpt deepseek llm-eval openai qwen guessarena benchmark evaluation-framework large-language-models reliable-evaluation gamearena knowledge-evaluation domain-specific-eval reasoning-evaluation
Language:Python 7
pyladiesams / eval-llm-based-apps-jan2025
Create an evaluation framework for your LLM based app. Incorporate it into your test suite. Lay the monitoring foundation.
llm llm-eval llm-evals llm-evaluation-framework llm-evaluation-metrics llm-monitoring llm-test llm-testing llmops llms workshop
Language:Jupyter Notebook 7
prompt-foundry / typescript-sdk
The prompt engineering, prompt management, and prompt evaluation tool for TypeScript, JavaScript, and NodeJS.
llm-test prompt-engineering prompt-management prompt-testing typescript llm-eval llm-evaluation open-ai prompt-evaluation prompt-manager gpt gpt-3 gpt-4 llm llm-ops llmops
Language:TypeScript 6
harshagrawal523 / GenerativeAgents
Generative agents — computational software agents that simulate believable human behavior and OpenAI LLM models. Our main focus was to develop a game - “Werewolves of Miller’s Hollow”, aiming to replicate human-like behavior.
docker generative-ai llm llm-eval mongodb-atlas openai pygame-gui transformers
Language:Python 5
harlev / eva-l
LLM Evaluation Framework
llm llm-eval llm-evaluation llms
Language:Python 4
llm-coup
khoj-ai / llm-coup
Let LLMs play coup with each other and see who's the best at deception & strategy
ai artificial-intelligence coup deception environment games llm-benchmarking llm-eval llm-evaluation llms simulation
Language:TypeScript 4
parea-ai / parea-sdk-ts
TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
llm llm-evaluation llm-evaluation-framework llm-evaluation-toolkit llm-tools llms llms-benchmarking llm-eval prompt-engineering
Language:TypeScript 4
yukinagae / genkitx-promptfoo
Community Plugin for Genkit to use Promptfoo
ai evaluation evaluation-framework firebase genkit genkit-plugin genkitx llm llm-eval llm-evaluation llm-evaluation-framework llmops plugin prompt prompt-testing promptfoo prompts testing
Language:TypeScript 4
jaaack-wang / multi-problem-eval-llm
Evaluating LLMs with Multiple Problems at once: A New Paradigm for Probing LLM Capabilities
explainable-ai large-language-models llm llm-eval llm-evaluation-framework llm-prompting
Language:Jupyter Notebook 3
yuzu-ai / ShinRakuda
Shin Rakuda is a comprehensive framework for evaluating and benchmarking Japanese large language models, offering researchers and developers a flexible toolkit for assessing LLM performance across diverse datasets.
llm llm-eval llm-evaluation llm-evaluation-framework japanese
Language:Python 3
artefactop / promptdev
A prompt evaluation framework that provides comprehensive testing for AI agents across multiple providers.
prompt prompt-engineering prompt-toolkit evaluation-framework llm llm-eval llm-evaluation llm-evaluation-framework red-team testing ci-cd
Language:Python 2
genia-dev / vibraniumdome-docs
LLM Security Platform Docs
adverarial-attacks chatgpt large-language-model llm llm-agent llm-eval llm-evaluation llm-firewall llm-framework llm-inference llm-security llm-serving llmops openai prompt-engineering prompt-injection prompt-injection-tool prompts security
Language:MDX 2
llm-SugarScape
yukincom / llm-SugarScape
Multi-agent simulation using LLMs. Agents autonomously decide actions for survival, reproduction, and social behavior in a grid world.This project aims to replicate a paper published in 2025 (arXiv:2508.12920).
agent-based-modeling grok grok-api llm python simulation sugarscape xai-api aisafety ai-testing llm-testing llm-eval llm-evaluation alignment
Language:Python 1