llm-eval

There are 3 repositories under llm-eval topic.

giskard
Giskard-AI / giskard
🐢 Open-Source Evaluation & Testing for LLMs and ML models
ai-red-team ai-safety ai-security ai-testing ethical-artificial-intelligence fairness-ai llm llm-eval llm-evaluation llm-security llmops ml-safety ml-testing ml-validation mlops model-monitoring rag-evaluation red-team-tools responsible-ai trustworthy-ai
Language:Python 3192
promptfoo / promptfoo
Test your prompts, models, and RAGs. Catch regressions and improve prompt quality. LLM evals for OpenAI, Azure, Anthropic, Gemini, Mistral, Llama, Bedrock, Ollama, and other local & private models with CI/CD integration.
ci ci-cd cicd evaluation evaluation-framework llm llm-eval llm-evaluation llm-evaluation-framework llmops prompt-engineering prompt-testing prompts rag testing
Language:TypeScript 2926
phoenix
Arize-ai / phoenix
AI Observability & Evaluation
ai-monitoring ai-observability ai-roi clustering llm-eval llmops ml-monitoring ml-observability mlops model-monitoring model-observability umap
Language:Jupyter Notebook 2757
uptrain-ai / uptrain
UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
autoevaluation evaluation experimentation hallucination-detection jailbreak-detection llm-eval llm-prompting llm-test llmops machine-learning monitoring openai-evals prompt-engineering root-cause-analysis
Language:Python 2016
athina-ai / athina-evals
Python SDK for running evaluations on LLM generated responses
evaluation evaluation-framework evaluation-metrics llm-eval llm-evaluation llm-evaluation-toolkit llm-ops llmops
Language:Python 136
Re-Align / just-eval
A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.
evaluation gpt4 llm llm-eval llm-evaluation llm-evaluation-toolkit
Language:Python 62
kuk / rulm-sbs2
Бенчмарк сравнивает русские аналоги ChatGPT: Saiga, YandexGPT, Gigachat
llm-eval russian-specific
Language:Jupyter Notebook 54
parea-ai / parea-sdk-py
Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
llm llm-evaluation llm-tools llmops llms-benchmarking llm-eval llm-evaluation-framework llm-evaluation-toolkit prompt-engineering generative-ai good-first-issue metrics
Language:Python 40
ragrank
Auto-Playground / ragrank
🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it understands context, its tone, and more. This helps you see how good your LLM applications are.
evaluation language-model llm llm-eval llmops machine-learning prompt-engineering rag
Language:Python 16
parea-ai / parea-sdk-ts
TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
llm llm-evaluation llm-evaluation-framework llm-evaluation-toolkit llm-tools llms llms-benchmarking llm-eval prompt-engineering
Language:TypeScript 4
Networks-Learning / prediction-powered-ranking
Code for the paper Prediction-Powered Ranking of Large Language Models, Arxiv 2024.
llm-eval llm-evaluation llm-evaluation-framework ranking-algorithm prediction-powered-inference rank-sets
Language:Python 3
harshagrawal523 / GenerativeAgents
Generative agents — computational software agents that simulate believable human behavior and OpenAI LLM models. Our main focus was to develop a game - “Werewolves of Miller’s Hollow”, aiming to replicate human-like behavior.
docker generative-ai llm llm-eval mongodb-atlas openai pygame-gui transformers
Language:Python 2
awesome-software / evals
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
large-language-models llm-eval
Language:Python
kdcyberdude / punjabi-llm-eval
First Punjabi LLM Eval.
eval llm llm-eval panjabi punjabi
Language:Python

llm-eval

Giskard-AI / giskard

promptfoo / promptfoo

Arize-ai / phoenix

uptrain-ai / uptrain

athina-ai / athina-evals

Re-Align / just-eval

kuk / rulm-sbs2

parea-ai / parea-sdk-py

Auto-Playground / ragrank

parea-ai / parea-sdk-ts

Networks-Learning / prediction-powered-ranking

harshagrawal523 / GenerativeAgents

awesome-software / evals

kdcyberdude / punjabi-llm-eval