llm-evaluation-framework

There are 1 repository under llm-evaluation-framework topic.

promptfoo / promptfoo
Test your prompts, agents, and RAGs. Use LLM evals to improve your app's quality and catch problems. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.
ci ci-cd cicd evaluation evaluation-framework llm llm-eval llm-evaluation llm-evaluation-framework llmops prompt-engineering prompt-testing prompts rag testing
Language:TypeScript 3171
confident-ai / deepeval
The LLM Evaluation Framework
evaluation-framework evaluation-metrics llm-evaluation llm-evaluation-framework llm-evaluation-metrics
Language:Python 2046
parea-ai / parea-sdk-py
Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
llm llm-evaluation llm-tools llmops llms-benchmarking llm-eval llm-evaluation-framework llm-evaluation-toolkit prompt-engineering generative-ai good-first-issue metrics
Language:Python 41
aws-samples / fm-leaderboarder
FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts
llm-benchmarking llm-evaluation llm-evaluation-framework
Language:Python 11
Networks-Learning / prediction-powered-ranking
Code for the paper Prediction-Powered Ranking of Large Language Models, Arxiv 2024.
llm-eval llm-evaluation llm-evaluation-framework prediction-powered-inference rank-sets ranking-algorithm
Language:Python 4
parea-ai / parea-sdk-ts
TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
llm llm-evaluation llm-evaluation-framework llm-evaluation-toolkit llm-tools llms llms-benchmarking llm-eval prompt-engineering
Language:TypeScript 4
Psycoy / MixEval
MixEval, a ground-truth-based dynamic benchmark derived from off-the-shelf benchmark mixtures, which evaluates LLMs with a highly capable model ranking (i.e., 0.96 correlation with Chatbot Arena) while running locally and quickly (6% the time and cost of running MMLU), with its queries being stably updated every month to avoid contamination.
benchmark benchmark-mixture benchmarking-framework benchmarking-suite deep-learning dynamic-benchmark evaluation evaluation-framework foundation-models large-language-model large-language-models large-multimodal-models llm-evaluation llm-evaluation-framework llm-inference llm-training machine-learning
Language:Python 3
stair-lab / villm-eval
Evaluation of Language Models in Non-English Languages
llm-evaluation-framework llms-benchmarking
Language:Python 1
nagababumo / Building-and-Evaluating-Advanced-RAG
llamaindex llm-evaluation llm-evaluation-framework python rag retrieval-augmented-generation
Language:Jupyter Notebook

llm-evaluation-framework

promptfoo / promptfoo

confident-ai / deepeval

parea-ai / parea-sdk-py

aws-samples / fm-leaderboarder

Networks-Learning / prediction-powered-ranking

parea-ai / parea-sdk-ts

Psycoy / MixEval

stair-lab / villm-eval

nagababumo / Building-and-Evaluating-Advanced-RAG