disler / elm-itv-benchmark

Simple, Opinionated benchmark for testing the viability of Efficient Language Models (ELMs) for personal use cases.

Efficient Language Model Personal Viability Benchmarking

Simple, Opinionated benchmark for testing the viability of Efficient Language Models (ELMs) for personal use cases.

Uses bun, promptfoo, and ollama for a minimalist, cross-platform, local LLM prompt testing & benchmarking experience.

Setup

• Install Bun
• Install Ollama
  • Install llama3 ollama run llama3
  • Install phi3 ollama run phi3
  • Install gemma ollama run gemma
• Setup .env variables
  • cp .env.sample .env
  • Add your OpenAI API key to the .env file
• Install dependencies: bun i
• Run the minimal tests: bun minimal
• Open test viewer: bun view
• Run the ELM-ITV tests: bun elm

Guide

• First, watch the video where we walk through ELMs and this codebase.
• To get started take a look at BENCH__minimal_test_suite/ to get an idea of how to structure a basic test suite.
• Next take a look at the BENCH__efficient_language_models/ test suite to get an idea of how you can setup tests for your own viability tests for ELMs.
• Explore other ollama based models you can test
  • Or OpenAI models
  • Or Anthropic models
  • Or Groq models
• Modify the BENCH__minimal_test_suite/ or BENCH__efficient_language_models/ to suit your needs
• Create a new test with the Create a new test suite script

Folder Structure

• /BENCH__<name of test suite>
  • /prompt.txt - the prompt(s) to test
  • /test.yaml - variables and assertions
  • /promptfooconfig.yaml - llm model config

Scripts

• Create a new test suite: bun run ./scripts/new_prompt_test
• Run a test prompt against a running ollama server bun run ./scripts/ollama_local_model_call

Resources

• Ollama model library
  • https://ollama.com/library
• LMSYS Chatbot Arena Leaderboard
  • https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
• Ollama api.md docs
  • https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-chat-completion
• Promptfoo Ollama Provider
  • https://promptfoo.dev/docs/providers/ollama
• Promptfoo LLM Providers
  • https://www.promptfoo.dev/docs/providers
• Promptfoo Assertions
  • https://www.promptfoo.dev/docs/configuration/expected-outputs/