This Github Action uses promptfoo to produce a before/after view of edit prompts.
When you change a prompt, an eval will automatically be posted on the pull request:
![pull request llm eval](https://private-user-images.githubusercontent.com/310310/248143799-ec75fb39-c6b1-4395-9e41-6d66a7bf8657.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI2Njc5MzcsIm5iZiI6MTcxMjY2NzYzNywicGF0aCI6Ii8zMTAzMTAvMjQ4MTQzNzk5LWVjNzVmYjM5LWM2YjEtNDM5NS05ZTQxLTZkNjZhN2JmODY1Ny5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQwOVQxMzAwMzdaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zNzZiMTkzY2E5ODQzZGYzYWVmYmNkMzZlNjY5ZDY4ZjgzNzFiYTU3NzM0MTc2Mjc2NzExODJhOGI0NTUwM2M3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.L-u1JCjsgHfLGiayyJu-XUhoxlI-S_zwv23PXzgOlcI)
The provided link opens the promptfoo web viewer, which allows you to interactively explore the before vs. after:
![promptfoo web viewer](https://private-user-images.githubusercontent.com/310310/248143860-d0ef0497-0c1a-4886-b115-1ee92680891b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI2Njc5MzcsIm5iZiI6MTcxMjY2NzYzNywicGF0aCI6Ii8zMTAzMTAvMjQ4MTQzODYwLWQwZWYwNDk3LTBjMWEtNDg4Ni1iMTE1LTFlZTkyNjgwODkxYi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQwOVQxMzAwMzdaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1jYTNmNTI1ZjAzOGY4MDJhYWUxNzE4M2RiYjQ5YmU5NzAxOTgwNmNmNzUwNjMxYjVmN2QyNjRmZTViMTE2OWJkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.YYY_bxeC4X_XqsVKEGHF8lufBgWVxv0D7KBLUDNbrD4)
The action can be configured using the following inputs:
Parameter | Description | Required |
---|---|---|
github-token |
The Github token. Used to authenticate requests to the Github API. | Yes |
prompts |
The glob patterns for the prompt files. These patterns are used to find the prompt files that the action should evaluate. | Yes |
config |
The path to the configuration file. This file contains settings for the action. | Yes |
cache-path |
The path to the cache. This is where the action stores temporary data. | No |
promptfoo-version |
The version of promptfoo to use. Defaults to latest |
No |
The following API key parameters are supported:
Parameter | Description |
---|---|
openai-api-key |
The API key for OpenAI. Used to authenticate requests to the OpenAI API. |
azure-api-key |
The API key for Azure OpenAI. Used to authenticate requests to the Azure OpenAI API. |
anthropic-api-key |
The API key for Anthropic. Used to authenticate requests to the Anthropic API. |
huggingface-api-key |
The API key for Hugging Face. Used to authenticate requests to the Hugging Face API. |
aws-access-key-id |
The AWS access key ID. Used to authenticate requests to AWS services. |
aws-secret-access-key |
The AWS secret access key. Used to authenticate requests to AWS services. |
replicate-api-key |
The API key for Replicate. Used to authenticate requests to the Replicate API. |
palm-api-key |
The API key for Palm. Used to authenticate requests to the Palm API. |
vertex-api-key |
The API key for Vertex. Used to authenticate requests to the Vertex AI API. |
Here is a generic Github Action configuration using "typpo/promptfoo-action@v1" with a cache step:
name: 'Prompt Evaluation'
on:
pull_request:
paths:
- 'prompts/**'
jobs:
evaluate:
runs-on: ubuntu-latest
permissions:
contents: read # Required for actions/checkout
pull-requests: write # Ability to post comments on Pull Requests
steps:
# Required for promptfoo-action's git usage
- uses: actions/checkout@v4
# This cache is optional, but you'll save money and time by setting it up!
- name: Set up promptfoo cache
uses: actions/cache@v3
with:
path: ~/.cache/promptfoo
key: ${{ runner.os }}-promptfoo-v1
restore-keys: |
${{ runner.os }}-promptfoo-
- name: Run promptfoo evaluation
uses: promptfoo/promptfoo-action@v1
with:
openai-api-key: ${{ secrets.OPENAI_API_KEY }}
github-token: ${{ secrets.GITHUB_TOKEN }}
prompts: 'prompts/**/*.json'
config: 'prompts/promptfooconfig.yaml'
cache-path: ~/.cache/promptfoo
If you are using an OpenAI model, remember to create the secret in Repository Settings > Secrets and Variables > Actions > New repository secret.
For more information on how to set up the promptfoo config, see documentation.