promptfoo / promptfoo

Test your prompts, agents, and RAGs. Use LLM evals to improve your app's quality and catch problems. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

Home Page:https://www.promptfoo.dev/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

defaultTest property gets deleted from .yaml file when it is edited through the web ui

remixer-dec opened this issue · comments

https://promptfoo.dev/docs/configuration/guide/#default-test-cases
when I try to add this property to the config file in the UI via the "edit yaml" button, it just disappears after clicking "save"

Thanks for catching, fixed in #629

Thanks for a quick fix! I wanted to try multi-attempt evaluations, but seems like "evaluateOptions" also disappears from the config if it is edited in the UI https://promptfoo.dev/docs/configuration/reference#config

added this one too 👍

Change is released in 0.51.0

Hi, I checked out the updated version and these options are indeed in the UI, but have no effect and are ignored during the evalutation.

Hi, I checked out the updated version and these options are indeed in the UI, but have no effect and are ignored during the evalutation.

#829