promptfoo / promptfoo

Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

Home Page:https://promptfoo.dev

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Python scripts in prompts does not work with anthropic as provider

harsh-writesonic opened this issue · comments

Stacktrace:

/usr/local/lib/node_modules/promptfoo/dist/src/prompts.js:225
                            args: [JSON.stringify(context)],
                                        ^

TypeError: Converting circular structure to JSON
    --> starting at object with constructor 'Anthropic'
    |     property 'completions' -> object with constructor 'Completions'
    --- property '_client' closes the circle
    at JSON.stringify (<anonymous>)

promptfooconfig.yaml

prompts:
  - file://prompt.py

providers:
  - openai:gpt-4o
  - anthropic:messages:claude-3-opus-20240229

Nodejs version - v20.12.2
Promptfoo version - 0.58.1
Please let me know if it is reproducible for you.

Thanks for flagging! Fix in #784

As a temporary workaround, you can switch to using a specific function in the python file, e.g. prompt.py:my_prompt. That should work alright in the current version.

Thanks for quick fix.

Released in 0.59.0!