Giters

promptfoo / promptfoo

Test your prompts, agents, and RAGs. Redteaming, pentesting, vulnerability scanning for LLMs. Improve your app's quality and catch problems. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

Home Page:https://www.promptfoo.dev/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

file-wise `prompts` for `evaluate()` (on javascript) is unusable

YiFanChen99 opened this issue · comments

commented

It seem that the evaluate() did not read the corrsponding given file properly?
For examples, I ran your examples/node-package/full-eval.js.
The results output will be:

{
      ...,
      "provider": {
        "id": "openai:gpt-3.5-turbo"
      },
      "prompt": {
        "raw": "file://prompt.txt",
        "display": "file://prompt.txt"
      },
      "vars": {
        "body": "Hello world"
      },
      "response": {
        "output": "I'm sorry, but I cannot access files on your device. How can I assist you today?",
        "tokenUsage": {
          "total": 31,
          "prompt": 11,
          "completion": 20
        },
        "cached": false,
        "cost": 0.0000355
      },
      ...
}

I have tried with promptfoo ver. 0.51.0 and 0.52.0.

commented

FYR: Seems that the evaluate() are still unsupporting the filepaths? index.ts#L40

commented

Thank you for flagging this. Yes, perhaps we should add handling for file:// prefix. As a workaround for now, you can load the file content in your code.