Question: Using a python or javascript prompt function when using promptfoo.evaluate()

[Anthony Ivan]

Hi, I am trying to understand on whether it is possible to use a prompt function defined within a separate python/javascript file as the prompts value that is passed to promptfoo.evaluate(). I am not seeing that functionality in the documentations, and I would like to use promptfoo.evaluate() directly instead as my test data will only be available in memory (not as local files).

[Ian Webster]

You can specify a separate python/js file using a path (e.g. file://path/to/file.py), which should follow the prompt functions documentation.

Or, when calling evaluate directly with node, you can supply a function directly in the prompts array that you pass to evaluate:

prompts: [
      ({ vars }) => {
        // Call your own code here
        const foo = require('foo');
        finalResult = await foo(vars);
        return finalResult;
      },
    ],

[Anthony Ivan]

Right now what I am seeing is that when I tried adding file://path/to/file.js to the prompts argument in evaluate(), the prompt used would just be the path instead of the content from the prompt function js file

[Anthony Ivan]

ok seems like I can proceed with the second method by importing the prompt function content and then adding it to prompts.

[Ian Webster]

👍

Looks like I misremembered the prompt function/file support for the node package, it was never actually added. I've PRed that here: #668