Using python script assertions file path is not relative to config file

[wild-vish]

I am trying to maintain multiple prompt evaluations in separate folders. Each folder contains promptfoo config files along with python scripts that are used for prompts, variables, and assertions.

When specifying the file path of python scripts for prompts and variables, the path relative to the config file is used. However, when the file path for python assertions is specified, it is relative to where the promptfoo eval is used. This behavior breaks the tests if the evaluation is not always run from a specific relative path to the python assertion scripts.

It would be helpful if the file path for python assertion scripts is relative to the config file specifying the path and not where the evaluation is run from.

Thanks in advance

[wild-vish]

I am not very familiar with JS but maybe it's related to using
runPythonCode (https://github.com/promptfoo/promptfoo/blob/main/src/assertions.ts#L31) which is used only there
Instead of
runPython (e.g. https://github.com/promptfoo/promptfoo/blob/main/src/prompts.ts#L10) which is used in other places I looked into.

[Ian Webster]

Thanks for flagging this. The problem was that basePath was not plumbed through to the assertions logic - something that I've been putting off a while. I've got a fix here #624

[wild-vish]

That's awesome :) thanks for the speedy response and fix

[Ian Webster]

Change is released in 0.51.0