Feature Request: Loading data for evaluate() from data loaded into memory

[Anthony Ivan]

Hi team,

I was reading through the docs and it seems like today promptfoo's evaluate() does not support the use of data that is already in memory (when running the node library). I have a use case where I would like to evaluate hundreds of new question answers on a daily basis through promptfoo, and I am not sure if that is something that is doable today.

[Ian Webster]

Hi @anthonyivn2, can you clarify what you're looking for? It seems you could do something like this and pass in the data you've already loaded separately:

const results = await promptfoo.evaluate(
  {
    prompts: ["First prompt with {{question}} and {{answer}}', "..."],
    providers: ["openai:gpt-3.5-turbo"],
    tests: [
      {
        vars: {
          question: "...",
          answer: "...",
        },
      },
      {
        vars: {
          question: "...",
          answer: "...",
        },
      },
    ],
    writeLatestResults: true, // write results to disk so they can be viewed in web viewer
  },
  {
    maxConcurrency: 2,
  },
);

[Anthony Ivan]

@typpo Ok so does evaluate() accepts different input formats other than the TestSuiteConfiguration object as specified in the Node Package Usage doc? That example you showed me above would definitely work for my use case if I can do that with evaluate().

Also I would like to confirm with you on another thing. Would writeLatestResults: true makes it so that I share the data (I don't want to share the data), or does it only write the result to my local disk so that its viewable in the web viewer?

[Ian Webster]

Yes, looks like that type definition in the doc is a bit out of date. I'll get it updated. Here is the actual type used in the code: https://github.com/promptfoo/promptfoo/blob/0337f1343a3fb21df457a1ced7c76734dc32f16f/src/types.ts#L560 Also, I can confirm that writeLatestResults only writes locally. Let me know if you have any other questions!

…

On Fri, Apr 12, 2024, 7:46 PM Anthony Ivan ***@***.***> wrote: @typpo <https://github.com/typpo> Ok so does evaluate() accepts inputs other than TestSuiteConfiguration object <https://www.promptfoo.dev/docs/configuration/reference#testsuiteconfiguration> as specified in the Node Package Usage doc <https://www.promptfoo.dev/docs/usage/node-package#usage>? To confirm, writeLatestResults: true is different from sharing the data right? — Reply to this email directly, view it on GitHub <#665 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACLYJW5SJQ75F5CUGDEYW3Y5CMCDAVCNFSM6AAAAABGE6BDBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJTGA3DQMBZHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Anthony Ivan]

Ok this should work for my use case. I will close this issue, thank you for the response!