"[FAIL] Error: Invalid provider definition for output type 'text'" when using model-graded metrics

[jkng10]

Hey all,

I'm using promptfoo 0.50.1 and would like to use model-graded metrics, wondering if something is wrong with my config.

Referring to #323 (comment) - I am using a text-embedding-ada-002 deployment, deployed via Azure OpenAI.

Below is my config

...
- type: llm-rubric
      value: is not apologetic
      provider:
        - id: azureopenai:embeddings:text-embedding-ada-002
          config:
            temperature: 0
            max_tokens: 128
            ...
...

When running promptfoo eval against the above, I get a

[Ian Webster]

It should be a single object, not an array:

- type: llm-rubric
      value: is not apologetic
      provider:
        id: azureopenai:embeddings:text-embedding-ada-002
        config:
          temperature: 0
          max_tokens: 128
          #...

[jkng10]

Thanks @typpo, appreciate the quick feedback, this error is gone! However now getting the below:

Error: Not implemented
    at AzureOpenAiEmbeddingProvider.callApi (...\npm\node_modules\promptfoo\dist\src\providers\azureopenai.js:76:15)
    at matchesLlmRubric (...\npm\node_modules\promptfoo\dist\src\matchers.js:227:38)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async runAssertion (...\npm\node_modules\promptfoo\dist\src\assertions.js:667:17)
    at async ...\npm\node_modules\promptfoo\dist\src\assertions.js:82:24

Is there any difference in between using

id: azureopenai:embedding:text-embedding-ada-002
as per https://www.promptfoo.dev/docs/configuration/expected-outputs/similar/

or

id: azureopenai:embeddings:text-embeddings-ada-002
as per #323 (comment)

[Ian Webster]

Ah, I didn't notice you were using llm-rubric, but that requires a text generation provider (like gpt-4), not an embedding provider. More details here

[jkng10]

Thanks @typpo ! It works now!

One last quick question - can provider details/config be also defined more globally on "assert" level?

I'd like to use the gpt-4 provider for all subsequent llm-rubric and context-* types and currently specify same provider details/config (e.g. apiKey and apiHost) again for each "type", which looks redundant.

currently:

assert:
  - type: llm-rubric
    value: does not describe self as an AI or chat assistant
    threshold: 0.85
    provider:
      id: azureopenai:chat:gpt-4
      config:
        apiKey: <key>
        apiHost: <url>
  - type: llm-rubric
    value: is not apologetic
    threshold: 0.85
    provider:
      id: azureopenai:chat:gpt-4
      config:
        apiKey: <key>
        apiHost: <url>
  - type: llm-rubric
    value: is confident and does not guess or speculate
    threshold: 0.85
    provider:
      id: azureopenai:chat:gpt-4
      config:
        apiKey: <key>
        apiHost: <url>

[Ian Webster]

Yep, you can put it on the defaultTest object. See here: https://promptfoo.dev/docs/configuration/expected-outputs/model-graded/#multiple-graders

[jkng10]

Works like a charm, thanks @typpo!