tmc / langchaingo

LangChain for Go, the easiest way to write LLM-based programs in Go

Home Page:https://tmc.github.io/langchaingo/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Unable to send BinaryPart to OpenAI Completions API

pilsnerbeer opened this issue · comments

Trying to directly send a image/png to OpenAI completions API.
(Model: Gpt-4o / Mini)

Snippet:

		if imageData != nil {
			historyWithBinary = append(historyWithBinary, llms.MessageContent{
				Role: llms.ChatMessageTypeHuman,
				Parts: []llms.ContentPart{
					llms.BinaryPart("image/png", imageData),
				},
			})
		}

		choices, err := client.GenerateContent(context.Background(), historyWithBinary, llms.WithTools([]llms.Tool{FileWriteTool}))

Returns Error:

Error generating content: API returned unexpected status code: 400: Invalid value:
'binary'. Supported values are: 'text', 'image_ url', and 'audio_url'.

This seems to be only problematic with OpenAI. Gemini and Ollama worked fine when i tested it with the same snippet.
According to https://platform.openai.com/docs/guides/vision doc sending images directly to vision API shoudl be possible so the error is not clear to me