Bug with GPT Vision ?

[ThibautPV]

Hello,

I don't know whether it's me who has a problem with my code or whether the API isn't yet working with GPT Vision:

Here is my code

$result = OpenAI::chat()->create([
            'model' => 'gpt-4-vision-preview',
            'messages' => [
                [
                    'role' => 'user',
                    'content' => [
                        ['type' => 'text', 'text' => 'Describe image'],
                        ['type' => 'image_url', 'image_url' => "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"],
                    ],
                ]
            ],
            'max_tokens' => 900,
        ]);

And here is the error returned by Laravel:
Undefined array key "finish_reason"

Thanks for you help.

[ThibautPV]

The problem is that Vision doesn't have a "finish_reason" key, but a "finish_details" key.

Here is the return of "$response" in the "create" function

  ["data":"OpenAI\ValueObjects\Transporter\Response":private]=>
  array(6) {
    ["id"]=>
    string(38) "chatcmpl-8IDwyCPKagOULoujiDqvNXssfs1s"
    ["object"]=>
    string(15) "chat.completion"
    ["created"]=>
    int(1699354220)
    ["model"]=>
    string(25) "gpt-4-1106-vision-preview"
    ["usage"]=>
    array(3) {
      ["prompt_tokens"]=>
      int(1114)
      ["completion_tokens"]=>
      int(184)
      ["total_tokens"]=>
      int(1298)
    }
    ["choices"]=>
    array(1) {
      [0]=>
      array(3) {
        ["message"]=>
        array(2) {
          ["role"]=>
          string(9) "assistant"
          ["content"]=>
          string(942) "This is an image of a wooden boardwalk traversing through a lush green field. The boardwalk is constructed with horizontal planks and has a simple design without handrails or barriers. It extends from the foreground into the middle of the image, guiding the viewer's eye towards the horizon. On either side of the walkway, tall green grasses are visible, suggesting the boardwalk is meant to protect the natural environment and provide a path for visitors to enjoy without disturbing the surrounding flora.

The sky is a vibrant blue with wispy white clouds scattered throughout, suggesting a fair weather day. In the distance, there's a line of trees or shrubs, dividing the grassy field from the sky. The natural lighting and composition give the scene a serene and inviting atmosphere, ideal for a peaceful walk or nature observation. The combination of the verdant fields and the expansive sky creates a sense of openness and tranquility."
        }
        ["finish_details"]=>
        array(2) {
          ["type"]=>
          string(4) "stop"
          ["stop"]=>
          string(14) "<|fim_suffix|>"
        }
        ["index"]=>
        int(0)
      }
    }
  }
  ["meta":"OpenAI\ValueObjects\Transporter\Response":private]=>
  object(OpenAI\Responses\Meta\MetaInformation)#331 (4) {
    ["requestId"]=>
    string(32) "22736336820a8a74cf0b5c3e91319a42"
    ["openai"]=>
    object(OpenAI\Responses\Meta\MetaInformationOpenAI)#322 (4) {
      ["model"]=>
      NULL
      ["organization"]=>
      string(9) "my-organization"
      ["version"]=>
      string(10) "2020-10-01"
      ["processingMs"]=>
      int(5843)
    }
    ["requestLimit"]=>
    object(OpenAI\Responses\Meta\MetaInformationRateLimit)#334 (3) {
      ["limit"]=>
      int(100)
      ["remaining"]=>
      int(93)
      ["reset"]=>
      string(11) "1h37m5.049s"
    }
    ["tokenLimit"]=>
    object(OpenAI\Responses\Meta\MetaInformationRateLimit)#323 (3) {
      ["limit"]=>
      int(40000)
      ["remaining"]=>
      int(39095)
      ["reset"]=>
      string(6) "1.357s"
    }
  }
}```

[ThibautPV]

Files to be modified :

• CreateResponse.php
• CreateResponseChoice.php

[Alexander Karlstad]

I am seeing this too. The problem is most likely in https://github.com/openai-php/client though and not the Laravel package I think.

[Sandro Gehri]

Support for vision added in the latest release: https://github.com/openai-php/client/releases/tag/v0.7.8