[JS] Support model streaming in JSON mode

[Pavel Jbanov]

Right now when you stream output from a generate call in JSON mode you still get raw model output that is hard to work with.

Ex:

const GameCharactersSchema = z.object({
  characters: z
    .array(
      z
        .object({
          name: z.string().describe('Name of a character'),
          abilities: z
            .array(z.string())
            .describe('Various abilities (strength, magic, archery, etc.)'),
        })
        .describe('Game character')
    )
    .describe('Characters'),
});

  const response = await generate({
    model: gemini15Flash,
    output: {
      schema: GameCharactersSchema,
    },
    prompt: `Respond as JSON only. Generate ${count} different RPG game characters.`,
    streamingCallback,
  });
  return response.text();

Output

{"index":0,"content":[{"text":"```"}]}
{"index":0,"content":[{"text":"json\n{\n  \"characters\": [\n    {\n      \"name"}]}
{"index":0,"content":[{"text":"\": \"Anya the Swift\",\n      \"abilities\": [\n        \""}]}
{"index":0,"content":[{"text":"Stealth\",\n        \"Swordsmanship\",\n        \"Acrobatics\",\n        \"Lockpicking\"\n      ],\n      \"background\": \"A"}]}
{"index":0,"content":[{"text":" cunning thief who roams the shadows, seeking adventure and riches.\",\n      \"alignment\": \"Chaotic Good\"\n    },\n    {\n      \""}]}
{"index":0,"content":[{"text":"name\": \"Grimbold the Stout\",\n      \"abilities\": [\n        \"Strength\",\n        \"Axe Mastery\",\n        \"Endurance\",\n        \"Intimidation\"\n      ],\n      \"background\": \"A"}]}
{"index":0,"content":[{"text":" seasoned warrior with a heart of gold, who fights for what he believes is right.\",\n      \"alignment\": \"Lawful Good\"\n    },\n    {\n      \"name\": \"Elara the Wise\",\n      \"abilities"}]}
{"index":0,"content":[{"text":"\": [\n        \"Arcane Magic\",\n        \"Healing\",\n        \"Divination\",\n        \"Lore\"\n      ],\n      \"background\": \"A powerful sorceress seeking knowledge and understanding of the world.\",\n      \"alignment\": \"Neutral Good\"\n    }\n  ]\n}\n```"}]}
{"index":0,"content":[{"text":""}]}
{"name":"93c286e9-d5be-4e1c-81df-06af117ad125","done":true,"result":{"response":"```json\n{\n  \"characters\": [\n    {\n      \"name\": \"Anya the Swift\",\n      \"abilities\": [\n        \"Stealth\",\n        \"Swordsmanship\",\n        \"Acrobatics\",\n        \"Lockpicking\"\n      ],\n      \"background\": \"A cunning thief who roams the shadows, seeking adventure and riches.\",\n      \"alignment\": \"Chaotic Good\"\n    },\n    {\n      \"name\": \"Grimbold the Stout\",\n      \"abilities\": [\n        \"Strength\",\n        \"Axe Mastery\",\n        \"Endurance\",\n        \"Intimidation\"\n      ],\n      \"background\": \"A seasoned warrior with a heart of gold, who fights for what he believes is right.\",\n      \"alignment\": \"Lawful Good\"\n    },\n    {\n      \"name\": \"Elara the Wise\",\n      \"abilities\": [\n        \"Arcane Magic\",\n        \"Healing\",\n        \"Divination\",\n        \"Lore\"\n      ],\n      \"background\": \"A powerful sorceress seeking knowledge and understanding of the world.\",\n      \"alignment\": \"Neutral Good\"\n    }\n  ]\n}\n```"}}

It would be much better if generate returned partial JSON, like this:

{"characters":[{"name":"Aella Stormrider"}]}
{"characters":[{"name":"Aella Stormrider","abilities":["Lightning Magic","Swordsmanship","Eagle Vision"]}]}
{"characters":[{"name":"Aella Stormrider","abilities":["Lightning Magic","Swordsmanship","Eagle Vision"]},{"name":"Torvin Stonefist","abilities":["Blacksmithing","Heavy Armor","Two-Handed Weapons"]},{}]}
{"characters":[{"name":"Aella Stormrider","abilities":["Lightning Magic","Swordsmanship","Eagle Vision"]},{"name":"Torvin Stonefist","abilities":["Blacksmithing","Heavy Armor","Two-Handed Weapons"]},{"name":"Lyra Whisperwind","abilities":["Stealth","Lockpicking","Poison Craft"]}]}
{"characters":[{"name":"Aella Stormrider","abilities":["Lightning Magic","Swordsmanship","Eagle Vision"]},{"name":"Torvin Stonefist","abilities":["Blacksmithing","Heavy Armor","Two-Handed Weapons"]},{"name":"Lyra Whisperwind","abilities":["Stealth","Lockpicking","Poison Craft"]}]}
{"name":"76efd20d-53bc-4364-91ac-7ffead5aed3c","done":true,"result":{"response":"```json\n{\"characters\": [{\"name\": \"Aella Stormrider\", \"abilities\": [\"Lightning Magic\", \"Swordsmanship\", \"Eagle Vision\"]}, {\"name\": \"Torvin Stonefist\", \"abilities\": [\"Blacksmithing\", \"Heavy Armor\", \"Two-Handed Weapons\"]}, {\"name\": \"Lyra Whisperwind\", \"abilities\": [\"Stealth\", \"Lockpicking\", \"Poison Craft\"]}]}\n```"}}

[Jacob Cable]

I've opened a draft PR here, let me know what you think of this approach?

https://github.com/firebase/genkit/pull/484/files

[Pavel Jbanov]

#484 is not merged yet, right?

[Jacob Cable]

Sorry, just saw this, not yet no