JSON output doesn't conform to pattern

[Samrat Man Singh]

Hello 👋,

When I try to generate JSON using the example in the basics_tutorial notebook with the phi2 model, the JSON generated doesn't conform to the structure specified:

Please summarize and extract key factors from the following text:
<text>
The 3 astronauts who flew to the International Space Station reached their destination today (May 31). The
crew successfully docked at 10:16 a.m. EDT, about 19 hours after blasting off from
NASA's Kennedy Space Center. The docking went smoothly, with the Crew Dragon's soft capture docking
mechanism attaching automatically at 10:16 a.m. EDT.</text>
Return as JSON:{shorttitle: 'Astronauts reach space station', who: '3 astronauts', what: 'docked at 10:16 a.m. EDT', when: 'May 31'}
## INPUT

##OUTPUT
{
    "shorttitle": "Astronauts reach space station",
    "who": "3 astronauts",
    "what": "docked at 10:16 a.m. EDT",
    "when": "May 31"
}
The text summarizes that three ast

(The generation doesn't stop-- I had to interrupt the kernel for it to stop)

There is JSON at the beginning, but it is missing some keys(quality_level, excitement_level).

I realize that phi2 is a relatively small model and limited in capability, but my understanding was that aici would constrain the output forcing specific tokens. In this case, shouldn't it have ensured that the keys specified are output?

[Emre Kıcıman]

Hi Samrat, thanks for catching this bug. You are correct that the JSON is being incorrectly enforced in that tutorial notebook.

Looking at the code, I see that the sample is setting the general structure of the JSON, but there is no stop token being enforced. This allows the underlying library to close the JSON object and continue with text generation. A quick fix is to add the ' quote character as a stop token for each of the gen calls for the fields.

Let me make that quick fix, and then I'll also think about whether there's a more elegant way to do this --- like having the first token of the next fixed string be a stop token (though that might create other surprises as an implicit behavior).

[superchargez]

It is fixed? Can aici work with tiny llama (even 70M models?). Please update.

[Michał Moskal]

Yes, AICI does not place limits on the model size. However, you do need to have it integrated in whatever system you're using for inference!