Streaming support?

[Guillaume Duteaud]

Feature request

Add support for streaming generated outputs. This appears to be supported in the transformers library: https://huggingface.co/docs/transformers/v4.38.2/en/generation_strategies#streaming

Motivation

Because outputs take a certain time to generate, it is desirable from a user experience standpoint to be able to display outputs "live" as they are being generated, as opposed to waiting until generation is completed before displaying the output.

Your contribution

I really wish I could but unfortunately this is well beyond my ability to implement.

[Joshua Lochner]

You can use the callback_function generation parameter. For example:

transformers.js/examples/demo-site/src/worker.js

Lines 161 to 175 in 314b7f0

     return await pipeline(data.text, { 
         ...data.generation, 
         callback_function: function (beams) { 
             const decodedText = pipeline.tokenizer.decode(beams[0].output_token_ids, { 
                 skip_special_tokens: true, 
             }) 
  
             self.postMessage({ 
                 type: 'update', 
                 target: data.elementIdToUpdate, 
                 data: decodedText 
             }); 
         } 
     }) 
 } 

[Guillaume Duteaud]

🤯 oh wow that's amazing, thank you!!

[Steven Roussey]

In typescript, the options give an error if you have callback_function in options.