Is OgaGenerator a one-time object?
MasterYi1024 opened this issue · comments
I'm writing a c++ demo to use OgaGenerator to generate text with chat history, so I rely on the ability of the kv cache to reduce the generation time. But I found (in the code genai already has) that OgaGenerator seems to be a one-time object and cannot be reused. And the KV-Cache class indirectly belongs to OgaGenerator. If OgaGenerator is disposable, then after creating a new generator, the original KV-Cache will no longer be useful.
May I ask if I understand correctly? I have read the document for onnxruntime.ai, is there anything I missed?
I found this discussion, but it doesn't seem to have much content.
#416
Thank you in advance：）
@MasterYi1024 , yes, you're right. OgaGenerator is a one-time object. We will add interactive decoding to support your scenario.
@yufenglee Thank you so much for your reply : )
Feel free to close this issue or something.