rustformers / llm

[Unmaintained, see README] An ecosystem of Rust libraries for working with large language models

Home Page:https://docs.rs/llm/latest/llm/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Ability to await the result of infer

Lordxan opened this issue · comments

I was wondering is it possible to use this library without callbacks?

No, this is not possible at this time. I believe the goal of this library is to leave "higher-level" abstractions such as that up to the user. For now, you'd probably want to use probably want to use use a combination of InferenceSession::feed_prompt and InferenceSession::infer_next_token to implement this. I'm going to close this Issue, but please feel free to re-open it or open another one if you have more questions.