Utilize sessions such that client doesn't need to upload megabytes per chat request
grymmy opened this issue · comments
Describe the bug
Right now the implementation of the LLM chat feature requires excessive uploads from the web browser client, making it not very useful on low bandwidth/high latency connections.
Expected behavior
Theoretically sessions could be used to ensure we don't need to reupload context multiple times across requests.
implemented since hackclub/llm-api#4