run-llama / llama_index

LlamaIndex is a data framework for your LLM applications

Home Page:https://docs.llamaindex.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Feature Request]: Please support stream_chat for vllm

DavidLetGo opened this issue · comments

Feature Description

Need to implement the stream_chat function.

class Vllm(LLM):
@llm_chat_callback()
def stream_chat(
self, messages: Sequence[ChatMessage], **kwargs: Any
) -> ChatResponseGen:
raise (ValueError("Not Implemented"))

code source:
https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-vllm/llama_index/llms/vllm/base.py

Reason

No response

Value of Feature

vLLM is popular and widely used. Is it possible to integrate it to this project?

Anyone will help?