Giters

LlamaEdge / LlamaEdge

The easiest & fastest way to run customized and fine-tuned LLMs locally or on the edge

Home Page:https://llamaedge.com/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Feature Request: Use `data: [DONE]` as a stop sign of streaming output

alabulei1 opened this issue · comments

commented

Summary

From OpenAI docs, the stream is terminated by a data: [DONE] message. Since LlamaEdge's goal is to make the aPI server compatible, I think we should use data: [DONE] as a stop sign of Streaming output too.

If we changed the stop sign in LlamaEdge, the chatbot UI also needs to be changed as well. And maybe langchain?

Appendix

https://platform.openai.com/docs/api-reference/chat/create#chat-create-stream