armbues / SiLLM

SiLLM simplifies the process of training and running Large Language Models (LLMs) on Apple Silicon by leveraging the MLX framework.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[FeatureRequest] Add streaming support to openAI compatible api.

s-kostyaev opened this issue · comments

For now api not fully compatible with openAI one, it lacks streaming support. I would like to use it with streaming support.This one https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/SERVER.md supports it and has MIT license.

Yes, the server is not fully implemented yet - good suggestion!