Giters

sgl-project / sglang

SGLang is a fast serving framework for large language models and vision language models.

Home Page:https://sglang.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Development Roadmap (Deprecated)

Ying1123 opened this issue · comments

commented

Function Calling

High-level Pythonic Interface

Inference Optimizations

  • Speculative decoding for local models
  • Speculative execution for OpenAI Chat API

Structured Decoding

Compiler

  • Support tracing and compiling sgl.fork
  • Support sending a full serialized SGL program to the server

LoRA Support

  • Port multi-LoRA batching and unified memory from S-LoRA

Model Coverage

Device Coverage

  • AMD support. Investigate AMD support in Trion and FlashInfer.
  • CPU support. This is better done by adding a llama.cpp backend.
commented

Are there still plans for a high level pythonic interface? #39 (comment)

commented

Are there still plans for a high level pythonic interface? #39 (comment)

Hi @AriMKatz, thanks for the reference. This is very important, I just added it.

commented

For the vision models support, is it possible to align with the openai gpt4v API?
https://platform.openai.com/docs/guides/vision

commented

Are there plans for loading models in 8bit or 4bit?

commented

For the vision models support, is it possible to align with the openai gpt4v API? https://platform.openai.com/docs/guides/vision

@nivibilla Yes, it is already aligned with the openai gpt4v API, see here.
You can also find a runnable example of serving it with Sky Serve here.

commented

Are there plans for loading models in 8bit or 4bit?

@aliencaocao Thanks for the question! The AWQ and GPTQ are already supported. But we do not support an automatic dtype translation yet. You are welcome to submit a PR for that.

commented

Are there plans for loading models in 8bit or 4bit?

@aliencaocao Thanks for the question! The AWQ and GPTQ are already supported. But we do not support an automatic dtype translation yet. You are welcome to submit a PR for that.

I'm looking to load llava 1.6 in 8bit, but it does not seem that llava series has AWQ or GPTQ quants, or did I miss out anything here?

EDIT: I saw 1.5 has but not 1.6 yet. Perhaps its just too new and no one did a calibration yet.

commented

Hi all - is anyone working on the S-LoRA integration currently? I see the branch, but it looks a few months old.

Would love to see this, happy to pick up from existing work or start fresh.

commented

Hi all - is anyone working on the S-LoRA integration currently? I see the branch, but it looks a few months old.

Would love to see this, happy to pick up from existing work or start fresh.

Hi @qeternity, I was working on it but have been blocked by other affairs. You are welcome to contribute, either continue on the branch or start fresh! I'll be happy to review and collaborate.

commented

Tools support is very important, which is necessary for many use cases.

commented

Is TinyLlama supported? TinyLlama/TinyLlama-1.1B-Chat-v1.0
generation seems a bit slow...

commented

I see llama.cpp integration is on the roadmap. When will this feature be delivered? It would be very nice to have it , since it will support running local LLMs, such as llama models, on Mac computers and experiment them with the powerful and expressive SGLang.

commented

I'd request to include support for Phi-3-mini

commented

Hi all - is anyone working on the S-LoRA integration currently? I see the branch, but it looks a few months old.
Would love to see this, happy to pick up from existing work or start fresh.

Hi @qeternity, I was working on it but have been blocked by other affairs. You are welcome to contribute, either continue on the branch or start fresh! I'll be happy to review and collaborate.

Hi which branch is it? looks like better start fresh

commented

I can help by getting rid of the vLLM in the dependencies.

commented

Moved to #634