sgl-project / sglang

SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.

sgl-project/sglang Issues

/generate request possibly hanging when `CUDA out of memory` is thrown
Updated 3 days ago2
OOM CUDA error on 8 * L4 machine when launching sglang server
Updated 4 days ago2
I can't use the OpenAI endpoint with images?
Updated 4 days ago1
Does sglang do automatic batching?
Updated 4 days ago2
How do I get logprobs
Closed 6 days ago1
Llama-3 regex generation can get stuck in infinite generation beyond max_tokens and crash server (reproduction example)
Updated 6 days ago2
Support for multimodal models
Updated 8 days ago
vLLM import error
Updated 10 days ago5
Use `__all__` to explictly control the APIs.
Closed 11 days ago
Please add Phi3 support
Updated 12 days ago2
run python3 test_httpserver_llava.py get ValueError: 64002 is not in list
Updated 12 days ago
Chat role prefixes for SRT backend broken in latest
Closed 12 days ago3
Support InternVL 1.5
Updated 13 days ago1
Logprobs are almost the same for all choices
Updated 14 days ago8
LLaVA-v1.6 RuntimeError in llava image encoding
Updated 15 days ago
Switch to non gated models
Updated 17 days ago1
Choices functionality breaking with images
Updated 18 days ago
Import Errors occurring even when dependencies are installed
Updated 19 days ago2
Launch server using local LLaVA checkpoints
Closed 20 days ago2
[Question] offload kv cache to disk/database ?
Closed 20 days ago
Repetitive zeros for structuring JSON (float type)
Updated 20 days ago
no batch run when using openai's format for calling.
Updated 20 days ago
How does RadixAttention implements multi-head/multi-query/grouped-query attention.
Updated 21 days ago
Is it possible to define the prompts for KV caching up-front?
Updated 21 days ago1
Support Datetime in JSON mode
Closed 21 days ago1
Don't get API response when sending images
Updated 21 days ago1
with default setting, LoRA has no improvement is normal? 默认设置下，LoRA 完全没效果正常吗？
Closed 22 days ago
Loading a BNB 4 bit model + adapter
Updated 22 days ago1
Prefill out of memory occur when deployed with servers
Updated 25 days ago3
Regenerate benchmark results for latest vLLM
Updated 25 days ago1
ImportError: cannot import name 'function' from partially initialized module 'sglang'
Closed a month ago
compatibility issues and memory leak problems --enable-flashinfer
Updated a month ago3
[BUG] srt throws KeyError when sgl.gen(...) regex parameter contains Chinese characters
Updated a month ago1
ImportError: cannot import name 'get_cuda_stream' from 'triton.runtime.jit' In triton-nightly(V100)
Updated a month ago
VLLM version
Updated a month ago3
How does or does sglang support multiple completions / samples given the same prompt?
Updated a month ago
Supported LLaVa models 请问还支持哪些LLaVa的model？Lora后的模型如何批量推理？
Closed a month ago
Loading Chat Template in a more flexible way?
Updated a month ago
JSON decoding result don't match regex
Updated a month ago
System message not supported for Anthropic (anthropic.BadRequestError)
Closed a month ago2
[Feature] Support CohereForCausalLM architectures
Closed a month ago1
Inference does not work in sglang with regex?
Updated a month ago
Fails with latest vllm
Closed a month ago1
sglang doesn't work with vllm versions above 0.3.3
Closed a month ago3
About parameter `max_tokens`
Updated a month ago3
How do I instantiate multiple sgl backends?
Closed a month ago2
Beam Search Support
Updated a month ago
Beginner here, how to load the model from checkpoint
Updated a month ago1
Source build instructions gives broken build
Closed a month ago3
PR review process standard
Closed 2 months ago