triton-inference-server / tensorrtllm_backend

The Triton TensorRT-LLM Backend

triton-inference-server/tensorrtllm_backend Issues

Bug in tensorrt_llm_bls
Updated 17 days ago2
API Usage Error: profileMinDims.d[i] <= dimensions.d[i]
Updated 17 days ago3
Assertion failed: Cannot determine if hopper is specialised without a selected config at runtime (Latest commit)
Closed 18 days ago1
Invalid argument: unable to find backend library for backend '${triton_backend}'
Updated 18 days ago3
How to pass request_id using requests.post?
Updated 18 days ago
[Question] Understanding Generation Logits & Context Logits
Closed 20 days ago2
The default configuration for the top_p parameter is what?
Closed 20 days ago1
Key 'lora_config' not found
Closed 20 days ago1
Deserializing Engine Version Mismatch
Closed 21 days ago1
How to Replicate and Serve Multiple Instances of a Model?
Closed 2 months ago2
how to set `ignore_eos` when benchmark TensorRT LLM
Closed 25 days ago2
Issue Mixtral 8x7b failed to load preprocessing model.
Closed 25 days ago1
3rd Tritonserver fails to respond
Closed a month ago1
Ensemble and tensorrt_llm_bls have different results when using accumulate_tokens
Closed a month ago
launch multi-gpu triton server and got an Error
Updated a month ago1
No 24.05-trtllm-python-py3 in NGC Repo
Closed 2 months ago2
Invalid argument: unable to find 'libtritoncache_redis.so' for cache. Searched: /opt/tritonserver/caches/redis/libtritoncache_redis.so
Closed a month ago
Error: terminate called after throwing an instance of 'boost::interprocess::lock_exception'
Updated a month ago
Deepseek model streaming mode with Chinese character �?
Updated a month ago15
KV Cache Reuse: Where are the prefixes/cache values stored?
Updated a month ago
Got repeated answer while deploying LLaMA3-Instruct-8B model in triton server
Closed 2 months ago3
Support python runtime
Updated a month ago
speculative decoding performance
Updated a month ago3
nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3 missing tritonserver binary?
Closed a month ago
Qwen 7B giving error internal: ValueError: Input None is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers
Updated a month ago
Accumulation of tokens while beam_width > 1
Updated a month ago
How to solve the problem of errors when loading qwen1.5-7B (using two GPUs) and llama3-8B (using two GPUs) models simultaneously using tritonserver?
Updated a month ago3
UNAVAILABLE: Internal: unexpected error when creating modelInstanceState: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal
Closed a month ago1
2x docker image size increase for trtllm: from 8.38 GB (24.03) to 18.46 GB (24.04)
Updated a month ago3
ailed to read text proto from tensorrtllm_backend/triton_model_repo/tensorrt_llm/config.pbtxt
Updated a month ago1
how to deploy multimodal model like llava on triton server? tensorrtllm_backend does not support multimodal
Closed a month ago1
Multiple outputs in sampling
Updated a month ago1
Exception when disabling "inflight_fused_batching"
Updated a month ago
"error":"Unable to parse 'data': Shape does not match true shape of 'data' field"
Closed a month ago1
Assertion failed: Invalid tensor name: decoder_input_lengths
Updated a month ago
No Text Output when hosted in Triton server
Updated 2 months ago
Do Triton Containers Support CentOS 7?
Updated 2 months ago
LTS for this Repo with NIMs on the horizon
Closed 2 months ago1
Track input and output token count for every request
Closed 2 months ago1
Error in streaming mode noting that execute function should return None
Closed 2 months ago2
[Bug] Output generation does not stop at stop token </s>
Closed 2 months ago2
[Question] Best practises to track inputs and predictions?
Updated 2 months ago2
Grafana Dashboard (Feature Request)
Updated 2 months ago
Triton Launches Model on Incorrect GPU
Closed 2 months ago6
tensorrt_llm_bls disregards top_k / temperature setting
Updated 2 months ago
missing nv_trt_llm_request_metrics from python backend
Updated 2 months ago
How to deploy Triton Inference Server Container (tritonserver:24.04-trtllm-python-py3) in K8S without launching Triton Server directly?
Closed 2 months ago
Cannot build Docker container on Grace Hopper
Closed 2 months ago2
Speculative decoding Assertion failed: Number of draft tokens (56) is larger than maximum number of draft tokens (0
Closed 2 months ago
Other backends are missing
Closed 2 months ago