triton-inference-server

There are 3 repositories under triton-inference-server topic.

NVIDIA / GenerativeAIExamples
Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
gpu-acceleration large-language-models llm llm-inference microservice nemo rag retrieval-augmented-generation tensorrt triton-inference-server
Language:Jupyter Notebook 3567
CoinCheung / BiSeNet
Add bisenetv2. My implementation of BiSeNet
ade20k bisenet cityscapes cocostuff ncnn openvino pytorch tensorrt triton-inference-server
Language:Python 1572
isarsoft / yolov4-triton-tensorrt
This repository deploys YOLOv4 as an optimized TensorRT engine to Triton Inference Server
deep-learning docker object-detection tensorrt triton-inference-server yolov4 yolov4-tiny
Language:C++ 287
npuichigo / openai_trtllm
OpenAI compatible API for TensorRT LLM triton backend
langchain llm openai-api tensorrt-llm triton-inference-server
Language:Rust 216
NetEase-Media / grps
Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offering scalability, extensibility, and high performance. It helps users quickly deploy models and provide services through HTTP/RPC interfaces.
dynamic-batching serving tensorflow tensorrt tensorrt-llm torch triton-inference-server vllm
Language:C++ 166
triton-inference-server / onnxruntime_backend
The Triton backend for the ONNX Runtime.
backend inference onnx-runtime triton-inference-server
Language:C++ 165
torchpipe / torchpipe
Serving Inside Pytorch
deployment inference pipeline-parallelism serving tensorrt triton-inference-server ray pytorch torch2trt serve llm-serving
Language:C++ 164
clearml / clearml-serving
ClearML - Model-Serving Orchestration and Repository Solution
machine-learning mlops devops deep-learning kubernetes ai clearml model-serving serving serving-pytorch-models serving-ml tensorflow-serving triton triton-inference-server
Language:Python 157
kamalkraj / stable-diffusion-tritonserver
Deploy stable diffusion model with onnx/tenorrt + tritonserver
docker nvidia stablediffusion transformers deploy fp16 onnx python3 triton-inference-server inference machine-learning pytorch tensorrt tensorrt-inference
Language:Jupyter Notebook 127
NVIDIA-ISAAC-ROS / isaac_ros_dnn_inference
NVIDIA-accelerated DNN model inference ROS 2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU
ros dnn tensorrt triton triton-inference-server tensorrt-inference tao deeplearning deep-learning nvidia ai ros2-humble ros2 gpu jetson
Language:C++ 122
Koldim2001 / TrafficAnalyzer
Анализ трафика на круговом движении с использованием компьютерного зрения
bytetrack bytetracker docker docker-compose flask grafana grafana-dashboard hydra influxdb multiple-object-tracking multiprocessing nginx object-detection oop-principles postgresql tensorrt-inference traffic-analysis transport-detection triton-inference-server yolov8
Language:Python 101
notAI-tech / fastDeploy
Deploy DL/ ML inference pipelines with minimal extra code.
deep-learning tensorflow-serving tf-serving pytorch serving falcon gevent docker model-deployment model-serving http-server gunicorn torchserve triton-inference-server python triton triton-server inference-server streaming-audio websocket
Language:Python 100
trinhtuanvubk / Diff-VC
Diffusion Model for Voice Conversion
diffusion-models gradio triton-inference-server voice-conversion
Language:Jupyter Notebook 61
akiragy / recsys_pipeline
Build Recommender System with PyTorch + Redis + Elasticsearch + Feast + Triton + Flask. Vector Recall, DeepFM Ranking and Web Application.
elasticsearch python pytorch recommender-system redis inverted-index ranking retrieval feast recommendation triton-inference-server vector-database flask
Language:Python 57
bug-developer021 / YOLOV5_optimization_on_triton
Compare multiple optimization methods on triton to imporve model service performance
gpu inference tensorrt triton-inference-server yolov5
Language:Jupyter Notebook 53
rtzr / tritony
Tiny configuration for Triton Inference Server
inference mlops triton-inference-server tritonclient
Language:Python 45
chiehpower / Setup-deeplearning-tools
Set up CI in DL/ cuda/ cudnn/ TensorRT/ onnx2trt/ onnxruntime/ onnxsim/ Pytorch/ Triton-Inference-Server/ Bazel/ Tesseract/ PaddleOCR/ NVIDIA-docker/ minIO/ Supervisord on AGX or PC from scratch.
tensorrt cuda cudnn installation onnxruntime onnx2trt onnx-simplifier agx pytorch tesseract-ocr triton-inference-server tensorrt-inference-server nvidia deep-learning ci docker triton-server paddleocr supervisord minio
Language:Python 43
omarabid59 / yolov8-triton
Provides an ensemble model to deploy a YoloV8 ONNX model to Triton
deployment triton-inference-server triton-server ultralytics yolov8
Language:Python 40
k9ele7en / Triton-TensorRT-Inference-CRAFT-pytorch
Advanced inference pipeline using NVIDIA Triton Inference Server for CRAFT Text detection (Pytorch), included converter from Pytorch -> ONNX -> TensorRT, Inference pipelines (TensorRT, Triton server - multi-format). Supported model format for Triton inference: TensorRT engine, Torchscript, ONNX
triton-inference-server tensorrt tensorrt-conversion onnx onnx-torch pytorch nvidia-docker inference-engine inference-server inference text-detection text-detection-from-image
Language:Python 33
Bobo-y / triton_ensemble_model_demo
triton server ensemble model demo
pipeline triton-inference-server
Language:Python 30
olibartfast / tritonic
C++ application to perform computer vision tasks using Nvidia Triton Server for model inference
object-detection triton-inference-server computer-vision classifier deim dfine keras-tensorflow rf-detr rt-detr torchvision vit yolo raft
Language:C++ 28
openhackathons-org / End-to-End-Computer-Vision
This repository is an AI bootcamp material that consist of a workflow for computer vision
computer-vision deep-learning deep-neural-networks deepstream image-processing image-recognition object-detection object-tracking opencv tao tensorrt triton-inference-server
Language:Jupyter Notebook 25
Curt-Park / serving-codegen-gptj-triton
Serving Example of CodeGen-350M-Mono-GPTJ on Triton Inference Server with Docker and Kubernetes
codegen docker fastertransformer huggingface-transformers kubernetes pytorch triton-inference-server
Language:Python 20
inferless / triton-co-pilot
Generate Glue Code in seconds to simplify your Nvidia Triton Inference Server Deployments
co-pilot nvidia triton-inference-server
Language:Python 20
Koldim2001 / Triton_example
Triton Inference Server + TensorRT + метрики
grafana onnx production prometheus tensorrt triton triton-inference-server
Language:Jupyter Notebook 17
tech4ai / t4ai-signature-detect-server
This project provides a pipeline for deploying and performing inference with the YOLOv8 object detection model using the Triton Inference Server. It supports integration with local systems, Docker-based setups, or Google Cloud’s Vertex AI. The repository includes scripts for automated deployment, benchmarks and GUI inference.
deep-learning python triton-inference-server yolov8
Language:Jupyter Notebook 17
tonhathuy / tensorrt-triton-magface
Magface Triton Inferece Server Using Tensorrt
face-recognition magface onnx tensorrt-engine triton-inference-server
Language:Jupyter Notebook 17
Biano-AI / serving-compare-middleware
FastAPI middleware for comparing different ML model serving approaches
tensorflow torchserve fastapi python triton-inference-server tensorflow-serving pytorch
Language:Python 15
duydvu / triton-inference-server-web-ui
Triton Inference Server Web UI
ai triton-inference-server web user-interface
Language:TypeScript 15
redis-applied-ai / redis-feast-gcp
A demo of Redis Enterprise as the Online Feature Store deployed on GCP with Feast and NVIDIA Triton Inference Server.
feast gcp redis bigquery feature-store machine-learning nvidia google-cloud google-cloud-platform triton-inference-server vertex-ai
Language:Jupyter Notebook 15
YeonwooSung / MLOps
Miscellaneous codes and writings for MLOps
ai ai-as-a-service aws llm llm-inference llm-ops ml-serving mlops multimodal bentoml triton-inference-server apache-iceberg data-intensive-applications docker kubernetes spark spark-nlp rag vector-database vectordb
Language:Jupyter Notebook 15
detail-novelist / novelist-triton-server
Deploy KoGPT with Triton Inference Server
fastertransformer gptj huggingface kogpt large-language-models transformers triton triton-inference-server
Language:Shell 14
ybai789 / yolov8-triton-tensorrt
Provides an ensemble model to deploy a YOLOv8 TensorRT model to Triton
deployment tensorrt triton-inference-server ultralytics yolov8
Language:Python 13
dpressel / reserve
FastAPI + WebSockets + SSE service to interface with Triton/Riva ASR
asr fastapi riva socketio sse triton-inference-server
Language:Python 12
levipereira / deepstream-yolo-triton-server-rtsp-out
The Purpose of this repository is to create a DeepStream/Triton-Server sample application that utilizes yolov7, yolov7-qat, yolov9 models to perform inference on video files or RTSP streams.
deepstream deepstream-python deepstream-python-apps deepstreamsdk triton-inference-server triton-server yolov7 yolov9
Language:Python 12
yas-sim / openvino-model-server-wrapper
Python wrapper class for OpenVINO Model Server. User can submit inference request to OVMS with just a few lines of code.
openvino openvino-docker model-serving deep-learning inference python intel openvino-model-server tensorflow-serving triton-inference-server grpc grpc-client ai serving cloud edge area-intrusion-detection line-crossing-detection object-tracking
Language:Python 10

triton-inference-server

NVIDIA / GenerativeAIExamples

CoinCheung / BiSeNet

isarsoft / yolov4-triton-tensorrt

npuichigo / openai_trtllm

NetEase-Media / grps

triton-inference-server / onnxruntime_backend

torchpipe / torchpipe

clearml / clearml-serving

kamalkraj / stable-diffusion-tritonserver

NVIDIA-ISAAC-ROS / isaac_ros_dnn_inference

Koldim2001 / TrafficAnalyzer

notAI-tech / fastDeploy

trinhtuanvubk / Diff-VC

akiragy / recsys_pipeline

bug-developer021 / YOLOV5_optimization_on_triton

rtzr / tritony

chiehpower / Setup-deeplearning-tools

omarabid59 / yolov8-triton

k9ele7en / Triton-TensorRT-Inference-CRAFT-pytorch

Bobo-y / triton_ensemble_model_demo

olibartfast / tritonic

openhackathons-org / End-to-End-Computer-Vision

Curt-Park / serving-codegen-gptj-triton

inferless / triton-co-pilot

Koldim2001 / Triton_example

tech4ai / t4ai-signature-detect-server

tonhathuy / tensorrt-triton-magface

Biano-AI / serving-compare-middleware

duydvu / triton-inference-server-web-ui

redis-applied-ai / redis-feast-gcp

YeonwooSung / MLOps

detail-novelist / novelist-triton-server

ybai789 / yolov8-triton-tensorrt

dpressel / reserve

levipereira / deepstream-yolo-triton-server-rtsp-out

yas-sim / openvino-model-server-wrapper