triton-inference-server

There are 3 repositories under triton-inference-server topic.

NVIDIA / GenerativeAIExamples
Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
gpu-acceleration large-language-models llm llm-inference microservice nemo rag retrieval-augmented-generation tensorrt triton-inference-server
Language:Python 1570
CoinCheung / BiSeNet
Add bisenetv2. My implementation of BiSeNet
bisenet cityscapes pytorch cocostuff tensorrt ncnn openvino triton-inference-server ade20k
Language:Python 1364
isarsoft / yolov4-triton-tensorrt
This repository deploys YOLOv4 as an optimized TensorRT engine to Triton Inference Server
triton-inference-server object-detection tensorrt yolov4 yolov4-tiny docker deep-learning
Language:C++ 276
allegroai / clearml-serving
ClearML - Model-Serving Orchestration and Repository Solution
ai clearml deep-learning devops kubernetes machine-learning mlops model-serving serving serving-ml serving-pytorch-models tensorflow-serving triton triton-inference-server
Language:Python 125
torchpipe / torchpipe
An Alternative for Triton Inference Server. Boosting DL Service Throughput 1.5-4x by Ensemble Pipeline Serving with Concurrent CUDA Streams for PyTorch/LibTorch Frontend and TensorRT/CVCUDA, etc., Backends
deployment inference pipeline-parallelism serving tensorrt ray-serve triton-inference-server ray pytorch torch2trt serve cvcuda
Language:C++ 125
kamalkraj / stable-diffusion-tritonserver
Deploy stable diffusion model with onnx/tenorrt + tritonserver
deploy docker fp16 inference machine-learning nvidia onnx python3 pytorch stablediffusion tensorrt tensorrt-inference transformers triton-inference-server
Language:Jupyter Notebook 115
triton-inference-server / onnxruntime_backend
The Triton backend for the ONNX Runtime.
triton-inference-server backend onnx-runtime inference
Language:C++ 113
NVIDIA-ISAAC-ROS / isaac_ros_dnn_inference
Hardware-accelerated DNN model inference ROS 2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU
ai deep-learning deeplearning dnn gpu jetson nvidia ros ros2 ros2-humble tao tensorrt tensorrt-inference triton triton-inference-server
Language:C++ 97
notAI-tech / fastDeploy
Deploy DL/ ML inference pipelines with minimal extra code.
deep-learning tensorflow-serving tf-serving pytorch serving falcon gevent docker model-deployment model-serving http-server gunicorn torchserve triton-inference-server python triton triton-server inference-server streaming-audio websocket
Language:Python 93
npuichigo / openai_trtllm
OpenAI compatible API for TensorRT LLM triton backend
langchain llm openai-api tensorrt-llm triton-inference-server
Language:Rust 84
chiehpower / Setup-deeplearning-tools
Set up CI in DL/ cuda/ cudnn/ TensorRT/ onnx2trt/ onnxruntime/ onnxsim/ Pytorch/ Triton-Inference-Server/ Bazel/ Tesseract/ PaddleOCR/ NVIDIA-docker/ minIO/ Supervisord on AGX or PC from scratch.
tensorrt cuda cudnn installation onnxruntime onnx2trt onnx-simplifier agx pytorch tesseract-ocr triton-inference-server tensorrt-inference-server nvidia deep-learning ci docker triton-server paddleocr supervisord minio
Language:Python 44
bug-developer021 / YOLOV5_optimization_on_triton
Compare multiple optimization methods on triton to imporve model service performance
gpu inference tensorrt triton-inference-server yolov5
Language:Jupyter Notebook 42
rtzr / tritony
Tiny configuration for Triton Inference Server
inference mlops triton-inference-server tritonclient
Language:Python 38
akiragy / recsys_pipeline
Build Recommender System with PyTorch + Redis + Elasticsearch + Feast + Triton + Flask. Vector Recall, DeepFM Ranking and Web Application.
elasticsearch python pytorch recommender-system redis inverted-index ranking retrieval feast recommendation triton-inference-server vector-database flask
Language:Python 32
k9ele7en / Triton-TensorRT-Inference-CRAFT-pytorch
Advanced inference pipeline using NVIDIA Triton Inference Server for CRAFT Text detection (Pytorch), included converter from Pytorch -> ONNX -> TensorRT, Inference pipelines (TensorRT, Triton server - multi-format). Supported model format for Triton inference: TensorRT engine, Torchscript, ONNX
inference inference-engine inference-server nvidia-docker onnx onnx-torch pytorch tensorrt tensorrt-conversion text-detection text-detection-from-image triton-inference-server
Language:Python 31
Bobo-y / triton_ensemble_model_demo
triton server ensemble model demo
triton-inference-server pipeline
Language:Python 29
trinhtuanvubk / Diff-VC
Diffusion Model for Voice Conversion
diffusion-models gradio triton-inference-server voice-conversion
Language:Jupyter Notebook 24
omarabid59 / yolov8-triton
Provides an ensemble model to deploy a YoloV8 ONNX model to Triton
deployment triton-inference-server triton-server ultralytics yolov8
Language:Python 21
Curt-Park / serving-codegen-gptj-triton
Serving Example of CodeGen-350M-Mono-GPTJ on Triton Inference Server with Docker and Kubernetes
codegen docker fastertransformer kubernetes triton-inference-server pytorch huggingface-transformers
Language:Python 20
tonhathuy / tensorrt-triton-magface
Magface Triton Inferece Server Using Tensorrt
face-recognition magface onnx tensorrt-engine triton-inference-server
Language:Jupyter Notebook 15
Biano-AI / serving-compare-middleware
FastAPI middleware for comparing different ML model serving approaches
tensorflow torchserve fastapi python triton-inference-server tensorflow-serving pytorch
Language:Python 14
detail-novelist / novelist-triton-server
Deploy KoGPT with Triton Inference Server
fastertransformer gptj huggingface kogpt large-language-models transformers triton triton-inference-server
Language:Shell 14
RedisVentures / redis-feast-gcp
A demo of Redis Enterprise as the Online Feature Store deployed on GCP with Feast and NVIDIA Triton Inference Server.
feast gcp redis bigquery feature-store machine-learning nvidia google-cloud google-cloud-platform triton-inference-server vertex-ai
Language:Jupyter Notebook 14
openhackathons-org / End-to-End-Computer-Vision
This repository is an AI bootcamp material that consist of a workflow for computer vision
computer-vision deep-learning deep-neural-networks deepstream image-processing image-recognition object-detection object-tracking opencv tao tensorrt triton-inference-server
Language:Jupyter Notebook 11
olibartfast / computer-vision-triton-cpp-client
C++ application to perform computer vision tasks using Nvidia Triton Server for model inference
computer-vision object-detection triton-inference-server
Language:C++ 10
dpressel / reserve
FastAPI + WebSockets + SSE service to interface with Triton/Riva ASR
asr fastapi riva socketio sse triton-inference-server
Language:Python 9
yas-sim / openvino-model-server-wrapper
Python wrapper class for OpenVINO Model Server. User can submit inference request to OVMS with just a few lines of code.
openvino openvino-docker model-serving deep-learning inference python intel openvino-model-server tensorflow-serving triton-inference-server grpc grpc-client ai serving cloud edge area-intrusion-detection line-crossing-detection object-tracking
Language:Python 8
hiennguyen9874 / triton-face-recognition
Triton face detection & recognition
deep-learning face-detection face-recognition triton-inference-server
Language:Jupyter Notebook 7
smarter-project / armnn_tflite_backend
TensorFlow Lite backend with ArmNN delegate support for Nvidia Triton
deep-learning gpu inference triton triton-inference-server
Language:C++ 7
duydvu / triton-inference-server-web-ui
Triton Inference Server Web UI
ai triton-inference-server web user-interface
Language:TypeScript 5
LeslieZhoa / Triton-Torch-Custom
Triton-Pytorch Custom operator tutorial
triton-inference-server pytorch custom-operators
Language:Python 5
viamrobotics / viam-mlmodelservice-triton
MLModelService wrapping Nvidia's Triton Server
cxx gpu gpu-acceleration inference-engine jetpack jetson jetson-orin ml nvidia nvidia-gpu onnx openvino pytorch robotics tensorflow tensorrt triton-inference-server viam cuda triton
Language:C++ 5
dev6699 / yolotriton
Go gRPC client for YOLO-NAS, YOLOv8 inference using the Triton Inference Server.
grpc-go triton-inference-server yolov8 yolo-nas tensorrt-int8
Language:Go 4
neuro-inc / mlops-pytorch-mlflow-triton
Example of deployment Pytorch model into the Triton inference server via MLFlow model registry
inference machine-learning mlflow mlops pytorch triton triton-inference-server
Language:Jupyter Notebook 4
rushai-dev / licence-plate-triton-server-ensemble
Triton backend that enables pre-processing, post-processing and other logic to be implemented in Python. In the repository, I use tech stack including YOLOv8, ONNX, EasyOCR, Triton Inference Server, CV2, Minio, Docker, and K8S. All of which we deploy on k80 and use CUDA 11.4
yolov8 cuda cv2 docker easyocr k8s onnx triton-inference-server minio
Language:Python 4
YeonwooSung / MLOps
Miscellaneous codes and writings for MLOps
ai ai-as-a-service aws llm llm-inference llm-ops ml-serving mlops multimodal bentoml triton-inference-server apache-iceberg data-intensive-applications docker kubernetes spark spark-nlp rag vector-database vectordb
Language:Jupyter Notebook 4

triton-inference-server

NVIDIA / GenerativeAIExamples

CoinCheung / BiSeNet

isarsoft / yolov4-triton-tensorrt

allegroai / clearml-serving

torchpipe / torchpipe

kamalkraj / stable-diffusion-tritonserver

triton-inference-server / onnxruntime_backend

NVIDIA-ISAAC-ROS / isaac_ros_dnn_inference

notAI-tech / fastDeploy

npuichigo / openai_trtllm

chiehpower / Setup-deeplearning-tools

bug-developer021 / YOLOV5_optimization_on_triton

rtzr / tritony

akiragy / recsys_pipeline

k9ele7en / Triton-TensorRT-Inference-CRAFT-pytorch

Bobo-y / triton_ensemble_model_demo

trinhtuanvubk / Diff-VC

omarabid59 / yolov8-triton

Curt-Park / serving-codegen-gptj-triton

tonhathuy / tensorrt-triton-magface

Biano-AI / serving-compare-middleware

detail-novelist / novelist-triton-server

RedisVentures / redis-feast-gcp

openhackathons-org / End-to-End-Computer-Vision

olibartfast / computer-vision-triton-cpp-client

dpressel / reserve

yas-sim / openvino-model-server-wrapper

hiennguyen9874 / triton-face-recognition

smarter-project / armnn_tflite_backend

duydvu / triton-inference-server-web-ui

LeslieZhoa / Triton-Torch-Custom

viamrobotics / viam-mlmodelservice-triton

dev6699 / yolotriton

neuro-inc / mlops-pytorch-mlflow-triton

rushai-dev / licence-plate-triton-server-ensemble

YeonwooSung / MLOps