inference-server

There are 2 repositories under inference-server topic.

inference
roboflow / inference
A fast, easy-to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models.
computer-vision inference-api inference-server vit yolact yolov5 yolov7 yolov8 jetson tensorrt classification instance-segmentation object-detection onnx deployment docker inference machine-learning python server
Language:Python 1018
truss
basetenlabs / truss
The simplest way to serve AI/ML models in production
machine-learning artificial-intelligence easy-to-use inference-api inference-server model-serving open-source packaging falcon stable-diffusion whisper wizardlm
Language:Python 831
pipeless-ai / pipeless
An open-source computer vision framework to build and deploy apps in minutes
artificial-intelligence cloud computer-vision deep-learning ffmpeg gstreamer inference inference-server machine-learning multimedia multimedia-applications object-detection perception pipeline-framework python stream-processing video video-processing vision-framework yolo
Language:Rust 644
underneathall / pinferencia
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
ai inference-server predict inference deep-learning modelserver machine-learning python serving model-deployment huggingface pytorch tensorflow transformers artificial-intelligence data-science model-serving computer-vision nlp paddlepaddle
Language:Python 557
NVIDIA / gpu-rest-engine
A REST API for Caffe using Docker and Go
caffe gpu inference inference-server docker deep-learning
Language:C++ 421
BMW-InnovationLab / BMW-YOLOv4-Inference-API-GPU
This is a repository for an nocode object detection inference API using the Yolov3 and Yolov4 Darknet framework.
yolov3 inference gpu api deep-learning computer-vision detection-inference-api bounding-boxes inference-server docker rest-api deeplearning yolo alexeyab-darknet yolo-gui neural-network dockerfile inference-gui yolov4 no-code
Language:Python 280
BMW-InnovationLab / BMW-YOLOv4-Inference-API-CPU
This is a repository for an nocode object detection inference API using the Yolov4 and Yolov3 Opencv.
yolov3 inference api cpu deep-learning computer-vision opencv object-detection cpu-inference-api docker detection-inference-api deep-neural-networks neural-network rest-api inference-server inference-gui bounding-boxes yolov4 yolov4-darknet no-code
Language:Python 219
BMW-InnovationLab / BMW-TensorFlow-Inference-API-CPU
This is a repository for an object detection inference API using the Tensorflow framework.
tensorflow inference api cpu deep-learning object-detection computer-vision detection-inference-api docker tensorflow-framework predictions bounding-boxes docker-image docker-container deeplearning computervision docker-ce inference-engine inference-server rest-api
Language:Python 185
orkhon
vertexclique / orkhon
Orkhon: ML Inference Framework and Server Runtime
inference-server machine-learning python3 tensorflow async multiprocessing data-parallelism
Language:Rust 145
autodeployai / ai-serving
Serving AI/ML models in the open standard formats PMML and ONNX with both HTTP (REST API) and gRPC endpoints
ai-serving api deep-learning grpc inference inference-server machine-learning onnx onnx-models pmml pmml-model real-time
Language:Scala 143
kf5i / k3ai
K3ai is a lightweight, fully automated, AI infrastructure-in-a-box solution that allows anyone to experiment quickly with Kubeflow pipelines. K3ai is perfect for anything from Edge to laptops.
kubeflow-pipelines kubernetes k3s machinelearning datascience artificial-intelligence edge kubeflow inference-server
Language:PowerShell 101
kibae / onnxruntime-server
ONNX Runtime Server: The ONNX Runtime Server is a server that provides TCP and HTTP/HTTPS REST APIs for ONNX inference.
ai machine-learning onnx onnxruntime deep-learning inference-server nueral-networks cuda contributions-welcome
Language:C++ 90
notAI-tech / fastDeploy
Deploy DL/ ML inference pipelines with minimal extra code.
deep-learning tensorflow-serving tf-serving pytorch serving falcon gevent docker model-deployment model-serving http-server gunicorn torchserve triton-inference-server python triton triton-server inference-server streaming-audio websocket
Language:Python 90
RubixML / Server
A standalone inference server for trained Rubix ML estimators.
machine-learning http-server model-server infrastructure api model-deployment microservice json-api php rest-api rubix-ml inference inference-engine php-ml ml-infrastructure php-machine-learning inference-server rubix-server
Language:PHP 61
wingman
curtisgray / wingman
Wingman is the fastest and easiest way to run Llama models on your PC or Mac.
ai chatbot chatgpt linux llama llamacpp llm local macos windows download downloader openai gpu gpu-acceleration gpu-monitoring inference inference-engine inference-server
Language:TypeScript 36
friendliai / friendli-client
Friendli: the fastest serving engine for generative AI
generative-ai llm llm-inference llmops serving gpt gpt3 inference llama2 llm-serving llms inference-engine inference-server ai llm-ops mistral ml mlops stable-diffusion
Language:Python 33
k9ele7en / Triton-TensorRT-Inference-CRAFT-pytorch
Advanced inference pipeline using NVIDIA Triton Inference Server for CRAFT Text detection (Pytorch), included converter from Pytorch -> ONNX -> TensorRT, Inference pipelines (TensorRT, Triton server - multi-format). Supported model format for Triton inference: TensorRT engine, Torchscript, ONNX
triton-inference-server tensorrt tensorrt-conversion onnx onnx-torch pytorch nvidia-docker inference-engine inference-server inference text-detection text-detection-from-image
Language:Python 31
haicheviet / fullstack-machine-learning-inference
Fullstack machine learning inference template
aws cloudformation fastapi full-stack inference-server infrastructure-as-code machine-learning machine-learning-template twitter-sentiment-analysis machine-learning-infrastructure
Language:Jupyter Notebook 28
tensorchord / inference-benchmark
Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)
benchmark inference-server llm stable-diffusion whisper
Language:Python 25
leimao / Simple-Inference-Server
Inference Server Implementation from Scratch for Machine Learning Models
inference-server
Language:Python 24
csy1204 / TripBigs_Web
Session Based Real-time Hotel Recommendation Web Application
inference-server redux react
Language:Python 10
pandruszkow / whisper-inference-server
A networked inference server for Whisper so you don't have to keep waiting for the audio model to reload for the x-hunderdth time.
flask inference-api inference-server python3 whisper-ai
Language:Python 8
tensorchord / modelz-docs
Modelz is a developer-first platform for prototyping and deploying machine learning models.
generative-ai llm modelz inference inference-server machine-learning serverless
Language:MDX 7
RedisVentures / loan-prediction-microservice
An example of using Redis + RedisAI for a microservice that predicts consumer loan probabilities using Redis as a feature and model store and RedisAI as an inference server.
redisai feature-store inference-server redis fastapi xgboost loan-prediction onnx onnxruntime
Language:Jupyter Notebook 6
roboflow / inference-dashboard-example
Roboflow's inference server to analyze video streams. This project extracts insights from video frames at defined intervals and generates informative visualizations and CSV outputs.
inference inference-server object-detection predictions
Language:Python 6
SABER-labs / torch_batcher
Serve pytorch inference requests using batching with redis for faster performance.
pytorch inference-server batch-inference gpu redis
Language:Python 5
dlzou / computron
Serving distributed deep learning models with model parallel swapping.
deep-learning inference-server model-parallelism
Language:Jupyter Notebook 4
StefanoLusardi / tiny_inference_engine
Client/Server system to perform distributed inference on high load systems.
inference-engine ai deep-neural-networks cpp cmake conan onnxruntime grpc docker inference-server kserve inference-client
Language:C++ 4
geniusrise / geniusrise-vision
Vision and vision-multi-modal components for geniusrise framework
huggingface inference inference-server mlops multimodal vision
Language:Python 3
ajinkyapuar / qis
job-queue celery inference-server rabbitmq
Language:Python 2
geniusrise / geniusrise-text
Text components powering LLMs & SLMs for geniusrise framework
ai huggingface inference-api llm inference inference-server
Language:Python 2
geniusrise / geniusrise-audio
Audio components for geniusrise framework
ai audio huggingface speech-recognition speech-to-text inference inference-server
Language:Python 1
liusy182 / sagemaker-run-your-own-inference
Run your own production inference code with Sagemaker
sagemaker keras machine-learning inference-server model-deployment
Language:Python 1
nikhiltadikonda / Kushagra-AI
An AI-powered mobile crop advisory app for farmers, gardeners that can provide information about crops using an image taken by the user. This supports 10 crops and 37 kinds of crop diseases. The AI model is a ResNet network that has been fine-tuned using crop images that were collected by web-scraping from Google Images and Plant-Village Dataset.
artificial-intelligence inference-server
Language:Python 1
TheFaheem / TGS
Effortlessly Deploy and Serve Large Language Models in the Cloud as an API Endpoint for Inference
inference-server llm-inference llmops
Language:Python 1
zhangjun / TensorRT-Server
TensorRT Server
inference-engine inference-server onnx tensorrt
Language:C++ 1

inference-server

roboflow / inference

basetenlabs / truss

pipeless-ai / pipeless

underneathall / pinferencia

NVIDIA / gpu-rest-engine

BMW-InnovationLab / BMW-YOLOv4-Inference-API-GPU

BMW-InnovationLab / BMW-YOLOv4-Inference-API-CPU

BMW-InnovationLab / BMW-TensorFlow-Inference-API-CPU

vertexclique / orkhon

autodeployai / ai-serving

kf5i / k3ai

kibae / onnxruntime-server

notAI-tech / fastDeploy

RubixML / Server

curtisgray / wingman

friendliai / friendli-client

k9ele7en / Triton-TensorRT-Inference-CRAFT-pytorch

haicheviet / fullstack-machine-learning-inference

tensorchord / inference-benchmark

leimao / Simple-Inference-Server

csy1204 / TripBigs_Web

pandruszkow / whisper-inference-server

tensorchord / modelz-docs

RedisVentures / loan-prediction-microservice

roboflow / inference-dashboard-example

SABER-labs / torch_batcher

dlzou / computron

StefanoLusardi / tiny_inference_engine

geniusrise / geniusrise-vision

ajinkyapuar / qis

geniusrise / geniusrise-text

geniusrise / geniusrise-audio

liusy182 / sagemaker-run-your-own-inference

nikhiltadikonda / Kushagra-AI

TheFaheem / TGS

zhangjun / TensorRT-Server