serving

There are 10 repositories under serving topic.

ray-project / ray
Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
automl data-science deep-learning deployment distributed hyperparameter-optimization hyperparameter-search java llm-serving machine-learning model-selection optimization parallel python pytorch ray reinforcement-learning rllib serving tensorflow
Language:Python 31326
tensorflow / serving
A flexible, high-performance serving system for machine learning models
cpp deep-learning deep-neural-networks machine-learning ml neural-network python serving tensorflow
Language:C++ 6088
vespa
vespa-engine / vespa
AI + Data, online. https://vespa.ai
vespa search-engine big-data ai serving serving-recommendation machine-learning server tensorflow java cpp vector-search
Language:Java 5371
ahkarami / Deep-Learning-in-Production
In this repository, I will share some useful notes and references about deploying deep learning-based models in production.
deep-learning deep-neural-networks python pytorch tesnorflow keras mxnet caffe2 production serving c-plus-plus model-serving tutorial flask rest-api react serving-pytorch-models convert-pytorch-models angularjs tensorflow-models
4247
SeldonIO / seldon-core
An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
aiops deployment kubernetes machine-learning machine-learning-operations mlops production-machine-learning serving
Language:HTML 4233
pytorch / serve
Serve, optimize and scale PyTorch models in production
cpu deep-learning docker gpu kubernetes machine-learning metrics mlops optimization pytorch serving
Language:Java 3977
PaddlePaddle / FastDeploy
⚡️An Easy-to-use and Fast Deep Learning Model Deployment Toolkit for ☁️Cloud 📱Mobile and 📹Edge. Including Image, Video, Text and Audio 20+ main stream scenarios and 150+ SOTA models with end-to-end optimization, multi-platform and multi-framework support.
android jetson tensorrt onnxruntime object-detection yolov5 openvino picodet uie graphcore intel kunlun rockchip serving onnx stable-diffusion yolov8
Language:C++ 2726
evadb
georgia-tech-db / evadb
Database system for AI-powered apps
eva video-analytics serving database labeling object-detection data-analysis ai chatgpt langchain auto-gpt gpt4all huggingface llm gpt-4 agent hacktoberfest
Language:Python 2582
tobegit3hub / tensorflow_template_application
TensorFlow template application for deep learning
tensorflow tfrecords libsvm csv deep-learning machine-learning mlp cnn lstm inference tensorboard wide-and-deep serving
Language:Python 1868
Delta-ML / delta
DELTA is a deep learning based natural language and speech processing platform.
nlp deep-learning tensorflow speech sequence-to-sequence seq2seq speech-recognition text-classification speaker-verification nlu text-generation emotion-recognition tensorflow-serving tensorflow-lite inference asr serving front-end custom-ops ops
Language:Python 1585
llm-applications
ray-project / llm-applications
A comprehensive guide to building RAG-based LLM applications for production.
anyscale fine-tuning llama2 llms machine-learning openai ray serving
Language:Jupyter Notebook 1516
ray-project / ray-llm
RayLLM - LLMs on Ray
distributed-systems large-language-models llm llm-inference llm-serving llmops ray serving transformers
Language:Python 1165
PaddlePaddle / Serving
A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）
paddle-serving rpc-service gpu python docker serving pipeline paddle deep-learning online-service prediction predictor dag micro-service microservice-toolkit
Language:C++ 878
tobegit3hub / simple_tensorflow_serving
Generic and easy-to-use serving service for machine learning models
tensorflow-models savedmodel tensorflow serving client http machine-learning deep-learning
Language:JavaScript 757
openvinotoolkit / model_server
A scalable inference server for models optimized with OpenVINO™
openvino inference ai edge cloud deep-learning serving dag kubernetes machine-learning model-serving
Language:C++ 640
meta-soul / MetaSpore
A unified end-to-end machine intelligence platform
abtesting ai deeplearning machinelearning serving training
Language:Python 628
underneathall / pinferencia
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
ai inference-server predict inference deep-learning modelserver machine-learning python serving model-deployment huggingface pytorch tensorflow transformers artificial-intelligence data-science model-serving computer-vision nlp paddlepaddle
Language:Python 558
polyaxon / haupt
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
tensorflow deep-learning jupyter python pytorch machine-learning models ui visualization matplotlib plotly bokeh mlops data-science data-visualization data-processing data-profiling tracking lineage serving
Language:Python 451
bodywork-core
bodywork-ml / bodywork-core
ML pipeline orchestration and model deployments on Kubernetes.
mlops python kubernetes data-science machine-learning pipeline serving continuous-deployment batch devops framework cicd orchestration
Language:Python 433
zzsza / Boostcamp-AI-Tech-Product-Serving
부스트캠프 AI Tech - Product Serving 자료
mlops serving
Language:Python 403
dingodb / dingo
A multi-modal vector database that supports upserts and vector queries using unified SQL (MySQL-Compatible) on structured and unstructured data, while meeting the requirements of high concurrency and ultra-low latency.
serving embedding-store vector-database mysql-compatibility embedding-search key-value-distributed-store vector-ocean unified-sql structured-data unstructured-data real-time-semantic-search
Language:Java 385
vectorch-ai / ScaleLLM
A high-performance inference system for large language models, designed for production environments.
cuda inference llm llm-inference model production serving speculative transformer efficiency performance gpu llama3 llama
Language:C++ 295
Hydrospheredata / hydro-serving
MLOps Platform
serving scoring spark tensorflow scikit-learn serverless pipelines realtime machine-learning models
Language:Mustache 270
outcaste-io / outserv
Blockchain Search with GraphQL APIs
graphql serving system decentralized web3 search search-engine
Language:Go 196
cap-ntu / ML-Model-CI
MLModelCI is a complete MLOps platform for managing, converting, profiling, and deploying MLaaS (Machine Learning-as-a-Service), bridging the gap between current ML training and serving systems.
serving deep-learning continuous-integration inference convert-models dispatcher profiler tensorflow-serving onnx pytorch tensorrt tensorrt-inference-server mlops
Language:Python 188
deepjavalibrary / djl-serving
A universal scalable machine learning model deployment solution
deep-learning deployment djl inference pytorch serving
Language:Java 172
krystianity / keras-serving
bring keras-models to production with tensorflow-serving and nodejs + docker :pizza:
cpp docker grpc keras network neuronal nodejs production python serving tensorflow
Language:Python 153
allegroai / clearml-serving
ClearML - Model-Serving Orchestration and Repository Solution
ai clearml deep-learning devops kubernetes machine-learning mlops model-serving serving serving-ml serving-pytorch-models tensorflow-serving triton triton-inference-server
Language:Python 125
torchpipe / torchpipe
An Alternative for Triton Inference Server. Boosting DL Service Throughput 1.5-4x by Ensemble Pipeline Serving with Concurrent CUDA Streams for PyTorch/LibTorch Frontend and TensorRT/CVCUDA, etc., Backends
deployment inference pipeline-parallelism serving tensorrt ray-serve triton-inference-server ray pytorch torch2trt serve cvcuda
Language:C++ 125
emacski / tensorflow-serving-arm
TensorFlow Serving ARM - A project for cross-compiling TensorFlow Serving targeting popular ARM cores
tensorflow serving arm aarch64 docker cross-compile armhf armv7 arm64 armv8
Language:C++ 98
notAI-tech / fastDeploy
Deploy DL/ ML inference pipelines with minimal extra code.
deep-learning tensorflow-serving tf-serving pytorch serving falcon gevent docker model-deployment model-serving http-server gunicorn torchserve triton-inference-server python triton triton-server inference-server streaming-audio websocket
Language:Python 93
model_deployment
balavenkatesh3322 / model_deployment
A collection of model deployment library and technique.
model model-deployment model-serving model-server tensorflow keras pytorch aws azure serving serving-pytorch-models serving-recommendation serving-tensors caffe mxnet machine-learning deep-learning neural-network data-science
72
Angel-ML / serving
A stand alone industrial serving system for angel.
machine-learning serving serving-recommendation
Language:Java 63
Lightning-AI / litserve
Deploy AI models at scale. High-throughput serving engine for AI/ML models that uses the latest state-of-the-art model deployment techniques.
ai api serving
Language:Python 61
aws / sagemaker-sparkml-serving-container
This code is used to build & run a Docker container for performing predictions against a Spark ML Pipeline.
machine-learning sparkml sagemaker serving spark pipeline inference-pipeline inference mleap-serialized-spark mleap
Language:Java 50
Hydrospheredata / spark-ml-serving
Spark ML Lib serving library
spark serving scoring inference
Language:Scala 48

serving

ray-project / ray

tensorflow / serving

vespa-engine / vespa

ahkarami / Deep-Learning-in-Production

SeldonIO / seldon-core

pytorch / serve

PaddlePaddle / FastDeploy

georgia-tech-db / evadb

tobegit3hub / tensorflow_template_application

Delta-ML / delta

ray-project / llm-applications

ray-project / ray-llm

PaddlePaddle / Serving

tobegit3hub / simple_tensorflow_serving

openvinotoolkit / model_server

meta-soul / MetaSpore

underneathall / pinferencia

polyaxon / haupt

bodywork-ml / bodywork-core

zzsza / Boostcamp-AI-Tech-Product-Serving

dingodb / dingo

vectorch-ai / ScaleLLM

Hydrospheredata / hydro-serving

outcaste-io / outserv

cap-ntu / ML-Model-CI

deepjavalibrary / djl-serving

krystianity / keras-serving

allegroai / clearml-serving

torchpipe / torchpipe

emacski / tensorflow-serving-arm

notAI-tech / fastDeploy

balavenkatesh3322 / model_deployment

Angel-ML / serving

Lightning-AI / litserve

aws / sagemaker-sparkml-serving-container

Hydrospheredata / spark-ml-serving