serving

There are 11 repositories under serving topic.

ray-project / ray
Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
ray distributed parallel machine-learning reinforcement-learning deep-learning python rllib hyperparameter-search optimization data-science automl hyperparameter-optimization model-selection java serving deployment pytorch tensorflow llm-serving
Language:Python 33088
tensorflow / serving
A flexible, high-performance serving system for machine learning models
machine-learning deep-learning deep-neural-networks python cpp neural-network ml serving tensorflow
Language:C++ 6161
vespa
vespa-engine / vespa
AI + Data, online. https://vespa.ai
ai big-data cpp java machine-learning search-engine server serving serving-recommendation tensorflow vector-search vespa
Language:Java 5627
SeldonIO / seldon-core
An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
aiops deployment kubernetes machine-learning machine-learning-operations mlops production-machine-learning serving
Language:HTML 4342
ahkarami / Deep-Learning-in-Production
In this repository, I will share some useful notes and references about deploying deep learning-based models in production.
deep-learning deep-neural-networks python pytorch tesnorflow keras mxnet caffe2 production serving c-plus-plus model-serving tutorial flask rest-api react serving-pytorch-models convert-pytorch-models angularjs tensorflow-models
4294
pytorch / serve
Serve, optimize and scale PyTorch models in production
cpu deep-learning docker gpu kubernetes machine-learning metrics mlops optimization pytorch serving
Language:Java 4158
PaddlePaddle / FastDeploy
⚡️An Easy-to-use and Fast Deep Learning Model Deployment Toolkit for ☁️Cloud 📱Mobile and 📹Edge. Including Image, Video, Text and Audio 20+ main stream scenarios and 150+ SOTA models with end-to-end optimization, multi-platform and multi-framework support.
android jetson tensorrt onnxruntime object-detection yolov5 openvino picodet uie graphcore intel kunlun rockchip serving onnx stable-diffusion yolov8
Language:C++ 2919
evadb
georgia-tech-db / evadb
Database system for AI-powered apps
eva video-analytics serving database labeling object-detection data-analysis ai chatgpt langchain auto-gpt gpt4all huggingface llm gpt-4 agent hacktoberfest
Language:Python 2627
Lightning-AI / LitServe
Lightning-fast serving engine for AI models. Flexible. Easy. Enterprise-scale.
ai api artificial-intelligence deep-learning developer-tools fastapi rest-api serving web
Language:Python 2094
tobegit3hub / tensorflow_template_application
TensorFlow template application for deep learning
cnn csv deep-learning inference libsvm lstm machine-learning mlp serving tensorboard tensorflow tfrecords wide-and-deep
Language:Python 1871
llm-applications
ray-project / llm-applications
A comprehensive guide to building RAG-based LLM applications for production.
llms machine-learning ray anyscale fine-tuning llama2 openai serving
Language:Jupyter Notebook 1671
Delta-ML / delta
DELTA is a deep learning based natural language and speech processing platform.
asr custom-ops deep-learning emotion-recognition front-end inference nlp nlu ops seq2seq sequence-to-sequence serving speaker-verification speech speech-recognition tensorflow tensorflow-lite tensorflow-serving text-classification text-generation
Language:Python 1590
dingodb / dingo
A multi-modal vector database that supports upserts and vector queries using unified SQL (MySQL-Compatible) on structured and unstructured data, while meeting the requirements of high concurrency and ultra-low latency.
serving embedding-store vector-database mysql-compatibility embedding-search key-value-distributed-store vector-ocean unified-sql structured-data unstructured-data real-time-semantic-search
Language:Java 1397
ray-project / ray-llm
RayLLM - LLMs on Ray
distributed-systems large-language-models ray serving transformers llm llm-inference llm-serving llmops
Language:Python 1222
PaddlePaddle / Serving
A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）
dag deep-learning docker gpu micro-service microservice-toolkit online-service paddle paddle-serving pipeline prediction predictor python rpc-service serving
Language:C++ 893
tobegit3hub / simple_tensorflow_serving
Generic and easy-to-use serving service for machine learning models
client deep-learning http machine-learning savedmodel serving tensorflow tensorflow-models
Language:JavaScript 757
openvinotoolkit / model_server
A scalable inference server for models optimized with OpenVINO™
openvino inference ai edge cloud deep-learning serving dag kubernetes machine-learning model-serving
Language:C++ 657
meta-soul / MetaSpore
A unified end-to-end machine intelligence platform
abtesting ai deeplearning machinelearning serving training
Language:Python 639
underneathall / pinferencia
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
ai inference-server predict inference deep-learning modelserver machine-learning python serving model-deployment huggingface pytorch tensorflow transformers artificial-intelligence data-science model-serving computer-vision nlp paddlepaddle
Language:Python 561
polyaxon / haupt
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
bokeh data-processing data-profiling data-science data-visualization deep-learning jupyter lineage machine-learning matplotlib mlops models plotly python pytorch serving tensorflow tracking ui visualization
Language:Python 453
bodywork-core
bodywork-ml / bodywork-core
ML pipeline orchestration and model deployments on Kubernetes.
mlops python kubernetes data-science machine-learning pipeline serving continuous-deployment batch devops framework cicd orchestration
Language:Python 435
zzsza / Boostcamp-AI-Tech-Product-Serving
부스트캠프 AI Tech - Product Serving 자료
mlops serving
Language:Python 404
vectorch-ai / ScaleLLM
A high-performance inference system for large language models, designed for production environments.
cuda inference llm llm-inference model production serving speculative transformer efficiency performance gpu llama3 llama
Language:C++ 372
Hydrospheredata / hydro-serving
MLOps Platform
serving scoring spark tensorflow scikit-learn serverless pipelines realtime machine-learning models
Language:Mustache 271
outcaste-io / outserv
Blockchain Search with GraphQL APIs
decentralized graphql search search-engine serving system web3
Language:Go 197
deepjavalibrary / djl-serving
A universal scalable machine learning model deployment solution
deep-learning deployment djl inference pytorch serving
Language:Java 192
cap-ntu / ML-Model-CI
MLModelCI is a complete MLOps platform for managing, converting, profiling, and deploying MLaaS (Machine Learning-as-a-Service), bridging the gap between current ML training and serving systems.
continuous-integration convert-models deep-learning dispatcher inference mlops onnx profiler pytorch serving tensorflow-serving tensorrt tensorrt-inference-server
Language:Python 188
FasterDecoding / BitDelta
llm quantization serving
Language:Jupyter Notebook 174
krystianity / keras-serving
bring keras-models to production with tensorflow-serving and nodejs + docker :pizza:
cpp docker grpc keras network neuronal nodejs production python serving tensorflow
Language:Python 153
NetEase-Media / grps
【深度学习模型部署框架】支持tf/torch/trt/trtllm/vllm以及更多nn框架，支持dynamic batching、streaming模式，支持python/c++双语言，可限制，可拓展，高性能。帮助用户快速地将模型部署到线上，并通过http/rpc接口方式提供服务。
tensorflow tensorrt torch vllm dynamic-batching serving triton-inference-server tensorrt-llm
Language:C++ 143
torchpipe / torchpipe
Serving Inside Pytorch
deployment inference llm-serving pipeline-parallelism pytorch ray serve serving tensorrt torch2trt triton-inference-server
Language:C++ 141
allegroai / clearml-serving
ClearML - Model-Serving Orchestration and Repository Solution
machine-learning mlops devops deep-learning kubernetes ai clearml model-serving serving serving-pytorch-models serving-ml tensorflow-serving triton triton-inference-server
Language:Python 137
emacski / tensorflow-serving-arm
TensorFlow Serving ARM - A project for cross-compiling TensorFlow Serving targeting popular ARM cores
tensorflow serving arm aarch64 docker cross-compile armhf armv7 arm64 armv8
Language:C++ 99
notAI-tech / fastDeploy
Deploy DL/ ML inference pipelines with minimal extra code.
deep-learning docker falcon gevent gunicorn http-server inference-server model-deployment model-serving python pytorch serving streaming-audio tensorflow-serving tf-serving torchserve triton triton-inference-server triton-server websocket
Language:Python 95
model_deployment
balavenkatesh3322 / model_deployment
A collection of model deployment library and technique.
model model-deployment model-serving model-server tensorflow keras pytorch aws azure serving serving-pytorch-models serving-recommendation serving-tensors caffe mxnet machine-learning deep-learning neural-network data-science
73
Angel-ML / serving
A stand alone industrial serving system for angel.
machine-learning serving serving-recommendation
Language:Java 62