inference-engine

There are 22 repositories under inference-engine topic.

FedML-AI / FedML
FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.
ai-agent deep-learning distributed-training edge-ai federated-learning inference-engine machine-learning mlops model-deployment model-serving on-device-training
Language:Python 3964
zjhellofss / KuiperInfer
校招、秋招、春招、实习好项目！带你从零实现一个高性能的深度学习推理库，支持大模型 llama2 、Unet、Yolov5、Resnet等模型的推理。Implement a high-performance deep learning inference library step by step
inference inference-engine deep-learning deep-neural-networks convolution relu sigmoid graph-algorithms maxpooling caffe pnnx pytorch ncnn diy resnet yolo yolov5
Language:C++ 3167
hyperjumptech / grule-rule-engine
Rule engine implementation in Golang
golang rule-engine rule inference-engine rule-based rule-based-engine hacktoberfest hacktoberfest2021
Language:Go 2429
onediff
siliconflow / onediff
OneDiff: An out-of-the-box acceleration library for diffusion models.
comfyui diffusers pytorch sdxl stable-diffusion aigc-serving comfyui-workflow cuda inference-engine lcm lcm-lora performance-optimization sd-webui stable-video-diffusion diffusion-models lora sdxl-turbo
Language:Jupyter Notebook 1941
aphrodite-engine / aphrodite-engine
Large-scale LLM inference engine
api-rest inference-engine machine-learning cuda inferentia rocm intel lora speculative-decoding tpu
Language:C++ 1580
Tencent / FeatherCNN
FeatherCNN is a high performance inference engine for convolutional neural networks.
convolutional-neural-networks inference-engine caffe android ios arm-neon
Language:C++ 1222
PaddlePaddle / Paddle.js
Paddle.js is a web project for Baidu PaddlePaddle, which is an open source deep learning framework running in the browser. Paddle.js can either load a pre-trained model, or transforming a model from paddle-hub with model transforming tools provided by Paddle.js. It could run in every browser with WebGL/WebGPU/WebAssembly supported. It could also run in Baidu Smartprogram and WX miniprogram.
webgl webgpu webassembly model inference-engine paddlepaddle ocr deep-learning
Language:JavaScript 1059
zhihu / ZhiLight
A highly optimized LLM inference acceleration engine for Llama and its variants.
inference-engine llm cuda gpt llama llm-serving pytorch llm-inference model-serving deepseek-r1
Language:C++ 902
quic / ai-hub-models
The Qualcomm® AI Hub Models are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
deeplearning demos inference inference-api inference-engine machine-learning machinelearning onnx pytorch qnn tensorflow-lite
Language:Python 827
Adlik / Adlik
Adlik: Toolkit for Accelerating Deep Learning Inference
deep-learning inference tensorflow-serving openvino tensorrt compiler inference-engine model-optimizer docker-images
Language:C++ 806
msnh2012 / Msnhnet
🔥 (yolov3 yolov4 yolov5 unet ...)A mini pytorch inference framework which inspired from darknet.
yolov3 yolov4 yolov5 pytorch inference-engine darknet jetson-nx mobilenetv2 mobilenetyolo
Language:C++ 745
insight-platform / Savant
Python Computer Vision & Video Analytics Framework With Batteries Included
computer-vision deepstream edge-computing inference-engine machine-learning nvidia-deepstream-sdk deep-learning nvidia object-detection video cuda opencv tensorrt instance-segmentation peoplenet yolo yolov5-face yolov8 yolov8-face
Language:Python 713
xllm
jd-opensource / xllm
A high-performance inference engine for LLMs, optimized for diverse AI accelerators.
deepseek inference inference-engine large-language-models llm-inference qwen
Language:C++ 676
ovg-project / kvcached
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
elastic-kvcache gpu-mutiplexing gpu-sharing inference-engine kvcache kvcache-optimization kvcached llm llm-framework llm-inference llm-serving ollama online-offline-coserve serverless sglang vllm
Language:Python 617
pylint-dev / astroid
A common base representation of python source code for pylint and other projects
ast closember hacktoberfest inference-engine parser static-analysis static-code-analysis
Language:Python 561
Forward
Tencent / Forward
A library for high performance deep learning inference on NVIDIA GPUs.
deep-learning forward tensorrt tensorflow pytorch keras neural-network gpu inference-engine inference cuda onnx
Language:C++ 557
PaddlePaddle / Anakin
High performance Cross-platform Inference-engine, you could run Anakin on x86-cpu,arm, nv-gpu, amd-gpu,bitmain and cambricon devices.
ai inference-engine arm nvidia amd intel bitmain cambricon high-performance cross-platform
Language:C++ 534
andrewkchan / yalm
Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O
cpp cuda inference-engine llama llamacpp llm llm-inference machine-learning mistral
Language:C++ 528
HoloClean / holoclean
A Machine Learning System for Data Enrichment.
machine-learning inference-engine pytorch data-science data-enrichment
Language:Python 528
buguroo / pyknow
PyKnow: Expert Systems for Python
python3 expert-system inference-engine
Language:Python 490
ulfurinn / wongi-engine
A rule engine written in Ruby.
ruby rule-engine rete inference-engine
Language:Ruby 488
zjhellofss / KuiperLLama
校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。
cpp cuda inference-engine llama2 llama3 llm llm-inference qwen qwen2
Language:C++ 445
chengzeyi / ParaAttention
https://wavespeed.ai/ Context parallel attention that accelerates DiT model inference with dynamic caching
attention diffusers inference inference-engine parallel-computing transformers flux hunyuan-video
Language:Python 386
ReactiveBayes / RxInfer.jl
Julia package for automated Bayesian inference on a factor graph with reactive message passing
bayesian-inference inference-engine julia-language machine-learning message-passing probabilistic-programming variational-inference
Language:Jupyter Notebook 371
quic / ai-hub-apps
The Qualcomm® AI Hub apps are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
deeplearning demos inference inference-api inference-engine machine-learning machinelearning onnx pytorch qnn tensorflow-lite
Language:Java 327
interestingLSY / swiftLLM
A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).
transformer transformers cuda inference inference-engine pytorch gpt llama llm llm-inference llm-serving llmops mlops model-serving
Language:Python 286
EfficientMoE / MoE-Infinity
PyTorch library for cost-effective, fast and easy serving of MoE models.
inference-engine large-language-models mixture-of-experts huggingface pytorch llm-inference
Language:Python 257
SearchSavior / OpenArc
Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.
agentic-ai fastapi inference-engine openvino-genai openvino-toolkit optimum-intel transformers
Language:Python 237
gottingen / kumo-search
docs for search system and ai infra
ai performance search-engine deep-learning neural-network python inference-engine tensorflow tensorflow2
219
MIVisionX
ROCm / MIVisionX
MIVisionX toolkit is a set of comprehensive computer vision and machine intelligence libraries, utilities, and applications bundled into a single toolkit. AMD MIVisionX also delivers a highly optimized open-source implementation of the Khronos OpenVX™ and OpenVX™ Extensions.
computer-vision openvx neural-network opencl rocm inference inference-engine onnx nnef winml windows-machine-learning openvx-neural-network machine-learning ryzen virtual-reality amd-opencl amd-openvx openvx-extensions khronos-openvx amd-opencv
Language:C++ 202
BMW-InnovationLab / BMW-TensorFlow-Inference-API-CPU
This is a repository for an object detection inference API using the Tensorflow framework.
tensorflow inference api cpu deep-learning object-detection computer-vision detection-inference-api docker tensorflow-framework predictions bounding-boxes docker-image docker-container deeplearning computervision docker-ce inference-engine inference-server rest-api
Language:Python 183
nilp0inter / experta
Expert Systems for Python
knowledge-base inference inference-engine clips python3
Language:Python 178
midea-ai / Aidget
Ai edge toolbox，专门面向边端设备尤其是嵌入式RTOS平台，AI模型部署工具链，包括模型推理引擎和模型压缩工具
inference inference-engine mcu pruning rtos simd tflm resrep asr wakeup dsp deep-learning hifi5
Language:Python 164
tuanlda78202 / gpt-oss-amd
implement GPT-OSS 20B & 120B C++ inference from scratch on AMD GPUs
gpt-oss hip amd inference-engine hpc
Language:C++ 156
matteocarnelos / microflow-rs
A robust and efficient TinyML inference engine.
inference-engine machine-learning rust tensorflow-lite tinyml
Language:Rust 152
CAS-CLab / CNN-Inference-Engine-Quick-View
A quick view of high-performance convolution neural networks (CNNs) inference engines on mobile devices.
inference-engines cnns cnn-inference-engine cnn ai-benchmarks speed-benchmarks inference-engine
151

inference-engine

FedML-AI / FedML

zjhellofss / KuiperInfer

hyperjumptech / grule-rule-engine

siliconflow / onediff

aphrodite-engine / aphrodite-engine

Tencent / FeatherCNN

PaddlePaddle / Paddle.js

zhihu / ZhiLight

quic / ai-hub-models

Adlik / Adlik

msnh2012 / Msnhnet

insight-platform / Savant

jd-opensource / xllm

ovg-project / kvcached

pylint-dev / astroid

Tencent / Forward

PaddlePaddle / Anakin

andrewkchan / yalm

HoloClean / holoclean

buguroo / pyknow

ulfurinn / wongi-engine

zjhellofss / KuiperLLama

chengzeyi / ParaAttention

ReactiveBayes / RxInfer.jl

quic / ai-hub-apps

interestingLSY / swiftLLM

EfficientMoE / MoE-Infinity

SearchSavior / OpenArc

gottingen / kumo-search

ROCm / MIVisionX

BMW-InnovationLab / BMW-TensorFlow-Inference-API-CPU

nilp0inter / experta

midea-ai / Aidget

tuanlda78202 / gpt-oss-amd

matteocarnelos / microflow-rs

CAS-CLab / CNN-Inference-Engine-Quick-View