inference

There are 48 repositories under inference topic.

ColossalAI
hpcaitech / ColossalAI
Making large AI models cheaper, faster and more accessible
ai big-model data-parallelism deep-learning distributed-computing foundation-models heterogeneous-training hpc inference large-scale model-parallelism pipeline-parallelism
Language:Python 38012
microsoft / DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
billion-parameters compression data-parallelism deep-learning gpu inference machine-learning mixture-of-experts model-parallelism pipeline-parallelism pytorch trillion-parameters zero
Language:Python 32986
whisper.cpp
ggerganov / whisper.cpp
Port of OpenAI's Whisper model in C/C++
openai speech-to-text transformer whisper inference speech-recognition
Language:C 31769
google-ai-edge / mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
mediapipe c-plus-plus computer-vision deep-learning android video-processing audio-processing mobile-development machine-learning inference graph-framework graph-based calculator framework pipeline-framework stream-processing perception
Language:C++ 25649
vllm-project / vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt llm pytorch llmops mlops model-serving transformer llm-serving inference llama amd rocm cuda inferentia trainium
Language:Python 19546
Tencent / ncnn
ncnn is a high-performance neural network inference framework optimized for the mobile platform
inference high-preformance simd arm-neon deep-learning artificial-intelligence android ios ncnn vulkan neural-network caffe mxnet pytorch onnx darknet tensorflow mlir keras riscv
Language:C++ 19340
ts-pattern
gvergnaud / ts-pattern
🎨 The exhaustive Pattern Matching library for TypeScript, with smart type inference.
branching conditions exhaustive inference javascript matching pattern pattern-matching ts type-inference typescript
Language:TypeScript 10988
aws / amazon-sagemaker-examples
Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
sagemaker aws reinforcement-learning machine-learning deep-learning examples jupyter-notebook mlops data-science training inference
Language:Jupyter Notebook 9552
NVIDIA / TensorRT
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
tensorrt nvidia deep-learning inference gpu-acceleration
Language:C++ 9209
SYSTRAN / faster-whisper
Faster Whisper transcription with CTranslate2
deep-learning inference quantization speech-recognition speech-to-text transformer whisper openai
Language:Python 9178
huggingface / text-generation-inference
Large Language Model Text Generation Inference
bloom nlp pytorch inference gpt deep-learning transformer falcon starcoder
Language:Python 8028
triton-inference-server / server
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
inference gpu machine-learning deep-learning cloud datacenter edge
Language:Python 7428
jetson-inference
dusty-nv / jetson-inference
Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
deep-learning inference computer-vision embedded image-recognition object-detection segmentation jetson jetson-tx1 jetson-tx2 jetson-xavier nvidia tensorrt digits caffe video-analytics robotics machine-learning jetson-nano jetson-xavier-nx
Language:C++ 7402
Linzaer / Ultra-Light-Fast-Generic-Face-Detector-1MB
💎1MB lightweight face detection model (1MB轻量级人脸检测模型)
arm face-detection inference mnn ncnn
Language:Python 7047
gcanti / io-ts
Runtime type system for IO decoding/encoding
typescript validation inference types runtime
Language:TypeScript 6613
openvinotoolkit / openvino
OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference
inference deep-learning openvino ai computer-vision diffusion-models generative-ai llm-inference natural-language-processing nlp performance-boost speech-recognition stable-diffusion deploy-ai optimize-ai transformers yolo recommendation-system good-first-issue
Language:C++ 6025
Trusted-AI / adversarial-robustness-toolbox
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams
python attack adversarial-machine-learning poisoning trusted-ai artificial-intelligence extraction adversarial-attacks adversarial-examples evasion inference privacy ai trustworthy-ai red-team blue-team machine-learning
Language:Python 4502
NVIDIA-AI-IOT / torch2trt
An easy to use PyTorch to TensorRT converter
jetson-nano jetson-tx2 jetson-xavier pytorch tensorrt inference classification
Language:Python 4411
superduperdb
SuperDuperDB / superduperdb
🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalable model training and vector search.
ai mlops torch transformers mongodb python pytorch ml database data inference distributed-ml llm-inference pretrained-models chatbot semantic-search llm-serving llmops vector-search rag
Language:Python 4407
Tencent / TNN
TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capability, high performance, model compression and code pruning. Based on ncnn and Rapidnet, TNN further strengthens the support and performance optimization for mobile devices, and also draws on the advantages of good extensibility and high performance from existed open source efforts. TNN has been deployed in multiple Apps from Tencent, such as Mobile QQ, Weishi, Pitu, etc. Contributions are welcome to work in collaborative with us and make TNN a better framework.
deep-learning mnn ncnn inference pytorch tensorflow coreml tensorrt tengine openvino face-detection hairsegmentaion ocr
Language:C++ 4297
openvinotoolkit / open_model_zoo
Pre-trained Deep Learning models and demos (high quality and extremely fast)
models caffemodel demo tensorflow-models model-zoo model deep-learning-models cnn-model openvino inference onnx-models pytorch-models openvino-model-zoo openvino-models openvino-toolkit
Language:Python 3966
AutoGPTQ / AutoGPTQ
An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
transformers deep-learning inference large-language-models llms nlp pytorch quantization transformer
Language:Python 3864
typedb
vaticle / typedb
TypeDB: the polymorphic database powered by types
database knowledge-base knowledge-representation reasoning logic inference typedb typeql type-system strongly-typed polymorphic polymorphism
Language:Java 3685
bytedance / lightseq
LightSeq: A High Performance Library for Sequence Processing and Generation
accelerate bart beam-search bert cuda diverse-decoding gpt inference multilingual-nmt sampling training transformer
Language:C++ 3108
deepsparse
neuralmagic / deepsparse
Sparsity-aware deep learning inference runtime for CPUs
machinelearning onnx inference computer-vision object-detection pruning quantization pretrained-models nlp cpus sparsification llm-inference performance deepsparse
Language:Python 2882
OpenNMT / CTranslate2
Fast inference engine for Transformer models
neural-machine-translation cpp mkl quantization cuda thrust opennmt deep-neural-networks openmp onednn intrinsics avx2 avx parallel-computing gemm neon transformer-models machine-translation deep-learning inference
Language:C++ 2852
xorbitsai / inference
Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
ggml pytorch chatglm deployment flan-t5 llm wizardlm artificial-intelligence machine-learning whisper inference openai-api mistral gemma llama llamacpp vllm qwen chatglm3 llama3
Language:Python 2804
pgmpy / pgmpy
Python Library for learning (Structure and Parameter), inference (Probabilistic and Causal), and simulations in Bayesian Networks.
python probabilistic-graphical-models bayesian-networks causal-inference structure-learning causal-discovery directed-acyclic-graph inference simulations
Language:Python 2626
cube-studio
tencentmusic / cube-studio
cube studio开源云原生一站式机器学习/深度学习AI平台，支持sso登录，多租户/多项目组，大数据平台对接，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU，边缘计算，serverless，标注平台，自动化标注，数据集管理，大模型微调，vllm大模型推理，llmops，私有知识库，AI模型应用商店，支持模型一键开发/推理/微调，支持国产cpu/gpu/npu芯片，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/spark/ray/volcano分布式
kubernetes inference mlops workflow ai pytorch spark argo kubeflow automl aihub gpt llmops notebook pipeline vgpu
Language:Jupyter Notebook 2573
HuaizhengZhang / Awesome-System-for-Machine-Learning
A curated list of research in machine learning systems (MLSys). Paper notes are also provided.
machine-learning system deep-learning optimization paper resouce-management deep-reinforcement-learning edge-computing computer-system distributed-systems infrastructure deep-neural-networks automl model-database inference
2442
argmaxinc / WhisperKit
Swift native on-device speech recognition with Whisper for Apple Silicon
inference ios pretrained-models speech-recognition swift whisper transformers macos visionos watchos
Language:Swift 2331
huggingface / optimum
🚀 Accelerate training and inference of 🤗 Transformers and 🤗 Diffusers with easy to use hardware optimization tools
onnx pytorch inference training intel graphcore onnxruntime transformers quantization habana optimization tflite
Language:Python 2190
zjhellofss / KuiperInfer
带你从零实现一个高性能的深度学习推理库，支持大模型 llama2 、Unet、Yolov5、Resnet等模型的推理。Implement a high-performance deep learning inference library step by step
inference inference-engine deep-learning deep-neural-networks convolution relu sigmoid graph-algorithms maxpooling caffe pnnx pytorch ncnn diy resnet yolo yolov5
Language:C++ 2026
openvino_notebooks
openvinotoolkit / openvino_notebooks
📚 Jupyter notebook tutorials for OpenVINO™
deep-learning machine-learning computer-vision openvino inference
Language:Jupyter Notebook 2000
tairov / llama2.mojo
Inference Llama 2 in one file of pure 🔥
inference llama llama2 modular mojo parallelize performance simd tensor transformer-architecture vectorization
Language:Mojo 1966
tobegit3hub / tensorflow_template_application
TensorFlow template application for deep learning
cnn csv deep-learning inference libsvm lstm machine-learning mlp serving tensorboard tensorflow tfrecords wide-and-deep
Language:Python 1867

inference

hpcaitech / ColossalAI

microsoft / DeepSpeed

ggerganov / whisper.cpp

google-ai-edge / mediapipe

vllm-project / vllm

Tencent / ncnn

gvergnaud / ts-pattern

aws / amazon-sagemaker-examples

NVIDIA / TensorRT

SYSTRAN / faster-whisper

huggingface / text-generation-inference

triton-inference-server / server

dusty-nv / jetson-inference

Linzaer / Ultra-Light-Fast-Generic-Face-Detector-1MB

gcanti / io-ts

openvinotoolkit / openvino

Trusted-AI / adversarial-robustness-toolbox

NVIDIA-AI-IOT / torch2trt

SuperDuperDB / superduperdb

Tencent / TNN

openvinotoolkit / open_model_zoo

AutoGPTQ / AutoGPTQ

vaticle / typedb

bytedance / lightseq

neuralmagic / deepsparse

OpenNMT / CTranslate2

xorbitsai / inference

pgmpy / pgmpy

tencentmusic / cube-studio

HuaizhengZhang / Awesome-System-for-Machine-Learning

argmaxinc / WhisperKit

huggingface / optimum

zjhellofss / KuiperInfer

openvinotoolkit / openvino_notebooks

tairov / llama2.mojo

tobegit3hub / tensorflow_template_application