inference

There are 31 repositories under inference topic.

ColossalAI
hpcaitech / ColossalAI
Making large AI models cheaper, faster and more accessible
ai big-model data-parallelism deep-learning distributed-computing foundation-models heterogeneous-training hpc inference large-scale model-parallelism pipeline-parallelism
Language:Python 30043
microsoft / DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
deep-learning pytorch gpu machine-learning billion-parameters data-parallelism model-parallelism inference pipeline-parallelism compression mixture-of-experts trillion-parameters zero
Language:Python 25472
google / mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
android audio-processing c-plus-plus calculator computer-vision deep-learning framework graph-based graph-framework inference machine-learning mediapipe mobile-development perception pipeline-framework stream-processing video-processing
Language:C++ 22025
whisper.cpp
ggerganov / whisper.cpp
Port of OpenAI's Whisper model in C/C++
inference openai speech-recognition speech-to-text transformer whisper
Language:C 19092
Tencent / ncnn
ncnn is a high-performance neural network inference framework optimized for the mobile platform
inference high-preformance simd arm-neon deep-learning artificial-intelligence android ios ncnn vulkan neural-network caffe mxnet pytorch onnx darknet tensorflow mlir keras riscv
Language:C++ 17280
aws / amazon-sagemaker-examples
Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
sagemaker aws reinforcement-learning machine-learning deep-learning examples jupyter-notebook mlops data-science training inference
Language:Jupyter Notebook 8436
nebuly-ai / nebuly
The next-generation platform to monitor and optimize your AI costs in one place 🚀
deep-learning pytorch onnx machine-learning tensorflow tvm openvino tensorrt huggingface compiler computing inference neural-network pypi quantization edge-computing gpu optimization transformers
Language:Python 8163
NVIDIA / TensorRT
NVIDIA® TensorRT™, an SDK for high-performance deep learning inference, includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for inference applications.
tensorrt nvidia deep-learning inference
Language:C++ 7279
ts-pattern
gvergnaud / ts-pattern
🎨 The exhaustive Pattern Matching library for TypeScript, with smart type inference.
pattern-matching typescript ts pattern matching inference type-inference exhaustive conditions branching javascript
Language:TypeScript 7045
Linzaer / Ultra-Light-Fast-Generic-Face-Detector-1MB
💎1MB lightweight face detection model (1MB轻量级人脸检测模型)
face-detection arm inference mnn ncnn
Language:Python 6724
jetson-inference
dusty-nv / jetson-inference
Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
deep-learning inference computer-vision embedded image-recognition object-detection segmentation jetson jetson-tx1 jetson-tx2 jetson-xavier nvidia tensorrt digits caffe video-analytics robotics machine-learning jetson-nano jetson-xavier-nx
Language:C++ 6449
gcanti / io-ts
Runtime type system for IO decoding/encoding
inference runtime types typescript validation
Language:TypeScript 6278
triton-inference-server / server
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
inference gpu machine-learning deep-learning cloud datacenter edge
Language:Python 5423
openvinotoolkit / openvino
OpenVINO™ Toolkit repository
inference-engine inference deep-learning performance openvino model-optimizer
Language:C++ 4331
Tencent / TNN
TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capability, high performance, model compression and code pruning. Based on ncnn and Rapidnet, TNN further strengthens the support and performance optimization for mobile devices, and also draws on the advantages of good extensibility and high performance from existed open source efforts. TNN has been deployed in multiple Apps from Tencent, such as Mobile QQ, Weishi, Pitu, etc. Contributions are welcome to work in collaborative with us and make TNN a better framework.
deep-learning mnn ncnn inference pytorch tensorflow coreml tensorrt tengine openvino face-detection hairsegmentaion ocr
Language:C++ 3988
NVIDIA-AI-IOT / torch2trt
An easy to use PyTorch to TensorRT converter
jetson-nano jetson-tx2 jetson-xavier pytorch tensorrt inference classification
Language:Python 3970
Trusted-AI / adversarial-robustness-toolbox
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams
python attack adversarial-machine-learning poisoning trusted-ai artificial-intelligence extraction adversarial-attacks adversarial-examples evasion inference privacy ai trustworthy-ai red-team blue-team machine-learning
Language:Python 3720
openvinotoolkit / open_model_zoo
Pre-trained Deep Learning models and demos (high quality and extremely fast)
models caffemodel demo tensorflow-models model-zoo model deep-learning-models cnn-model openvino inference onnx-models pytorch-models openvino-model-zoo openvino-models openvino-toolkit
Language:Python 3629
typedb
vaticle / typedb
TypeDB: a strongly-typed database
database enterprise-knowledge-graph graph graph-database graph-theory graph-visualisation graphdb hyper-relational inference knowledge-base knowledge-graph knowledge-representation logic reasoning relational strongly-typed type-system typedb typeql
Language:Java 3470
bytedance / lightseq
LightSeq: A High Performance Library for Sequence Processing and Generation
inference transformer beam-search bert cuda sampling diverse-decoding multilingual-nmt training bart gpt accelerate
Language:C++ 2780
guillaumekln / faster-whisper
Faster Whisper transcription with CTranslate2
deep-learning inference quantization speech-recognition speech-to-text transformer whisper openai
Language:Python 2390
HuaizhengZhang / Awesome-System-for-Machine-Learning
A curated list of research in machine learning systems (MLSys). Paper notes are also provided.
machine-learning system deep-learning optimization paper resouce-management deep-reinforcement-learning edge-computing computer-system distributed-systems infrastructure deep-neural-networks automl model-database inference
2009
tobegit3hub / tensorflow_template_application
TensorFlow template application for deep learning
tensorflow tfrecords libsvm csv deep-learning machine-learning mlp cnn lstm inference tensorboard wide-and-deep serving
Language:Python 1869
huggingface / text-generation-inference
Large Language Model Text Generation Inference
bloom nlp pytorch inference gpt deep-learning transformer falcon starcoder
Language:Python 1660
Maratyszcza / NNPACK
Acceleration package for neural networks on multi-core CPUs
neural-network neural-networks convolutional-layers inference high-performance high-performance-computing simd cpu multithreading fast-fourier-transform winograd-transform matrix-multiplication
Language:C 1613
Delta-ML / delta
DELTA is a deep learning based natural language and speech processing platform.
nlp deep-learning tensorflow speech sequence-to-sequence seq2seq speech-recognition text-classification speaker-verification nlu text-generation emotion-recognition tensorflow-serving tensorflow-lite inference asr serving front-end custom-ops ops
Language:Python 1549
deepsparse
neuralmagic / deepsparse
Inference runtime offering GPU-class performance on CPUs and APIs to integrate ML into your application
ml machinelearning pytorch onnx deepsparse-engine inference computer-vision object-detection pruning quantization pretrained-models nlp auto-ml cpus sparsification cpu-inference-api
Language:Python 1496
google / XNNPACK
High-efficiency floating-point neural network inference operators for mobile, server, and Web
convolutional-neural-network convolutional-neural-networks cpu inference inference-optimization matrix-multiplication mobile-inference multithreading neural-network neural-networks simd
Language:C 1442
transformer-deploy
ELS-RD / transformer-deploy
Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀
deep-learning deployment inference machine-learning natural-language-processing server
Language:Python 1407
Tencent / TurboTransformers
a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.
nlp transformer bert decoder gpu machine-translation inference huggingface-transformers pytorch albert roberta gpt2
Language:C++ 1320
budgetml
ebhy / budgetml
Deploy a ML inference service on a budget in less than 10 lines of code.
mlops deployment machine-learning api fastapi inference data-science
Language:Python 1312
huggingface / optimum
🚀 Accelerate training and inference of 🤗 Transformers and 🤗 Diffusers with easy to use hardware optimization tools
onnx pytorch inference training intel graphcore onnxruntime transformers quantization habana optimization tflite
Language:Python 1216
cube-studio
tencentmusic / cube-studio
云原生一站式机器学习平台，多租户，数据资产，notebook在线开发，拖拉拽任务流编排，多机多卡分布式训练，超参搜索，推理服务，多集群调度，多项目组资源组，边缘计算，大模型实时训练, ai应用商店
kubernetes inference mlops workflow ai pytorch spark argo kubeflow automl
Language:Jupyter Notebook 1185
openvino_notebooks
openvinotoolkit / openvino_notebooks
📚 Jupyter notebook tutorials for OpenVINO™
deep-learning machine-learning computer-vision openvino inference
Language:Jupyter Notebook 1160
kamalkraj / BERT-NER
Pytorch-Named-Entity-Recognition-with-BERT
bert named-entity-recognition pytorch conll-2003 cpp11 bert-ner inference curl postman pretrained-models
Language:Python 1133
tucan9389 / awesome-ml-demos-with-ios
The challenge projects for Inferencing machine learning models on iOS
ios machine-learning coreml mlkit tensorflow tensorflow-lite demo awesome inference
Language:Python 1058

inference

hpcaitech / ColossalAI

microsoft / DeepSpeed

google / mediapipe

ggerganov / whisper.cpp

Tencent / ncnn

aws / amazon-sagemaker-examples

nebuly-ai / nebuly

NVIDIA / TensorRT

gvergnaud / ts-pattern

Linzaer / Ultra-Light-Fast-Generic-Face-Detector-1MB

dusty-nv / jetson-inference

gcanti / io-ts

triton-inference-server / server

openvinotoolkit / openvino

Tencent / TNN

NVIDIA-AI-IOT / torch2trt

Trusted-AI / adversarial-robustness-toolbox

openvinotoolkit / open_model_zoo

vaticle / typedb

bytedance / lightseq

guillaumekln / faster-whisper

HuaizhengZhang / Awesome-System-for-Machine-Learning

tobegit3hub / tensorflow_template_application

huggingface / text-generation-inference

Maratyszcza / NNPACK

Delta-ML / delta

neuralmagic / deepsparse

google / XNNPACK

ELS-RD / transformer-deploy

Tencent / TurboTransformers

ebhy / budgetml

huggingface / optimum

tencentmusic / cube-studio

openvinotoolkit / openvino_notebooks

kamalkraj / BERT-NER

tucan9389 / awesome-ml-demos-with-ios