inference

There are 53 repositories under inference topic.

ColossalAI
hpcaitech / ColossalAI
Making large AI models cheaper, faster and more accessible
ai big-model data-parallelism deep-learning distributed-computing foundation-models heterogeneous-training hpc inference large-scale model-parallelism pipeline-parallelism
Language:Python 38628
microsoft / DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
deep-learning pytorch gpu machine-learning billion-parameters data-parallelism model-parallelism inference pipeline-parallelism compression mixture-of-experts trillion-parameters zero
Language:Python 34810
whisper.cpp
ggerganov / whisper.cpp
Port of OpenAI's Whisper model in C/C++
inference openai speech-recognition speech-to-text transformer whisper
Language:C 34530
vllm-project / vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt llm pytorch llmops mlops model-serving transformer llm-serving inference llama amd rocm cuda inferentia trainium tpu xpu
Language:Python 27066
google-ai-edge / mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
android audio-processing c-plus-plus calculator computer-vision deep-learning framework graph-based graph-framework inference machine-learning mediapipe mobile-development perception pipeline-framework stream-processing video-processing
Language:C++ 26826
Tencent / ncnn
ncnn is a high-performance neural network inference framework optimized for the mobile platform
android arm-neon artificial-intelligence caffe darknet deep-learning high-preformance inference ios keras mlir mxnet ncnn neural-network onnx pytorch riscv simd tensorflow vulkan
Language:C++ 20165
ts-pattern
gvergnaud / ts-pattern
🎨 The exhaustive Pattern Matching library for TypeScript, with smart type inference.
branching conditions exhaustive inference javascript matching pattern pattern-matching ts type-inference typescript
Language:TypeScript 11859
SYSTRAN / faster-whisper
Faster Whisper transcription with CTranslate2
deep-learning inference openai quantization speech-recognition speech-to-text transformer whisper
Language:Python 11467
NVIDIA / TensorRT
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
deep-learning gpu-acceleration inference nvidia tensorrt
Language:C++ 10571
aws / amazon-sagemaker-examples
Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
sagemaker aws reinforcement-learning machine-learning deep-learning examples jupyter-notebook mlops data-science training inference
Language:Jupyter Notebook 9986
huggingface / text-generation-inference
Large Language Model Text Generation Inference
bloom deep-learning falcon gpt inference nlp pytorch starcoder transformer
Language:Python 8789
triton-inference-server / server
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
cloud datacenter deep-learning edge gpu inference machine-learning
Language:Python 8072
jetson-inference
dusty-nv / jetson-inference
Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
deep-learning inference computer-vision embedded image-recognition object-detection segmentation jetson jetson-tx1 jetson-tx2 jetson-xavier nvidia tensorrt digits caffe video-analytics robotics machine-learning jetson-nano jetson-xavier-nx
Language:C++ 7735
Linzaer / Ultra-Light-Fast-Generic-Face-Detector-1MB
💎1MB lightweight face detection model (1MB轻量级人脸检测模型)
arm face-detection inference mnn ncnn
Language:Python 7144
openvinotoolkit / openvino
OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference
ai computer-vision deep-learning deploy-ai diffusion-models generative-ai good-first-issue inference llm-inference natural-language-processing nlp openvino optimize-ai performance-boost recommendation-system speech-recognition stable-diffusion transformers yolo
Language:C++ 6864
gcanti / io-ts
Runtime type system for IO decoding/encoding
inference runtime types typescript validation
Language:TypeScript 6684
sgl-project / sglang
SGLang is a fast serving framework for large language models and vision language models.
cuda inference llama llava llm llm-serving moe pytorch transformer vlm llama2 llama3 llama3-1
Language:Python 5187
xorbitsai / inference
Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
artificial-intelligence chatglm deployment flan-t5 gemma ggml glm4 inference llama llama3 llamacpp llm machine-learning mistral openai-api pytorch qwen vllm whisper wizardlm
Language:Python 4842
Trusted-AI / adversarial-robustness-toolbox
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams
adversarial-attacks adversarial-examples adversarial-machine-learning ai artificial-intelligence attack blue-team evasion extraction inference machine-learning poisoning privacy python red-team trusted-ai trustworthy-ai
Language:Python 4764
superduper
superduper-io / superduper
Superduper: Integrate AI models and machine learning workflows with your database to implement custom AI applications, without moving your data. Including streaming inference, scalable model hosting, training and vector search.
ai mlops torch transformers mongodb python pytorch ml database data inference distributed-ml llm-inference pretrained-models chatbot semantic-search llm-serving llmops vector-search rag
Language:Jupyter Notebook 4657
NVIDIA-AI-IOT / torch2trt
An easy to use PyTorch to TensorRT converter
classification inference jetson-nano jetson-tx2 jetson-xavier pytorch tensorrt
Language:Python 4568
Tencent / TNN
TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capability, high performance, model compression and code pruning. Based on ncnn and Rapidnet, TNN further strengthens the support and performance optimization for mobile devices, and also draws on the advantages of good extensibility and high performance from existed open source efforts. TNN has been deployed in multiple Apps from Tencent, such as Mobile QQ, Weishi, Pitu, etc. Contributions are welcome to work in collaborative with us and make TNN a better framework.
coreml deep-learning face-detection hairsegmentaion inference mnn ncnn ocr openvino pytorch tengine tensorflow tensorrt
Language:C++ 4382
AutoGPTQ / AutoGPTQ
An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
transformers deep-learning inference large-language-models llms nlp pytorch quantization transformer
Language:Python 4342
openvinotoolkit / open_model_zoo
Pre-trained Deep Learning models and demos (high quality and extremely fast)
caffemodel cnn-model deep-learning-models demo inference model model-zoo models onnx-models openvino openvino-model-zoo openvino-models openvino-toolkit pytorch-models tensorflow-models
Language:Python 4071
typedb
typedb / typedb
TypeDB: one giant leap for databases
database inference knowledge-base knowledge-representation logic polymorphic polymorphism reasoning strongly-typed type-system typedb typeql
Language:Rust 3762
cube-studio
tencentmusic / cube-studio
cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，支持sso登录，多租户，大数据平台对接，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU，边缘计算，serverless，标注平台，自动化标注，数据集管理，大模型微调，vllm大模型推理，llmops，私有知识库，AI模型应用商店，支持模型一键开发/推理/微调，支持国产cpu/gpu/npu芯片，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/spark/ray/volcano分布式
ai aihub argo automl gpt inference kubeflow kubernetes llmops mlops notebook pipeline pytorch spark vgpu workflow
Language:Jupyter Notebook 3482
OpenNMT / CTranslate2
Fast inference engine for Transformer models
avx avx2 cpp cuda deep-learning deep-neural-networks gemm inference intrinsics machine-translation mkl neon neural-machine-translation onednn openmp opennmt parallel-computing quantization thrust transformer-models
Language:C++ 3231
bytedance / lightseq
LightSeq: A High Performance Library for Sequence Processing and Generation
accelerate bart beam-search bert cuda diverse-decoding gpt inference multilingual-nmt sampling training transformer
Language:C++ 3171
argmaxinc / WhisperKit
On-device Speech Recognition for Apple Silicon
inference ios macos pretrained-models speech-recognition swift transformers visionos watchos whisper
Language:Swift 3146
deepsparse
neuralmagic / deepsparse
Sparsity-aware deep learning inference runtime for CPUs
machinelearning onnx inference computer-vision object-detection pruning quantization pretrained-models nlp cpus sparsification llm-inference performance deepsparse
Language:Python 2979
pgmpy / pgmpy
Python Library for learning (Structure and Parameter), inference (Probabilistic and Causal), and simulations in Bayesian Networks.
bayesian-networks causal-discovery causal-identification causal-inference inference python simulations structure-learning
Language:Python 2705
huggingface / optimum
🚀 Accelerate training and inference of 🤗 Transformers and 🤗 Diffusers with easy to use hardware optimization tools
onnx pytorch inference training intel graphcore onnxruntime transformers quantization habana optimization tflite
Language:Python 2463
zjhellofss / KuiperInfer
校招、秋招、春招、实习好项目！带你从零实现一个高性能的深度学习推理库，支持大模型 llama2 、Unet、Yolov5、Resnet等模型的推理。Implement a high-performance deep learning inference library step by step
inference inference-engine deep-learning deep-neural-networks convolution relu sigmoid graph-algorithms maxpooling caffe pnnx pytorch ncnn diy resnet yolo yolov5
Language:C++ 2402
openvino_notebooks
openvinotoolkit / openvino_notebooks
📚 Jupyter notebook tutorials for OpenVINO™
computer-vision deep-learning inference machine-learning openvino
Language:Jupyter Notebook 2302
tairov / llama2.mojo
Inference Llama 2 in one file of pure 🔥
inference llama llama2 modular mojo parallelize performance simd tensor transformer-architecture vectorization
Language:Mojo 2093
microsoft / aici
AICI: Prompts as (Wasm) Programs
ai inference language-model llm llm-framework llm-inference llm-serving llmops model-serving rust transformer wasm wasmtime
Language:Rust 1901

inference

hpcaitech / ColossalAI

microsoft / DeepSpeed

ggerganov / whisper.cpp

vllm-project / vllm

google-ai-edge / mediapipe

Tencent / ncnn

gvergnaud / ts-pattern

SYSTRAN / faster-whisper

NVIDIA / TensorRT

aws / amazon-sagemaker-examples

huggingface / text-generation-inference

triton-inference-server / server

dusty-nv / jetson-inference

Linzaer / Ultra-Light-Fast-Generic-Face-Detector-1MB

openvinotoolkit / openvino

gcanti / io-ts

sgl-project / sglang

xorbitsai / inference

Trusted-AI / adversarial-robustness-toolbox

superduper-io / superduper

NVIDIA-AI-IOT / torch2trt

Tencent / TNN

AutoGPTQ / AutoGPTQ

openvinotoolkit / open_model_zoo

typedb / typedb

tencentmusic / cube-studio

OpenNMT / CTranslate2

bytedance / lightseq

argmaxinc / WhisperKit

neuralmagic / deepsparse

pgmpy / pgmpy

huggingface / optimum

zjhellofss / KuiperInfer

openvinotoolkit / openvino_notebooks

tairov / llama2.mojo

microsoft / aici