tensorrt-llm

There are 0 repository under tensorrt-llm topic.

DefTruth / Awesome-LLM-Inference
📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.
awesome-llm deepseek flash-attention flash-attention-2 flash-attention-3 llm llm-inference llms open-sora paged-attention sora tensorrt-llm vllm
2293
janhq / cortex
Drop-in, local AI alternative to the OpenAI stack. Multi-engine (llama.cpp, TensorRT-LLM, ONNX). Powers 👋 Jan
gguf llama2 llamacpp tensorrt-llm accelerated ai inference-engine openai-api stable-diffusion cuda llama llm llms
Language:C++ 1860
collabora / WhisperLive
A nearly-live implementation of OpenAI's Whisper.
dictation obs openai tensorrt tensorrt-llm text-to-speech translation voice-recognition whisper whisper-tensorrt
Language:Python 1699
shashikg / WhisperS2T
An Optimized Speech-to-Text Pipeline for the Whisper Model Supporting Multiple Inference Engine
asr deep-learning speech-recognition speech-to-text whisper tensorrt-llm tensorrt vad voice-activity-detection
Language:Jupyter Notebook 267
huggingface / optimum-benchmark
🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.
benchmark onnxruntime openvino pytorch neural-compressor tensorrt-llm text-generation-inference
Language:Python 221
npuichigo / openai_trtllm
OpenAI compatible API for TensorRT LLM triton backend
langchain llm openai-api tensorrt-llm triton-inference-server
Language:Rust 136
openhackathons-org / End-to-End-LLM
This repository is an AI Bootcamp material that consist of a workflow for LLM
deep-learning natural-language-processing p-tuning prompt-tuning nemo-megatron llm nemo-guardrails question-answering tensorrt-llm genai
Language:Jupyter Notebook 56
rpehkone / Chat-With-RTX-python-api
Chat With RTX Python API
llama2-13b llm llm-inference mistral-7b chat-with-rtx tensorrt tensorrt-llm nvidia-chat-with-rtx
Language:Python 53
janhq / cortex.tensorrt-llm
Cortex.Tensorrt-LLM is a C++ inference library that can be loaded by any server at runtime. It submodules NVIDIA’s TensorRT-LLM for GPU accelerated inference on NVIDIA's GPUs.
jan llm tensorrt tensorrt-llm nvidia
Language:C++ 34
fgblanch / OutlookLLM
Add-in for new Outlook that adds LLM new features (Composition, Summarizing, Q&A). It uses a local LLM via Nvidia TensorRT-LLM
outlook-addin tensorrt-llm
Language:Python 15
EdVince / whisper-trtllm
Whisper in TensorRT-LLM
asr cuda huggingface openai tensorrt tensorrt-llm transformers whisper
Language:C++ 14
zRzRzRzRzRzRzR / lm-fly
大模型推理框架加速，让 LLM 飞起来
llm llm-inference mlx openvino tensorrt-llm tgi vllm
Language:Python 14
lix19937 / llm-deploy
AI Infra LLM infer/ tensorrt-llm/ vllm
llm llm-inference tensorrt-llm
Language:Python 6
CactusQ / TensorRT-LLM-Tutorial
Getting started with TensorRT-LLM using BLOOM as a case study
deeplearning jupyter-notebook llm-inference llms tensorrt tensorrt-inference tensorrt-llm
Language:Jupyter Notebook 4
j3soon / LLM-Tutorial
LLM tutorial materials include but not limited to NVIDIA NeMo, TensorRT-LLM, Triton Inference Server, and NeMo Guardrails.
llm nemo nemo-guardrails nvidia-nemo tensorrt-llm
Language:Jupyter Notebook 2
nyunAI / TensorRT-LLM
tensorrt-engine tensorrt-llm
Language:C++ 0
cyanff / nyxt
ai extension-chrome tensorrt tensorrt-llm
Language:TypeScript

tensorrt-llm

DefTruth / Awesome-LLM-Inference

janhq / cortex

collabora / WhisperLive

shashikg / WhisperS2T

huggingface / optimum-benchmark

npuichigo / openai_trtllm

openhackathons-org / End-to-End-LLM

rpehkone / Chat-With-RTX-python-api

janhq / cortex.tensorrt-llm

fgblanch / OutlookLLM

EdVince / whisper-trtllm

zRzRzRzRzRzRzR / lm-fly

lix19937 / llm-deploy

CactusQ / TensorRT-LLM-Tutorial

j3soon / LLM-Tutorial

nyunAI / TensorRT-LLM

cyanff / nyxt