vllm

There are 1 repository under vllm topic.

meta-llama / llama-recipes
Scripts for fine-tuning Meta Llama3 with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization and Q&A. Supporting a number of candid inference solutions such as HF TGI, VLLM for local or cloud deployment. Demo apps to showcase Meta Llama3 for WhatsApp & Messenger.
ai finetuning langchain llama llama2 llm machine-learning python pytorch vllm
Language:Jupyter Notebook 9670
xorbitsai / inference
Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
artificial-intelligence chatglm chatglm3 deployment flan-t5 gemma ggml inference llama llama3 llamacpp llm machine-learning mistral openai-api pytorch qwen vllm whisper wizardlm
Language:Python 2799
DefTruth / Awesome-LLM-Inference
📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.
awesome-llm deepseek flash-attention flash-attention-2 llm llm-inference llms open-sora paged-attention sora streaming-llm tensorrt-llm vllm
1437
OpenLLMAI / OpenRLHF
An Easy-to-use, Scalable and High-performance RLHF Framework (Support 70B+ full tuning & LoRA & Mixtral & KTO)
deepspeed transformers vllm large-language-models raylib reinforcement-learning-from-human-feedback reinforcement-learning
Language:Python 1168
atfortes / Awesome-LLM-Reasoning
Reasoning in Large Language Models: Papers and Resources, including Chain-of-Thought, Instruction-Tuning and Multimodality.
language-models reasoning prompt question-answering in-context-learning chatgpt chain-of-thought prompt-engineering cot awesome gpt gpt-4 mllm multimodal papers vllm
1126
bricks-cloud / BricksLLM
🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, vLLM, and open-source LLMs.
golang llm openai ai anthropic azure gpt postgresql rest-api ycombinator api docker privacy security artificial-intelligence generative-ai open-source self-hosted vllm
Language:Go 755
varunshenoy / super-json-mode
Low latency JSON generation using LLMs ⚡️
huggingface-transformers llm openai vllm
Language:Jupyter Notebook 343
runpod-workers / worker-vllm
The RunPod worker template for serving our large language model endpoints. Powered by vLLM.
language-model llm runpod vllm
Language:Python 159
gotzmann / collider
Large Model Collider - The Platform for serving LLM models
llm chatgpt gpt llama alpaca vicuna stablelm open-assistant openai llama-cpp koboldai ggml wizardlm exllama vllm llamacpp oobabooga
Language:C++ 119
nbasyl / DoRA
Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
dora lora peft-fine-tuning-llm vllm
105
jasonacox / TinyLLM
Setup and run a local LLM and Chatbot using consumer grade hardware.
artificial-intelligence chatbot large-language-models llama-cpp-python llm openai rag retrieval-augmented-generation vllm
Language:JavaScript 98
Trainy-ai / llm-atc
Fine-tuning and serving LLMs on any cloud
finetuning llama2 llms vllm
Language:Python 82
wangcx18 / llm-vscode-inference-server
An endpoint server for efficiently serving quantized open-source LLMs for code.
llm vscode-extension llm-inference vllm
Language:Python 44
OpenCSGs / llm-inference
llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource management, monitoring, and more.
deepspeed llama-cpp llm-inference ray transformer vllm
Language:Python 41
gameofdimension / vllm-cn
演示 vllm 对中文大语言模型的神奇效果
vllm
Language:Jupyter Notebook 31
asprenger / ray_vllm_inference
A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
inference llm llm-serving llmops mlops model-serving pytorch ray transformer vllm
Language:Python 30
microsoft / vidur
A large-scale simulation framework for LLM inference
inference llm simulation transformer vllm
Language:Python 28
phospho-app / fastassert
Dockerized LLM inference server with constrained output (JSON mode), built on top of vLLM and outlines. Faster, cheaper and without rate limits. Compare the quality and latency to your current LLM API provider.
llm llm-inference docker outlines vllm
Language:Jupyter Notebook 26
OSS-Pole-Emploi / happy_vllm
A REST API for vLLM, production ready
api-rest llm llm-serving mlops production serving transformers vllm
Language:Python 21
YY0649 / ICE-PIXIU
ICE-PIXIU：A Cross-Language Financial Megamodeling Framework
nlp internlm large-language-models llama pixiu vllm
Language:Python 15
zRzRzRzRzRzRzR / lm-fly
大模型推理框架加速，让 LLM 飞起来
llm llm-inference mlx openvino tensorrt-llm tgi vllm
Language:Python 11
Climatik-Project / Climatik-Project
Carbon Limiting Auto Tuning for Kubernetes
keda kepler kubernetes llm llm-inference power-capping kserve vllm kubernetes-operator green-computing sustainability
Language:Python 9
mddunlap924 / LLM-Inference-Serving
This repository demonstrates LLM execution on CPUs using packages like llamafile, emphasizing low-latency, high-throughput, and cost-effective benefits for inference and serving.
deepspeed large-language-models llamacpp llamafile llm-inference llm-serving llms vllm
Language:Jupyter Notebook 8
yoziru / nextjs-vllm-ui
Fully-featured, beautiful web interface for vLLM - built with NextJS.
nextjs ui vllm openai-api tailwindcss typescript llm-ui vllm-ui llm-webui webui ai self-hosted
Language:TypeScript 8
iNeil77 / vllm-code-harness
Run code inference-only benchmarks quickly using vLLM
code-generation nlp-machine-learning transformers vllm
Language:Python 7
joydeb28 / llm-lab
LLM, Fine Tuning, Llama 2, Gemma, Mixtral, vLLM, LangChain, RAG, ChromaDB, FAISS
finetune-llm gemma genai llama2 llm mixtral nlp openllm vllm langchain rag chromadb faiss
Language:Jupyter Notebook 6
LLM-inference-router / vllm-router
vLLM Router
huggingface kubernetes llama2 llm llm-inference vllm
Language:Python 6
ivangabriele / docker-functionary
Ready-to-deploy Docker image for Functionary LLM served as an OpenAI-Compatible API.
ai docker docker-hub docker-image functions large-language-models llama2 llm openai openai-api server vllm functionary
Language:Dockerfile 5
ivangabriele / docker-llm
Pre-loaded LLMs served as an OpenAI-Compatible API via Docker images.
llm llms api docker openai openai-api openorca orca server vicuna vllm lmsys runpod vast llong docker-image
Language:Dockerfile 5
kyegomez / SimpleUnet
An simple implementation of Unet because all the implementations i've seen are wayy tooo complicated.
artificial-intelligence biomedical biomedical-image-processing computer-vision gpt4 image image-classification image-segmentation unet vllm texttovide
Language:Python 5
esmailza / Llama2-vLLM-LangChain-knowledge-graph
Preserving entities through the integration of knowledge graphs, Llama 2, vLLM, and LangChain.
distributed information-extraction knowledge-graph langchain llama2 named-entity-recognition python summarization vllm
Language:Python 3
lklivingstone / sih_2023
A Large Language Model based tool for generating human like responses to natural language inputs for network not connected over internet.
django linux llm reactjs scylladb vllm
Language:Python 3
lucataco / cog-deepseek-67b-base
Cog wrapper for deepseek-ai/deepseek-67b-base
cog vllm
Language:Python 2
lucataco / cog-Hermes-2-Pro-Llama-3-8B
Cog wrapper for NousResearch/Hermes-2-Pro-Llama-3-8B
cog vllm
Language:Python 2
lucataco / cog-Qwen1.5-32b
Cog wrapper for vllm implementation of Qwen/Qwen1.5-32B
cog vllm
Language:Python 2
lucataco / cog-Wizard-Vicuna-13B-Uncensored
Cog wrapper for cognitivecomputations/Wizard-Vicuna-13B-Uncensored
cog vllm
Language:Python 2

vllm

meta-llama / llama-recipes

xorbitsai / inference

DefTruth / Awesome-LLM-Inference

OpenLLMAI / OpenRLHF

atfortes / Awesome-LLM-Reasoning

bricks-cloud / BricksLLM

varunshenoy / super-json-mode

runpod-workers / worker-vllm

gotzmann / collider

nbasyl / DoRA

jasonacox / TinyLLM

Trainy-ai / llm-atc

wangcx18 / llm-vscode-inference-server

OpenCSGs / llm-inference

gameofdimension / vllm-cn

asprenger / ray_vllm_inference

microsoft / vidur

phospho-app / fastassert

OSS-Pole-Emploi / happy_vllm

YY0649 / ICE-PIXIU

zRzRzRzRzRzRzR / lm-fly

Climatik-Project / Climatik-Project

mddunlap924 / LLM-Inference-Serving

yoziru / nextjs-vllm-ui

iNeil77 / vllm-code-harness

joydeb28 / llm-lab

LLM-inference-router / vllm-router

ivangabriele / docker-functionary

ivangabriele / docker-llm

kyegomez / SimpleUnet

esmailza / Llama2-vLLM-LangChain-knowledge-graph

lklivingstone / sih_2023

lucataco / cog-deepseek-67b-base

lucataco / cog-Hermes-2-Pro-Llama-3-8B

lucataco / cog-Qwen1.5-32b

lucataco / cog-Wizard-Vicuna-13B-Uncensored