anttttti

linux

Linux kernel source tree

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

LocalAI

:robot: The free, Open Source OpenAI alternative. Self-hosted, community-driven and local-first. Drop-in replacement for OpenAI running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. It allows to generate Text, Audio, Video, Images. Also with voice cloning capabilities.

FinGPT

FinGPT: Open-Source Financial Large Language Models! Revolutionize 🔥 We release the trained model on HuggingFace.

qlora

QLoRA: Efficient Finetuning of Quantized LLMs

WizardLM

LLMs build upon Evol Insturct: WizardLM, WizardCoder, WizardMath

text-generation-inference

Large Language Model Text Generation Inference

axolotl

Go ahead and axolotl questions

skypilot

SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

CTranslate2

Fast inference engine for Transformer models

optimum

🚀 Accelerate training and inference of 🤗 Transformers and 🤗 Diffusers with easy to use hardware optimization tools

llm-awq

[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

lightllm

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

AutoAWQ

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

evalplus

Rigourous evaluation of LLM-synthesized code - NeurIPS 2023

llmtools

Finetuning Large Language Models on One Consumer GPU in Under 4 Bits

attention_sinks

Extend existing LLMs way beyond the original training length with constant memory usage, without retraining

Wordbatch

Python library for distributed AI processing pipelines, using swappable scheduler backends.

langstream

Build robust LLM applications with true composability 🔗

text-generation-inference

speculative-decoding

Explorations into some recent techniques surrounding speculative decoding

text-generation-inference

IBM development fork of https://github.com/huggingface/text-generation-inference

babelcode

hf-hub-ctranslate2

Connecting Transformers on HuggingFace Hub with CTranslate2

Pytorch_Merge

Merge LLM that are split in to parts

text-generation-inference

Large Language Model Text Generation Inference

PromptGPT

Prompt Engineer Agent + (GPT-4,) = PromptGPT. Autonomous, self-supportive & objective-agnostic agents.

open-text-generation-inference

Open Large Language Model Text Generation Inference - will remain Apache-2.0