rawsh

nm-vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

litgpt

Load, pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.

lit-llama

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

ragas

Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines

REMEDI

Inspecting and Editing Knowledge Representations in Language Models

gpt-prompt-engineer

litellm

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)

dspy-rag-fastapi

FastAPI wrapper around DSPy

memory-compressed-attention

Implementation of Memory-Compressed Attention, from the paper "Generating Wikipedia By Summarizing Long Sequences"

optimum-nvidia

snorkel

A system for quickly generating training data with weak supervision

aphrodite-engine

PygmalionAI's large-scale inference engine

tensorrtllm_backend

The Triton TensorRT-LLM Backend

raptor

The official implementation of RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval

neural-cherche

Neural Search

llama2-burn

Llama2 LLM ported to Rust burn

burn

Burn is a new comprehensive dynamic Deep Learning Framework built using Rust with extreme flexibility, compute efficiency and portability as its primary goals.

token-hawk

WebGPU LLM inference tuned by hand

refunction

Reusing containers for faster serverless function execution - Masters Project @ Imperial College

serverless-dns

The RethinkDNS resolver that deploys to Cloudflare Workers, Deno Deploy, Fastly, and Fly.io

workerd

The JavaScript / Wasm runtime that powers Cloudflare Workers

wonnx

A WebGPU-accelerated ONNX inference run-time written 100% in Rust, ready for native and the web

LongMamba

Some preliminary explorations of Mamba's context scaling.

mamba

Mamba SSM architecture

MEGABYTE-pytorch

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

danswer

Gen-AI Chat for Teams - Think ChatGPT if it had access to your team's unique knowledge.

recurrent-memory-transformer

[NeurIPS 22] [AAAI 24] Recurrent Transformer-based long-context architecture.

compressive-transformer-pytorch

Pytorch implementation of Compressive Transformers, from Deepmind

block-recurrent-transformer-pytorch

Implementation of Block Recurrent Transformer - Pytorch

llmperf-leaderboard