lliquid

LLMs-from-scratch

Implementing a ChatGPT-like LLM in PyTorch from scratch, step by step

tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

lm-evaluation-harness

A framework for few-shot evaluation of language models.

DiT

Official PyTorch Implementation of "Scalable Diffusion Models with Transformers"

gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

sglang

SGLang is a fast serving framework for large language models and vision language models.

tiny-cuda-nn

Lightning fast C++/CUDA neural network framework

mamba-minimal

Simple, minimal implementation of the Mamba SSM in one file of PyTorch.

Medusa

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Awesome-Text2SQL

Curated tutorials and resources for Large Language Models, Text2SQL, Text2DSL、Text2API、Text2Vis and more.

smoothquant

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

streaming

A Data Streaming Library for Efficient Neural Network Training

flashinfer

FlashInfer: Kernel Library for LLM Serving

blocksparse

Efficient GPU kernels for block-sparse matrix multiplication and convolution

extension-cpp

C++ extensions in PyTorch

dolma

Data and tools for generating and inspecting OLMo pre-training data.

safari

Convolutions for Sequence Modeling

llama-moe

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

marlin

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

megalodon

Reference implementation of Megalodon 7B model

NVTX

The NVIDIA® Tools Extension SDK (NVTX) is a C-based Application Programming Interface (API) for annotating events, code ranges, and resources in your applications.

QuaRot

Code for QuaRot, an end-to-end 4-bit inference of large language models.

optimum-neuron

Easy, fast and very cheap training and inference on AWS Trainium and Inferentia chips.

CrossNER

CrossNER: Evaluating Cross-Domain Named Entity Recognition (AAAI-2021)

MoEfication

zeno-hub

AI Evaluation Platform

hyena

JAX/Flax implementation of the Hyena Hierarchy

cfx-article-src

PromptNER

Prompting For Named Entity Recognition