yzhangcs

LMFlow

An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.

torchtune

A Native-PyTorch Library for LLM Fine-tuning

LLMTest_NeedleInAHaystack

Doing simple retrieval from LLM models at various context lengths to measure accuracy

nanotron

Minimalistic large language model 3D-parallelism training

JetMoE

Reaching LLaMA2 Performance with 0.1M Dollars

tensor_parallel

Automatically split your PyTorch models on multiple GPUs for training & inference

ByteTransformer

optimized BERT transformer inference on NVIDIA GPU. https://arxiv.org/abs/2210.03052

InfiniTransformer

Unofficial PyTorch/🤗Transformers(Gemma/Llama3) implementation of Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

long-context-attention

Sequence Parallel Attention for Long Context LLM Model Training and Inference

Rewrite-the-Stars

[CVPR 2024] Rewrite the Stars

minicons

Utility for behavioral and representational analyses of Language Models

DiJiang

[ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear attention mechanism.

mad-lab

A MAD laboratory to improve AI architecture designs 🧪

LLM-Inheritune

This is the official repository for Inheritune.

LASP

Linear Attention Sequence Parallelism (LASP)

LinearAttentionArena

Here we will test various linear attention designs.

infini-mini-transformer

This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and training code.

gpt-accelera

Simple and efficient pytorch-native transformer training and inference (batched)

ran

HGRN2

HGRN2: Gated Linear RNNs with State Expansion

rwkv

RWKV model implementation

GORU-tensorflow

Gated Orthogonal Recurrent Unit implementation in tensorflow

mamba-primer

flash-rwkv

rnn-icrag

Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"

heinsen_attention

Reference implementation of "Softmax Attention with Constant Cost per Token" (Heinsen, 2024)

hgru2-pytorch

megablocks

MLP-Attention

metaseq