yzhangcs

minGPT

A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training

LMFlow

An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.

lit-llama

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

grok

JetMoE

Reaching LLaMA2 Performance with 0.1M Dollars

punica

Serving multiple LoRA finetuned LLM as one

flash-attention-minimal

Flash Attention in ~100 lines of CUDA (forward pass only)

Rewrite-the-Stars

[CVPR 2024] Rewrite the Stars

long-context-attention

Sequence Parallel Attention for Long Context LLM Model Training and Inference

minicons

Utility for behavioral and representational analyses of Language Models

DiJiang

[ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear attention mechanism.

hippogriff

Griffin MQA + Hawk Linear RNN Hybrid

mixinglaws

mad-lab

A MAD laboratory to improve AI architecture designs 🧪

rotational-unit-of-memory

RUM

stk

DenseFormer

Counting-Stars

Counting-Stars (★)

flash_attn_jax

JAX bindings for Flash Attention v2

LASP

Linear Attention Sequence Parallelism (LASP)

gpt-accelera

Simple and efficient pytorch-native transformer training and inference (batched)

GORU-tensorflow

Gated Orthogonal Recurrent Unit implementation in tensorflow

rnn-icrag

Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"

ParallelTokenizer

Use the tokenizer in parallel to achieve superior acceleration

based-evaluation-harness

A framework for few-shot evaluation of language models.

flash-attention-lookahead

NBCE-master

LLMTest_NeedleInAHaystack_HFModel

Support huggingface model to do simple retrieval from LLM models at various context lengths to measure accuracy

metaseq

seq-test

Understand and test language model architectures on synthetic tasks.