yzhangcs

The-Art-of-Linear-Algebra

Graphic notes on Gilbert Strang's "Linear Algebra for Everyone"

LLaMA-Adapter

[ICLR 2024] Fine-tuning LLaMA to follow Instructions within 1 Hour and 1.2M Parameters

gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

gradient-checkpointing

Make huge neural nets fit in memory

gptq

Code for the ICLR 2023 paper "GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers".

S-LoRA

S-LoRA: Serving Thousands of Concurrent LoRA Adapters

segment-anything-fast

A batched offline inference oriented version of segment-anything

LookaheadDecoding

Yuan-2.0

Yuan 2.0 Large Language Model

linear-attention-transformer

Transformer based on a variant of attention that is linear complexity in respect to sequence length

NBCE

Naive Bayes-based Context Extension

AdaLoRA

AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR 2023).

RetNet

Huggingface compatible implementation of RetNet (Retentive Networks, https://arxiv.org/pdf/2307.08621.pdf) including parallel, recurrent, and chunkwise forward.

long-llms-learning

A repository sharing the literatures about long-context large language models, including the methodologies and the evaluation benchmarks

Mega-pytorch

Implementation of Mega, the Single-head Attention with Multi-headed EMA architecture that currently holds SOTA on Long Range Arena

LightSeq

Official repository for LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers

LongQLoRA

LongQLoRA: Extent Context Length of LLMs Efficiently

checkmate

Training neural networks in TensorFlow 2.0 with 5x less memory

lq-lora

GAU-alpha

基于Gated Attention Unit的Transformer模型（尝鲜版）

grouped-query-attention-pytorch

(Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (https://arxiv.org/pdf/2305.13245.pdf)

scaling_mlps

minimal-LRU

Non official implementation of the Linear Recurrent Unit (LRU, Orvieto et al. 2023)

QKNorm

Code for the paper "Query-Key Normalization for Transformers"

make-lstm-great-again

Donald Trump's tweets generator

fast_feedforward_computation

Official code for "Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving", ICML 2021

mRNN-mLSTM

Code for ICML 2020 paper: Do RNN and LSTM have Long Memory?

not-from-scratch

llama2Rnn.c

Decoding-as-a-dynamic-program-for-recurrent-autoregressive-models