sustcsonglin

flash-linear-attention

Efficient implementations of state-of-the-art linear attention models in Pytorch and Triton

TN-PCFG

source code of NAACL2021 "PCFGs Can Do Better: Inducing Probabilistic Context-Free Grammars with Many Symbols“ and ACL2021 main conference "Neural Bilexicalized PCFG Induction"

flash-linear-rnn

Implementations of various linear RNN layers using pytorch and triton

mamba-triton

gated_linear_attention_layer

disco-pointer

Official Implementation of ACL2023: Don't Parse, Choose Spans! Continuous and Discontinuous Constituency Parsing via Autoregressive Span Selection

TN-LCFRS

Official Implementation of ACL2023: Unsupervised Discontinuous Constituency Parsing with Mildly Context-Sensitive Grammars

FlagAttention

A collection of memory efficient attention operators implemented in the Triton language.

lit-gpt

Hackable implementation of state-of-the-art open-source LLMs based on nanoGPT. Supports flash attention, 4-bit and 8-bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

cuda-playground

mamba.py

An efficient Mamba implementation in PyTorch and MLX.

nanokitchen

Parallel Associative Scan for Language Models

safari

Convolutions for Sequence Modeling

stk

streaming-llm

Efficient Streaming Language Models with Attention Sinks

sustcsonglin.github.io

:page_facing_up: Elegant & friendly homepage (bio, tech portfolio, resume, doc...) template with Markdown and VuePress

sustcsonglin_old.github.io

:page_facing_up: Elegant & friendly homepage (bio, tech portfolio, resume, doc...) template with Markdown and VuePress

TinyLlama

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.

zoology

Understand and test language model architectures on synthetic tasks.

Academic-project-page-template

A project page template for academic papers. Demo at https://eliahuhorwitz.github.io/Academic-project-page-template/

BeamTreeRecursiveCells

cutlass-kernels

hyena-dna

Official implementation for HyenaDNA, a long-range genomic foundation model built with Hyena

m2

Monarch Mixer

mamba

S5

s5-pytorch

Pytorch implementation of Simplified Structured State-Spaces for Sequence Modeling (S5)

SGEMM_CUDA

Fast CUDA matrix multiplication from scratch

stack-attention

Code for the paper "Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns"

state-spaces

Sequence Modeling with Structured State Spaces