lilhongxy

makemore

An autoregressive character-level language model for making more things

hotpot

LongLM

[ICML'24 Spotlight] LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

yarn

YaRN: Efficient Context Window Extension of Large Language Models

Parallel-Context-Windows

ChunkLlama

[ICML'24] Data and code for our paper "Training-Free Long-Context Scaling of Large Language Models"

attention_with_linear_biases

Code for the ALiBi method for transformer language models (ICLR 2022)

landmark-attention

Landmark Attention: Random-Access Infinite Context Length for Transformers

streaming-llm

[ICLR 2024] Efficient Streaming Language Models with Attention Sinks

CLEX

[ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models

LongBench

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding