long-context

There are 1 repository under long-context topic.

InternLM / InternLM
Official release of InternLM2 7B and 20B base and chat models. 200K context support
chatbot gpt large-language-model long-context rlhf fine-tuning-llm llm chinese flash-attention pretrained-models
Language:Python 5433
dvlab-research / LongLoRA
Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)
fine-tuning-llm large-language-models llm long-context lora
Language:Python 2507
lucidrains / MEGABYTE-pytorch
Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch
artificial-intelligence deep-learning learned-tokenization attention-mechanisms long-context transformers
Language:Python 593
lhao499 / RingAttention
Transformers with Arbitrarily Large Context
large-language-models long-context memory-efficient transformers
Language:Python 552
THUDM / LongBench
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
benchmark llm long-context longtext
Language:Python 509
lucidrains / ring-attention-pytorch
Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
attention-mechanism distributed-attention efficient-attention long-context
Language:Python 386
lucidrains / recurrent-memory-transformer-pytorch
Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch
artificial-intelligence attention-mechanisms deep-learning transformers long-context memory recurrence
Language:Python 384
dingo-actual / infini-transformer
PyTorch implementation of Infini-Transformer from "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" (https://arxiv.org/abs/2404.07143)
attention-mechanism deep-learning infini-attention long-context mixture-of-depths pytorch transformers
Language:Python 226
thunlp / InfLLM
The code of our paper "InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory"
large-language-models llm long-context training-free
Language:Python 215
OpenBMB / InfiniteBench
Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718
benchmark large-language-models long-context
Language:Python 200
yangjianxin1 / LongQLoRA
LongQLoRA: Extent Context Length of LLMs Efficiently
llm long-context longlora lora qlora
Language:Python 145
Infini-AI-Lab / TriForce
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
acceleration llm long-context speculative-decoding llm-inference efficiency inference
Language:Python 129
THUDM / LongAlign
LongAlign: A Recipe for Long Context Alignment Encompassing Data, Training, and Evaluation
alignment llm long-context longtext
Language:Python 121
bigai-nlco / LooGLE
LooGLE: Long Context Evaluation for Long-Context Language Models
large-language-models long-context
Language:Python 117
nightdessert / Retrieval_Head
open-source code for paper: Retrieval Head Mechanistically Explains Long-Context Factuality
large-language-models long-context
Language:Python 111
jeffreysijuntan / lloco
The official repo for "LLoCo: Learning Long Contexts Offline"
context-compression finetune llm long-context pytorch
Language:Python 93
lucidrains / infini-transformer-pytorch
Implementation of Infini-Transformer in Pytorch
artificial-intelligence attention-mechanism deep-learning long-context memory transformers
Language:Python 91
lucidrains / perceiver-ar-pytorch
Implementation of Perceiver AR, Deepmind's new long-context attention network based on Perceiver architecture, in Pytorch
artficial-intelligence attention-mechanism deep-learning transformer long-context
Language:Python 85
Glaciohound / LM-Infinite
Implementation of paper "LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models"
language-model long-context model-diagnostics
Language:Python 74
metame-ai / awesome-llm-plaza
awesome llm plaza: daily tracking all sorts of awesome topics of llm, e.g. llm for coding, robotics, reasoning, multimod etc.
awesome llm awesome-code-llm awesome-llm-plaza awesome-robotics-llm awesome-rlhf long-context awesome-llm-agents awesome-llm-prompt awesome-llm-reasoning awesome-llm-security awesome-multimod-llm llm-application
68
nick7nlp / Counting-Stars
Counting-Stars (★)
evaluation-metrics large-language-model long-context
Language:Jupyter Notebook 60
lucidrains / flash-genomics-model
My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other hierarchical methods)
artificial-intelligence attention-mechanisms deep-learning genomics long-context transformers
Language:Python 52
open-compass / Ada-LEval
The official implementation of "Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks"
gpt4 llm long-context
Language:Python 43
4AI / RAN
RAN: Recurrent Attention Networks for Long-text Modeling | Findings of ACL23
acl2023 recurrent-networks long-document-modeling long-context long-context-attention long-context-transformers acl recurrent-attention-networks
Language:Python 20
Heptabit-Music-Transformer
asigalov61 / Heptabit-Music-Transformer
[DEPRECIATED] Very fast, large music transformer with 8k sequence length, efficient heptabit MIDI notes encoding, true full MIDI instruments range, chords counters and outro tokens
artificial-intelligence heptagon heptagram midi music-ai music-transformer heptabit sota-model long-context
Language:Python 15
VITA-Group / Ms-PoE
"Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding" Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang.
large-language-models long-context lost-in-the-middle positional-encoding
Language:Python 15
davendw49 / Awesome-Long-Context-Language-Modeling
Papers of Long Context Language Model
awesome-list llm long-context nlp
8
lucaslingle / e-lra
Streamlined variant of Long-Range Arena with pinned dependencies, automated data downloads, and deterministic shuffling.
long-context long-range-arena transformers
Language:Python 3
melvinebenezer / Liah-Lie_in_a_haystack
needle in a haystack for LLMs
llm llm-inference llms-benchmarking long-context needle-in-haystack
Language:Python 2
sayhitosandy / Mamba_SSM
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
generative-ai llm long-context mamba ssm
1
nopperl / corporate_emission_reports
Finetuning and evaluating LLMs to extract GHG emissions from PDF reports using RAG and grammar-based decoding.
data-extraction evaluation information-extraction llm long-context lora
Language:TeX 0

long-context

InternLM / InternLM

dvlab-research / LongLoRA

lucidrains / MEGABYTE-pytorch

lhao499 / RingAttention

THUDM / LongBench

lucidrains / ring-attention-pytorch

lucidrains / recurrent-memory-transformer-pytorch

dingo-actual / infini-transformer

thunlp / InfLLM

OpenBMB / InfiniteBench

yangjianxin1 / LongQLoRA

Infini-AI-Lab / TriForce

THUDM / LongAlign

bigai-nlco / LooGLE

nightdessert / Retrieval_Head

jeffreysijuntan / lloco

lucidrains / infini-transformer-pytorch

lucidrains / perceiver-ar-pytorch

Glaciohound / LM-Infinite

metame-ai / awesome-llm-plaza

nick7nlp / Counting-Stars

lucidrains / flash-genomics-model

open-compass / Ada-LEval

4AI / RAN

asigalov61 / Heptabit-Music-Transformer

VITA-Group / Ms-PoE

davendw49 / Awesome-Long-Context-Language-Modeling

lucaslingle / e-lra

melvinebenezer / Liah-Lie_in_a_haystack

sayhitosandy / Mamba_SSM

nopperl / corporate_emission_reports