couragelfyang / Awesome-LLM-Long-Context-Modeling

📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥

Large Language Model Based Long Context Modeling Papers and Blogs

📝 Papers | 📚 Notions

This repo includes papers and blogs about Efficient Transformers, Length Extrapolation, Long Term Memory, Retrieval Augmented Generation(RAG), and Evaluation for Long Context Modeling.

🔥 Must-read papers for LLM-based Long Context Modeling.

Thanks for all the great contributors on GitHub!🔥⚡🔥

Contents

• 1. Survey Papers
• 2. Efficient Attention
  • 2.1 Sparse Attention
  • 2.2 Linear Attention
  • 2.3 Hierarchical Attention
  • 2.4 IO-Aware Attention
• 3. Recurrent Transformers
• 4. State Space Models
• 5. Length Extrapolation 🔥RoPE🔥
• 6. Long Term Memory
• 7. RAG and ICL
• 8. Agent
• 9. Compress
• 10. Long Video and Image
• 11. Benchmark and Evaluation
  • 11.1 LLM
  • 11.2 MLLM
• 12. Blogs
• Acknowledgements

📢 News

Week Papers

• [2024.06.26]

  • Paper: LongIns: A Challenging Long-context Instruction-based Exam for LLMs
  • Paper: Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

• [2024.06.25]

  • Paper: Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts
  • Paper: Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
  • Paper: USDC: A Dataset of $\underline{U}$ser $\underline{S}$tance and $\underline{D}$ogmatism in Long $\underline{C}$onversations
  • Paper: Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers
  • Paper: One Thousand and One Pairs: A "novel" challenge for long-context language models
  • Paper: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention

• [2024.06.24]

  • Paper: LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs
  • Paper: Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell
  • Paper: MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokens
  • Paper: MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression
  • Paper: Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning

• [2024.06.21]

  • Paper: In-Context Former: Lightning-fast Compressing Context for Large Language Model
  • Paper: Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
  • Paper: GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models
  • Paper: Can Few-shot Work in Long-Context? Recycling the Context to Generate Demonstrations
  • Paper: Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective
  • Paper: D2O: Dynamic Discriminative Operations for Efficient Generative Inference of Large Language Models
  • Paper: Towards Event-oriented Long Video Understanding
  • Paper: VisualRWKV: Exploring Recurrent Neural Networks for Visual Language Models

Month Papers

• [2024.06.19]

  • Paper: Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters
  • Paper: CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling
  • Paper: Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understanding

• [2024.06.18]

  • Paper: A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression
  • Paper: InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models
  • Paper: Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactions
  • Paper: Can Many-Shot In-Context Learning Help Long-Context LLM Judges? See More, Judge Better!
  • Paper: What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modeling
  • Paper: Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens
  • Paper: Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference ICML 2024
  • Paper: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration MLSys 2024 Best Paper Award
  • Paper: Investigating Video Reasoning Capability of Large Language Models with Tropes in Movies
  • Paper: Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models

• [2024.06.17]

  • Paper: BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
  • Paper: 3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding
  • Paper: HiP Attention: Sparse Sub-Quadratic Attention with Hierarchical Attention Pruning
  • Paper: Short Film Dataset (SFD): A Benchmark for Story-Level Video Understanding

• [2024.06.13]

  • Paper: An Empirical Study of Mamba-based Language Models

• [2024.06.12]

  • Paper: Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
  • Paper: When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
  • Paper: Effectively Compress KV Heads for LLM
  • Paper: Evaluating Zero-Shot Long-Context LLM Compression
  • Paper: Never Miss A Beat: An Efficient Recipe for Context Window Extension of Large Language Models with Consistent "Middle" Enhancement

• [2024.06.11]

  • Paper: Enhancing Long-Term Memory using Hierarchical Aggregate Tree for Retrieval Augmented Generation
  • Paper: SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Models
  • Paper: Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  • Paper: RepoQA: Evaluating Long Context Code Understanding
  • Paper: LoCoCo: Dropping In Convolutions for Long Context Compression
  • Blog: Generalizing an LLM from 8k to 1M Context using Qwen-Agent

• [2024.06.10]

  • Paper: Multi-Head RAG: Solving Multi-Aspect Problems with LLMs
  • Paper: CRAG -- Comprehensive RAG Benchmark

• [2024.06.07]

  • Paper: MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding

• [2024.06.06]

  • Paper: Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
  • Paper: FragRel: Exploiting Fragment-level Relations in the External Memory of Large Language Models
  • Paper: PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

• [2024.06.05]

  • Paper: Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding
  • Paper: Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs
  • Paper: PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
  • Paper: Position Debiasing Fine-Tuning for Causal Perception in Long-Term Dialogue
  • Paper: Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Tasks

• [2024.06.04]

  • Paper: LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models
  • Paper: Toward Conversational Agents with Context and Time Sensitive Long-term Memory

• [2024.06.03]

  • Paper: In-context Autoencoder for Context Compression in a Large Language Model ICLR 2024
  • Paper: You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet
  • Paper: Position Coupling: Leveraging Task Structure for Improved Length Generalization of Transformers
  • Paper: Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

• [2024.05.31]

  • Paper: Language Models Need Inductive Biases to Count Inductively
  • Paper: The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving
  • Paper: Is In-Context Learning Sufficient for Instruction Following in LLMs?
  • Paper: Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model
  • Paper: PostDoc: Generating Poster from a Long Multimodal Document Using Deep Submodular Optimization

• [2024.05.30]

  • Paper: Contextual Position Encoding: Learning to Count What's Important
  • Paper: EasyAnimate: A High-Performance Long Video Generation Method based on Transformer Architecture
  • Paper: VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos
  • Blog: Transformer升级之路：18、RoPE的底数设计原则

• [2024.05.29]

  • Paper: Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models
  • Paper: XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inference
  • Blog: 2024.5 A Side-by-Side Comparison of the Long Context of Various LLMs (32k articles)

• [2024.05.28]

  • Paper: Zamba: A Compact 7B SSM Hybrid Model
  • Paper: SelfCP: Compressing Long Prompt to 1/12 Using the Frozen Large Language Model Itself
  • Paper: Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention
  • Paper: Compressing Lengthy Context With UltraGist
  • Paper: Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective
  • Paper: Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection
  • Paper: Attention as an RNN

• [2024.05.27]

  • Paper: Are Long-LLMs A Necessity For Long-Context Tasks?
  • Blog: 2024.5 A Side-by-Side Comparison of the Long Context of Various LLMs (128k articles)

• [2024.05.24]

  • Paper: HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
  • Paper: Can LLMs Solve longer Math Word Problems Better?
  • Paper: CAPE: Context-Adaptive Positional Encoding for Length Extrapolation
  • Paper: Base of RoPE Bounds Context Length
  • Paper: MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
  • Paper: xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
  • Paper: Implicit In-context Learning

• [2024.05.22]

  • Paper: Your Transformer is Secretly Linear
  • Paper: Reducing Transformer Key-Value Cache Size with Cross-Layer Attentionr
  • Paper: PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference
  • Paper: SirLLM: Streaming Infinite Retentive LLM
  • Paper: Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
  • Paper: OLAPH: Improving Factuality in Biomedical Long-form Question Answering

• [2024.05.17]

  • Paper: KG-RAG: Bridging the Gap Between Knowledge and Creativity
  • Paper: A Multi-Perspective Analysis of Memorization in Large Language Models
  • Paper: Imagination Augmented Generation: Learning to Imagine Richer Context for Question Answering over Large Language Models

• [2024.05.17]

  • Paper: DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
  • Paper: Layer-Condensed KV Cache for Efficient Inference of Large Language Models
  • Paper: Feature-Adaptive and Data-Scalable In-Context Learning
  • Blog: 缓存与效果的极限拉扯：从MHA、MQA、GQA到MLA
  • Blog: Towards 100x Speedup: Full Stack Transformer Inference Optimization

• [2024.05.19]

  • Blog: Full Stack Transformer Inference Optimization Season 2: Deploying Long-Context Models

• [2024.05.17]

  • Paper: Many-Shot In-Context Learning in Multimodal Foundation Models
  • Paper: FinTextQA: A Dataset for Long-form Financial Question Answering

• [2024.05.16]

  • Paper: Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis

• [2024.05.14]

  • Paper: Evaluation of Retrieval-Augmented Generation: A Survey

• [2024.05.13]

  • Paper: Linearizing Large Language Models
  • Paper: A Survey on RAG Meets LLMs: Towards Retrieval-Augmented Large Language Models
  • Paper: HMT: Hierarchical Memory Transformer for Long Context Language Processing

📜 Papers

You can directly click on the title to jump to the corresponding PDF link location

1. Survey Papers

1. Efficient Transformers: A Survey. Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler. Arxiv 2022.

2. A Survey on Long Text Modeling with Transformers. Zican Dong, Tianyi Tang, Lunyi Li, Wayne Xin Zhao. Arxiv 2023.

3. Neural Natural Language Processing for Long Texts: A Survey of the State-of-the-Art. Dimitrios Tsirmpas, Ioannis Gkionis, Ioannis Mademlis, Georgios Papadopoulos. Arxiv 2023.

4. Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey. Yunpeng Huang, Jingwei Xu, Zixu Jiang, Junyu Lai, Zenan Li, Yuan Yao, Taolue Chen, Lijuan Yang, Zhou Xin, Xiaoxing Ma. Arxiv 2023.

       

5. Length Extrapolation of Transformers: A Survey from the Perspective of Position Encoding. Liang Zhao, Xiaocheng Feng, Xiachong Feng, Bing Qin, Ting Liu. Arxiv 2024.

6. The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey. Saurav Pawar, S.M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Aman Chadha, Amitava Das. Arxiv 2024.

7. State Space Model for New-Generation Network Alternative to Transformers: A Survey. Xiao Wang, Shiao Wang, Yuhe Ding, Yuehang Li, Wentao Wu, Yao Rong, Weizhe Kong, Ju Huang, Shihao Li, Haoxiang Yang, Ziwen Wang, Bo Jiang, Chenglong Li, Yaowei Wang, Yonghong Tian, Jin Tang. Arxiv 2024.

       

8. A Survey on Efficient Inference for Large Language Models. Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, Yu Wang. Arxiv 2024.

9. A Survey on RAG Meets LLMs: Towards Retrieval-Augmented Large Language Models. Yujuan Ding, Wenqi Fan, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, Qing Li. Arxiv 2024.

10. Evaluation of Retrieval-Augmented Generation: A Survey. Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, Zhaofeng Liu. Arxiv 2024.

       

11. The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving. Pai Zeng, Zhenyu Ning, Jieru Zhao, Weihao Cui, Mengwei Xu, Liwei Guo, Xusheng Chen, Yizhou Shan. Arxiv 2024.

2. Efficient Attention

2.1 Sparse Attention

1. Generating Long Sequences with Sparse Transformers. Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever. Arxiv 2019.

2. Blockwise selfattention for long document understanding. Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, Jie Tang. EMNLP 2020.

       

3. Longformer: The Long-Document Transformer. Iz Beltagy, Matthew E. Peters, Arman Cohan. Arxiv 2020.

       

4. ETC: Encoding Long and Structured Inputs in Transformers. Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, Li Yang. EMNLP 2020.

5. Big Bird: Transformers for Longer Sequences. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed. NeurIPS 2020.

       

6. Reformer: The efficient transformer. Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya. ICLR 2020.

       

7. Sparse Sinkhorn Attention. Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, Da-Cheng Juan. ICML 2020.

       

8. Sparse and continuous attention mechanisms. André F. T. Martins, António Farinhas, Marcos Treviso, Vlad Niculae, Pedro M. Q. Aguiar, Mário A. T. Figueiredo. NIPS 2020.

9. Efficient Content-Based Sparse Attention with Routing Transformers. Aurko Roy, Mohammad Saffar, Ashish Vaswani, David Grangier. TACL 2021.

       

10. LongT5: Efficient text-to-text transformer for long sequences. Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang. NAACL 2022.

       

11. Efficient Long-Text Understanding with Short-Text Models. Maor Ivgi, Uri Shaham, Jonathan Berant. TACL 2023.

       

12. Parallel Context Windows for Large Language Models. Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham. ACL 2023.

       

13. Unlimiformer: Long-Range Transformers with Unlimited Length Input. Amanda Bertsch, Uri Alon, Graham Neubig, Matthew R. Gormley. Arxiv 2023.

       

14. Landmark Attention: Random-Access Infinite Context Length for Transformers. Amirkeivan Mohtashami, Martin Jaggi Arxiv 2023.

       

15. LONGNET: Scaling Transformers to 1,000,000,000 Tokens. Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, Furu Wei. Arxiv 2023.

       

16. Adapting Language Models to Compress Contexts. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen. Arxiv 2023.

       

17. Blockwise Parallel Transformer for Long Context Large Models. Hao Liu, Pieter Abbeel. Arxiv 2023.

       

18. MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers. Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, Mike Lewis. Arxiv 2023.

       

19. Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers. Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, Thomas Hofmann. Arxiv 2023.

20. Long-range Language Modeling with Self-retrieval. Ohad Rubin, Jonathan Berant. Arxiv 2023.

21. Max-Margin Token Selection in Attention Mechanism. Davoud Ataee Tarzanagh, Yingcong Li, Xuechen Zhang, Samet Oymak. Arxiv 2023.

22. Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers. Jiawen Xie, Pengyu Cheng, Xiao Liang, Yong Dai, Nan Du. Arxiv 2023.

23. Sparse Token Transformer with Attention Back Tracking. Heejun Lee, Minki Kang, Youngwan Lee, Sung Ju Hwang. ICLR 2023.

24. Empower Your Model with Longer and Better Context Comprehension. YiFei Gao, Lei Wang, Jun Fang, Longhua Hu, Jun Cheng. Arxiv 2023.

       

25. Ring Attention with Blockwise Transformers for Near-Infinite Context. Hao Liu, Matei Zaharia, Pieter Abbeel. Arxiv 2023.

26. Efficient Streaming Language Models with Attention Sinks. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis. Arxiv 2023.

       

27. HyperAttention: Long-context Attention in Near-Linear Time. Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David P. Woodruff, Amir Zandieh. Arxiv 2023.

28. Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention. Ziwei He,Jian Yuan,Le Zhou,Jingwen Leng,Bo Jiang. Arxiv 2023.

       

29. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition. Lu Ye, Ze Tao, Yong Huang, Yang Li. Arxiv 2024.

30. Training-Free Long-Context Scaling of Large Language Models. Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, Lingpeng Kong. Arxiv 2024.

       

31. LongHeads: Multi-Head Attention is Secretly a Long Context Processor. Yi Lu, Xin Zhou, Wei He, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang. Arxiv 2024.

32. Zebra: Extending Context Window with Layerwise Grouped Local-Global Attention. Kaiqiang Song, Xiaoyang Wang, Sangwoo Cho, Xiaoman Pan, Dong Yu. Arxiv 2023.

33. SnapKV: LLM Knows What You are Looking for Before Generation. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen. Arxiv 2024.

       

34. Sequence can Secretly Tell You What to Discard. Jincheng Dai, Zhuowei Huang, Haiyun Jiang, Chen Chen, Deng Cai, Wei Bi, Shuming Shi. Arxiv 2024.

35. SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Models. Hengyu Zhang. Arxiv 2024.

       

36. HiP Attention: Sparse Sub-Quadratic Attention with Hierarchical Attention Pruning. Heejun Lee, Geon Park, Youngwan Lee, Jina Kim, Wonyoung Jeong, Myeongjae Jeon, Sung Ju Hwang. Arxiv 2024.

37. Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens. Weiyao Luo, Suncong Zheng, Heming Xia, Weikang Wang, Yan Lei, Tianyu Liu, Shuang Chen, Zhifang Sui. Arxiv 2024.

38. MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression. Weiyao Luo, Suncong Zheng, Heming Xia, Weikang Wang, Yan Lei, Tianyu Liu, Shuang Chen, Zhifang Sui. Arxiv 2024.

39. Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers. Chao Lou, Zixia Jia, Zilong Zheng, Kewei Tu. Arxiv 2024.

40. Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention. Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Huanqi Cao, Xiao Chuanfu, Xingcheng Zhang, Dahua Lin, Chao Yang. Arxiv 2024.

2.2 Linear Attention

1. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret. ICML 2020.

       

2. Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations. Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, Christopher Ré. Arxiv 2019.

       

3. Masked language modeling for proteins via linearly scalable long-context transformers. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, David Belanger, Lucy Colwell, Adrian Weller. Arxiv 2020.

4. Rethinking attention with performers. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller. Arxiv 2020.

       

5. Linformer: Self-attention with linear complexity. Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, Hao Ma. Arxiv 2020.

       

6. Random Feature Attention. Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, Lingpeng Kong. Arxiv 2021.

       

7. Luna: Linear unified nested attention. Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, Luke Zettlemoyer. Arxiv 2021.

       

8. Fnet: Mixing tokens with fourier transforms. James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. Arxiv 2021.

       

9. Gated Linear Attention Transformers with Hardware-Efficient Training. Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, Yoon Kim. Arxiv 2023.

       

10. Latent Attention for Linear Time Transformers. Rares Dolga, Marius Cobzarenco, David Barber. Arxiv 2024.

11. Simple linear attention language models balance the recall-throughput tradeoff. Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, Christopher Ré. Arxiv 2024.

       

12. Linear Attention Sequence Parallelism. Weigao Sun, Zhen Qin, Dong Li, Xuyang Shen, Yu Qiao, Yiran Zhong. Arxiv 2024.

       

13. Softmax Attention with Constant Cost per Token. Franz A. Heinsen. Arxiv 2024.

        ![GitHub Repo stars](https://img.shields.io/github/stars/glassroom/heinsen_attention

14. Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length. Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, Chunting Zhou. Arxiv 2024.

        ![GitHub Repo stars](https://img.shields.io/github/stars/XuezheMax/megalodon

15. Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, Yiran Zhong. Arxiv 2024.

16. Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective. Zhen Qin, Xuyang Shen, Weigao Sun, Dong Li, Stan Birchfield, Richard Hartley, Yiran Zhong. Arxiv 2024.

17. Attention as an RNN. Leo Feng, Frederick Tung, Hossein Hajimirsadeghi, Mohamed Osama Ahmed, Yoshua Bengio, Greg Mori. Arxiv 2024.

18. You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet. Zhen Qin, Yuxin Mao, Xuyang Shen, Dong Li, Jing Zhang, Yuchao Dai, Yiran Zhong. Arxiv 2024.

        ![GitHub Repo stars](https://img.shields.io/github/stars/OpenNLPLab/LightNet

19. When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models. Haoran You, Yichao Fu, Zheng Wang, Amir Yazdanbakhsh, Yingyan (Celine)Lin. Arxiv 2024.

        ![GitHub Repo stars](https://img.shields.io/github/stars/GATECH-EIC/Linearized-LLM

2.3 Hierarchical Attention

1. Neural Legal Judgment Prediction in English. Ilias Chalkidis, Ion Androutsopoulos, Nikolaos Aletras. ACL 2019.

       

2. Hierarchical Neural Network Approaches for Long Document Classification. Snehal Khandve, Vedangi Wagh, Apurva Wani, Isha Joshi, Raviraj Joshi. ICML 2022.

3. Hi-transformer: Hierarchical interactive transformer for efficient and effective long document modeling. Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang. ACL-IJCNLP 2021

4. Erniesparse: Learning hierarchical efficient transformer through regularized self-attention. Yang Liu, Jiaxiang Liu, Li Chen, Yuxiang Lu, Shikun Feng, Zhida Feng, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang. Arxiv 2022.

2.4 IO-Aware Attention

1. Self-attention Does Not Need O(n^2) Memory. Markus N. Rabe, Charles Staats. Arxiv 2021.

2. Faster Causal Attention Over Large Sequences Through Sparse Flash Attention. Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, François Fleuret. Arxiv 2023.

3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré. Arxiv 2022.

       

4. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. Tri Dao. Arxiv 2023.

       

5. Efficient Memory Management for Large Language Model Serving with PagedAttention. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica. Arxiv 2023.

       

6. TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer. Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Xiao Luo, Yu Qiao, Yiran Zhong. Arxiv 2023.

       

7. Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, Yiran Zhong. Arxiv 2024.

       

8. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition. Lu Ye, Ze Tao, Yong Huang, Yang Li. Arxiv 2024.

9. SnapKV: LLM Knows What You are Looking for Before Generation. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen. Arxiv 2024.

       

10. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, Jianfeng Gao. ICLR 2024 Oral.

11. Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference. Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, Purushotham Kamath. Arxiv 2024.

12. Efficient LLM Inference with Kcache. Qiaozhi He, Zhihua Wu. Arxiv 2024.

13. You Only Cache Once: Decoder-Decoder Architectures for Language Models. Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei. Arxiv 2024.

       

14. Fast Transformer Decoding: One Write-Head is All You Need. Noam Shazeer. Arxiv 2019.

15. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai. Arxiv 2023.

16. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. DeepSeek-AI. Arxiv 2024.

       

17. Layer-Condensed KV Cache for Efficient Inference of Large Language Models. Haoyi Wu, Kewei Tu. Arxiv 2024.

       

18. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention. William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan Kelly. Arxiv 2024.

19. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference. William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan Kelly. Arxiv 2024.

       

20. Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression. Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Yipeng Ma, Tao Wang, Ji-Rong Wen. Arxiv 2024.

21. MiniCache: KV Cache Compression in Depth Dimension for Large Language Models. Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, Bohan Zhuang. Arxiv 2024.

22. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. Zefan Cai., Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao. Arxiv 2024.

23. Effectively Compress KV Heads for LLM. Hao Yu, Zelan Yang, Shen Li, Yong Li, Jianxin Wu. Arxiv 2024.

24. A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression. Alessio Devoto, Yu Zhao, Simone Scardapane, Pasquale Minervini. Arxiv 2024.

25. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han. ICML 2024.

       

26. Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters. Zhiyu Guo, Hidetaka Kamigaito, Taro Watanabe. Arxiv 2024.

27. CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling. Yu Bai, Xiyuan Zou, Heyan Huang, Sanxing Chen, Marc-Antoine Rondeau, Yang Gao, Jackie Chi Kit Cheung. Arxiv 2024.

28. D2O: Dynamic Discriminative Operations for Efficient Generative Inference of Large Language Models. Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, Mi Zhang. Arxiv 2024.

29. MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression. Weiyao Luo, Suncong Zheng, Heming Xia, Weikang Wang, Yan Lei, Tianyu Liu, Shuang Chen, Zhifang Sui. Arxiv 2024.

3. Recurrent Transformers

1. Transformer-XL: Attentive language models beyond a fixed-length context. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov. ACL 2019.

       

2. Compressive Transformers for Long-Range Sequence Modelling. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Timothy P. Lillicrap. Arxiv 2019.

       

3. Memformer: The memory-augmented transformer. Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, Zhou Yu. Arxiv 2020.

       

4. ERNIE-Doc: A Retrospective Long-Document Modeling Transformer. SiYu Ding, Junyuan Shang, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang. ACL-IJCNLP 2021.

5. Memorizing Transformers. Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, Christian Szegedy. Arxiv 2022.

       

6. Recurrent Attention Networks for Long-text Modeling. Xianming Li, Zongxi Li, Xiaotian Luo, Haoran Xie, Xing Lee, Yingbin Zhao, Fu Lee Wang, Qing Li. ACL 2023.

       

7. RWKV: Reinventing RNNs for the Transformer Era. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Xiangru Tang, Bolun Wang, Johan S. Wind, Stansilaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Jian Zhu, Rui-Jie Zhu. Arxiv 2023.

               

8. Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model. Yinghan Long, Sayeed Shafayet Chowdhury, Kaushik Roy. Arxiv 2023.

9. Scaling Transformer to 1M tokens and beyond with RMT. Aydar Bulatov, Yuri Kuratov, Mikhail S. Burtsev. Arxiv 2023.

10. Block-Recurrent Transformers. DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, Behnam Neyshabur. Arxiv 2023.

       

11. TRAMS: Training-free Memory Selection for Long-range Language Modeling. Haofei Yu, Cunxiang Wang, Yue Zhang, Wei Bi. Arxiv 2023.

       

12. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models. Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, Caglar Gulcehre. Arxiv 2024.

13. Extensible Embedding: A Flexible Multipler For LLM's Context Length. Ninglu Shao, Shitao Xiao, Zheng Liu, Peitian Zhang. Arxiv 2024.

       

14. Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence. Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Stanisław Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, Rui-Jie Zhu. Arxiv 2024.

                       

15. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention. Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal. Arxiv 2024.

16. RecurrentGemma: Moving Past Transformers for Efficient Open Language Models. Aleksandar Botev, Soham De, Samuel L Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Sertan Girgin, Olivier Bachem, Alek Andreev, Kathleen Kenealy, Thomas Mesnard, Cassidy Hardin, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Armand Joulin, Noah Fiedel, Evan Senter, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, David Budden, Arnaud Doucet, Sharad Vikram, Adam Paszke, Trevor Gale, Sebastian Borgeaud, Charlie Chen, Andy Brock, Antonia Paterson, Jenny Brennan, Meg Risdal, Raj Gundluru, Nesh Devanathan, Paul Mooney, Nilay Chauhan, Phil Culliton, Luiz GUStavo Martins, Elisa Bandy, David Huntsperger, Glenn Cameron, Arthur Zucker, Tris Warkentin, Ludovic Peran, Minh Giang, Zoubin Ghahramani, Clément Farabet, Koray Kavukcuoglu, Demis Hassabis, Raia Hadsell, Yee Whye Teh, Nando de Frietas. Arxiv 2024.

17. Linearizing Large Language Models. Jean Mercat, Igor Vasiljevic, Sedrick Keh, Kushal Arora, Achal Dave, Adrien Gaidon, Thomas Kollar. Arxiv 2024.

       

18. VisualRWKV: Exploring Recurrent Neural Networks for Visual Language Models. Haowen Hou, Peigen Zeng, Fei Ma, Fei Richard Yu. Arxiv 2024.

       

4. State Space Models

1. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Albert Gu, Tri Dao. Arxiv 2023.

       

2. MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts. Maciej Pióro, Kamil Ciebiera, Krystian Król, Jan Ludziejewski, Sebastian Jaszczur. Arxiv 2024.

3. MambaByte: Token-free Selective State Space Model. Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, Alexander M Rush. Arxiv 2024.

4. LOCOST: State-Space Models for Long Document Abstractive Summarization. Florian Le Bronnec, Song Duong, Mathieu Ravaut, Alexandre Allauzen, Nancy F. Chen, Vincent Guigue, Alberto Lumbreras, Laure Soulier, Patrick Gallinari. Arxiv 2024.

5. State Space Models as Foundation Models: A Control Theoretic Overview. Carmen Amo Alonso, Jerome Sieber, Melanie N. Zeilinger. Arxiv 2024.

6. Jamba: A Hybrid Transformer-Mamba Language Model. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, Yoav Shoham. Arxiv 2024.

7. Robustifying State-space Models for Long Sequences via Approximate Diagonalization. Annan Yu, Arnur Nigmetov, Dmitriy Morozov, Michael W. Mahoney, N. Benjamin Erichson. ICLR 2024 Spotlight.

8. Zamba: A Compact 7B SSM Hybrid Model. Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, Beren Millidge. Arxiv 2024.

9. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. Tri Dao, Albert Gu. Arxiv 2024.

       

10. Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling. Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, Weizhu Chen. Arxiv 2024.

       

11. An Empirical Study of Mamba-based Language Models. Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, Bryan Catanzaro. Arxiv 2024.

       

5. Length Extrapolation

1. RoFormer: Enhanced Transformer with Rotary Position Embedding. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu. Arxiv 2021.

       

2. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. Ofir Press, Noah A. Smith, Mike Lewis. ICLR 2022.

       

3. KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation. Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge, Alexander I. Rudnicky. Arxiv 2022.

4. Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis. Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky, Peter J. Ramadge. ACL 2023.

5. A Length-Extrapolatable Transformer. Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, Furu Wei. ACL 2023.

       

6. Randomized Positional Encodings Boost Length Generalization of Transformers. Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, Joel Veness. ACL 2023.

       

7. The Impact of Positional Encoding on Length Generalization in Transformers. Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, Siva Reddy. Arxiv 2023.

       

8. Focused Transformer: Contrastive Training for Context Scaling. Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, Piotr Miłoś. Arxiv 2023.

       

9. Extending Context Window of Large Language Models via Positional Interpolation. Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian. Arxiv 2023.

10. Exploring Transformer Extrapolation. Zhen Qin, Yiran Zhong, Hui Deng. Arxiv 2023.

       

11. LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models. Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang. Arxiv 2023.

       

12. YaRN: Efficient Context Window Extension of Large Language Models. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole. Arxiv 2023.

       

13. PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training. Dawei Zhu,Nan Yang,Liang Wang,Yifan Song,Wenhao Wu,Furu Wei,Sujian Li. Arxiv 2023.

       

14. LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, Jiaya Jia. ICLR 2024 Oral.

       

15. Scaling Laws of RoPE-based Extrapolation. Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, Dahua Lin. Arxiv 2023.

16. Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation. Ta-Chung Chi,Ting-Han Fan,Alexander I. Rudnicky. Arxiv 2023.

       

17. CoCA: Fusing position embedding with Collinear Constrained Attention for fine-tuning free context window extending. Shiyi Zhu, Jing Ye, Wei Jiang, Qi Zhang, Yifan Wu, Jianguo Li. Arxiv 2023.

       

18. Structured Packing in LLM Training Improves Long Context Utilization. Konrad Staniszewski, Szymon Tworkowski, Sebastian Jaszczur, Henryk Michalewski, Łukasz Kuciński, Piotr Miłoś. Arxiv 2024.

19. LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, Xia Hu. Arxiv 2024.

20. Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache. Bin Lin, Tao Peng, Chen Zhang, Minmin Sun, Lanbo Li, Hanyu Zhao, Wencong Xiao, Qi Xu, Xiafei Qiu, Shen Li, Zhigang Ji, Yong Li, Wei Lin. Arxiv 2024.

21. Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, Yiran Zhong. Arxiv 2024.

       

22. Extending LLMs' Context Window with 100 Samples. Yikai Zhang, Junlong Li, Pengfei Liu. Arxiv 2024.

       

23. E^2-LLM: Efficient and Extreme Length Extension of Large Language Models. Jiaheng Liu, Zhiqi Bai, Yuanxing Zhang, Chenchen Zhang, Yu Zhang, Ge Zhang, Jiakai Wang, Haoran Que, Yukang Chen, Wenbo Su, Tiezheng Ge, Jie Fu, Wenhu Chen, Bo Zheng. Arxiv 2024.

24. With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation. Y. Wang, D. Ma, D. Cai. Arxiv 2024.

       

25. Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation. Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Di He, Jingjing Xu, Zhi Zhang, Hongxia Yang, Liwei Wang. ICML 2024.

       

26. Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens. Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, Hannaneh Hajishirzi. Arxiv 2024.

       

27. LongRoPE: Extending LLM ContextWindow Beyond 2 Million Tokens. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, Mao Yang. Arxiv 2024.

28. Data Engineering for Scaling Language Models to 128K Context. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, Hao Peng. Arxiv 2024.

       

29. Transformers Can Achieve Length Generalization But Not Robustly. Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, Denny Zhou. Arxiv 2024.

30. Long-Context Language Modeling with Parallel Context Encoding. Howard Yen, Tianyu Gao, Danqi Chen. Arxiv 2024.

       

31. CLEX: Continuous Length Extrapolation for Large Language Models. Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, Lidong Bing. Arxiv 2023.

       

32. Resonance RoPE: Improving Context Length Generalization of Large Language Models. Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, Bang Liu. Arxiv 2024.

       

33. Can't Remember Details in Long Documents? You Need Some R&R. Devanshu Agrawal, Shang Gao, Martin Gajek. Arxiv 2024.

       

34. Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding. Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang. Arxiv 2024.

       

35. InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory. Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, Maosong Sun. Arxiv 2024.

36. Naive Bayes-based Context Extension for Large Language Models. Jianlin Su, Murtadha Ahmed, Wenbo, Luo Ao, Mingren Zhu, Yunfeng Liu. Arxiv 2024.

       

37. Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference. Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, Purushotham Kamath. Arxiv 2024.

38. In-Context Pretraining: Language Modeling Beyond Document Boundaries. Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen-tau Yih, Mike Lewis. ICLR 2024 Spotlight.

       

39. Effective Long-Context Scaling of Foundation Models. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, Hao Ma. Arxiv 2023.

40. Fewer Truncations Improve Language Modeling. Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, Stefano Soatto. Arxiv 2024.

41. Length Generalization of Causal Transformers without Position Encoding. Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, Xiaoling Wang. Arxiv 2024.

       

42. Extending Llama-3's Context Ten-Fold Overnight. Peitian Zhang, Ninglu Shao, Zheng Liu, Shitao Xiao, Hongjin Qian, Qiwei Ye, Zhicheng Dou. Arxiv 2024.

       

43. Long Context Alignment with Short Instructions and Synthesized Positions. Wenhao Wu, Yizhong Wang, Yao Fu, Xiang Yue, Dawei Zhu, Sujian Li. Arxiv 2024.

       

44. xLSTM: Extended Long Short-Term Memory. Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter. Arxiv 2024.

45. CAPE: Context-Adaptive Positional Encoding for Length Extrapolation. Chuanyang Zheng, Yihang Gao, Han Shi, Minbin Huang, Jingyao Li, Jing Xiong, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, Yu Li. Arxiv 2024.

       

46. Contextual Position Encoding: Learning to Count What's Important. Olga Golovneva, Tianlu Wang, Jason Weston, Sainbayar Sukhbaatar. Arxiv 2024.

47. Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model. Chaochen Gao, Xing Wu, Qi Fu, Songlin Hu. Arxiv 2024.

48. Position Coupling: Leveraging Task Structure for Improved Length Generalization of Transformers. Hanseul Cho, Jaeyoung Cha, Pranjal Awasthi, Srinadh Bhojanapalli, Anupam Gupta, Chulhee Yun. Arxiv 2024.

       

49. LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models. Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang, Peng Cheng, Lijie Wang, Chenxia Li, Xuejie Wu, Bo Zhu, Yimeng Gan, Rui Hu, Shuicheng Yan, Han Fang, Yahui Zhou. Arxiv 2024.

50. Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Tasks. Mahdi Sabbaghi, George Pappas, Hamed Hassani, Surbhi Goel. Arxiv 2024.

51. Never Miss A Beat: An Efficient Recipe for Context Window Extension of Large Language Models with Consistent "Middle" Enhancement. Tong Wu, Yanpeng Zhao, Zilong Zheng. Arxiv 2024.

52. 3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding. Xindian Ma, Wenyuan Liu, Peng Zhang, Nan Xu. Arxiv 2024.

6. Long Term Memory

1. Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System. Xinnian Liang, Bing Wang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, Zhoujun Li. Arxiv 2023.

       

2. MemoryBank: Enhancing Large Language Models with Long-Term Memory. Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, Yanlin Wang. Arxiv 2023.

       

3. Improve Long-term Memory Learning Through Rescaling the Error Temporally. Shida Wang, Zhanglu Yan. Arxiv 2023.

4. Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models. Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, Li Guo. Arxiv 2023.

5. Empowering Working Memory for Large Language Model Agents. Jing Guo, Nan Li, Jianchuan Qi, Hang Yang, Ruiqiao Li, Yuzhen Feng, Si Zhang, Ming Xu. Arxiv 2024.

6. Evolving Large Language Model Assistant with Long-Term Conditional Memory. Ruifeng Yuan, Shichao Sun, Zili Wang, Ziqiang Cao, Wenjie Li. Arxiv 2024.

7. Commonsense-augmented Memory Construction and Management in Long-term Conversations via Context-aware Persona Refinement. Hana Kim, Kai Tzu-iunn Ong, Seoyeon Kim, Dongha Lee, Jinyoung Yeo. Arxiv 2024.

8. A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts. Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John Canny, Ian Fischer. Arxiv 2024.

9. Steering Conversational Large Language Models for Long Emotional Support Conversations. Navid Madani, Sougata Saha, Rohini Srihari. Arxiv 2024.

10. SPAR: Personalized Content-Based Recommendation via Long Engagement Attention. Chiyu Zhang, Yifei Sun, Jun Chen, Jie Lei, Muhammad Abdul-Mageed, Sinong Wang, Rong Jin, Sem Park, Ning Yao, Bo Long. Arxiv 2024.

11. Compress to Impress: Unleashing the Potential of Compressive Memory in Real-World Long-Term Conversations. Nuo Chen, Hongguang Li, Juhua Huang, Baoyuan Wang, Jia Li. Arxiv 2024.

       

12. StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses. Jia-Nan Li, Quan Tu, Cunli Mao, Zhengtao Yu, Ji-Rong Wen, Rui Yan. Arxiv 2024.

13. Prompts As Programs: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimization. Tobias Schnabel, Jennifer Neville. Arxiv 2024.

       

14. HMT: Hierarchical Memory Transformer for Long Context Language Processing. Tobias Schnabel, Jennifer Neville. Arxiv 2024.

       

15. SirLLM: Streaming Infinite Retentive LLM. Yao Yao, Zuchao Li, Hai Zhao. Arxiv 2024.

       

16. Toward Conversational Agents with Context and Time Sensitive Long-term Memory. Nick Alonso, Tomás Figliolia, Anthony Ndirango, Beren Millidge. Arxiv 2024.

       

16. Position Debiasing Fine-Tuning for Causal Perception in Long-Term Dialogue. Shixuan Fan, Wei Wei, Wendi Li, Xian-Ling Mao, Wenfeng Xie, Dangyang Chen. Arxiv 2024.

17. Enhancing Long-Term Memory using Hierarchical Aggregate Tree for Retrieval Augmented Generation. Aadharsh Aadhithya A, Sachin Kumar S, Soman K.P. Arxiv 2024.

7. RAG and ICL

1. Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading. Howard Chen, Ramakanth Pasunuru, Jason Weston, Asli Celikyilmaz. Arxiv 2023.

2. Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processing. Zi Yang, Nan Hua. Arxiv 2024.

3. BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models. Kun Luo, Zheng Liu, Shitao Xiao, Kang Liu. Arxiv 2024.

       

4. Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, Jong C. Park. Arxiv 2024.

       

5. RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation. Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, Jie Fu. Arxiv 2024.

       

6. Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts. Zhuo Chen, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, Kewei Tu. Arxiv 2024.

7. Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation. Thomas Merth, Qichen Fu, Mohammad Rastegari, Mahyar Najibi. Arxiv 2024.

8. Multi-view Content-aware Indexing for Long Document Retrieval. Kuicai Dong, Derrick Goh Xin Deik, Yi Quan Lee, Hao Zhang, Xiangyang Li, Cong Zhang, Yong Liu. Arxiv 2024.

9. Retrieval Head Mechanistically Explains Long-Context Factuality. Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, Yao Fu. Arxiv 2024.

       

10. FlashBack:Efficient Retrieval-Augmented Language Modeling for Long Context Inference. Runheng Liu, Xingchen Xiao, Heyan Huang, Zewen Chi, Zhijing Wu. Arxiv 2024.

11. Feature-Adaptive and Data-Scalable In-Context Learning. Jiahao Li, Quan Wang, Licheng Zhang, Guoqing Jin, Zhendong Mao. Arxiv 2024.

       

12. KG-RAG: Bridging the Gap Between Knowledge and Creativity. Diego Sanmartin. Arxiv 2024.

       

13. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, Yu Su. Arxiv 2024.

       

14. Implicit In-context Learning. Zhuowei Li, Zihao Xu, Ligong Han, Yunhe Gao, Song Wen, Di Liu, Hao Wang, Dimitris N. Metaxas. Arxiv 2024.

       

15. Are Long-LLMs A Necessity For Long-Context Tasks?. Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Yujia Zhou, Xu Chen, Zhicheng Dou. Arxiv 2024.

16. Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection. Yun Zhu, Jia-Chen Gu, Caitlin Sikora, Ho Ko, Yinxiao Liu, Chu-Cheng Lin, Lei Shu, Liangchen Luo, Lei Meng, Bang Liu, Jindong Chen. Arxiv 2024.

17. Is In-Context Learning Sufficient for Instruction Following in LLMs?. Hao Zhao, Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion. Arxiv 2024.

       

18. FragRel: Exploiting Fragment-level Relations in the External Memory of Large Language Models. Xihang Yue, Linchao Zhu, Yi Yang. Arxiv 2024.

19. Multi-Head RAG: Solving Multi-Aspect Problems with LLMs. Maciej Besta, Ales Kubicek, Roman Niggli, Robert Gerstenberger, Lucas Weitzendorf, Mingyuan Chi, Patrick Iff, Joanna Gajda, Piotr Nyczyk, Jürgen Müller, Hubert Niewiadomski, Marcin Chrapek, Michał Podstawski, Torsten Hoefler. Arxiv 2024.

       

20. Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactions. Yiming Tang, Bin Dong. Arxiv 2024.

21. Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understanding. Weizhi Fei, Xueyan Niu, Guoqing Xie, Yanhua Zhang, Bo Bai, Lei Deng, Wei Han. Arxiv 2024.

22. FoRAG: Factuality-optimized Retrieval Augmented Generation for Web-enhanced Long-form Question Answering. Tianchi Cai, Zhiwen Tan, Xierui Song, Tao Sun, Jiyan Jiang, Yunqi Xu, Yinger Zhang, Jinjie Gu. Arxiv 2024.

23. Can Few-shot Work in Long-Context? Recycling the Context to Generate Demonstrations. Arie Cattan, Alon Jacovi, Alex Fabrikant, Jonathan Herzig, Roee Aharoni, Hannah Rashkin, Dror Marcus, Avinatan Hassidim, Yossi Matias, Idan Szpektor, Avi Caciularu. Arxiv 2024.

24. LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs. Ziyan Jiang, Xueguang Ma, Wenhu Chen. Arxiv 2024.

       

25. Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning. Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, Roei Herzig. Arxiv 2024.

8. Agent

1. LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration. Jun Zhao, Can Zu, Hao Xu, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, Xuanjing Huang. Arxiv 2024.

2. A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis. Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, Aleksandra Faust. ICLR 2024 Oral.

3. PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents. Simeng Sun, Yang Liu, Shuohang Wang, Dan Iter, Chenguang Zhu, Mohit Iyyer. EACL 2024.

       

4. AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents. Jake Grigsby, Linxi Fan, Yuke Zhu. ICLR 2024 Spotlight.

       

5. Chain of Agents: Large Language Models Collaborating on Long-Context Tasks. Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, Sercan Ö. Arik. Arxiv 2024.

6. GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models. Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jiaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, Wenbo Su, Bo Zheng. Arxiv 2024.

9. Compress

1. Adapting Language Models to Compress Contexts. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen. Arxiv 2023.

       

2. Compressing Context to Enhance Inference Efficiency of Large Language Models. Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin. Arxiv 2023.

       

3. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu. Arxiv 2023.

       

4. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu. Arxiv 2023.

       

5. System 2 Attention (is something you might need too). Jason Weston, Sainbayar Sukhbaatar. Arxiv 2023.

6. DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization. Rahul Chand, Yashoteja Prabhu, Pratyush Kumar. Arxiv 2023.

7. Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, Zhicheng Dou. Arxiv 2024.

       

8. Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization. Ninglu Shao, Shitao Xiao, Zheng Liu, Peitian Zhang. Arxiv 2024.

       

9. Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression. Xinze Li, Zhenghao Liu, Chenyan Xiong, Shi Yu, Yukun Yan, Shuo Wang, Ge Yu. Arxiv 2024.

       

10. Learning to Compress Prompt in Natural Language Formats. Yu-Neng Chuang, Tianwei Xing, Chia-Yuan Chang, Zirui Liu, Xun Chen, Xia Hu. Arxiv 2024.

11. Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference. Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan, Edoardo M. Ponti. Arxiv 2024.

12. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Dongmei Zhang. Arxiv 2024.

       

13. PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models. Jinyi Li, Yihuai Lan, Lei Wang, Hao Wang. Arxiv 2024.

       

14. Compressed Context Memory for Online Language Model Interaction. Jang-Hyun Kim, Junyoung Yeom, Sangdoo Yun, Hyun Oh Song. ICLR 2024.

       

15. Compressing Large Language Models by Streamlining the Unimportant Layer. Xiaodong Chen, Yuxuan Hu, Jing Zhang. Arxiv 2024.

16. PROMPT-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression. Muhammad Asif Ali, Zhengping Li, Shu Yang, Keyuan Cheng, Yang Cao, Tianhao Huang, Lijie Hu, Lu Yu, Di Wang. Arxiv 2024.

17. Training LLMs over Neurally Compressed Text. Brian Lester, Jaehoon Lee, Alex Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl-Dickstein, Noah Constant. Arxiv 2024.

18. Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models. Taiqiang Wu, Chaofan Tao, Jiahao Wang, Zhe Zhao, Ngai Wong. Arxiv 2024.

19. Adapting LLMs for Efficient Context Processing through Soft Prompt Compression. Cangqing Wang, Yutian Yang, Ruisi Li, Dan Sun, Ruicong Cai, Yuzhu Zhang, Chengqian Fu, Lillian Floyd. Arxiv 2024.

20. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, Jianfeng Gao. ICLR 2024 Oral.

21. LLoCO: Learning Long Contexts Offline. Sijun Tan, Xiuyu Li, Shishir Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E. Gonzalez, Raluca Ada Popa. Arxiv 2024.

       

22. In-Context Learning State Vector with Inner and Momentum Optimization. Dongfang Li, Zhenyu Liu, Xinshuo Hu, Zetian Sun, Baotian Hu, Min Zhang. Arxiv 2024.

       

23. Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation. Kaize Shi, Xueyao Sun, Qing Li, Guandong Xu. Arxiv 2024.

24. Improving Long Text Understanding with Knowledge Distilled from Summarization Model. Yan Liu, Yazheng Yang, Xiaokang Chen. Arxiv 2024.

25. OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruning. Dan Qiao, Yi Su, Pinzheng Wang, Jing Ye, Wenjing Xie, Yuechi Zhou, Yuyang Ding, Zecheng Tang, Jikai Wang, Yixin Ji, Yue Wang, Pei Guo, Zechen Sun, Zikang Zhang, Juntao Li, Pingfu Chao, Wenliang Chen, Guohong Fu, Guodong Zhou, Qiaoming Zhu, Min Zhang. Arxiv 2024.

       

26. Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization. Yixin Ji, Yang Xiang, Juntao Li, Wei Chen, Zhongyi Liu, Kehai Chen, Min Zhang. Arxiv 2024.

       

27. Imagination Augmented Generation: Learning to Imagine Richer Context for Question Answering over Large Language Models. Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, Shengping Liu, Jun Zhao. Arxiv 2024.

       

28. Your Transformer is Secretly Linear. Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov. Arxiv 2024.

       

29. xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token. Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, Dongyan Zhao. Arxiv 2024.

       

30. SelfCP: Compressing Long Prompt to 1/12 Using the Frozen Large Language Model Itself. Jun Gao. Arxiv 2024.

31. Compressing Lengthy Context With UltraGist. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, Zhicheng Dou. Arxiv 2024.

       

32. XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inference. Shengnan Wang, Youhui Bai, Lin Zhang, Pingyi Zhou, Shixiong Zhao, Gong Zhang, Sen Wang, Renhai Chen, Hua Xu, Hongwei Sun. Arxiv 2024.

33. In-context Autoencoder for Context Compression in a Large Language Model. Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, Furu Wei. ICLR 2024.

       

34. Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs. Zhiwei Cao, Qian Cao, Yu Lu, Ningxin Peng, Luyang Huang, Shanbo Cheng, Jinsong Su. Arxiv 2024.

       

35. Recurrent Context Compression: Efficiently Expanding the Context Window of LLM. Chensen Huang, Guibo Zhu, Xuepeng Wang, Yifei Luo, Guojing Ge, Haoran Chen, Dong Yi, Jinqiao Wang. Arxiv 2024.

       

36. LoCoCo: Dropping In Convolutions for Long Context Compression. Ruisi Cai, Yuandong Tian, Zhangyang Wang, Beidi Chen. Arxiv 2024.

       

37. Evaluating Zero-Shot Long-Context LLM Compression. Chenyu Wang, Yihan Wang. Arxiv 2024.

38. InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models. Juseon-Do, Jingun Kwon, Hidetaka Kamigaito, Manabu Okumura. Arxiv 2024.

       

39. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, Song Han. MLSys 2024 Best Paper Award.

       

39. In-Context Former: Lightning-fast Compressing Context for Large Language Model. Xiangfeng Wang, Zaiyi Chen, Zheyong Xie, Tong Xu, Yongyi He, Enhong Chen. Arxiv 2024.

10. Long Video and Image

1. EasyAnimate: A High-Performance Long Video Generation Method based on Transformer Architecture. Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, Jun Huang. Arxiv 2024.

       

2. VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos. Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, Mohit Bansal. Arxiv 2024.

3. PostDoc: Generating Poster from a Long Multimodal Document Using Deep Submodular Optimization. Vijay Jaisankar, Sambaran Bandyopadhyay, Kalp Vyas, Varre Chaitanya, Shwetha Somasundaram. Arxiv 2024.

4. Investigating Video Reasoning Capability of Large Language Models with Tropes in Movies. Hung-Ting Su, Chun-Tong Chao, Ya-Ching Hsu, Xudong Lin, Yulei Niu, Hung-Yi Lee, Winston H. Hsu. Arxiv 2024.

       

5. Towards Event-oriented Long Video Understanding. Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, Ji-Rong Wen. Arxiv 2024.

       

11. Benchmark and Evaluation

11.1 LLM

1. Long Range Arena : A Benchmark for Efficient Transformers. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, Donald Metzler. ICLR 2021.

       

2. LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation. Jian Guan, Zhuoer Feng, Yamei Chen, Ruilin He, Xiaoxi Mao, Changjie Fan, Minlie Huang. TACL 2022.

       

3. SCROLLS: Standardized CompaRison Over Long Language Sequences. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy. EMNLP 2022.

       

4. MuLD: The Multitask Long Document Benchmark. George Hudson, Noura Al Moubayed. LREC 2022.

       

5. Lost in the Middle: How Language Models Use Long Contexts. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang. Arxiv 2023.

       

6. L-Eval: Instituting Standardized Evaluation for Long Context Language Models. Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, Xipeng Qiu. Arxiv 2023.

       

7. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li. Arxiv 2023.

       

8. Content Reduction, Surprisal and Information Density Estimation for Long Documents. Shaoxiong Ji, Wei Sun, Pekka Marttinen. Arxiv 2023.

9. BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models. Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, Ji-Rong Wen. Arxiv 2023.

       

10. Retrieval meets Long Context Large Language Models. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, Bryan Catanzaro. Arxiv 2023.

11. LooGLE: Long Context Evaluation for Long-Context Language Models. Jiaqi Li, Mengmeng Wang, Zilong Zheng, Muhan Zhang. Arxiv 2023.

       

12. The Impact of Reasoning Step Length on Large Language Models. Mingyu Jin, Qinkai Yu, Dong shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, Mengnan Du. Arxiv 2024.

13. DocFinQA: A Long-Context Financial Reasoning Dataset. Varshini Reddy, Rik Koncel-Kedziorski, Viet Dac Lai, Chris Tanner. Arxiv 2024.

14. LongFin: A Multimodal Document Understanding Model for Long Financial Domain Documents. Ahmed Masry, Amir Hajian. Arxiv 2024.

15. PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models. Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, Linqi Song. Arxiv 2024.

16. LongHealth: A Question Answering Benchmark with Long Clinical Documents. Lisa Adams, Felix Busch, Tianyu Han, Jean-Baptiste Excoffier, Matthieu Ortala, Alexander Löser, Hugo JWL. Aerts, Jakob Nikolas Kather, Daniel Truhn, Keno Bressem. Arxiv 2024.

17. Long-form evaluation of model editing. Domenic Rosati, Robie Gonzales, Jinkun Chen, Xuemin Yu, Melis Erkan, Yahya Kayani, Satya Deepika Chavatapalli, Frank Rudzicz, Hassan Sajjad. Arxiv 2024.

18. In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs Miss. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, Mikhail Burtsev. Arxiv 2024.

       

19. ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, Maosong Sun. Arxiv 2024.

20. Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models. Mosh Levy, Alon Jacoby, Yoav Goldberg. Arxiv 2024.

       

21. Evaluating Very Long-Term Conversational Memory of LLM Agents. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang. Arxiv 2024.

       

22. Language Models as Science Tutors. Alexis Chevalier, Jiayi Geng, Alexander Wettig, Howard Chen, Sebastian Mizera, Toni Annala, Max Jameson Aragon, Arturo Rodríguez Fanlo, Simon Frieder, Simon Machado, Akshara Prabhakar, Ellie Thieu, Jiachen T. Wang, Zirui Wang, Xindi Wu, Mengzhou Xia, Wenhan Jia, Jiatong Yu, Jun-Jie Zhu, Zhiyong Jason Ren, Sanjeev Arora, Danqi Chen. Arxiv 2024.

       

23. Needle in a haystack - pressure testing llms. Kamradt, G. Github 2024.

       

24. In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, Mikhail Burtsev. Arxiv 2024.

       

25. LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K. Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, Yu Wang. Arxiv 2024.

       

26. Counting-Stars: A Simple, Efficient, and Reasonable Strategy for Evaluating Long-Context Large Language Models. Mingyang Song, Mao Zheng, Xuan Luo. Arxiv 2024.

       

27. NovelQA: A Benchmark for Long-Range Novel Question Answering. Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Qian Wang, Yue Zhang. Arxiv 2024.

       

28. Long-form factuality in large language models. Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, Quoc V. Le. Arxiv 2024.

       

29. LUQ: Long-text Uncertainty Quantification for LLMs. JCaiqi Zhang, Fangyu Liu, Marco Basaldella, Nigel Collier. Arxiv 2024.

30. CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models. Zexuan Qiu, Jingjing Li, Shijue Huang, Wanjun Zhong, Irwin King. Arxiv 2024.

       

31. Long-context LLMs Struggle with Long In-context Learning. Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, Wenhu Chen. Arxiv 2024.

       

32. CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems. Sara Rosenthal, Avirup Sil, Radu Florian, Salim Roukos. Arxiv 2024.

       

33. XL2Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies. Xuanfan Ni, Hengyi Cai, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Piji Li. Arxiv 2024.

       

34. Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors. Ido Amos, Jonathan Berant, Ankit Gupta. ICLR 2024 Oral.

35. Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks. Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, Kai Chen. Arxiv 2024.

       

36. RULER: What's the Real Context Size of Your Long-Context Language Models?. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Boris Ginsburg. Arxiv 2024.

       

37. LongEmbed: Extending Embedding Models for Long Context Retrieval. Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li. Arxiv 2024.

       

38. Make Your LLM Fully Utilize the Context. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou. Arxiv 2024.

       

39. S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models. Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, Kang Liu. NAACL 2024.

       

40. In-Context Learning with Long-Context Models: An In-Depth Exploration. Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, Graham Neubig. Arxiv 2024.

       

42. Many-shot Jailbreaking. Anthropic 2024.

43. DOLOMITES: Domain-Specific Long-Form Methodical Tasks. Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev, Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar, Dipanjan Das, Mirella Lapata, Chris Alberti. Arxiv 2024.

44. Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis. Yao Fu. Arxiv 2024.

45. FinTextQA: A Dataset for Long-form Financial Question Answering. Jian Chen, Peilin Zhou, Yining Hua, Yingxin Loh, Kehui Chen, Ziyuan Li, Bing Zhu, Junwei Liang. Arxiv 2024.

46. A Multi-Perspective Analysis of Memorization in Large Language Models. Bowen Chen, Namgi Han, Yusuke Miyao. Arxiv 2024.

47. OLAPH: Improving Factuality in Biomedical Long-form Question Answering. Minbyul Jeong, Hyeon Hwang, Chanwoong Yoon, Taewhoo Lee, Jaewoo Kang. Arxiv 2024.

       

48. Can LLMs Solve longer Math Word Problems Better?. Xin Xu, Tong Xiao, Zitong Chao, Zhenya Huang, Can Yang, Yang Wang. Arxiv 2024.

       

49. Base of RoPE Bounds Context Length. Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, Weipeng Chen. Arxiv 2024.

50. Many-shot In-Context Learning. Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, Hugo Larochelle. Arxiv 2024.

51. Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models. Longze Chen, Ziqiang Liu, Wanwei He, Yunshui Li, Run Luo, Min Yang. Arxiv 2024.

       

52. Language Models Need Inductive Biases to Count Inductively. Yingshan Chang, Yonatan Bisk. Arxiv 2024.

       

53. Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding. Zhihan Zhang, Yixin Cao, Chenchen Ye, Yunshan Ma, Lizi Liao, Tat-Seng Chua. Arxiv 2024.

54. CRAG -- Comprehensive RAG Benchmark. Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue Liu, Nirav Shah, Rakesh Wanga, Anuj Kumar, Wen-tau Yih, Xin Luna Dong. Arxiv 2024.

55. An Empirical Study of Mamba-based Language Models. Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, Bryan Catanzaro. Arxiv 2024.

       

56. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, Mikhail Burtsev. Arxiv 2024.

       

57. Can Many-Shot In-Context Learning Help Long-Context LLM Judges? See More, Judge Better!. Mingyang Song, Mao Zheng, Xuan Luo. Arxiv 2024.

       

58. What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modeling. Yutong Hu, Quzhe Huang, Kangcheng Luo, Yansong Feng. Arxiv 2024.

59. Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective. Meizhi Zhong, Chen Zhang, Yikun Lei, Xikai Liu, Yan Gao, Yao Hu, Kehai Chen, Min Zhang. Arxiv 2024.

60. Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?. Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, Kelvin Guu. Arxiv 2024.

       

61. Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell. Taiming Lu, Muhan Gao, Kuai Yu, Adam Byerly, Daniel Khashabi. Arxiv 2024.

       

62. MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokens. Yongqi Fan, Hongli Sun, Kui Xue, Xiaofan Zhang, Shaoting Zhang, Tong Ruan. Arxiv 2024.

       

63. USDC: A Dataset of $\underline{U}$ser $\underline{S}$tance and $\underline{D}$ogmatism in Long $\underline{C}$onversations. Mounika Marreddy, Subba Reddy Oota, Venkata Charan Chinni, Manish Gupta, Lucie Flek. Arxiv 2024.

64. Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization. Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T. Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, Tomas Pfister. Arxiv 2024.

65. One Thousand and One Pairs: A "novel" challenge for long-context language models. Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, Mohit Iyyer. Arxiv 2024.

       

66. LongIns: A Challenging Long-context Instruction-based Exam for LLMs. Shawn Gavin, Tuney Zheng, Jiaheng Liu, Quehry Que, Noah Wang, Jian Yang, Chenchen Zhang, Wenhao Huang, Wenhu Chen, Ge Zhang. Arxiv 2024.

67. Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA. Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li. Arxiv 2024.

       

11.2 MLLM

1. MileBench: Benchmarking MLLMs in Long Context. Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, Benyou Wang. Arxiv 2024.

       

2. Many-Shot In-Context Learning in Multimodal Foundation Models. Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H. Chen, Andrew Y. Ng. Arxiv 2024.

       

3. MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, Zheng Liu. Arxiv 2024.

       

4. RepoQA: Evaluating Long Context Code Understanding. Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, Lingming Zhang. Arxiv 2024.

       

5. Short Film Dataset (SFD): A Benchmark for Story-Level Video Understanding. Ridouane Ghermi, Xi Wang, Vicky Kalogeiton, Ivan Laptev. Arxiv 2024.

       

6. Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models. Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, Hao Wang. Arxiv 2024.

       

7. Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts. Aditya Sharma, Michael Saxon, William Yang Wang. Arxiv 2024.

12. Blogs

1. Extending Context is Hard…but not Impossible†. kaiokendev. 2023.

2. NTK-Aware Scaled RoPE. u/bloc97 . 2023.

3. The Secret Sauce behind 100K context window in LLMs: all tricks in one place. Galina Alperovich. 2023.

4. Transformer升级之路：7、长度外推性与局部注意力. 苏剑林(Jianlin Su). 2023.

5. Transformer升级之路：9、一种全局长度外推的新思路. 苏剑林(Jianlin Su). 2023.

6. Transformer升级之路：12、无限外推的ReRoPE. 苏剑林(Jianlin Su). 2023.

7. Transformer升级之路：14、当HWFA遇见ReRoPE. 苏剑林(Jianlin Su). 2023.

8. Transformer升级之路：15、Key归一化助力长度外推. 苏剑林(Jianlin Su). 2023.

9. Transformer升级之路：16、“复盘”长度外推技术. 苏剑林(Jianlin Su). 2024.

10. Introducing RAG 2.0. Contextual AI Team. 2024.

11. How Do Language Models put Attention Weights over Long Context?. Yao Fu. 2024.

12. An open-source and open-access RAG platform. Yunfan Gao. 2024.

13. Many-shot Jailbreaking. Anthropic. 2024.

14. Full Stack Transformer Inference Optimization Season 2: Deploying Long-Context Models. Yao Fu. 2024.

15. 缓存与效果的极限拉扯：从MHA、MQA、GQA到MLA. 苏剑林(Jianlin Su). 2024.

16. Towards 100x Speedup: Full Stack Transformer Inference Optimization. Yao Fu. 2024.

17. 2024.5 A Side-by-Side Comparison of the Long Context of Various LLMs (128k articles). SomeoneKong. 2024.

       

18. 2024.5 A Side-by-Side Comparison of the Long Context of Various LLMs (32k articles). SomeoneKong. 2024.

       

19. Transformer升级之路：18、RoPE的底数设计原则. 苏剑林(Jianlin Su). 2024.

20. Generalizing an LLM from 8k to 1M Context using Qwen-Agent. Qwen Team. 2024.

Acknowledgements

Please contact me if I miss your names in the list, I will add you back ASAP!