Efficient Streaming Language Models with Attention Sinks
Open accessAbstract
Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a "sink" even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous work on KV cache compression often allocates a fixed KV cache size across LLM layers.
“Previous work on KV cache compression (Li et al., 2024; Zhang et al., 2024; Xiao et al., 2023) often allocates a fixed KV cache size across LLM layers.”
From PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024) · cites this paper as Xiao et al., 2023
The Introduction (p. 2) describes a fixed-size sliding window for cached KV states, and Section 3.2 (p. 5) describes StreamingLLM retaining initial and recent tokens. These passages support the fixed-size cache aspect, but they do not explicitly say that the same cache size is allocated across LLM layers. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe last α tokens of the input contain the most immediate task-related information.
“Following the common practice in KV cache compression (Li et al., 2024; Xiao et al., 2023), we first retain the KV cache for the last α tokens of the input across all layers, as these tokens have been shown to contain the most immediate task-related information”
From PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024) · cites this paper as Xiao et al., 2023
Section 3.2 (p. 5) says the rolling KV cache retains the most recent tokens because they are crucial for language modeling. Section 1 (p. 2) also reports that recent tokens receive more attention in the first two layers. The paper does not establish the stronger claim that the last α tokens contain the most immediate task-related information. (LLM, conf=0.82)
Experimental — accuracy not yet measuredAttention matrices in LLMs are typically sparse, focusing disproportionately on a few tokens.
“attention matrices in LLMs are typically sparse (Chen et al., 2024a; Xiao et al., 2023; Zhang et al., 2024), focusing disproportionately on a few tokens.”
From PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024) · cites this paper as Xiao et al., 2023
The Introduction (p. 2) and quantitative analysis (p. 18) report that a few initial tokens attract disproportionate attention; the first token often receives more than half the attention in most layers and heads. The paper does not establish the broader claim that LLM attention matrices are typically sparse. (LLM, conf=0.82)
Experimental — accuracy not yet measuredExisting methods have not effectively addressed the challenges encountered in real-world applications, where prompts are exceptionally long yet require accurate information retrieval.
“In summary, existing methods have not effectively addressed the challenges encountered in realworld applications, where prompts are exceptionally long yet require accurate information retrieval.”
From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [5]
Related Work (pp. 3–4) says that extending context does not ensure effective use of long text and that using prolonged contexts remains a challenge. Discussions (p. 15) says StreamingLLM is unsuitable for long-document question answering, but the paper does not establish that existing methods generally fail at accurate retrieval from exceptionally long prompts in real-world applications. (LLM, conf=0.82)
Experimental — accuracy not yet measuredExisting KV cache compression methods mainly focus on compressing the KV cache appended during decoding steps, while overlooking compression of prompt KV cache.
“they mainly focus on compressing the KV cache appended during decoding steps, while overlooking the realistic problem of compressing KV cache for prompts, which is typically the bottleneck in memory efficiency.”
From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [5]
The Introduction (p. 2) describes KV-cache memory growth during decoding and discusses sliding-window approaches, supporting the decoding-focused part of the claim. It does not establish that existing compression methods mainly focus on decoding, that they overlook prompt-cache compression, or that prompt caches are typically the memory-efficiency bottleneck. (LLM, conf=0.83)
Experimental — accuracy not yet measured
The efficacy of long-context LLMs and RAG techniques in very long-term dialogues remains unexplored.
“Despite advancements in longcontext large language models (LLMs) and retrieval augmented generation (RAG) techniques, their efficacy in very long-term dialogues remains unexplored.”
From Evaluating Very Long-Term Conversational Memory of LLM Agents (2024) · cites this paper as Xiao et al., 2023
Related Work (p. 3–4) says that extending an LLM’s context does not ensure effective use of long text and that using prolonged contexts effectively remains a challenge. The Introduction (p. 1) identifies day-long conversations as a desirable application, but the paper does not say that efficacy in very long-term dialogues remains unexplored, and the retrieved passages do not address RAG techniques. (LLM, conf=0.84)
Experimental — accuracy not yet measuredContext window adaptation techniques seek to extend the usable input length of LLMs by modifying attention mechanisms, positional encoding schemes, or input structures.
“Context window adaptation techniques seek to extend the usable input length of LLMs by modifying attention mechanisms, positional encoding schemes, or input structures.”
From From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms (2026) · cites this paper as Xiao et al., 2023
Related Work (p. 3) describes extending LLM context windows through approximate attention methods and RoPE position interpolation. It does not identify changes to input structure as a context-window extension technique; instead, Related Work (p. 3) distinguishes StreamingLLM’s approach to longer input streams from expanding the attention window. (LLM, conf=0.86)
Experimental — accuracy not yet measuredOver time as the context grows, vital character information gets lost due to the "attention sink," leading to degraded NPC responses.
“Over time as the context grows, vital character information gets lost due to a phenomenon known as the “attention sink” [7], leading to degraded NPC responses.”
From Post-Thinking in NPC Dialogue: A Paradigm for Reflective Character Models (2026) · cites this paper as [7]
Section 3.1 (p. 4) reports that long-text performance deteriorates when window attention excludes initial tokens from the cache, and explains this using attention sinks. It does not report that attention sinks cause vital character information to be lost as context grows or that NPC responses degrade. The same section finds that replacing the initial tokens with linebreaks largely restores performance, pointing to their position rather than their semantic content. (LLM, conf=0.85)
Experimental — accuracy not yet measuredAttention focuses most on tokens in the beginning and the end of the prompt, while the context in the middle receives disproportionately less attention, a phenomenon termed "attention sink."
“Xiao et al. demonstrates that attention focuses most on tokens in the beginning and the end of the prompt, while the context in the middle receives disproportionately less attention, a phenomenon termed “attention sink” [7].”
From Post-Thinking in NPC Dialogue: A Paradigm for Reflective Character Models (2026) · cites this paper as [7]
The Introduction (p. 2) identifies initial tokens that attract disproportionate attention as “attention sinks.” Section 4.2 (p. 8) also reports local attention in early layers, but the paper does not establish that attention is highest at both ends of a prompt or define reduced attention to the middle as the attention-sink phenomenon. (LLM, conf=0.88)
Experimental — accuracy not yet measuredXiao et al. (2023) identified an 'attention sink' phenomenon, where maintaining the KV states of the first few tokens can substantially restore the performance of windowed attention, despite these tokens not being semantically crucial.
“Xiao et al. (2023) identified an 'attention sink' phenomenon, where maintaining the Key and Value (KV) states of the first few tokens can substantially restore the performance of windowed attention, despite these tokens not being semantically crucial.”
From PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024) · cites this paper as Xiao et al., 2023
StreamingLLM shows that rolling KV cache with the correct relative position is crucial for maintaining performance.
“StreamingLLM (Xiao et al., 2023) shows that rolling kv cache with the correct relative position is crucial for maintaining performance.”
From PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024) · cites this paper as Xiao et al., 2023
StreamLLM loses important information carried by discarded middle tokens.
“In StreamLLM [5], only the most recent tokens and attention sinks (first few tokens) are retained to reduce the KV cache size, making it lose the important information carried by the discarded middle tokens 2 .”
From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [5]
StreamingLLM and FastGen keep only a fixed window plus a few important tokens for long-sequence serving.
“StreamingLLM [33] and FastGen [6] keep only a fixed window plus a few important tokens for long-sequence serving.”
From TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing (2026) · cites this paper as [33]
Sink tokens were excluded from the KL loss by masking out a fixed set of sink positions on the key axis to stabilize optimization, since their consistently large attention weights can dominate gradients.
“To stabilize optimization, we exclude sink tokens (Xiao et al., 2023) from the KL loss by masking out a fixed set of sink positions on the key axis, since their consistently large attention weights can dominate gradients.”
From IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference (2026) · cites this paper as Xiao et al., 2023
The transformer attention patterns are noisy, with a strong 'attention sink' at position 0.
“The transformer attention patterns are noisy, with a strong”
From Memory Mosaics (2024) · cites this paper as Xiao et al., 2023
PyramidKV runs at a comparable speed to baselines (SnapKV, StreamingLLM, H2O), i.e., does not sacrifice inference speed.
“PyramidKV does not sacrifice the speed.”
From PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024) · cites this paper as Xiao et al., 2023
Section 4.5 (p. 9) benchmarks StreamingLLM’s decoding speed against sliding-window re-computation, not PyramidKV against SnapKV, StreamingLLM, and H2O. Searches for PyramidKV and inference speed found no passage in our copy addressing PyramidKV’s speed, so the cited paper cannot verify this claim. (LLM, conf=0.76)
Experimental — accuracy not yet measuredMost existing KV cache eviction methods lack detailed evaluation in long-context settings.
“However, most of these methods lack a detailed evaluation in long-context settings.”
From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [5]
The Introduction (p. 2) discusses performance problems with window attention, and the Long-Range Benchmark Evaluation (p. 17) evaluates StreamingLLM. Neither passage assesses whether most existing KV-cache eviction methods lack detailed long-context evaluations. Searches for that evaluation gap and for prior eviction methods found no passage addressing it. (LLM, conf=0.70)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 29
- Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508 , 2023. · arXiv
- Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. arXiv:2004.05150. · arXiv
- Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence , 2020. bloc97. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation., 2023. URL https://www.reddit.com/ r/LocalLLaMA/comments/14lz7j5/ntkaware\_scaled\_rope\_allows\_llama\_models\_to\_have/.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
- Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. arXiv: 2306.15595. · arXiv
- Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1 , 2018. · arXiv
- Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness, 2022. arXiv:2205.14135. · arXiv
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics , 2019. URL https://api.semanticscholar.org/CorpusID:52967399.
- Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2019.
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020. · arXiv
- Tanya Goyal and Greg Durrett. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020 , Online, 2020. Association for Computational Linguistics.
- Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multihop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics , December 2020.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020 . OpenReview.net, April 2020.
- Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP , 2018.
- Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102 , 2022. · arXiv
- Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=R8sQPpGCv0.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations , 2020.
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641 , 2019. · arXiv
- Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864 , 2021. · arXiv
- Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys , 55(6), dec 2022. ISSN 0360-0300.
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023a. · arXiv
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023b. · arXiv
- Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. HPCA , 2021.
- Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning , 2023.
- Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2018.
- Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big Bird: Transformers for longer sequences. In Proc. of NeurIPS , volume 33, 2020a.
- Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020 . Curran Associates, Inc., 2020b.
- Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? CoRR , abs/1905.07830, 2019. URL http://arxiv.org/abs/1905.07830. · arXiv
Cited by
Sorted by citations
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
“With ongoing efforts to expand LLM context windows to virtually unlimited sizes, the adaptation of RAG to these changes presents a significant research question [Xu et al. , 2023c, Packer et al. , 2023, Xiao et al. , 2023].”
BackgroundPoint Transformer V3: Simpler, Faster, Stronger
Wu, X., Jiang, L., Wang, P., Liu, Z., Liu, X., Qiao, Y., Ouyang, W., He, T., & Zhao, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00463 · Cited by 727 · see citing works
EPlus-LLM: A large language model-based computing platform for automated building energy modeling
Jiang, G., Ma, Z., Zhang, L., & Chen, J. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.123431 · Cited by 204 · see citing works
Friha, O., Ferrag, M. A., Kantarcı, B., Çakmak, B., Ozgun, A., & Ghoualmi‐Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 209 · see citing works
Evaluating Very Long-Term Conversational Memory of LLM Agents
Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.747 · Cited by 96 · see citing works
VideoLLM-online: Online Video Large Language Model for Streaming Video
Chen, J., Lv, Z., Wu, S., Lin, K. Q., Song, C., Gao, D., Liu, J.-W., Gao, Z., Mao, D., & Shou, M. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01742 · Cited by 83 · see citing works
Irfan, B., Kuoppamäki, S., Hosseini, A., & Skantze, G. (2025) · Autonomous Robots · doi:10.1007/s10514-025-10190-y · Cited by 75 · see citing works
A review of AI edge devices and lightweight CNN and LLM deployment
Sun, K., Wang, X., Miao, X., & Zhao, Q. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128791 · Cited by 76 · see citing works
Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
Cao, C., Sang, J., Arora, R., Chen, D., Kloosterman, R., Cecere, M., Gorla, J., Saleh, R., Drennan, I., Teja, B., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-24-02189 · Cited by 67 · see citing works
“Finally, for lengthy documents (that is, full-text articles), our ISO-ScreenPrompt approach addresses the lost-in-the-middle phenomenon (42 – 44) by repeating and numbering instructions at the start and end of the prompt (43).”
Understanding LLMs: A comprehensive overview from training to inference
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129190 · Cited by 68 · see citing works
Zhu, H., Gu, J., Wang, H., Jiang, Z., Zhang, Z., Tang, R., Feng, C., Han, S., Chen, R. T., & Pan, D. Z. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00059 · Cited by 51 · see citing works
Self-citationLLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang, Y., Lin, C.-Y., et al. (2024) · Findings of the Association for Computational Linguistics: ACL 2024 · doi:10.18653/v1/2024.findings-acl.57 · Cited by 58 · see citing works
Resource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 153 · see citing works
Pashangpour, S., & Nejat, G. (2024) · Robotics · doi:10.3390/robotics13080112 · Cited by 37 · see citing works
Learning Visual Prompt for Gait Recognition
Ma, K., Fu, Y., Cao, C., Hou, S., Huang, Y., & Zheng, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00063 · Cited by 34 · see citing works
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.814 · Cited by 27 · see citing works
InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference
Pan, X., Li, E., Li, Q., Liang, S., Shan, Y., Zhou, K., Luo, Y., Wang, X., & Zhang, J. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00113 · Cited by 38 · see citing works
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.936 · Cited by 26 · see citing works
Luo, X., Liu, D., Kong, H., Huai, S., Chen, H., Xiong, G., & Liu, W. (2024) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3701728 · Cited by 24 · see citing works
Titans: Learning to Memorize at Test Time
Behrouz, A., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2501.00663 · Cited by 24 · see citing works
“From the technical perspective, these learnable parameters at the start of the sequence can mitigate such effect by redistributing the attention weights more effectively [120, 121].”
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · doi:10.31219/osf.io/ac4sp · Cited by 51 · see citing works
Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format
Fang, C., Shi, M., Geens, R., Symons, A., Wang, Z., & Verhelst, M. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00110 · Cited by 16 · see citing works
Prompt Highlighter: Interactive Control for Multi-Modal LLMs
Zhang, Y., Qian, S., Peng, B., Liu, S., & Jia, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01255 · Cited by 14 · see citing works
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · Cited by 13 · see citing works
“Various sparse attention mechanisms (Child et al., 2019; Jaszczur et al., 2021) such as shifted sparse attention (Chen et al., 2024), dilated attention (Ding et al., 2023), and attention sinks (Han et al., 2023; Xiao et al., 2024b) were employed to enable efficient context scaling.”
MethodReimagining Self-Adaptation in the Age of Large Language Models
Donakanti, R., Jain, P., Kulkarni, S., & Vaidhyanathan, K. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00036 · Cited by 12 · see citing works
Yang, Y., Yuan, G., & Li, J. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3544736 · Cited by 12 · see citing works
HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
Li, Y., Gu, R., Huan, C., Wang, Z., Yao, R., Tian, C., & Chen, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3749168 · Cited by 9 · see citing works
TightLLM: Maximizing Throughput for LLM Inference via Adaptive Offloading Policy
Hu, Y., Liu, X., Yang, G., Li, L., Zeng, K., Zhao, Z., Chen, S., Zhao, L., Li, W., & Li, K. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3558009 · Cited by 8 · see citing works
Qin, Z., Bai, Y., Sohrabizadeh, A., Ding, Z., Hu, Z., Sun, Y., & Cong, J. (2024) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1109/mlcad62225.2024.10740204 · Cited by 7 · see citing works
SnapKV: LLM Knows What You are Looking for Before Generation
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., & Chen, D. (2024) · arXiv · doi:10.48550/arxiv.2404.14469 · Cited by 74 · see citing works
“There are many perspectives to mitigate these problems, including KV cache eviction during token generation [5–8].”
BackgroundLAD: Efficient Accelerator for Generative Inference of LLM with Locality Aware Decoding
Wang, H., Li, Y., Xu, H., Wang, Y., Liu, L., Yang, J., & Han, Y. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00111 · Cited by 6 · see citing works
XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models
Kumar, S., Madikeri, S. R., Zuluaga-Gómez, J., Villatoro-Tello, E., Thorbecke, I., Motlíček, P., Manjunath, K. E., & Ganapathiraju, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888110 · Cited by 6 · see citing works
Anchor Attention, Small Cache: Code Generation With Large Language Models
Zhang, X., Zhou, Y., Yang, G., Gall, H. C., & Chen, T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3570680 · Cited by 6 · see citing works
FINCH: Prompt-guided Key-Value Cache Compression for Large Language Models
Corallo, G., & Papotti, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00716 · Cited by 5 · see citing works
2-D Transformer: Extending Large Language Models to Long-Context With Few Memory
He, X., Liu, J., & Duan, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3548047 · Cited by 5 · see citing works
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
Liu, X., Tang, Z., Dong, P., Li, Z., Liu, Y., Li, B., Hu, X., & Chu, X. (2025) · arXiv · doi:10.48550/arxiv.2502.00299 · Cited by 4 · see citing works
“Recent advances in long-context technology within the field of machine learning (ML) systems [6–8] have significantly improved computational throughputs and reduced latency of LLMs to process increasingly large input context lengths [9, 10] with historical KV cache (key value attentions).”
Efficient Streaming LLM for Speech Recognition
Jia, J., Keren, G., Zhou, W. J., Lakomkin, E., Zhang, X., Wu, C., Seide, F. T. B., Mahadeokar, J., & Kalinli, O. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890853 · Cited by 4 · see citing works
WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models
Yuan, J., He, Z., Bai, H., Leng, J., & Jiang, B. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889583 · Cited by 4 · see citing works
End-to-End Acceleration of Generative Models With Runtime Regularized KV Cache Management
Moradifirouzabadi, A., & Kang, M. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3568716 · Cited by 4 · see citing works
ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughput
Kim, J., Lee, H., Ko, G., Choi, G., Ham, S., Hong, S., & Kim, J.-Y. (2025) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass64960.2025.00012 · Cited by 4 · see citing works
Proceedings of the Workshop on Edge and Mobile Foundation Models
(2024) · doi:10.1145/3662006 · Cited by 4 · see citing works
Evaluating Very Long-Term Conversational Memory of LLM Agents
Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024) · arXiv · doi:10.48550/arxiv.2402.17753 · Cited by 3 · see citing works
“Despite recent advancements in dialogue models based on LLMs for extended contexts (Bertsch et al., 2023; Xiao et al., 2023), as well as the integration of retrieval augmented generation (RAG) Figure 1: An example in L O C O M O .”
BackgroundFrom Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms
Luo, J., Tian, Y., Cao, C., Luo, Z., Lin, H., Li, K., Kong, C., Yang, R., & Ma, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.2069 · Cited by 3 · see citing works
“…focuses on the extension of the window of context via modifications to the mechanism of attention or the encoding of position (Ratner et al., 2022; Xiao et al., 2023; Jin et al., 2024), as well as the achievement of information sparsification through the mechanical reduction of noise (Zhang et…”
Self-citationOASIS: Outlier-Aware KV Cache Clustering for Scaling LLM Inference in CXL Memory Systems
Seo, M., Hyun, J., Jeong, S., Nguyen, X. T., Lee, H.-J., & Lee, H. (2025) · IEEE Computer Architecture Letters · doi:10.1109/lca.2025.3567844 · Cited by 3 · see citing works
When Medical Imaging Met Self-Attention: A Love Story That Didn’t Quite Work out
Piater, T., Penzel, N., Stein, G., & Denzler, J. (2024) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0012382600003660 · Cited by 3 · see citing works
Enhancing Large Language Model Inference Efficiency via Lookahead Cache Filtering
Ou, J., Chen, Y., Jiang, S., & Tian, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888646 · Cited by 3 · see citing works
The Continuous Language of Protein StructurePreprint
Billera, L., Oresten, A., Stålmarck, A., Sato, K., Kaduk, M., & Murrell, B. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.05.11.593685 · Cited by 2 · see citing works
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
Cai, Z., Zhang, Y., Gao, B., Liu, Y., Li, Y., Liu, T., Lu, K., Xiong, W., Dong, Y., Hu, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.02069 · Cited by 2 · see citing works
“Previous work on KV cache compression [22, 34, 32] often allocates a fixed KV cache size across LLM layers.”
BackgroundMethodInfluentialFrom Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs
Wu, Y., Liang, S., Zhang, C., Wang, Y., Zhang, Y., Guo, H., Tang, R., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.15965 · Cited by 3 · see citing works
“LookupFFN [109], ChunkKV [110], vLLM [111], FastServe [112], StreamingLLM [113], Orca [114], DistServe [115], LLM.int8() [116], FastGen [117], Train Large, Then Compress [118], Scissorhands [119], , Mooncake [121], MemServe [122], SLM Serving [123], IMPRESS [124], AdaServe [125], MPIC [126],…”
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang, Y., Lin, C.-Y., et al. (2024) · arXiv · Cited by 2 · see citing works
“An alternative approach is to compress or trim the context hidden or KV caches (Chevalier et al., 2023; Ge et al., 2024; Zhang et al., 2023; Liu et al., 2023; Xiao et al., 2024).”
BackgroundDroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving
Liu, Y., Huang, Y., Yao, J., Feng, S., Gu, Z., Du, K., Li, H., Cheng, Y., Jiang, J., Lu, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.02820 · Cited by 2 · see citing works
Beyond KV caching: Shared attention for efficient LLMs
Liao, B., & Vargas, D. V. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130587 · Cited by 2 · see citing works
LongSight: Compute-Enabled Memory to Accelerate Large-Context LLMs via Sparse Attention
Quinn, D., Yücel, E. E., Kim, J., Martínez, J. F., & Alian, M. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756062 · Cited by 2 · see citing works
The Locality and Symmetry of Positional Encodings
Chen, L., Varoquaux, G. P., & Suchanek, F. M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.955 · Cited by 3 · see citing works
LVMScissor: Split and Schedule Large Vision Model Inference on Mobile Edges via Salp Swarm Algorithm
Liu, Y., Lu, R., & Wang, D. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3550519 · Cited by 2 · see citing works
Stabilization and Approximation in Recurrent Neural Computation: An Enchan Field ReadingPreprint
Kobayashi, M. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20328350 · Cited by 2 · see citing works
Kobayashi, M. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20347694 · Cited by 2 · see citing works
Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores
Chari, V., & Van Durme, B. (2025) · arXiv · doi:10.48550/arxiv.2507.08143 · Cited by 1 · see citing works
“We aim to retain only a fraction r ∈ (0 , 1] of the tokens while preserving a model’s predictive quality.”
How Sparse Attention Approximates Exact Attention? Your Attention is Naturally $n^C$-Sparse
Deng, Y., Song, Z., Xiong, J., & Yang, C. (2024) · arXiv · doi:10.48550/arxiv.2404.02690 · Cited by 1
Fusing AI: Multimodal Language Models Inference Across Diverse Inputs
Jovanovic, M., & Campbell, M. (2024) · Computer · doi:10.1109/mc.2024.3445515 · Cited by 1 · see citing works
ALAS: A Stateful Multi-LLM Agent Framework for Disruption-Aware Planning
Chang, E. Y. (2025) · ACM eBooks · doi:10.1145/3749421.3749436 · Cited by 1 · see citing works
Sheng, C. Y., Xin, C. Y., & Jie, T. G. (2026) · Cureus Journal of Computer Science. · doi:10.7759/s44389-026-00047-5 · Cited by 1 · see citing works
Tan, J., Yu, G., Li, J., Ma, X., Bao, F., Pan, E., Bian, D., Li, Y., Du, Y., Du, L., et al. (2024) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas59952.2024.10595886 · Cited by 1 · see citing works
Chen, S. (2026) · arXiv
“Long-context degradation is a known confound: models under-use mid-context information [17], exhibit “context rot” as distractors grow [13], and show unstable refusal in long-context agents [11].”
Post-Thinking in NPC Dialogue: A Paradigm for Reflective Character Models
Carey, K., & Wang, H. (2026) · Proceedings of the 21st International Conference on the Foundations of Digital Games · doi:10.1145/3815598.3815681
Zhang, J., Nolte, N., Sadhukhan, R., Chen, B., & Bottou, L. (2024) · arXiv · doi:10.48550/arxiv.2405.06394
“The transformer attention patterns are noisy, with a strong “attention sink” at position 0 (Xiao et al., 2023).”
Self-citationAn, Y., Lu, C., Zhu, K., Yu, T., Zhao, C., Wu, H., Tang, M., & Wang, J. (2026) · arXiv · doi:10.48550/arxiv.2605.08840
“We compare ReST-KV with five baselines: (1) Fixed Attention Patterns: StreamingLLM (Xiao et al., 2023); (2) Eviction Indicator: H2O (Zhang et al., 2023), TOVA (Oren et al., 2024), SnapKV (Li et al., 2024b), LaCache (Shi et al., 2025).”
InfluentialMake Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
Bui, N., Nguyen, H. T., Cohan, A., & Ying, R. (2026) · arXiv · doi:10.48550/arxiv.2605.09649
“In these scenarios, simple token-eviction heuristics [Xiao et al., 2023; Li et al., 2024c; Cai et al., 2025] are typically preferred due to their low overhead, at the cost of potential performance degradation.”
InfluentialSelf-GC: Self-Governing Context for Long-Horizon LLM Agents
Hao, X., Meng, H., Yin, X., Zhu, J., & Cao, C. (2026) · arXiv · doi:10.48550/arxiv.2607.00692
“…while model-internal long-context systems compress or evict KV cache through heavy-hitter selection, attention sinks, query-aware selection, pyramidal information funneling, or recoverable channel pruning (Zhang et al. 2023; Xiao et al. 2023; Li et al. 2024; Cai et al. 2024; Liao et al. 2026).”
How Query Visibility Changes KV-Cache Compression Rankings: A Matched-Budget Audit
Luo, D., Liang, C., & Xuan, J. (2026) · arXiv · doi:10.48550/arxiv.2607.11942
Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective
Wu, F., Silwal, S., Qiuyi, & Zhang (2026) · arXiv · doi:10.48550/arxiv.2601.18999
SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving
Zhang, Q., Chen, K., Liao, N., Lin, Z., Tang, B., Xiong, F., Li, Z., & Wang, X. (2026) · arXiv · doi:10.48550/arxiv.2606.01751
“StreamingLLM [18] reveals the attention-sink phenomenon: retaining the KV Cache tensors of a small number of initial tokens can substantially recover the performance of sliding-window attention under very long sequences, supporting streaming long-context inference.”
TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
Bian, Z., Wu, F., Zhang, C., Dong, H., Liang, Y., & Zhuo, Y. (2026) · arXiv · doi:10.48550/arxiv.2604.03143
Inference-Time Attention Steering for Vision-Language-Action Driving Models
Prasad, D. N., Ullrich, L., & Graichen, K. (2026) · arXiv · doi:10.48550/arxiv.2608.17095
“Recent work shows that multimodal attention flow is layer-dependent, with late layers carrying the bulk of action-relevant signal [25], and that LLMs concentrate disproportionate attention mass on initial-token “sinks” whose disruption destabilizes generation [24].”
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
Yang, X., Gu, H., Xu, B., Li, L., Liu, B., Liu, J., Zhu, Q., Guo, Y., & Han, S. (2026) · arXiv · doi:10.48550/arxiv.2605.25475
“We then minimize To stabilize optimization, we exclude sink tokens (Xiao et al., 2023) from the KL loss by masking out a fixed set of sink positions on the key axis, since their consistently large attention weights can dominate gradients.”
Self-citationFound in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16008
“…to handle longer input contexts, including designing efficient training and finetuning schemes (Dao et al., 2022; Li et al., 2023b,a; Shi et al., 2023b) and inference-time methods that extend an LLM’s context length (Press et al., 2021; Ratner et al., 2023; Xiao et al., 2023; Bertsch et al., 2023).”
Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression
Miao, R., Wang, Z., Li, W., Wu, S., Liu, S., Jiang, Y., & Yang, T. (2026) · arXiv · doi:10.48550/arxiv.2603.20616
“StreamingLLM (Xiao et al., 2023) marks the initial tokens and the tokens in the recent window as important.”
VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
Esmat, S., Shah, D., & Jannesari, A. (2026) · arXiv · doi:10.48550/arxiv.2607.15498
“The cache, however, grows linearly with context length, and at long context it—not the model’s weights—comes to dominate GPU memory at decode time (Xiao et al. 2024).”
An Empirical Study of Harness Design for Coding Agents
Fan, R.-Z., Zhang, Z., Ma, S., Hu, Y., Wang, S., Song, K., Liu, F., Zamani, H., & Wang, X. (2026) · arXiv · doi:10.48550/arxiv.2609.20804
“Lossy methods such as elision and summarization reduce context but may remove information that becomes useful later (Xiao et al., 2024; Jiang et al., 2023; Wu et al., 2021), whereas lossless methods preserve recoverability through external storage and retrieval but require additional machinery and…”
Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching
Ma, Q., Tang, Z., Cui, H., Yao, Z., & Jia, W. (2026) · arXiv · doi:10.48550/arxiv.2606.07684
“Beyond quantization, pruning strategies such as H2O (Zhang et al., 2023), SnapKV (Li et al., 2024a), Quest (Tang et al., 2024), and PyramidKV (Cai et al., 2024b) evict less salient tokens, while StreamingLLM (Xiao et al., 2023) uses attention sinks for infinite-length inference.”
Design and Implementation of a Lightweight Neural Controller for LLM Agent Systems
Langay, B. B. (2026) · International Journal of Latest Technology in Engineering Management & Applied Science · doi:10.51583/ijltemas.2026.150500172
Tamarin (HSQ): Reversible Hierarchical KV-Cache Compression for LLM Inference
Bazakov, A., & Kiselev, K. (2026) · doi:10.21203/rs.3.rs-10297225/v1
EvolKV: Evolutionary KV Cache Compression for LLM Inference
Yu, B., & Chai, Y. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.88
OjaKV: Context-Aware Online Low-Rank KV Cache Compression
Zhu, Y., Yang, D. H., Amiri, M. M., Murugesan, K., Pedapati, T., & Chen, P.-Y. (2025) · arXiv · doi:10.48550/arxiv.2509.21623
AI in emergency medicine: Building literacy or castles in the air
Niset, A., Hadwe, S. E., Englebert, A., & Barrit, S. (2024) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2024.08.020
Pearl, D., Murphy, M., & Intriligator, J. M. (2025) · Research Square · doi:10.21203/rs.3.rs-6397317/v1
Gim, I., & Zhong, L. (2025) · Workshop on Hot Topics in Operating Systems · doi:10.1145/3713082.3730398
Corm: Cache Optimization with Recent Message for Large Language Model InferencePreprint
Huang, Z., Jiang, H., & Dai, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5388897
EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse
Guo, T., Dong, H., Leng, Y., Liu, F., Lin, C., Xiao, N., & Zhang, X. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-99857-7_24
SagaLLM: Persistent Context Management, Constraint Validation, and Transaction Guarantees
Chang, E. Y. (2025) · ACM eBooks · doi:10.1145/3749421.3749435
Zhang, C., Shen, L., & Li, Y. (2026) · Complex & Intelligent Systems · doi:10.1007/s40747-026-02290-8
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., & Hu, X. (2024) · arXiv · doi:10.48550/arxiv.2402.02750
Self-citationPre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets
Berenbaum, T., & Venkatachalam, M. (2026) · arXiv · doi:10.48550/arxiv.2608.19147
Zerhoudi, S., Mitrović, J., & Granitzer, M. (2026) · ACM · doi:10.1145/3799682.3840567
- Authors
- Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2309.17453
- OpenAlex
- W4387294617
- Cited by
- 55
- References
- 0
- Citations by year
- 2026: 5 · 2025: 25 · 2024: 22 · 2023: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- attention sinks, key-value caching, window attention, multi-turn dialogue, sliding window decoding, LLM deployment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
