The DOI matches its DataCite record; 1 of the 6 references it cites were found in a registry or an index, and 5 are not checked yet.
Extending Context Window of Large Language Models via Positional Interpolation
Open accessAbstract
We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least $\sim 600 \times$ smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Recent works extend context windows to 100K+ tokens [18], [19], [23].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“unlimited context remains computationally infeasible and doesn't address selective retention or consistency.”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, many model developments do not incorporate instruction tuning (Chen et al., 2023; Fu et al., 2024), leaving these models reliant on perplexity-based evaluation or synthetic tasks.”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the fundamental tension persists: more capacity does not eliminate the need for selective curation.”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Extending context windows to 100k+ tokens [Chen et al., 2023] delays the problem but cannot eliminate it, and longer contexts incur quadratic cost increases in attention.”
From Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers (2026) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite context windows stretching to 200k tokens [Chen et al., 2023], long-context models consistently underperform purpose-built memory systems on tasks requiring selective retrieval and active management.”
From Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers (2026) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Some methods, such as PI [103] and SelfExtend [104], reorganize positional indices to ensure they remain within context lengths.”
From A Survey of Large Language Models (2026) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“they address extrapolation of the positional code, a different concern from the entropy growth we analyse”
From Curvature-Corrected Rotary Position Embeddings: An Entropy-Invariant Temperature for Mitigating Numerical-Rank Collapse in Long-Context Attention (2026) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“Position Interpolation [4] and YaRN [5] modify the RoPE frequencies to keep evaluation positions within the trained distribution”
From Curvature-Corrected Rotary Position Embeddings: An Entropy-Invariant Temperature for Mitigating Numerical-Rank Collapse in Long-Context Attention (2026) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“Position interpolation techniques enable models to process sequences beyond original context window limitations by intelligently rescaling position indices rather than extrapolating to unseen positions [153].”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [153]
The cited source says what the claim attributes to it.
In the citing paper
“Despite this growing importance, LLMs consistently face challenges in processing long contexts, primarily due to the substantial computational resources required for long sequence training (Dao et al., 2022; Dao, 2023) as well as the apparent inability to generalize to sequences longer than those encountered during training (Chen et al., 2023; Peng et al., 2023b).”
From ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024) · cites this paper as Chen et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Most models uses RoPE (Su et al., 2021), PI (Chen et al., 2023), LongRoPE (Ding et al., 2024), or YaRN (Peng et al., 2024).”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Chen et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Recent large language models (LLMs) support long contexts ranging from 128K to 1M tokens.”
From NoLiMa: Long-Context Evaluation Beyond Literal Matching (2025) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 19
- Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with per- formers. In 9th International Conference on Learning Representations, ICLR 2021. OpenRe- view.net, May 2021.
- Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Process- ing Systems, 2022.
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openreview.net/forum?id=YicbFdNTTy.
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
- Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. 2022.
- Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. · arXiv
- Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guided supervision for openqa with colbert. Transactions of the Association for Computational Linguistics, 9:929-944, 2021. doi: 10.1162/tacl a 00405. · DOI
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, April 2020.
- Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https://aclanthology.org/W04-1013.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https://openreview.net/forum?id= Bkg6RiCqY7.
- Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300, 2023. · arXiv
- Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=R8sQPpGCv0.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lilli- crap. Compressive transformers for long-range sequence modelling. In International Confer- ence on Learning Representations, 2020. URL https://openreview.net/forum?id= SylKikSYDH.
- Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, and Bo Dai. Combiner: Full attention transformer with sparse computation cost. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, pp. 22470-22482.
- Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 12007-12021, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022. emnlp-main.823.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30.
- Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/ file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
- Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing trans- formers. In The Tenth International Conference on Learning Representations, ICLR 2022. Open- Review.net, April 2022.
- Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020.
Cited by
Sorted by citations
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
Zheng, Y., Zhang, R., Zhang, J., YeYanhan, Y., & Luo, Z. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-demos.38 · Cited by 1,249
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531
“However, instead of downscaling frequencies linearly as Chen et al. (2023b), we change the base period from which they are derived.”
BackgroundMethodInfluentialGenome modelling and design across all domains of life with Evo 2
Brixi, G., Durrant, M. G., Ku, J., Naghipourfar, M., Poli, M., Sun, G., Brockman, G., Chang, D., Fanton, A., Gonzalez, G. A., et al. (2026) · Nature · doi:10.1038/s41586-026-10176-5 · Cited by 544
Pre-Trained Language Models for Text Generation: A Survey
Li, J., Tang, T., Zhao, W. X., Nie, J., & Wen, J.-R. (2024) · ACM Computing Surveys · doi:10.1145/3649449 · Cited by 383
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 193
From text to insight: large language models for chemical data extraction
Schilling-Wilhelmi, M., Ríos-García, M., Shabih, S., Gil, M. V., Miret, S., Koch, C. T., Marquez, J. A., & Jablonka, K. M. (2024) · Chemical Society Reviews · doi:10.1039/d4cs00913d · Cited by 178
Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 126
Self-citationInterpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models
Louis, A., DIJCK, G. V., & Spanakis, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30232 · Cited by 132
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., & Jia, Z. (2025) · ACM Computing Surveys · doi:10.1145/3754448 · Cited by 176
Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations
Yan, L., Echeverría, V., Fernandez-Nieto, G. M., Jin, Y., Swiecki, Z., Zhao, L., Gašević, D., & Martínez‐Maldonado, R. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650732 · Cited by 124
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.91 · Cited by 400
Large language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16789 · Cited by 71
Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
Wang, X., Salmani, M., Omidi, P., Ren, X., Rezagholizadeh, M., & Eshaghi, A. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/917 · Cited by 132
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 71
MemGPT: Towards LLMs as Operating Systems
Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2310.08560 · Cited by 64
“We generated the DMR question-answer (QA) pairs using a separate LLM that was instructed to write a question from one user to another that could only be answered correctly using knowledge gained from the past sessions (see Appendix for further details).”
MethodRecursively summarizing enables long-term dialogue memory in large language models
Wang, Q., Fu, Y., Cao, Y., Wang, S., Tian, Z., & Ding, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130193 · Cited by 55
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., & Jia, J. (2023) · arXiv · doi:10.48550/arxiv.2309.12307 · Cited by 50
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30205 · Cited by 66
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
Li, Z., Li, C., Zhang, M., Mei, Q., & Bendersky, M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track · doi:10.18653/v1/2024.emnlp-industry.66 · Cited by 49
Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
Zhang, Y., Sun, R., Chen, Y., Pfister, T., Zhang, R., & Arik, S. Ö. (2024) · arXiv · doi:10.48550/arxiv.2406.02818 · Cited by 47
“Window extension extends the context window of LLMs via finetuning to consume the whole input [13, 44, 48].”
On Context Utilization in Summarization with Large Language Models
Ravaut, M., Sun, A., Chen, N. F., & Joty, S. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.153 · Cited by 65
LongAlign: A Recipe for Long Context Alignment of Large Language Models
Bai, Y., Lv, X., Zhang, J., He, Y., Qi, J., Hou, L., Tang, J., Dong, Y., & Li, J. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.74 · Cited by 134
Weng, L., Wang, X., Lu, J., Feng, Y., Liu, Y., Feng, H., Huang, D., & Chen, W. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3567131 · Cited by 64
Hadi, M. U., Tashi, Q. A., Qureshi, R., Shah, A., Muneer, A., Irfan, M. I., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2025) · doi:10.36227/techrxiv.23589741.v8 · Cited by 49
Annotating the genome at single-nucleotide resolution with DNA foundation models
de Almeida, B. P., Dalla-Torre, H., Richard, G., Blum, C. F., Hexemer, L., Gélard, M., Mendoza‐Revilla, J., Tang, Z., Marin, F. I., Emms, D., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02881-2 · Cited by 39
Application of large language models to intelligently analyze long construction contract texts
Gao, Y., Gan, Y., Chen, Y., & Chen, Y. (2024) · Construction Management and Economics · doi:10.1080/01446193.2024.2415676 · Cited by 34
A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Qiu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2025) · ACM Computing Surveys · doi:10.1145/3729218 · Cited by 45
Transformers are Multi-State RNNs
Oren, M., Hassid, M., Yarden, N., Adi, Y., & Schwartz, R. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1043 · Cited by 143
Zhao, Q., Wang, R., Cen, Y., Zha, D., Tan, S., Dong, Y., & Tang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1259 · Cited by 32
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
Wang, M., Chen, L., Cheng, F., Liao, S., Zhang, X., Wu, B., Yu, H., Xu, N., Zhang, L., Luo, R., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.322 · Cited by 31
Self-citationHow to Train Long-Context Language Models (Effectively)
Gao, T., Wettig, A., Yen, H., & Chen, D. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.366 · Cited by 124
Ke, W., Zheng, Y., Li, Y., Xu, H., Nie, D., Wang, P., & He, Y. (2025) · ACM Transactions on Information Systems · doi:10.1145/3768156 · Cited by 36
AdaPipe: Optimizing Pipeline Parallelism with Adaptive Recomputation and Partitioning
Sun, Z., Cao, H., Wang, Y., Feng, G., Chen, S., Wang, H., & Chen, W. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651359 · Cited by 45
Self-citationA Continued Pretrained LLM Approach for Automatic Medical Note Generation
Yuan, D., Rastogi, E., Naik, G., Rajagopal, S. P., Goyal, S., Zhao, F., Chintagunta, B., & Ward, J. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) · doi:10.18653/v1/2024.naacl-short.47 · Cited by 28
Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling
Mu, Y., Dong, C., Bontcheva, K., & Song, X. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2x489fw7wi5m · Cited by 30
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.814 · Cited by 27
TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text
Lu, S., Wang, H., Rong, Y., Chen, Z., & Tang, Y. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.334 · Cited by 68
GPIoT: Tailoring Small Language Models for IoT Program Synthesis and Development
Shen, L., Yang, Q., Huang, X., Ma, Z., & Zheng, Y. (2025) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/3715014.3722064 · Cited by 24
LongEmbed: Extending Embedding Models for Long Context Retrieval
Zhu, D., Wang, L., Yang, N., Song, Y., Wu, W., Wei, F., & Li, S. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.47 · Cited by 50
Extending Context Window of Large Language Models via Semantic Compression
Fei, W., Niu, X., Zhou, P., Hou, L., Bai, B., Deng, L., & Han, W. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.306 · Cited by 20
LLM4MSR: An LLM-Enhanced Paradigm for Multi-Scenario Recommendation
Wang, Y., Wang, Y., Fu, Z., Li, X., Wang, W., Ye, Y., Zhao, X., Guo, H., & Tang, R. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679743 · Cited by 20
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E. P., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.14089 · Cited by 65
Salomão, P. E. A. (2026) · Journal International Review of Research Studies · doi:10.66104/kxf7hk05 · Cited by 19
Can ChatGPT support software verification?
Janßen, C., Richter, C., & Wehrheim, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57259-3_13 · Cited by 19
L-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.776 · Cited by 18
Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding
Zhao, L., Feng, X., Feng, X., Zhong, W., Xu, D., Yang, Q., Liu, H., Qin, B., & Liu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.582 · Cited by 44
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
“This approach is problematic as (1) closed-source APIs keep changing and have unpredictable availability (Pozzobon et al., 2023; Chen et al., 2023a), (2) it relies on the assumption that a more capable model exists (3) it can reinforce model hallucination (Gudibande et al., 2023) and (4) ,…”
BackgroundGRAM: Global Reasoning for Multi-Page VQA
Blau, T., Fogel, S., Ronen, R., Golts, A., Tsiper, S., Avraham, E. B., Aberdam, A., Ganz, R., & Litman, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01477 · Cited by 17
Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs
Lin, S., Chen, Y., Yang, S., Ding, X., Ge, Y., Chen, Y.-C., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01306 · Cited by 20
TeleOracle: Fine-Tuned Retrieval-Augmented Generation With Long-Context Support for Networks
Alabbasi, N., Erak, O., Alhussein, O., Lotfi, I., Muhaidat, S., & Debbah, M. A. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3553161 · Cited by 17
Wu, W., Pan, Z., Fu, K., Wang, C., Chen, L., BAI, Y., Wang, T., Wang, Z., & Xiong, H. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1079 · Cited by 34
Self-citationLiu, F., Farkiani, B., & Crowley, P. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111614 · Cited by 17
Zhong, W., Feng, X., Zhao, L., Li, Q., Huang, L., Gu, Y., Ma, W., Xu, Y., & Qin, B. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.648 · Cited by 38
Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models
Chen, L., Liu, Z., He, W., Zheng, Y., Sun, H., Li, Y., Luo, R., & Yang, M. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.447 · Cited by 27
Self-citationDing, R., Chen, Y., Lan, Y.-T., & Zhang, W. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679724 · Cited by 17
GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models
Li, S., He, Y., Guo, H., Bu, X., Bai, G., Liu, J., Liu, J., Qu, X., Li, Y., Ouyang, W., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.746 · Cited by 27
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · Cited by 1,068
“Recent advancements in AI system engineering (Dao et al., 2022; Jacobs et al., 2023; Fu et al., 2024) and language model designs (Chen et al., 2023; Xiong et al., 2023) have enabled efficient scaling up of context length for language models (Liu et al., 2024a; Young et al., 2024).”
BackgroundA Survey of Context Engineering for Large Language Models
Mei, L., Yao, J., Ge, Y., Wang, Y., Bi, B., Cai, Y., Liu, J., Li, M., Li, Z.-Z., Zhang, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.13334 · Cited by 281
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models
Diao, S., Pan, R. N., Dong, H., Shum, K., Zhang, J., Xiong, W., & Zhang, T. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-demo.12 · Cited by 18
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 12
“Add the subsequence x i,j = { x i , x i +1 , ..., x j } to the response y rec .”
BackgroundLarge Search Model: Redefining Search Stack in the Era of LLMs
Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., & Wei, F. (2023) · ACM SIGIR Forum · doi:10.1145/3642979.3643006 · Cited by 19
Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks
Wang, C., Duan, H., Zhang, S., Lin, D., & Chen, K. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.205 · Cited by 28
NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time
Chen, Y., Wang, G., Shang, J., Cui, S., Zhang, Z., Liu, T., Wang, S., Sun, Y., Yu, D., & Wu, H. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.428 · Cited by 16
Large Language Models for In-File Vulnerability Localization Can Be “Lost in the End”
Sovrano, F., Bauer, A., & Bacchelli, A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715758 · Cited by 12
Luo, K., Liu, Z., Xiao, S., Zhou, T., Chen, Y., Zhao, J. C., & Liu, K. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.180 · Cited by 17
ToolPlanner: A Tool Augmented LLM for Multi Granularity Instructions with Path Planning and Feedback
Wu, Q., Liu, W., Luan, J., & Wang, B. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1018 · Cited by 24
Hybe: GPU-NPU Hybrid System for Efficient LLM Inference with Million-Token Context Window
Moon, S., Cha, J., Park, H., & Kim, J.-Y. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731051 · Cited by 11
AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies
Xiao, Y., Wang, A., Choi, J., Lu, Y., Sharma, S., Shen, L., Tiyyala, V. M., Andrews, N., & Khashabi, D. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.725 · Cited by 10
Structured Packing in LLM Training Improves Long Context Utilization
Staniszewski, K., Tworkowski, S., Jaszczur, S., Zhao, Y., Michalewski, H., Kuciński, Ł., & Miłoś, P. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34706 · Cited by 10
HyperPIE: Hyperparameter Information Extraction from Scientific Publications
Saier, T., Ohta, M., Asakura, T., & Färber, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_17 · Cited by 11
Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-Context Models
Liu, X., Zhao, R., Huang, P., Xiao, C., Li, B., Wang, J., Xiao, T., & Zhu, J. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.269 · Cited by 13
LLoCO: Learning Long Contexts Offline
Tan, S., Li, X., Patil, S. G., Wu, Z., Zhang, T., Keutzer, K., Gonzalez, J. E., & Popa, R. A. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.975 · Cited by 28
ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage
Lee, T., Yoon, C., Jang, K., Lee, D., Song, M., Kim, H., & Kang, J. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.283 · Cited by 18
ReMamba: Equip Mamba with Effective Long-Sequence Modeling
Yuan, D., Liu, J., Li, B., Zhang, H., Wang, J., Cai, X., & Zhao, D. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.361 · Cited by 20
Real-GPT: Efficiently Tailoring LLMs for Informed Decision-Making in the Real Estate Industry
Gloria, B., Melsbach, J. W., Bienert, S., & Schoder, D. (2024) · Journal of Real Estate Portfolio Management · doi:10.1080/10835547.2024.2372748 · Cited by 10
Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis
Yu, J., Ding, Z., Tan, J., Luo, K., Weng, Z., Gong, C., Zeng, L., Cui, R., Han, C., Sun, Q., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.595 · Cited by 20
Sahu, G., Vechtomova, O., & Laradji, I. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.86 · Cited by 9
MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs
Alyafeai, Z., Al-shaibani, M. S., & Ghanem, B. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.655 · Cited by 8
Benchmarking and Improving Long-Text Translation with Large Language Models
Wang, L., Du, Z., Jiao, W., Lyu, C., Pang, J., Cui, L., Song, K., Wong, D., Shi, S., & Tu, Z. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.428 · Cited by 8
Sheng, B., Yao, J., Zhang, M., & He, G. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1538 · Cited by 10
ProConSuL: Project Context for Code Summarization with LLMs
Lomshakov, V., Podivilov, A., Savin, S. D., Baryshnikov, O., Lisevych, A., & Nikolenko, S. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-industry.65 · Cited by 12
Context-Aware Hierarchical Merging for Long Document Summarization
Ou, L., & Lapata, M. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.289 · Cited by 17
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1
InfiniPot: Infinite Context Processing on Memory-Constrained LLMs
Kim, M., Shim, K., Choi, J., & Chang, S. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.897 · Cited by 24
What Are the Odds? Language Models Are Capable of Probabilistic Reasoning
Paruchuri, A., Garrison, J., Liao, S., Hernandez, J., Sunshine, J. E., Althoff, T., Liu, X., & McDuff, D. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.654 · Cited by 19
The Role of Accuracy and Validation Effectiveness in Conversational Business Analytics
Alparslan, A. (2025) · IEEE Access · doi:10.1109/access.2025.3540975 · Cited by 7
Prompt engineering for bibliographic web-scraping
Ochando, M. B., Gutiérrez, J. J. P., & Ovalle-Perandones, M.-A. (2025) · Scientometrics · doi:10.1007/s11192-025-05372-5 · Cited by 7
NexusSum: Hierarchical LLM Agents for Long-Form Narrative Summarization
Kim, H., & Kim, B. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.500 · Cited by 13
How is language intelligence evolving? A multi-dimensional survey of large language models
Chen, X., Zhou, L., Chen, J., Wang, G., & Li, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130637 · Cited by 10
NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents
Czinczoll, T., Hönes, C. J., Schall, M., & de Melo, G. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.256 · Cited by 7
Evaluating Multilingual Long-Context Models for Retrieval and Reasoning
Agrawal, A., Dang, A., Nezhad, S. B., Pokharel, R., & Scheinberg, R. (2024) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2024.mrl-1.18 · Cited by 11
FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation
Tanzer, G. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.314 · Cited by 7
LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs
Chen, J., Wu, J., Xu, Y., & Zhang, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.154 · Cited by 11
Divide-Then-Aggregate: An Efficient Tool Learning Method via Parallel Tool Invocation
Zhu, D., Shi, W., Shi, Z., Ren, Z., Wang, S., Yan, L., & Yin, D. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1401 · Cited by 19
CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
Qiu, Z., Li, J., Huang, S., Jiao, X., Zhong, W., & King, I. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.230 · Cited by 20
Metadata
- Authors
- Chen, S., Wong, S., Chen, L., & Tian, Y.
- Affiliations
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2306.15595
- OpenAlex
- W4382603041
- Cited by
- 50
- References
- 0
- Citations by year
- 2026: 3 · 2025: 19 · 2024: 22 · 2023: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- context window extension, rope, large language models, self-attention, long-context modeling, long document summarization, llamas
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
