The DOI matches its DataCite record; 1 of the 6 references it cites were found in a registry or an index, and 5 are not checked yet.
Extending Context Window of Large Language Models via Positional Interpolation
Open accessAbstract
We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least $\sim 600 \times$ smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Recent works extend context windows to 100K+ tokens [18], [19], [23].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“unlimited context remains computationally infeasible and doesn't address selective retention or consistency.”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, many model developments do not incorporate instruction tuning (Chen et al., 2023; Fu et al., 2024), leaving these models reliant on perplexity-based evaluation or synthetic tasks.”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the fundamental tension persists: more capacity does not eliminate the need for selective curation.”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Extending context windows to 100k+ tokens [Chen et al., 2023] delays the problem but cannot eliminate it, and longer contexts incur quadratic cost increases in attention.”
From Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers (2026) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite context windows stretching to 200k tokens [Chen et al., 2023], long-context models consistently underperform purpose-built memory systems on tasks requiring selective retrieval and active management.”
From Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers (2026) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Some methods, such as PI [103] and SelfExtend [104], reorganize positional indices to ensure they remain within context lengths.”
From A Survey of Large Language Models (2026) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“they address extrapolation of the positional code, a different concern from the entropy growth we analyse”
From Curvature-Corrected Rotary Position Embeddings: An Entropy-Invariant Temperature for Mitigating Numerical-Rank Collapse in Long-Context Attention (2026) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“Position Interpolation [4] and YaRN [5] modify the RoPE frequencies to keep evaluation positions within the trained distribution”
From Curvature-Corrected Rotary Position Embeddings: An Entropy-Invariant Temperature for Mitigating Numerical-Rank Collapse in Long-Context Attention (2026) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“Position interpolation techniques enable models to process sequences beyond original context window limitations by intelligently rescaling position indices rather than extrapolating to unseen positions [153].”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [153]
The cited source says what the claim attributes to it.
In the citing paper
“Despite this growing importance, LLMs consistently face challenges in processing long contexts, primarily due to the substantial computational resources required for long sequence training (Dao et al., 2022; Dao, 2023) as well as the apparent inability to generalize to sequences longer than those encountered during training (Chen et al., 2023; Peng et al., 2023b).”
From ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024) · cites this paper as Chen et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Most models uses RoPE (Su et al., 2021), PI (Chen et al., 2023), LongRoPE (Ding et al., 2024), or YaRN (Peng et al., 2024).”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Chen et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Recent large language models (LLMs) support long contexts ranging from 128K to 1M tokens.”
From NoLiMa: Long-Context Evaluation Beyond Literal Matching (2025) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 19
- Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with per- formers. In 9th International Conference on Learning Representations, ICLR 2021. OpenRe- view.net, May 2021.
- Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Process- ing Systems, 2022.
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openreview.net/forum?id=YicbFdNTTy.
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
- Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. 2022.
- Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. · arXiv
- Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guided supervision for openqa with colbert. Transactions of the Association for Computational Linguistics, 9:929-944, 2021. doi: 10.1162/tacl a 00405. · DOI
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, April 2020.
- Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https://aclanthology.org/W04-1013.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https://openreview.net/forum?id= Bkg6RiCqY7.
- Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300, 2023. · arXiv
- Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=R8sQPpGCv0.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lilli- crap. Compressive transformers for long-range sequence modelling. In International Confer- ence on Learning Representations, 2020. URL https://openreview.net/forum?id= SylKikSYDH.
- Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, and Bo Dai. Combiner: Full attention transformer with sparse computation cost. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, pp. 22470-22482.
- Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 12007-12021, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022. emnlp-main.823.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30.
- Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/ file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
- Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing trans- formers. In The Tenth International Conference on Learning Representations, ICLR 2022. Open- Review.net, April 2022.
- Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020.
Cited by
Sorted by citations
Items 101–200 of 271
DeepSeek Models: A Comprehensive Survey of Methods and ApplicationsPreprint
Puspitasari, F. D., Zhang, C., Dam, S. K., Zhang, M., Kim, T.-H., Hong, C. S., Bae, S.-H., Qin, C., Wei, J., Wang, G., et al. (2025) · doi:10.36227/techrxiv.174198511.15158242/v1 · Cited by 8
L-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2307.11088 · Cited by 6
“Moreover, position interpolation (PI) (Chen et al., 2023) and NTK-aware (LocalLLaMA, 2023b;a) are the most popular approaches based on RoPE (Su et al., 2022) to efficiently and effectively extend the context length.”
MethodLongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data
Yang, C., Lin, X., Xu, C., Jiang, X., Ma, S., Liu, A. Q., Xiong, H., & Guo, J. G. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.169 · Cited by 19
Long context window-based zero-shot legal interpretation of building codes and regulations
Lee, J., & Lee, G. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106450 · Cited by 6
Fundamental Limitations of Generative LLMs
Kucharavy, A. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_5 · Cited by 7
Efficient Transformer Adaptation with Soft Token Merging
Yuan, X., Fei, H., & Baek, J. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00369 · Cited by 8
Anchor Attention, Small Cache: Code Generation With Large Language Models
Zhang, X., Zhou, Y., Yang, G., Gall, H. C., & Chen, T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3570680 · Cited by 6
Long Context vs. RAG: Strategies for Processing Long Documents in LLMs
Li, X., Bai, Y., Jin, B., Zhu, F., Pan, L., & Cao, Y. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3731690 · Cited by 9
Abdari, A., Falcon, A., & Serra, G. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00464 · Cited by 7
Teng, Z., Song, Y. I., Ye, X., & Ouyang, Y. (2024) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3651671.3651702 · Cited by 6
Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs
Cao, Z., Cao, Q., Lu, Y. F., Peng, N., Huang, L., Cheng, S., & Su, J. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.685 · Cited by 15
Firm-Level Tax Audits: A Generative AI-Based MeasurementPreprint
Choi, G., & Kim, A. G. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4645865 · Cited by 6
U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack
Gao, Y., Xiong, Y., Wu, W., Li, B., Zhong, Y., & Wang, H. (2026) · ACM Transactions on Information Systems · doi:10.1145/3786609 · Cited by 6
Token Weighting for Long-Range Language Modeling
Helm, F., Daheim, N., & Gurevych, I. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.79 · Cited by 14
Moreno-Cediel, A., García‐López, E., García‐Cabot, A., & de-Fitero-Dominguez, D. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114896 · Cited by 5
Investigating the Effects of Sparse Attention on Cross-Encoders
Schlatt, F., Fröbe, M., & Hagen, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_11 · Cited by 5
A Concise Review of Long Context in Large Language Models
Huang, H., Liang, Z., Fang, Z., Wang, Z., Chen, M., Hong, Y., Liu, K., & Shang, P. (2024) · International Conference on Algorithms, Software Engineering and Network Security (ASENS) · doi:10.1145/3677182.3677282 · Cited by 6
Fashion-VDM: Video Diffusion Model for Virtual Try-On
Karras, J., Li, Y., Liu, N., Zhu, L., Yoo, I., Lugmayr, A., Lee, C., & Kemelmacher-Shlizerman, I. (2024) · SIGGRAPH Asia Conference Papers · doi:10.1145/3680528.3687623 · Cited by 9
Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Tasks
Dehghankar, M., & Asudeh, A. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737099 · Cited by 7
LongAttn: Selecting Long-context Training Data via Token-level Attention
Wu, L., Zhu, D., Zhao, G., Yu, Z., Ran, J., Wong, X., Sun, L., & Li, S. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.991 · Cited by 12
FocusLLM: Precise Understanding of Long Context by Dynamic Condensing
Li, Z., Zhang, Y., Pan, T., Sun, Y., Duan, Z., Fang, J., Han, R., Wang, Z., & Wang, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1500 · Cited by 16
End-to-End Long Document Summarization using Gradient Caching
Saxena, R., Tang, H., & Keller, F. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.40 · Cited by 7
Yang, D., Liu, T., Zhang, D., Simoulin, A., Liu, X., Cao, Y., Teng, Z., Qian, X., Yang, G., Luo, J., et al. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.130 · Cited by 9
2-D Transformer: Extending Large Language Models to Long-Context With Few Memory
He, X., Liu, J., & Duan, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3548047 · Cited by 5
Fundamental Approaches to Software Engineering
Beyer, D., & Cavalcanti, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57259-3 · Cited by 7
Liu, F., Farkiani, B., & Crowley, P. (2024) · doi:10.36227/techrxiv.173386065.57486944/v1 · Cited by 5
NoLiMa: Long-Context Evaluation Beyond Literal Matching
Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., & Schütze, H. (2025) · arXiv · doi:10.48550/arxiv.2502.05167 · Cited by 3
“In recent years, large language models (LLMs) have made remarkable advancements in handling long-context inputs (Chen et al., 2023; Xiong et al., 2024; Peng et al., 2024).”
Can’t Remember Details in Long Documents? You Need Some R&R
Agrawal, D., Gao, S., & Gajek, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.742 · Cited by 8
Tian, J., Zheng, D., Chen, Y., Wang, R., Zhang, C., & Zhang, D. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.62 · Cited by 11
Reinforcement Learning Enhanced Muti-hop Reasoning for Temporal Knowledge Question Answering
Wen, W., Xue, C., Pan, S., Sun, Y., & Peng, M. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i40.40680 · Cited by 4
On the Power of Convolution-Augmented Transformer
Li, M., Zhang, X., Huang, Y., & Oymak, S. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i17.34024 · Cited by 9
Zhang, Z., Hu, Y., Pan, B., Ling, C., & Zhao, L. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761357 · Cited by 7
Expanding Shareholder Voice: The Impact of SEC Guidance on Environmental and Social ProposalsPreprint
Khoo, K., & Tallarita, R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4913660 · Cited by 4
Dong, Z., Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2309.13345 · Cited by 7
“Furthermore, a series of studies attempted to directly extend the context windows of LLMs with RoPE (Su et al., 2021) positional embeddings via Position Interpolation (kaiokendev, 2023; Chen et al., 2023; Peng et al., 2023).”
BackgroundMethodNeurocache: Efficient Vector Retrieval for Long-range Language Modeling
Safaya, A., & Yüret, D. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.50 · Cited by 5
Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs
Zhao, Y., Wu, H., & Xu, B. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34800 · Cited by 5
Select, Read, and Write: A Multi-Agent Framework of Full-Text-based Related Work Generation
Liu, X., Song, R., Wang, X., & Chen, X. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.366 · Cited by 3
Lee, G., Kim, S., Lee, D., Goh, K., & Kim, H. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108900 · Cited by 4
AI-assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations
Hartman, J., Sagtani, H., Tibshirani, J., & Mehrotra, R. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688060 · Cited by 3
Benchmarking Long-Context Language Models on Long Code Understanding
Li, J., Guo, X., Li, L., Zhang, K., Li, G., Li, J., Tao, Z., Liu, F., Tao, C., Zhu, Y. S., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1324 · Cited by 5
MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models
Huang, Z., Ling, G., Zhong, S., Wu, H., & Lin, L. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.560 · Cited by 3
When Small Wins Big: Classification Tasks Where Compact Models Outperform Original GPT-4Preprint
Lefort, B., Benhamou, E., Ohana, J., Guez, B., Saltiel, D., & Challet, D. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4780454 · Cited by 3
Toward conversational agents with context and time sensitive long-term memory
Alonso, N., Figliolia, T., Ndirango, A., & Millidge, B. (2024) · arXiv · doi:10.48550/arxiv.2406.00057 · Cited by 2
Improving Content-Based Data Product Retrieval in Federated Environments with LLM and Sampling
Falconi, M., & Plebani, P. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-94931-9_23 · Cited by 2
SEAL: Scaling to Emphasize Attention for Long-Context Retrieval
Lee, C., Seok, M., Jin, J.-G., Cho, Y., & Park, E. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1405 · Cited by 3
Toward the Use of LLMs to Support Curriculum Mapping to Established Frameworks
Brown, E. L., Talbert, D. A., & Roberts, J. (2025) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--57295 · Cited by 2
A collaborative reasoning framework for large language models in long-context Q&A
Yao, J., He, G., & Xu, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129960 · Cited by 3
Joint Enhancement of Relational Reasoning for Long-Context LLMs
Chen, Z., Shen, W., Huang, J., & Shao, L. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.462 · Cited by 2
Yu, Y., Zhang, Q., Qiao, L., Yin, D., Li, F., Wang, J., Xi, C., Zheng, S., Liang, X., & Sun, X. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1497 · Cited by 2
Prediction, syntax and semantic grounding in the brain and large language models
Kölbl, N., Rampp, S., Kaltenhäuser, M., Tziridis, K., Maier, A., Kinfe, T. M., Chavarriaga, R., Krauß, P., & Schilling, A. (2026) · Scientific Reports · doi:10.1038/s41598-026-41532-0 · Cited by 2
Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning
Ma, Z., Gou, C., Hu, Y., Wang, Y., Zhuang, B., & Cai, J. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i10.37733 · Cited by 2
Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learning
Feng, A., Ying, R., & Tassiulas, L. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.858 · Cited by 5
Dai, C., Lü, L., & Zhou, P. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.707 · Cited by 2
Visual Context Window Extension: A New Perspective for Long Video Understanding
Wei, H., & Chen, Z. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755383 · Cited by 15
Prompting, Retrieval, and Fine-Tuning: Foundations of Enterprise Language Model Adaptation
Nisar, K. (2024) · International Journal of Engineering & Extended Technologies Research · doi:10.15662/ijeetr.2024.0606030 · Cited by 2
3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding
Ma, X., Liu, W., Zhang, P., & Xu, N. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i23.34662 · Cited by 4
DLFR-VAE: Dynamic Latent Frame Rate VAE for Video Generation
Yuan, Z., Wang, S., Shang, Y., Zhang, H., Fang, T., Xie, R., Yan, S., Dai, G., & Wang, Y. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755638 · Cited by 4
NeedleChain: Measuring Intact Context Comprehension Capability of Large Language Models
Moon, H., & Lim, H. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1637 · Cited by 4
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2
“…with longer input lengths (Dubey et al., 2024; Fu et al., 2024; AI et al., 2024), explore position extrapolation techniques (Peng et al., 2024; Chen et al., 2023; Ding et al., 2024), and experiment with efficient architectures (Gu & Dao, 2023; Dao & Gu, 2024; Bertsch et al., 2023; Beltagy et…”
BackgroundTiMem: Temporal-Hierarchical Memory Consolidation for Long-Horizon Conversational Agents
Li, K., Yu, X., Ni, Z., Zeng, Y., Xu, Y., Zhang, Z., Li, X., Sang, J., Duan, X., Wang, X., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.02845 · Cited by 1
CoWPE: Adaptive Context Window Adjustment in LLMs for Complex Input Queries
Tamanampudi, V. M. (2024) · Journal of Artificial Intelligence General science (JAIGS) · doi:10.60087/jaigs.v5i1.221 · Cited by 1
SpecialtyScribe: Enhancing SOAP note Scribing for Medical Specialties using LLM’s
Goyal, S., Rastogi, E., Zhao, F., Yuan, D., & Beinstein, A. (2025) · Workshop on Patient-Oriented Language Processing (CL4Health) · doi:10.18653/v1/2025.cl4health-1.4 · Cited by 1
An, S., Sung, J., Park, W., Park, C., & Seo, P. H. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.524 · Cited by 2
Que, H., & Rong, W. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.347 · Cited by 2
Analyzing Retrieval Scaling in RAG Systems for Complex QA Benchmarks
Mosquera, J. S. R., & Manrique, R. (2025) · SN Computer Science · doi:10.1007/s42979-025-04302-9 · Cited by 1
From General Reward to Targeted Reward: Improving Open-ended Long-context Generation Models
Guo, Z., Wu, J., Cui, W., Zhang, Y., Hu, M., Wang, Y., & King, I. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.260 · Cited by 1
A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation
Li, Y., Zhang, T., Li, Z., & Han, C. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.443 · Cited by 1
A KG-LLM-Enabled Inspection Work Packaging Approach in Module Prefabrication
Dong, Q., Li, X., Liu, G., Chengke, W., & Chen, T. (2025) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-6595 · Cited by 1
Scientific knowledge graph and ontology generation using open large language models
Bran, A. M., Hart, M., Bran, A. C. M. M., Lederbauer, M., & Schwaller, P. (2026) · Digital Discovery · doi:10.1039/d5dd00275c · Cited by 1
Flora: Effortless Context Construction to Arbitrary Length and Scale
Chen, T., Tan, Z., Bo, X., Wu, Y., Gong, T., Chu, Q., & Ye, J. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i36.40280 · Cited by 1
ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting
Tian, Y., Mo, F., Zhang, W., Qi, Y., & Nie, J. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.66 · Cited by 1
Self-citationEnhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning
Yu, Q., He, Z., Li, S., Xun, Z., Zhang, J., Xu, J., & He, D. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.97 · Cited by 1
Yadav, I., Schindler, S., Peters, D., & Klinger, R. (2026) · Semantic Web · doi:10.1177/22104968261453132 · Cited by 2
Expanding the Context of Large Language Models Via Linear Interpolation of Positional Embeddings
Kilbas, I., Gribanov, D., Mukhin, A., Александрович, П. Р., & Kupriyanov, A. (2024) · International Conference on Information Technology and Nanotechnology (ITNT) · doi:10.1109/itnt60778.2024.10582292 · Cited by 2
CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling
Bai, Y., Zou, X., Huang, H., Chen, S., Rondeau, M.-A., Gao, Y., & Cheung, J. C. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.338 · Cited by 5
Self-citationOptimal RoPE extension via Bayesian Optimization for training-free length generalization
Zhang, X., Hu, S., Zhao, W., Wang, H., Han, X., He, C., Zeng, G., Liu, Z., & Sun, M. (2025) · AI Open · doi:10.1016/j.aiopen.2025.01.002 · Cited by 1
Beyond Sequences: Two-dimensional Representation and Dependency Encoding for Code Generation
Zhang, X., Zhou, Y. L., Yang, G., Cheng, W., & Chen, T. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.308 · Cited by 1
Pop, A., & Spörer, J. (2025) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds66131.2025.00025 · Cited by 1
Temporal AI model predicts drivers of cell state trajectories across human agingPreprint
Ortega, J. G., Nadadur, R. D., Kunitomi, A., Kothen-Hill, S., Wagner, J. U. G., Kurtoglu, S., Kim, B., Reid, M. M., Lu, T., Washizu, K., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.30.715396 · Cited by 3
Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers
Du, P. (2026) · arXiv · Cited by 1
Cao, S., Zou, K., & Wang, L. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1707
“…2023; Tay et al., 2021) are gradually falling behind the advancements of LLMs with long context windows (Ainslie et al., 2023; Liu et al., 2023a; Chen et al., 2023; Peng et al., 2023; Team et al., 2024), due to the insufficient coverage of model context lengths (128K and longer) by the included…”
Pollertlam, N., & Kornsuwannawit, W. (2026) · arXiv · doi:10.48550/arxiv.2603.04814
“Long-context LLMs. Extending the effective context window of transformer-based models has been studied through architectural changes (Guo et al., 2022; Beltagy et al., 2020) and positional encoding techniques (Chen et al., 2023b).”
Shared Selective Persistent Memory for Agentic LLM Systems
Pedada, S., Dhavala, A., & Patil, N. (2026) · arXiv · doi:10.48550/arxiv.2607.09493
“Context window extension (Press et al., 2022; Chen et al., 2023) increases the token budget without addressing what to persist.”
Mtetwa, J. T., Ogudo, K. A., & Pudaruth, S. (2026) · Mathematics · doi:10.3390/math14142637
Kummara, H. (2026) · Cambridge Open Engage (preprint) · doi:10.33774/coe-2026-xpkvb
Mixing Financial Stress with GPT-4 News Sentiment Analysis for Optimal Risk-On/Risk-Off DecisionsPreprint
Lefort, B., Benhamou, E., Ohana, J., Saltiel, D., Guez, B., & Jacquot, T. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4781752
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
Chi, T.-C., Fan, T.-H., & Rudnicky, A. I. (2024) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2024.findings-naacl.10 · Cited by 1
Context Length Extension via Generalized Extrapolation Scale
Li, L., & Huaping, Z. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.249
DAA:Dynamic attention allocation improves large-scale model reasoningPreprint
Yang, Y., Yang, T., Gao, S., ning, J., & Kong, L. (2024) · Research Square · doi:10.21203/rs.3.rs-5025111/v1
MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks
Bueno, M. C., Lotufo, R., & Nogueira, R. F. (2024) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2024.genbench-1.6
A KG-LLM-Based Product2inspection Mapping Approach for Module Prefabrication
Dong, Q., Li, X., Chen, T., & Chengke, W. (2025) · International Conference on Construction and Real Estate Management (ICCREM) · doi:10.1061/9780784485910.066
How Do Position Encodings Affect Length Generalization? Case Studies On In-Context Function Learning
Lin, D.-N., Jui-Feng, Y., Wu, K.-D., Xu, H., Huang, C.-H., & Kao, H. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i23.34637
Research on Technical Route of AI Large Models for Electric Power Knowledge Services
Peng, S., Xiao, J., Zhang, W., Zuo, Y., Liu, S., Long, Y., Huang, Z., & Wu, S. (2024) · IEEE Conference on Energy Internet and Energy System Integration (EI2) · doi:10.1109/ei264398.2024.10990616
LongLeader: A Comprehensive Leaderboard for Large Language Models in Long-context Scenarios
Chen, P., Jin, H., Lee, C.-C., Shao, R., Yang, J., Zhao, M., Zhang, Z., Lu, Q., Men, K., Xie, N., et al. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.439 · Cited by 1
A Resolution Extrapolation Method for Transformers Oriented Towards Scientific Computing
Wang, Z., Shi, Z., Ye, Q., Wang, Y., Yu, Z. H., Zhou, F., Zhou, H., Sun, Q., & Tai, Z. (2024) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait63902.2024.11022276
Examine the Opportunities and Challenges of Large Language Model (LLM) For Indic Languages
Panchal, B. Y. (2024) · Journal of Information Systems Engineering & Management · doi:10.52783/jisem.v9i4s.11887
Lee, J., & Lee, G. (2025) · International Symposium on Automation and Robotics in Construction (ISARC) · doi:10.22260/isarc2025/0143
100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
Yang, V., Jin, H., Zhong, S., Jiang, S., Wang, Q., Chaudhary, V., & Han, X. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.903 · Cited by 1
Metadata
- Authors
- Chen, S., Wong, S., Chen, L., & Tian, Y.
- Affiliations
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2306.15595
- OpenAlex
- W4382603041
- Cited by
- 50
- References
- 0
- Citations by year
- 2026: 3 · 2025: 19 · 2024: 22 · 2023: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- context window extension, rope, large language models, self-attention, long-context modeling, long document summarization, llamas
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
