Extending Context Window of Large Language Models via Semantic Compression
Open accessAbstract
Transformer based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses due to the quadratic complexity.These constraints restrict their applicability in long text scenarios.In this paper, we propose a novel semantic compression method that enables generalization to texts that are 6-8 times longer without incurring significant computational costs or requiring finetuning.Our proposed framework draws inspiration from source coding in information theory and employs a pre-trained model to reduce the semantic redundancy of long inputs before passing them to the LLMs for downstream tasks.Experimental results demonstrate that our method effectively extends the context window of LLMs across a range of tasks including question answering, summarization, few-shot learning, and information retrieval.Furthermore, the proposed semantic compression method exhibits consistent fluency in text generation while reducing the associated computational overhead.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Global semantic modeling reduces reasoning errors by 15% in processing lengthy Wikipedia articles (e.g., exceeding 64k tokens).
“in processing lengthy Wikipedia articles (e.g., exceeding 64k tokens), global semantic modeling reduces reasoning errors by 15% [7]”
From U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack (2026) · cites this paper as [7]
The Results section reports semantic compression extending the context window to up to 60,000 with YaRN, not processing articles exceeding 64k tokens. The Future Work section says evaluating long-text reasoning remains to be explored; it does not report a 15% reduction in reasoning errors on Wikipedia articles. Page numbers were not available in our copy. (LLM, conf=0.84)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Extending Context Window of Large Language Models via Positional Interpolation
Chen, S., Wong, S., Chen, L., & Tian, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.15595 · Cited by 50 · see citing works
Flagged first, then by citations
Research works printed but not linked: 33
- Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Am- brose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. 2022. Exploring length generalization in large language models. Advances in Neural Informa- tion Processing Systems, 35:38546-38556.
- Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508. · arXiv
- Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150. · arXiv
- Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. 2023a. Clex: Continuous length extrapolation for large language models. arXiv preprint arXiv:2310.16450. · DOI
- Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023c. Lon- glora: Efficient fine-tuning of long-context large lan- guage models. arXiv.
- Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. Adapting language models to compress contexts. ArXiv, abs/2305.14788. · arXiv
- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Asso- ciation for Computational Linguistics. · DOI · Paper page
- Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Sys- tems.
- Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers an- chored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4599-4610.
- Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. 2023. Longnet: Scaling transformers to 1,000,000,000 to- kens. In Proceedings of the 10th International Con- ference on Learning Representations.
- Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large- scale multi-document summarization dataset and ab- stractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 1074-1084.
- Weizhi Fei, Zihao Wang, Hang Yin, Yang Duan, Hang- hang Tong, and Yangqiu Song. 2024. Soft Reasoning on Uncertain Knowledge Graphs. ArXiv:2403.01508 · DOI
- Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 6609-6625, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics. · DOI · Paper page
- Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436, Online. Association for Computational Linguistics. · DOI · Paper page
- Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358-13376. Association for Computational Linguistics. · DOI · Paper page
- Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Ed- ward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Asso- ciation for Computational Linguistics, 6:317-328.
- Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
- Amirkeivan Mohtashami and Martin Jaggi. 2023. Landmark attention: Random-access infinite con- text length for transformers. arXiv preprint arXiv:2305.16300. · arXiv
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
- Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan. 2023. Evaluation metrics in the era of gpt-4: reli- ably evaluating large language models on sequence to sequence tasks. arXiv preprint arXiv:2310.13800. · arXiv
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971. · arXiv
- Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554. · DOI · Paper page
- Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. 2024. Rethinking the bounds of llm reasoning: Are multi-agent discussions the key? arXiv preprint arXiv:2402.18272. · arXiv
- Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, and Li Guo. 2023a. Re- cursively summarizing enables long-term dialogue memory in large language models. arXiv preprint arXiv:2308.15022. · arXiv
- Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural In- formation Processing Systems, 33:5776-5788.
- Zihao Wang, Weizhi Fei, Hang Yin, Yangqiu Song, Ginny Y Wong, and Simon See. 2023b. Wasserstein- Fisher-Rao Embedding: Logical Query Embeddings with Local Comparison and Global Transport. arXiv preprint arXiv:2305.04034. · arXiv
- Zihao Wang, Hang Yin, and Yangqiu Song. 2021. Benchmarking the Combinatorial Generalizability of Complex Query Answering on Knowledge Graphs. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1.
- David Wingate, Mohammad Shoeybi, and Taylor Sorensen. 2022. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5621-5634, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics. · DOI · Paper page
- Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answer- ing. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
- Hang Yin, Zihao Wang, Weizhi Fei, and Yangqiu Song. 2023. EFO k -CQA: Towards knowledge graph com- plex query answering beyond set operation. arXiv preprint arXiv:2307.13701. · arXiv
- Hang Yin, Zihao Wang, and Yangqiu Song. 2024a. Meta Operator for Complex Query Answering on Knowl- edge Graphs. ArXiv:2403.10110 [cs. · arXiv
- Hang Yin, Zihao Wang, and Yangqiu Song. 2024b. Re- thinking complex queries on knowledge graphs with neural link predictors. In The Twelfth International Conference on Learning Representations.
- Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyil- maz, Yang Liu, Xipeng Qiu, et al. 2021. Qmsum: A new benchmark for query-based multi-domain meet- ing summarization. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 5905-5921.
Cited by
Sorted by citations
U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack
Gao, Y., Xiong, Y., Wu, W., Huang, Z., Li, B., & Wang, H. (2025) · arXiv · doi:10.48550/arxiv.2503.00353 · Cited by 6 · see citing works
U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack
Gao, Y., Xiong, Y., Wu, W., Li, B., Zhong, Y., & Wang, H. (2026) · ACM Transactions on Information Systems · doi:10.1145/3786609 · Cited by 6 · see citing works
Chain-of-Memory: Lightweight Memory Construction with Dynamic Evolution for LLM Agents
Xu, X., Xu, B., Xueyun, T., Huang, Z., Chen, R., Yunfan, L., & Shen, H. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.534
- Authors
- Fei, W., Niu, X., Zhou, P., Hou, L., Bai, B., Deng, L., & Han, W.
- Affiliations
- Huawei Technologies (China) (CN)
- Venue
- Findings of the Association for Computational Linguistics ACL 2024
- Published
- 2024
- Pages
- 5169-5181
- OpenAlex
- W4402670549
- Cited by
- 20
- References
- 1
- Citations by year
- 2026: 8 · 2025: 10 · 2024: 2
- FWCI
- 5.77
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- semantic compression, context window extension, large language models, source coding, quadratic complexity, question answering, text summarization, few-shot learning, information retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
