LooGLE: Can Long-Context Language Models Understand Long Contexts?
Open accessAbstract
Large language models (LLMs) are typically limited to processing texts within contextwindow size, which has spurred significant research efforts into enhancing LLMs' longcontext understanding as well as developing high-quality benchmarks to evaluate the ability.However, prior datasets suffer from shortcomings like short length compared to the context window of modern LLMs; outdated documents that might have data leakage problems; and an emphasis on short dependency tasks only.In this paper, we present ooGLE , a Long Context Generic Language Evaluation benchmark.It features documents post-2022, with over 24,000 tokens per document and 6,000 newly generated questions spanning varying dependency ranges in diverse domains.Human annotators meticulously crafted over 1,100 high-quality question-answer (QA) pairs with thorough cross-validation for a most precise assessment of LLMs' long dependency capabilities.We conduct a comprehensive evaluation of representative LLMs on ooGLE .The results indicate that most LLMs have shockingly bad long context ability and fail to capture long dependencies in the context, even when their context window size is enough to fit the entire document.Our results shed light on enhancing the "true long-context understanding" ability of LLMs instead of merely enlarging their context window.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs exhibit significant limitations in long context scenarios, especially when relevant information appears in the middle of a prompt or when code completion requires cross-function or structural dependencies.
“Despite increasingly longer context windows, LLMs exhibit significant limitations in long context scenarios, especially when relevant information appears in the middle of a prompt [19], [20] or when code completion requires crossfunction or structural dependencies [21], [61].”
From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [20]
The Discussion and §4.2 report substantial limitations in LLMs’ understanding of long texts despite extended context windows. The cited passages do not establish the more specific claims about information appearing in the middle of a prompt or code completion requiring cross-function or structural dependencies. (LLM, conf=0.86)
Experimental — accuracy not yet measuredModern LLMs suffer from "contextual forgetting": as the text length increases, model performance worsens and relevant information gets lost.
“As is well known, modern LLMs suffer from “contextual forgetting”: as the text length increases, model performance worsens and relevant information gets lost [32,33,34].”
From Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study (2025) · cites this paper as [33]
LLMs struggle to identify and utilize relevant content amid lengthy inputs.
“Second, LLMs struggle to identify and utilize relevant content amid lengthy inputs [19], [20].”
From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 14
- Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320-335.
- Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2.
- Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213.
- Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lian- min Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023a. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Fol- lowing.
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023a. Lost in the middle: How language models use long contexts. Transactions of the Asso- ciation for Computational Linguistics, 12:157-173.
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730-27744.
- Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. Advances in neural information processing systems, 33:16857- 16867.
- Gaurav Suri, Lily R Slater, Ali Ziaee, and Morgan Nguyen. 2024. Do large language models show deci- sion heuristics similar to humans? a case study using gpt-3.5. Journal of Experimental Psychology: Gen- eral.
- Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Met- zler. 2022. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-28.
- Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. musique: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554.
- Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. 2024. Focused transformer: Contrastive train- ing for context scaling. Advances in Neural Informa- tion Processing Systems, 36.
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in neural information processing systems, 35:24824-24837.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judg- ing llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.
Cited by
Sorted by citations
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 159 · see citing works
Clinical entity augmented retrieval for clinical information extraction
López, I., Swaminathan, A., Vedula, K. S., Narayanan, S., Haredasht, F. N., Ma, S. P., Liang, A. S., Tate, S., Maddali, M. V., Gallo, R. J., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01377-1 · Cited by 82 · see citing works
Recursively summarizing enables long-term dialogue memory in large language models
Wang, Q., Fu, Y., Cao, Y., Wang, S., Tian, Z., & Ding, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130193 · Cited by 55 · see citing works
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.183 · Cited by 29 · see citing works
Self-citationThe evolution of language models: From N-Grams to LLMs, and beyond
Raeini, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100168 · Cited by 22 · see citing works
Statically Contextualizing Large Language Models with Typed Holes
Blinn, A., Li, X., Kim, J. H., & Omar, C. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689728 · Cited by 10 · see citing works
VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridges
Wang, Y., Song, Y., Xie, C., Liu, Y., & Zheng, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02240 · Cited by 8 · see citing works
Self-citationEnhancing Protocol Fuzzing via Diverse Seed Corpus Generation
Luo, Z., Du, Q., Wang, Y., Roychoudhury, A., & Jiang, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3595396 · Cited by 8 · see citing works
Vasilev, Y., Raznitsyna, I., Pamova, A., Burtsev, T., Bobrovskaya, T., Kosov, P., Vladzymyrskyy, A., Omelyanskaya, O., & Arzamasov, K. (2025) · Diagnostics · doi:10.3390/diagnostics16010003 · Cited by 5 · see citing works
Structured Memory Mechanisms for Stable Context Representation in Large Language Models
Xing, Y., Yang, T., Qi, Y., Wei, M., Cheng, Y., & Xin, H. (2025) · International Conference on Artificial Intelligence and Digital Ethics (ICAIDE) · doi:10.1109/icaide65466.2025.11189651 · Cited by 5 · see citing works
LongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00020 · Cited by 7 · see citing works
Instruct Large Language Models to Generate Scientific Literature Survey Step by Step
Lai, Y., Wu, Y., Wang, Y., Hu, W., & Zheng, C. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9443-0_43 · Cited by 5 · see citing works
Preacher: Paper-to-Video Agentic System
Liu, J.-W., Yang, L., Luo, H., Wang, F., Li, H.-Y., & Wang, M.-D. (2025) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.01591 · Cited by 9 · see citing works
Self-citationBlockPIM: Optimizing Memory Management for PIM-enabled Long-Context LLM Inference
Li, Z., Zhou, J., Li, X., & Sun, N. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11133193 · Cited by 3 · see citing works
CROP: Circuit Retrieval and Optimization with Parameter Guidance using LLMs
Pan, J., Jacobson, I., Zhao, Z., Chen, T.-C., Zhou, G.-S., Chang, C.-C., Rashingkar, V., & Chen, Y. (2025) · doi:10.1109/iccad66269.2025.11240643 · Cited by 3 · see citing works
Combining LLMs and Simheuristics: An Application to the Project Portfolio Selection Problem
Sáiz, M., & Calvet, L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78238-1_1 · Cited by 3 · see citing works
Li, Z., Xia, T., Chang, Y., & Wu, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130711 · Cited by 3 · see citing works
Discovering Inconsistencies in Documents with Long-Context LLMs
Martin, A., Witschel, H. F., Stockhecke, M., Nydegger, P., & Buga, K. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-94931-9_9 · Cited by 2 · see citing works
Adaptive Request Scheduling and Load Balancing for Edge Deployed Large Language Models
Mou, F., Tang, Z., Jia, W., & Zhao, W. (2026) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2026.3665250 · Cited by 2 · see citing works
HALO: A heterogeneous accelerator for low-latency and energy-efficient edge LLM inference
Zhang, K., Fan, Z., Liu, Y., Wang, L., Liu, Y., Wu, H., Ye, X., & Li, W. (2026) · Future Generation Computer Systems · doi:10.1016/j.future.2026.108646 · Cited by 2 · see citing works
A Distributed Learned Hash Table
Wang, S., Liu, Y., Zhang, X., Hu, L., & Qian, C. (2025) · IEEE International Conference on Network Protocols (ICNP) · doi:10.1109/icnp65844.2025.11192384 · Cited by 2 · see citing works
Chang, Y., Li, Z., Zhang, H., Kong, Y., Wu, Y., So, H. K.-H., Guo, Z., Zhu, L., & Wong, N. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.790 · Cited by 2 · see citing works
Leveraging electronic health records for atrial fibrillation cohort generation
Domingo-Aldama, A. G., Prado, M. M., Olea, A. G., Goikoetxea, J., Gojenola, K., & Atutxa, A. (2026) · Health Information Science and Systems · doi:10.1007/s13755-025-00415-w · Cited by 1 · see citing works
Shen, M.-T., & Joung, Y.-J. (2026) · International Workshop on Agentic Engineering · doi:10.1145/3786167.3788424 · Cited by 1 · see citing works
Kohita, R., & Kasuga, A. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.09.247 · Cited by 2 · see citing works
Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective
Wu, F., Silwal, S., Qiuyi, & Zhang (2026) · arXiv · doi:10.48550/arxiv.2601.18999
ElMallah, R., Zamani, N., & Lee, C.-G. (2024) · International Conference on Mechatronics - Mechatronika · doi:10.1109/me61309.2024.10789747
Pirenne, L., Mokeddem, S., Ernst, D., & Louppe, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_1
Wang, H., Ding, Z., & Huang, H. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0020-8_18
DataSculpt: A Holistic Data Management Framework for Long-Context LLMs Training
Lu, K., Nie, X., Liang, Z., Pan, D., Zhang, S., Zhao, K., Chen, W., Zhou, Z., Dong, G., Cui, B., et al. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00316
Zhao, J., Wan, B., Peng, Y., Lin, H., & Wu, C. (2025) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster59342.2025.11186491
SmartCache: Two-Dimensional KV-Cache Similarity for Efficient Long-Context LLM Decoding
Shen, C., Chen, H., Hui, K., Zhao, Z., Yang, C., Hu, Y., Chen, S., Liu, X., & Li, K. (2025) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc67675.2025.00065
FinLBench: A Benchmark for Evaluating Large Language Models on Long-Text Financial Documents
Hu, S., Yu, D., Zhang, Y., Yang, Z., Zhou, L., Jiang, X., Li, Y., Fei, B., & Liu, J. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4091-4_17
Long Math Reasoning Problem Generation
Li, C., Huang, G., Zhou, Z., & Wang, Q. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-09368-4_23
RiCoRecA: rich cooking recipe annotation schema
Ventirozos, F., Jacobo-Romero, M., Alrdahi, H., Clinch, S., & Batista-Navarro, R. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1550604
A Graph Enhanced Memory System for Long-Turn Dialogue Question Answering
Tang, Q., Gu, M., Xie, M., Tao, Z., Liu, Y., & Zhang, Z. (2025) · International Conference on Machine Learning and Computer Application (ICMLCA) · doi:10.1109/icmlca66850.2025.11336286
Multiple Schema-Conformant Declarative Code Generation
Kammakomati, M., & Tamilselvam, S. G. (2025) · International Conference on Automated Software Engineering · doi:10.1109/ase63991.2025.00344
Lan, K., Li, G. Y., Huang, W., Li, J., & Yu, Y. (2026) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2026.3687554 · Cited by 3 · see citing works
Self-citationvan de Beek, V. D., & Yoshimura, T. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807653
Unveiling LLMs for Advancing Long-Context Environmental Data Analysis
Samineni, B., Athota, V. S., Kona, T., & Doddipatla, M. (2026) · International Conference on Emerging Technologies and Future Innovations (ETFI) · doi:10.1109/etfi68128.2026.11484901
Beyond True Label: Label-Assumed Evidence Extraction for Personality Prediction
Ji, Y., Huang, Z., Liu, X., Shi, Y., & Wu, W. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0372-7_4
Hierarchical Attribution Analysis of Deficiencies in the Reasoning Ability of Large Language Models
Jiang, Z., Zhang, H., Wang, F., Yan, X., & Yang, Y. (2026) · International Conference on Computer, Artificial Intelligence and Control Engineering (CAICE) · doi:10.1145/3804601.3804629
Shi, F., Xu, W., Shi, J., Ji, X., & Wang, J. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-9507-5_12
EntiTractor: entity extraction for coherent hallucination correction in paragraph-level text
Yao, J., Yao, J., Zang, W., Zhao, K., Zhang, H. M., & Wang, J. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133359
Near-Zero Cost KV Cache Compression for Large Language Model Inference
Zhang, B., Zhou, D., Huang, Y., Song, S., Feng, H., Jia, J., Zhang, C., & Zhang, Z. (2026) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps65963.2026.00058
HydraPIM: A Heterogeneous PIM Architecture for Efficient Attention in Long-Context LLMs
Zhao, S., Liu, L., An, X., Zhou, Y., He, Y., Cheng, L., Wang, M., Li, X., Han, Y., & Wang, Y. (2026) · IEEE Transactions on Computers · doi:10.1109/tc.2026.3710733
Self-citationZhou, Z., He, Y., Guo, Y., Yu, H., Fang, L., & Li, F. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809572
Data Foundations of Long-Context Language Models: A Survey
Sun, Z., Sun, Y., Su, Z., Tang, Z., Li, J., Zhou, A., Chen, W., & Zhang, M. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.775
Self-citationTowards Efficient Serving of Network-intensive LLM Inferences
Wang, W., Chen, C., Zhang, J., Wang, Z., Yuan, H., Guan, Z., Zheng, X., Weng, Q., Chen, Y., & Guo, M. (2026) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3820441.3820479
Xu, S.-W. (2026) · World wide web (Bussum) · doi:10.1007/s11280-026-01427-6
Tekale, A., & Masum, M. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00037
FairCache : Demystifying Cache-Induced Unfairness in Multi-Tenant Large Language Model Serving
Bai, Z., Gao, B., Xu, F., Zhao, H., Zhou, Z., & Wong, W. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3729988
Bohemian Graphsody: Building Living Graphs for Storytelling with Shrecknet
Barros, P. V. A., Gravino, P., Pincione, E., & Sciutti, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38407-2_14
PolarKV: Tier Locally, Serve Globally-A KV Cache over Cloud Memory and Storage
Zhang, Y., Yang, X., Chen, H., Li, F., Wang, S., Sun, C., Wu, S., & Weng, N. (2026) · Proceedings of the VLDB Endowment · doi:10.14778/3827998.3828047
Yang, J., Liu, G., Wang, Y., Jian, S., Zhang, J., Tan, S., & Li, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38410-2_16
Wiranata, J. A., & Girsang, A. S. (2026) · International Conference on Smart Science and Technology (IncoSST) · doi:10.1109/incosst69426.2026.11709604
M-DocSum: Do LVLMs Genuinely Comprehend Interleaved Image-Text in Document Summarization?
Yan, H., Tan, K., Shen, Y., Huang, X., Wang, J., Ge, Z., Zhang, X., Li, S., & Jiang, D. (2026) · doi:10.1109/cvprf71909.2026.00266
- Authors
- Li, J., Wang, M., Zheng, Z., & Zhang, M.
- Affiliations
- Beijing Institute for General Artificial Intelligence (CN)
- Peking University (CN)
- Venue
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 16304-16333
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4402671258
- Cited by
- 55
- References
- 0
- Citations by year
- 2026: 26 · 2025: 26 · 2024: 3
- FWCI
- 16.21
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62376031)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- long-context understanding, large language models, context window size, benchmark dataset, question-answer pairs, LLM evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
