Lost in the Middle: How Language Models Use Long Contexts
Open accessThe DOI matches its Crossref record; 48 of the 59 references it cites were found in a registry or an index; 15 of the 23 sources its claims cite were checked for support.
Abstract
While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.
Overview
CoChat Summary
TL;DR
Language models’ performance on tasks requiring access to long input contexts is strongly affected by where relevant information appears, with a U-shaped curve: best at the beginning or end and worst in the middle. Extended-context models are not necessarily better at using context, and reader performance saturates before retriever recall in open-domain QA.
Key findings
- In multi-document QA, performance is highest when the answer-containing document is at the start or end of the context and degrades in the middle; GPT-3.5-Turbo can drop by more than 20%, and in 20- and 30-document settings middle-position performance is lower than its closed-book performance of 56.1% (oracle: 88.3%).
- Extended-context models are not necessarily better at using input context: when the input fits both windows, GPT-3.5-Turbo and GPT-3.5-Turbo(16K) perform nearly identically as a function of relevant-information position.
- In synthetic key-value retrieval, Claude-1.3 and Claude-1.3(100K) do nearly perfectly, while other models struggle especially with 140 or 300 key-value pairs; query-aware contextualization gives GPT-3.5-Turbo(16K) perfect performance at 300 pairs, versus worst-case 45.6% without it.
- In open-domain QA, reader performance saturates long before retriever recall: using 50 instead of 20 retrieved documents only marginally improves performance (∼1.5% for GPT-3.5-Turbo and ∼1% for Claude-1.3).
Methods
- Controlled multi-document QA on NaturalQuestions-Open: 2,655 queries whose annotated long answer is a paragraph; each context has exactly one answer-containing Wikipedia passage plus k−1 Contriever/MS-MARCO distractor passages; the paper varies answer position and context length (10, 20, 30 documents) and measures accuracy.
- Synthetic key-value retrieval: inputs are serialized JSON objects with unique random UUID key-value pairs; models must return the value for a specified key; experiments use 75, 140, and 300 key-value pairs (500 examples each), varying key position and number of pairs, with accuracy as the metric.
- Models include open MPT-30B-Instruct and LongChat-13B(16K), closed GPT-3.5-Turbo, GPT-3.5-Turbo(16K), Claude-1.3, and Claude-1.3(100K), plus additional architecture/instruction experiments with Flan-T5-XXL, Flan-UL2, MPT-30B, Llama-2, and GPT-4; an open-domain QA case study uses a Contriever retriever-reader setup.
Limitations
- Stated: the paper uses greedy decoding and leaves exploration of other decoding methods to future work; the investigations into why models are not robust are described as preliminary; GPT-4 is evaluated only on a subset of 500 multi-document QA examples because full evaluation would cost upwards of $6000.
- Stated: NaturalQuestions-Open has ambiguity, so the authors run an unambiguous subset and also test random distractors and randomized distractor order, finding similar trends.
- Evident: the controlled multi-document QA setting always contains exactly one document that answers the question, unlike open-domain QA; the synthetic key-value task removes natural language semantics; results may not generalize to all tasks, models, or decoding settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 0 Not valid · 5 Not checked · 13 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Language models have become an important and flexible building block in a variety of user-facing language technologies, including conversational interfaces, search and summarization, and collaborative writing (Shuster et al., 2022;Thoppilan et al., 2022;Lee et al., 2022, inter alia).”
§ Lost in the Middle: How Language Models Use Long Contexts > 1 Introduction
What the sources say
- Partly supportedcites Shuster et al., 2022 matched to “BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage” (2022) sourceExperimental — accuracy not yet measured
Section 3.1.1 (p. 4) describes a conversational agent built on a language model that generates internet search queries, supporting conversational interfaces and search as part of dialogue. Section 2 (p. 3) mentions summarization only as related work, and the passages do not establish collaborative writing as an application of this paper’s model. (LLM, conf=0.85)
- Supportedcites Thoppilan et al., 2022 matched to “LaMDA: Language Models for Dialog Applications” (2022) source
The Abstract (p. 1) identifies LaMDA as a family of language models specialized for dialog. In this sentence's list of applications, Thoppilan et al. supports the conversational-interface example; the other cited works need not each cover every example. (LLM, conf=0.92)
- Supportedcites Lee et al., 2022 matched to “CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities” (2022) source
Lee et al. is cited alongside other works, and its contribution to the sentence is the collaborative-writing example. Section 4.2.2 describes writers receiving and selecting model-generated suggestions in a writing interface; Section 2.2.1 also describes language models as applicable to dialogue and summarization. The paper need not establish every example in the jointly cited list. (LLM, conf=0.89)
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These observations complement prior work, which found that non-instruction fine-tuned language models are biased towards recent tokens (i.e., the end of the input context; Khandelwal et al., 2018;Press et al., 2021).”
§ Lost in the Middle: How Language Models Use Long Contexts > 4.3 Effect of Instruction Fine-Tuning
What the sources say
- Partly supportedcites Khandelwal et al., 2018 matched to “Sharp nearby, fuzzy far away: How neural language models use context” (2018) sourceExperimental — accuracy not yet measured
The Introduction and Section 5 report that the studied LSTM is much more sensitive to recent word order than to distant word order, supporting a recency effect. The paper studies a standard LSTM on two datasets, however, rather than establishing this tendency across non-instruction-fine-tuned language models generally; it also finds that distant words remain relevant. (LLM, conf=0.82)
- Partly supportedcites Press et al., 2021 matched to “Shortformer: Better language modeling using shorter inputs” (2021) sourceExperimental — accuracy not yet measured
Section 3.1 (page not available in our copy) finds that very short contexts hurt performance, while increasing subsequence length from 1,024 to 3,072 tokens gives negligible improvement. That is consistent with limited benefit from more distant context, but the paper does not establish that the models are biased toward recent tokens within an input context. (LLM, conf=0.78)
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
§ Lost in the Middle: How Language Models Use Long Contexts > 2.1 Experimental Setup
What the sources say
- Supportedcites Lee et al., 2019 matched to “Latent retrieval for weakly supervised open domain question answering” (2019) source
Section 7.1 says the paper creates an open version of Natural Questions and uses it for training and evaluation. The Introduction describes the task as answering questions by retrieving evidence from an open corpus, supporting the cited use of NaturalQuestions-Open for this task. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites Kwiatkowski et al., 2019 matched to “Natural Questions: A benchmark for question answering research” (2019) source
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Following Kandpal et al. (2022) and Mallen et al. (2023), we use accuracy as our primary evaluation metric, judging whether any of the cor- rect answers (as taken from the NaturalQuestions annotations) appear in the predicted output.”
§ Lost in the Middle: How Language Models Use Long Contexts > 2.1 Experimental Setup
What the sources say
- Supportedcites Kandpal et al. (2022) matched to “Large Language Models Struggle to Learn Long-Tail Knowledge” (2022) source
Section 3 (p. 4) says the paper evaluates answers using exact match against a ground-truth answer set, and Section 3.1 (p. 4) reports QA accuracies as its evaluation results. This supports citing Kandpal et al. for using accuracy to evaluate question answering. (LLM, conf=0.90)
- Supportedcites Mallen et al. (2023) matched to “When not to trust language models: Investigating effectiveness of parametric and non-parametric memories” (2023) source
Section 3.1 (‘Focus and Task’) explicitly names accuracy as the metric and says a prediction is correct when it contains an exact match to any gold answer. This supports the evaluation approach attributed to Mallen et al.; the NaturalQuestions annotations describe the citing authors’ answer source. (LLM, conf=0.94)
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“or when language models are augmented with external information (relevant documents from a search engine, database query results, etc.; Petroni et al., 2020;Ram et al., 2023;Shi et al., 2023;Mallen et al., 2023;Schick et al., 2023, inter alia).”
§ Lost in the Middle: How Language Models Use Long Contexts > 1 Introduction
What the sources say
- Partly supportedcites Petroni et al., 2020 matched to “How Context Affects Language Models' Factual Predictions” (2020) sourceExperimental — accuracy not yet measured
The Abstract (p. 1) says the paper integrates information from a retrieval system with a pre-trained language model, and §3.4.2 (p. 5) specifies a retrieved paragraph as context. This supports augmentation with retrieved documents, but our copy does not show language models being augmented with database query results. (LLM, conf=0.84)
- Partly supportedcites Ram et al., 2023 matched to “In-Context Retrieval-Augmented Language Models” (2023) sourceExperimental — accuracy not yet measured
The Introduction describes augmenting language models with relevant documents retrieved from an external knowledge source, and Section 5 describes using a BM25 retriever to find those documents. The passages do not establish database query results as another type of external information used by this paper. (LLM, conf=0.82)
- Partly supportedcites Shi et al., 2023 matched to “REPLUG: Retrieval-augmented black-box language models” (2023) sourceExperimental — accuracy not yet measured
Section 3.1 (page unavailable) describes retrieving relevant documents from a corpus, and the Conclusion says REPLUG uses those documents to augment a language model. The paper supports the document-retrieval example, but our copy does not describe augmenting models with database query results or documents from a search engine specifically. (LLM, conf=0.82)
- Partly supported
20 more claims · 2 with their cited sources checked
Create a free account to see all 25 claims and every source verdict.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In vanilla multi-document or NIAH-based benchmarks (Kamradt, 2023; Liu et al., 2024), models perform consistently well when the needle (or gold document) appears at the very beginning or end of the context window, with minimal impact from context length.”
From NoLiMa: Long-Context Evaluation Beyond Literal Matching (2025) · cites this paper as Liu et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Even within a single long sequence, attention effectiveness degrades with distance due to attention dilution , positional encoding limitations, and token interference, leading to the well-known 'lost-in-the-middle' and context-decay phenomena (Liu et al., 2024; Press et al., 2021a).”
From MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents (2026) · cites this paper as Liu et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current large language models (LLMs) exhibit significant deficiencies in episodic memory tasks including encoding, storing, and retrieving specific information from temporally dependent events over a long period of time.”
From ARTEM: Enhancing Large Language Model Agents with Spatial-Temporal Episodic Memory (2026) · cites this paper as Liu et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent approaches to handle memory tasks in LLMs, such as in-context learning, retrieval-augmented generation (RAG), and fine-tuning, may resolve the long-term retention issues, but are still inadequate to handle tasks requiring chronological awareness of the stored information.”
From ARTEM: Enhancing Large Language Model Agents with Spatial-Temporal Episodic Memory (2026) · cites this paper as Liu et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“If not ranked at the top, the key information may be neglected by the generative module, resulting in incomplete or inaccurate responses to the user queries24.”
From Leveraging long context in retrieval augmented language models for medical question answering (2025) · cites this paper as [24]
Experimental — accuracy not yet measured
86 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 2×
“Recent improvements in hardware (e.g., faster GPUs with more memory) and algorithms (Dai et al., 2019; Dao et al., 2022; Poli et al., 2023; Rubin and Berant, 2023, inter alia ) have resulted in language models with larger context windows (e.g., 4096, 32K, and even 100K tokens), but it remains…”
BackgroundLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2020) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2007.01282 · Cited by 2,281
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 1×
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019; Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
MethodThe serial position effect of free recall.
Murdock, B. B. (1962) · Journal of Experimental Psychology · doi:10.1037/h0045106 · Cited by 1,717 · cited 1×
“The U-shaped curve we observe in this work has a connection in psychology known as the serial-position effect (Ebbinghaus, 1913; Murdock Jr, 1962), that states that in free-association recall of elements from a list, humans tend to best re-member the first and last elements of the list.”
MethodMemory: A contribution to experimental psychology.
Ebbinghaus, H. (1913) · doi:10.1037/10011-000 · Cited by 1,645 · cited 1×
“The U-shaped curve we observe in this work has a connection in psychology known as the serial-position effect (Ebbinghaus, 1913; Murdock Jr, 1962), that states that in free-association recall of elements from a list, humans tend to best re-member the first and last elements of the list.”
Method
Research works printed but not linked: 1
- Sharan, 2018, Prediction with a short memory, 10.18653/v1/K18-3013 · DOI
Cited by
Items 1,101–1,200 of 1,597
Automated Meta-Analysis of Transcranial Magnetic Stimulation Based on GraphRAG: A Methodology StudyPreprint
Zhong, Y., Jin, J., Wang, X., He, W., & Yin, T. (2026) · Research Square · doi:10.21203/rs.3.rs-10200494/v1
Resource-Aware Evidence Selection with Dynamic Context Budgeting for Retrieval-Augmented Systems
Yu, Q., Niu, H., & Wang, X. (2026) · International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) · doi:10.1109/aiotc71250.2026.11688398
“Conversely, using more passages can introduce redundant or weakly related text and can dilute useful evidence in a long context [3].”
Herding CATs: ALARA for Agent Harness Engineering in Portable Composable Multi-Agent Teams
Agostino, C. J., & D'Souza, N. (2026) · International Conference on Human-AI Interaction and Experience Design (HAXD) · doi:10.1109/haxd70072.2026.11621027 · Cited by 5
“…tools, but the model is free to ignore or misinterpret the instruction, and attention dilution makes this increasingly likely over long contexts [10]; tools not on the Jinx list do not exist in the agentŠs schema, and no amount of prompt injection or attention drift can invoke what was never…”
MMKRF: A Domain-Specific Knowledge Retrieval Framework for RAG Systems in Materials Mechanics
Hu, Z., Duan, Q., Liu, B., & Li, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5637-3_8
The Developer as Curator: A Canvas for Coding Agent Context
Böhme, L., Beckmann, T., Perscheid, M., & Hirschfeld, R. (2026) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3840586.3843212
“Irrelevant information in the context window has been shown to degrade generation quality [22, 27, 39, 46].”
InfluentialTang, W., Gui, T., Ma, H., & Chen, M. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11581585
StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systems
Zhu, T., Wu, Y., Jin, W., Li, Y., & Huang, G. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817991 · Cited by 7
“While single LLM-based agents can independently perform tasks such as document analysis, question answering, and tool invocation [7, 30, 34, 40], they struggle with heterogeneous information integration, long-horizon reasoning, and multi-objective coordination [22, 33].”
Nyay Mitra: An AI Powered Legal Assistance Platform Using Advanced RAG and FAISS Architecture
Patel, F., Patil, T., Pandey, U., & Bhangale, P. (2026) · International Conference on Communication, Computing and Emerging Technologies (IC3ET) · doi:10.1109/ic3et64989.2026.11467539
“The system also needs to be capable of going through many legal documents at once [1].”
Position Bias in LLMs for Critical Decision Support: A Case Study on Multiple Casualty Triage
Bolin, U. W., Cohen, K., Björnesjö, H., & Tegen, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-14107-1_42 · Cited by 1
Xiao, L. (2026) · Journal of Education and Educational Research · doi:10.54097/2hcyvp15
“These aspects were backed by studies discussing the limitations of the language models used, long-context instability, clinical accountability, effectiveness of chatbots, and therapeutic relations [1, 3, 4, 6, 7].”
InfluentialWang, N., Zhao, X., Lin, X., Liu, X., & Sun, Q. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356417
“Large language models struggle with long-form context utilization, exhibiting the "Lost in the Middle" problem where information in the middle of long inputs is neglected [19].”
Khan, U. A. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.05.723102
“A recent survey by Gao et al. [19] organizes the field into Naive, Advanced, and Modular RAG taxonomies and catalogs documented limitations including retrieval noise, position-in-context effects [20], and grounding failures [21].”
Santos, E., Pimenta, C. M. S., & Tavares, F. O. (2026) · Journal of risk and financial management · doi:10.3390/jrfm19090654
Explainable Numerical Claim Verification
Aarnes, P. R. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761666
Optimal Rope Extension Via Bayesian Optimization for Training-Free Length GeneralizationPreprint
Zhang, X., Hu, S., Zhao, W., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4765732
DRMD: Explainable Depression Detection Based on Metaphorical Conceptual Mapping
Zhang, D., Liao, W., Hu, W., & Lin, H. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792505
DPS: Diverse Prototype Selection for Adaptive In-Context Learning
Fan, X., Li, K., Sun, H., Peng, B., Cheng, Z., & Zhang, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06078-5_13
Wang, Z., Zhu, X., Wang, H., Yu, Y., & Lan, Y. (2025) · PLoS ONE · doi:10.1371/journal.pone.0338821 · Cited by 1
“Recent research indicates that models suffer from middle dropout phenomena [22], paying little attention to middle parts when the input content is too long.”
DIVAgent: A Diversified Search Agent that Mimics the Human Search Process
Deng, Z., Qiao, J., Dou, Z., Wen, J.-R., & de Rijke, M. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761059
Lee, D. Y. H., Parker, A. J., Norbury, C., & Shanks, D. R. (2026) · Royal Society Open Science · doi:10.1098/rsos.250381 · Cited by 2
Divide and summarize: improve SLM text summarization
Bailly, A., Saubin, A., Kocevar, G., & Bodin, J. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1604034 · Cited by 3
Evaluating RAG Accuracy Enhancement via Page Merging for Non-Ideal Documents
Shinoda, T., Satō, H., & Kubo, M. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.09.445
Reply: Methodological clarifications of AI-powered research trend analytics in bronchiectasis
Narayana, J. K., & Chotirmall, S. H. (2026) · European Respiratory Journal · doi:10.1183/13993003.01968-2025
EFGCLS: A Cross-Lingual Summarization Method Based on Element Fact-Relationship Generation
Huang, Y., Jiushun, M. A., Li, T., Yu, Z., Xian, Y., & Xiang, Y. (2025) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2024edp7274 · Cited by 1
“The study by [39] suggests that the performance of large models is typically highest when relevant information appears at the beginning or end of the input context.”
Pal, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02240-w
“Long-context studies also show that simply supplying relevant context does not guarantee that a model will use evidence uniformly across context positions [27].”
Adaptive Web Resource Parsing System Based on Large Language Models Using Semantic Caching
Muradyan, D. S., Shutko, O., & Bushemelev, F. V. (2026) · IEEE International Conference on Soft Computing and Measurements (SCM) · doi:10.1109/scm71573.2026.11625872
“This step is necessary to reduce the size of the input context during model inference, which makes it possible to lower computational costs and processing cost, and in some cases to reduce response latency [7].”
AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems
Wang, Z., Yu, G., & Lyu, M. R.-T. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3821576 · Cited by 6
“This context saturation leads to a distinct degradation in instruction-following capabilities [30, 37].”
Evaluating Large Language Models in Cybersecurity: A Systematic Taxonomy and Empirical Analysis
Chen, M., Cheng, H., Su, T., Chen, M., Cai, W., & Zou, H. (2026) · Electronics · doi:10.3390/electronics15102222 · Cited by 2
“By prioritizing semantic correlation over deterministic execution, the architecture exhibits a critical state-tracking deficit—a recognized limitation where Transformers fail to inherently simulate deterministic state machines [5,103].”
InfluentialDeng, Y. (2026) · 2026 3rd World Conference on Computer and Information Security (WCCIS) · doi:10.1109/wccis70285.2026.11651016
“However, constrained by the context length (Context Window), the capacity of input and output windows in native LLMs still presents significant limitations [3].”
Towards Optimized Retrieval-Augmented Generation: A Comprehensive Survey
Houria, A. W., & Faiçal, A. (2025) · 2025 Fourth International Conference on Theoretical and Applicative Aspects of Computer Science (ICTAACS) · doi:10.1109/ictaacs69003.2025.11399325 · Cited by 2
“Selecting the appropriate LLM involves considering QA benchmark performance, context length, instruction-following ability, hallucination rate, multilingual support, licensing, and inference cost [8], [83], [84].”
CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMs
Lv, J., Feng, Y., Xie, X., Jia, X., Peng, Q., & Xie, G. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887916 · Cited by 8
“Additionally, the widely-used ”Needle-in-a-Haystack” [34] test, a synthetic QA task designed to assess long-context retrieval capabilities, is employed to facilitate a detailed evaluation across various context lengths.”
Enhancing RAPTOR with semantic chunking and adaptive graph clustering
Liu, Y., Xie, X., Wan, X., Pan, Y., & Wang, C. (2026) · Frontiers in Computer Science · doi:10.3389/fcomp.2025.1710121 · Cited by 2
Empowering 9-1-1 Calltaking Training with Generative AI: Experiences and Lessons Learned
Chen, Z., & Ma, M. (2026) · IEEE International Conference on Smart Computing · doi:10.1109/smartcomp69968.2026.00032 · Cited by 1
“Supplying full context either exceeded model capacity or produced intertwined evidence where reasoning became unstable [28], [41].”
de Faria, J. P., Castro, C., Brilha, J., & Leiva, V. (2026) · Geoheritage · doi:10.1007/s12371-026-01450-z
“Long inputs may also be used unevenly by the model, and struc - tured responses still require validation (Liu et al. 2024a, b; Geng et al. 2023).”
AGRAG: Advanced Graph-Based Retrieval-Augmented Generation for LLMs
Wang, Y., Li, H.-Y., Teng, F., & Chen, L. (2025) · IEEE International Conference on Data Engineering · doi:10.1109/icde65706.2026.00110 · Cited by 9
“However, the retrieval procedures of these models remain unstructured and struggle to capture the nuanced semantic and structured relationship within the text chunks, which impedes the effectiveness of LLMs [15]–[18].”
FlowRAG: Continual Learning for Dynamic Retriever in Retrieval-Augmented Generation
Zhang, S., Shi, T., Song, D., Zhang, L., Zhang, S., Liao, X., & Jin, H. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792361 · Cited by 2
Mapping the dynamics of corporate climate innovation strategiesPreprint
Stolz, N., Hoffmann, V. H., & Probst, B. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6592488
ELARA: A Framework for Empirical Legal Research with AI AgentsPreprint
Choi, J. H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7349078
Tail-Aware Adaptive-k: Query-Adaptive Context Selection for Retrieval-Augmented Generation
Song, Z., Fang, J., Li, K., Xia, T., & Wang, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37667-1_12 · Cited by 2
MedShard: Privacy-Preserving EMR QC with Rule Sharding and Multi-agent Collaboration
Dang, S., Lai, X., Li, Y., Chen, P., & Wang, H. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-7299-1_23
Lin, B., Mao, S., Wang, P., Zhang, L., & Haosu, Z. (2026) · International Conference on Information Science, Electrical and Automation Engineering (ISEAE) · doi:10.1109/iseae69422.2026.11544810
“During extended reasoning chains or cross-agent communication, models frequently dissociate target identifiers from their physical attributes and threat levels, leading to catastrophic target drift or friendly fire risks [12].”
Hierarchical Attribution Analysis of Deficiencies in the Reasoning Ability of Large Language Models
Jiang, Z., Zhang, H., Wang, F., Yan, X., & Yang, Y. (2026) · International Conference on Computer, Artificial Intelligence and Control Engineering (CAICE) · doi:10.1145/3804601.3804629
MENTOR: a multi-agent framework for event and narrative trend prediction with optimized reasoning
Chen, L., Jia, G., Gu, D., Yan, J., Jiang, Y., Li, X., & Zeng, X. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2500608 · Cited by 1
A visualization framework for hierarchical structure enhanced mind mapping
Liu, Y., Wang, H., Lu, W., Chen, W., & Ji, Y. (2026) · Visual Informatics · doi:10.1016/j.visinf.2026.100325
Survey Table Generation from Academic Articles
Chen, P.-C., Huang, H., & Chen, H. (2024) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat62293.2024.00033 · Cited by 2
“Previous research has shown that even though LLMs claim to have sufficiently large context lengths, overly long contexts can lead to performance degradation[39] and [40].”
SieveKV: Semantic Multi-signal Eviction for Long-Context Retrieval
Li, K. W. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3417-2_19
Sato, K., Niimi, Y., Ihara, R., Suzuki, K., Iwata, A., & Iwatsubo, T. (2026) · Journal of Alzheimer s Disease · doi:10.1177/13872877261464182 · Cited by 1
“40 Furthermore, these models are constantly being updated, meaning our speci fi c results may not be perfectly reproducible with future model versions, although the general principles and observed biases are likely to persist in some form.”
Training a Team of Language Models as Options to Build an SQL-Based Memory
Lee, S., & Ko, H. (2025) · Applied Sciences · doi:10.3390/app152111399
“Studies have shown that the longer the context input to a language model, the more difficult it becomes for the language model to reflect the information it contains [16–19].”
Han, Z., Chneg, C., & Yonglei, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5899045
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.171.79.1
Nian, F., & Fu, Y. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02677-0
“First, scientific papers are long, and their relevant evidence is unevenly distributed across abstracts, introductions, method sections, figures, tables, and experiments; direct long-context prompting may dilute local evidence and produce unstable outputs [11 ].”
Bai, G., Luo, X., Wang, Y., et al. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-09694-4_45
Hard Problems, Soft Guards: Softening LLM Guards with Olympiad-Grade Problems
Zhong, T., Xu, M., & Yang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2859-1_13
Steering Semantic Data Processing With DocWrangler
Shankar, S., Chopra, B., Hasan, M., Lee, S., Hartmann, B., Hellerstein, J. M., Parameswaran, A., & Wu, E. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747625 · Cited by 12
“Even with clear, unambiguous prompts, LLMs may fail to generalize correctly to the user’s actual data, struggling with long documents [49] or complex operations requiring simultaneous reasoning about multiple elements [33].”
Scene-Conditioned Capability Mounting as a Visible and Executable Boundary for Situated LLM Agents
Jiang, S., Cheng, Y., Wang, J., & Cui, S. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831213
“Large catalogs create context pressure [10, 15].”
Saßmannshausen, T. M., & Wagener, S. (2026) · Qeios · doi:10.32388/gag6kd.2
Bounded Descriptions for Knowledge Graph Traversal by Language Models
Henselmann, D., & Harth, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-25156-5_20
Chain-of-Summary: An Efficient Multi-Clustering Framework for Hierarchical Abstraction
Yang, C., Zhu, W., Yang, Z., Liu, C., & Hong, J. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00090 · Cited by 1
“Existing large language models often encounter the ”lost in the middle” [16] problem when handling long contexts.”
Reimagining Educational Techniques with Artificial Intelligence 1
Kloos, C. D., Alario‐Hoyos, C., Muñoz-Merino, P. J., Ayres, I. E., Moreno-Marcos, P. M., Callejo, P., & Kemcha, R. (2026) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon67543.2026.11574341 · Cited by 1
“Liu et al. [44] demonstrated that placing key documents in the middle of the prompt can degrade accuracy due to the wrong attention distribution in LLMs. Similarly, Cuconasu et al. [45] observed that both document type and retrieval position influence the effectiveness of RAG systems.”
MW-CoT-RAG: Multi-source Weighted Chain-of-Thought RAG for Trustworthy Equipment QA
Zhang, B. M., Pan, H., Lin, C., & Jiang, X. (2026) · International Symposium on Computer Technology and Information Science (ISCTIS) · doi:10.1109/isctis70043.2026.11571970
“A key limitation of standard RAG is the “ Lost-in-the-Middle ” effect, where models struggle to effectively utilize information located in the middle of long context windows [3].”
Synergistic multi-stage processing for enhanced context relevance in RAG architecture
Tien, L. N., & Tieu, B. H. (2026) · Discover Computing · doi:10.1007/s10791-026-10319-2
“This metric becomes increasingly challenging with longer contexts, as the probability of including irrelevant content increases [35].”
Yang, M., Ren, Y., Wang, H. B., Hou, W. Z., Li, W., Chun, Z., Zheng, W., & Ren, D. G. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3426-4_5
Prompting LLMs for Game Learning Analytics: A Case Study with Computer Science Students
Honda, F., Pires, F., Oliveira, E., & Pessoa, M. (2026) · doi:10.5753/educomp.2026.18681
Attunement Costs: How Interaction Locks Users into Specific LLMsPreprint
Hanschur, L., Obermeier, D., & Henkel, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7354642
Conditional diffusion transformer enabled few-shot spatiotemporal modeling
Zhao, Z., Jiang, X., Shen, G., Weng, W., Zhou, W., Kong, X., & Feng, X. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104499
Exploring Causal Effect of Social Bias on Faithfulness Hallucinations in Large Language Models
Zhang, Z., Zhang, J., Hu, X., Zhang, H., & Wan, X. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761298
“To address this gap, we leverage causal inference theory [21] to investigate the causal relationship between bias and hallucinations systematically.”
ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs
Li, X., Li, H., Zhou, Y., Ai, Q., & Liu, Y. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769499 · Cited by 6
Opening the ‘black box’ of AI for credible IRS: A rapid review
Igbinovia, M. O. (2025) · IFLA Journal · doi:10.1177/03400352251392575
“This implies that LLM-based IRS will often neglect items not well positioned in the input sequence, especially words in the middle of a list (Liu et al., 2024).”
Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools
Arat, B., & Sefer, E. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809852 · Cited by 2
“This controls for position bias, as LLMs are known to favor documents at certain positions independently of document relevance [8, 23].”
Jiang, Z. (2026) · Design and Artificial Intelligence · doi:10.1016/j.daai.2026.100087 · Cited by 3
Tekale, A., & Masum, M. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00037
“This provides empirical validation that longer inputs systematically degrade classification performance, consistent with the “lost - in-the-middle” phenomenon documented in long -context LLM literature [12], [14].”
InfluentialJoint Embeddings for Graph Instruction Tuning
HAAG, A. L., Argatu, V., & Lohse, O. (2024) · International Conference on Intelligent Computing in Data Sciences (ICDS) · doi:10.1109/icds62089.2024.10756351 · Cited by 2
“For graphs having complex node features and a large amount of edges, this approach ends-up in generating large prompts on which most LLMs have limited understanding capabilities [28].”
CoFiLoc: A Coarse-to-fine LLM-based framework for method-level bug localization
Cao, N., Nguyen, N. T., & Nguyen, T. B. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00649-2
“Prior work has shown that LLMs are sensitive to the position of relevant items within long contexts (Liu et al. 2024 ), potentially introducing position bias into the final ranking.”
LLM-based Specification-Driven Test Oracle Enhancement
Fu, R., Zhao, Y., Wang, M., Wang, Z., & Chen, J. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805616
“In addition to facilitating human triage, test case reduction mitigates the performance degradation of LLMs under long contexts, which is often referred to as the lost-in-the-middle phenomenon[18].”
From Retrieval to Faithful Memory Use: Context-Grounded Evidence Chains for Music Education Agents
Mao, J., Chen, Y., Wan, W., Liu, K., & Wang, R.-F. (2026) · Information · doi:10.3390/info17080783
Smart MRAG: Hybrid Retrieval-Augmented Multimodal Q&A Technique
Narvekar, A., Bajaj, R., Mukherjee, A., & Katoch, A. (2026) · Journal of Computer Information Systems · doi:10.1080/08874417.2026.2707153
Pwnagent: a knowledge-guided multi-agent system for automatic exploit generation
Wei, C., Geng, Y., Wang, Y., Wu, Q., Huang, J., Wu, Q., & Wei, Q. (2026) · Cybersecurity · doi:10.1186/s42400-026-00649-5
SPKG: A Two-Stage Structured Prompting Framework for Knowledge Graph Construction
Sun, Z., Liu, Z., Wang, Y., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3435-6_30
SHEA-SLM: Sustainable High-Efficiency Architecture for Small Language Models
Papakonstantinou, P., Nyktaris, S., Anagnostou, P., Tasoulis, S. K., Georgakopoulos, S. V., & Plagianakos, V. P. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-31141-2_16
Li, X., Yang, Z., Li, D., Jiang, Y., & Li, J. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00078
“(2) Severe Attention Dilution: Their visual attention severely dilutes over continuous long video streams, causing them to miss critical intermediate steps [13].”
Tani, T., Joseph, M. H., Metso, L., & Tynninen, L. (2026) · Computer Applications in Engineering Education · doi:10.1002/cae.70263
Language-Guided Adaptive Vision Token Pruning for Efficient Multimodal Large Language Models
Deniz, O. F., Arici, T., Sheikholeslami, F., Gozluklu, B., Trabelsi, A., Khan, S. A., Tian, Y., & Khan, L. R. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8186-0_9
Comparing Lightweight Gemma3 Models for SAT Math Under Resource Constraints
Lertyosbordin, C., Jearaborival, T., Tangprapa, M., et al. (2025) · International Symposium on Educational Technology · doi:10.1109/iset65607.2025.00041
“…enabling it to process text and translate natural language data into computable mathematical equations, facilitating work with complex problems [12,13]; and its support for symbolic computation, complementing numerical computation by organizing equations, factorizing, and performing logical…”
Pretrained language models for selection of cohorts from electronic health records
Spiero, I., Damen, J. A., Hooft, L., Moons, K. G. M., & Leeuwenberg, A. (2026) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2026.101803
Lay Theories About Artificial Versus Artificial Intelligence: Rethinking Theory of MachinePreprint
Rebholz, T. R. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ekz9a · Cited by 1
A Multi-stage Rumor Detection Framework Based on Retrieval-Augmented Generation Optimization
He, Z., Yang, Y., Yu, W., & Chen, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4861-3_2
EntiTractor: entity extraction for coherent hallucination correction in paragraph-level text
Yao, J., Yao, J., Zang, W., Zhao, K., Zhang, H. M., & Wang, J. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133359
In-Context Learning in Large Language Models: Mechanisms, Challenges, and Frontiers
Yuan, Y. (2026) · ITM Web of Conferences · doi:10.1051/itmconf/20268403003 · Cited by 1
“Without segmentation, attention weight diffuses and middle tokens receive minimal focus, leading to the well-known “lost-in-the-middle” effect [9].”
ARGUS: A Large Language Model Reasoning System for Machine Learning Task Identification
Gonçalves, C., Pereira, P. J., & Cortez, P. C. (2026) · ACM Symposium on Applied Computing · doi:10.1145/3748522.3779804 · Cited by 2
Fang, X., Yuan, H. R., Li, H., Kou, J., Gu, C., Zhang, W., Duan, X., & Fang, Y. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189217 · Cited by 4
“Studies reveal that LLMs often neglect the middle portion of long input sequences, attending primarily to the beginning and end [88], [89].”
Indexação Semântica Modular para Recuperação Colaborativa de Conhecimento em Sistemas RAG
Freire, E. C., de Azevedo, K. S., Silva, S. N., & Fernandes, M. A. C. (2026) · Simpósio Brasileiro de Sistemas Colaborativos (SBSC) · doi:10.5753/sbsc.2026.20073
Reader comes first: A demand-oriented readability controllable summarization
Han, Q., Sun, Y., Yang, Z., Liu, W., Luo, L., & Lin, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130858
Aziz, B. A., Rushdi, M. A., Yoshida, S., Dief, T. N., Shaban, I. A., & Kamra, M. M. (2026) · Applied Sciences · doi:10.3390/app16178774 · Cited by 1
Nguyen, V. A., Duc, T. M., Trung, N. V., Tuyet, N. T. A., Duc, P. M., & Chau, V. B. (2025) · International Conference on Education Technology and Computer · doi:10.1109/icetc66579.2025.11387670
“Studies on model performance with varying input contexts, sizes, and information extraction positions reveal that models perform best when information is extracted from the beginning (primacy bias) and the end of the context (recency bias) [6].”
Ke, W., Yang, L., & Xu, M. (2026) · AI · doi:10.3390/ai7080320 · Cited by 1
Zhong, S., Lan, X., Yi, W., & Nie, S. (2026) · Information · doi:10.3390/info17080778
A large language model-based agent system for interactive large-scale project scheduling
Seo, H., & Choi, M. (2026) · Journal of Computational Design and Engineering · doi:10.1093/jcde/qwag082 · Cited by 1
Research on Legal Question Answering System with Retrieval-Augmented Large Language Models
Xu, N., Li, S., & Xia, Y. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4276-2_10 · Cited by 1
Pampel, B., Martin, S., & Padó, U. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-39106-3_14 · Cited by 1
Metadata
- Authors
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Samaya AI, UK
- Corresponding author
- Liu, Nelson F.
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 157-173
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4391876619
- Semantic Scholar
- 1733eb7792f7a43dd21f51f4d1017a1bffd217b5
- Cited by
- 1,612
- References
- 59
- Citations by year
- 2026: 901 · 2025: 516 · 2024: 172
- FWCI
- 233.95
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- long-context language models, multi-document question answering, position bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-27 (prompt extract-v2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.