Lost in the Middle: How Language Models Use Long Contexts
Open accessThe DOI matches its Crossref record; 48 of the 59 references it cites were found in a registry or an index; 15 of the 23 sources its claims cite were checked for support.
Abstract
While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.
Overview
CoChat Summary
TL;DR
Language models’ performance on tasks requiring access to long input contexts is strongly affected by where relevant information appears, with a U-shaped curve: best at the beginning or end and worst in the middle. Extended-context models are not necessarily better at using context, and reader performance saturates before retriever recall in open-domain QA.
Key findings
- In multi-document QA, performance is highest when the answer-containing document is at the start or end of the context and degrades in the middle; GPT-3.5-Turbo can drop by more than 20%, and in 20- and 30-document settings middle-position performance is lower than its closed-book performance of 56.1% (oracle: 88.3%).
- Extended-context models are not necessarily better at using input context: when the input fits both windows, GPT-3.5-Turbo and GPT-3.5-Turbo(16K) perform nearly identically as a function of relevant-information position.
- In synthetic key-value retrieval, Claude-1.3 and Claude-1.3(100K) do nearly perfectly, while other models struggle especially with 140 or 300 key-value pairs; query-aware contextualization gives GPT-3.5-Turbo(16K) perfect performance at 300 pairs, versus worst-case 45.6% without it.
- In open-domain QA, reader performance saturates long before retriever recall: using 50 instead of 20 retrieved documents only marginally improves performance (∼1.5% for GPT-3.5-Turbo and ∼1% for Claude-1.3).
Methods
- Controlled multi-document QA on NaturalQuestions-Open: 2,655 queries whose annotated long answer is a paragraph; each context has exactly one answer-containing Wikipedia passage plus k−1 Contriever/MS-MARCO distractor passages; the paper varies answer position and context length (10, 20, 30 documents) and measures accuracy.
- Synthetic key-value retrieval: inputs are serialized JSON objects with unique random UUID key-value pairs; models must return the value for a specified key; experiments use 75, 140, and 300 key-value pairs (500 examples each), varying key position and number of pairs, with accuracy as the metric.
- Models include open MPT-30B-Instruct and LongChat-13B(16K), closed GPT-3.5-Turbo, GPT-3.5-Turbo(16K), Claude-1.3, and Claude-1.3(100K), plus additional architecture/instruction experiments with Flan-T5-XXL, Flan-UL2, MPT-30B, Llama-2, and GPT-4; an open-domain QA case study uses a Contriever retriever-reader setup.
Limitations
- Stated: the paper uses greedy decoding and leaves exploration of other decoding methods to future work; the investigations into why models are not robust are described as preliminary; GPT-4 is evaluated only on a subset of 500 multi-document QA examples because full evaluation would cost upwards of $6000.
- Stated: NaturalQuestions-Open has ambiguity, so the authors run an unambiguous subset and also test random distractors and randomized distractor order, finding similar trends.
- Evident: the controlled multi-document QA setting always contains exactly one document that answers the question, unlike open-domain QA; the synthetic key-value task removes natural language semantics; results may not generalize to all tasks, models, or decoding settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 0 Not valid · 5 Not checked · 13 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Language models have become an important and flexible building block in a variety of user-facing language technologies, including conversational interfaces, search and summarization, and collaborative writing (Shuster et al., 2022;Thoppilan et al., 2022;Lee et al., 2022, inter alia).”
§ Lost in the Middle: How Language Models Use Long Contexts > 1 Introduction
What the sources say
- Partly supportedcites Shuster et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.1.1 (p. 4) describes a conversational agent built on a language model that generates internet search queries, supporting conversational interfaces and search as part of dialogue. Section 2 (p. 3) mentions summarization only as related work, and the passages do not establish collaborative writing as an application of this paper’s model. (LLM, conf=0.85)
- Supportedcites Thoppilan et al., 2022 source
The Abstract (p. 1) identifies LaMDA as a family of language models specialized for dialog. In this sentence's list of applications, Thoppilan et al. supports the conversational-interface example; the other cited works need not each cover every example. (LLM, conf=0.92)
- Supportedcites Lee et al., 2022 source
Lee et al. is cited alongside other works, and its contribution to the sentence is the collaborative-writing example. Section 4.2.2 describes writers receiving and selecting model-generated suggestions in a writing interface; Section 2.2.1 also describes language models as applicable to dialogue and summarization. The paper need not establish every example in the jointly cited list. (LLM, conf=0.89)
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These observations complement prior work, which found that non-instruction fine-tuned language models are biased towards recent tokens (i.e., the end of the input context; Khandelwal et al., 2018;Press et al., 2021).”
§ Lost in the Middle: How Language Models Use Long Contexts > 4.3 Effect of Instruction Fine-Tuning
What the sources say
- Partly supportedcites Khandelwal et al., 2018 sourceExperimental — accuracy not yet measured
The Introduction and Section 5 report that the studied LSTM is much more sensitive to recent word order than to distant word order, supporting a recency effect. The paper studies a standard LSTM on two datasets, however, rather than establishing this tendency across non-instruction-fine-tuned language models generally; it also finds that distant words remain relevant. (LLM, conf=0.82)
- Partly supportedcites Press et al., 2021 sourceExperimental — accuracy not yet measured
Section 3.1 (page not available in our copy) finds that very short contexts hurt performance, while increasing subsequence length from 1,024 to 3,072 tokens gives negligible improvement. That is consistent with limited benefit from more distant context, but the paper does not establish that the models are biased toward recent tokens within an input context. (LLM, conf=0.78)
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
§ Lost in the Middle: How Language Models Use Long Contexts > 2.1 Experimental Setup
What the sources say
- Supportedcites Lee et al., 2019 source
Section 7.1 says the paper creates an open version of Natural Questions and uses it for training and evaluation. The Introduction describes the task as answering questions by retrieving evidence from an open corpus, supporting the cited use of NaturalQuestions-Open for this task. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites Kwiatkowski et al., 2019 source
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Following Kandpal et al. (2022) and Mallen et al. (2023), we use accuracy as our primary evaluation metric, judging whether any of the cor- rect answers (as taken from the NaturalQuestions annotations) appear in the predicted output.”
§ Lost in the Middle: How Language Models Use Long Contexts > 2.1 Experimental Setup
What the sources say
- Supportedcites Kandpal et al. (2022) source
Section 3 (p. 4) says the paper evaluates answers using exact match against a ground-truth answer set, and Section 3.1 (p. 4) reports QA accuracies as its evaluation results. This supports citing Kandpal et al. for using accuracy to evaluate question answering. (LLM, conf=0.90)
- Supportedcites Mallen et al. (2023) source
Section 3.1 (‘Focus and Task’) explicitly names accuracy as the metric and says a prediction is correct when it contains an exact match to any gold answer. This supports the evaluation approach attributed to Mallen et al.; the NaturalQuestions annotations describe the citing authors’ answer source. (LLM, conf=0.94)
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“or when language models are augmented with external information (relevant documents from a search engine, database query results, etc.; Petroni et al., 2020;Ram et al., 2023;Shi et al., 2023;Mallen et al., 2023;Schick et al., 2023, inter alia).”
§ Lost in the Middle: How Language Models Use Long Contexts > 1 Introduction
What the sources say
- Partly supportedcites Petroni et al., 2020 sourceExperimental — accuracy not yet measured
The Abstract (p. 1) says the paper integrates information from a retrieval system with a pre-trained language model, and §3.4.2 (p. 5) specifies a retrieved paragraph as context. This supports augmentation with retrieved documents, but our copy does not show language models being augmented with database query results. (LLM, conf=0.84)
- Partly supportedcites Ram et al., 2023 sourceExperimental — accuracy not yet measured
The Introduction describes augmenting language models with relevant documents retrieved from an external knowledge source, and Section 5 describes using a BM25 retriever to find those documents. The passages do not establish database query results as another type of external information used by this paper. (LLM, conf=0.82)
- Partly supportedcites Shi et al., 2023 sourceExperimental — accuracy not yet measured
Section 3.1 (page unavailable) describes retrieving relevant documents from a corpus, and the Conclusion says REPLUG uses those documents to augment a language model. The paper supports the document-retrieval example, but our copy does not describe augmenting models with database query results or documents from a search engine specifically. (LLM, conf=0.82)
- Partly supported
20 more claims · 2 with their cited sources checked
Create a free account to see all 25 claims and every source verdict.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In vanilla multi-document or NIAH-based benchmarks (Kamradt, 2023; Liu et al., 2024), models perform consistently well when the needle (or gold document) appears at the very beginning or end of the context window, with minimal impact from context length.”
From NoLiMa: Long-Context Evaluation Beyond Literal Matching (2025) · cites this paper as Liu et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Even within a single long sequence, attention effectiveness degrades with distance due to attention dilution , positional encoding limitations, and token interference, leading to the well-known 'lost-in-the-middle' and context-decay phenomena (Liu et al., 2024; Press et al., 2021a).”
From MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents (2026) · cites this paper as Liu et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current large language models (LLMs) exhibit significant deficiencies in episodic memory tasks including encoding, storing, and retrieving specific information from temporally dependent events over a long period of time.”
From ARTEM: Enhancing Large Language Model Agents with Spatial-Temporal Episodic Memory (2026) · cites this paper as Liu et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent approaches to handle memory tasks in LLMs, such as in-context learning, retrieval-augmented generation (RAG), and fine-tuning, may resolve the long-term retention issues, but are still inadequate to handle tasks requiring chronological awareness of the stored information.”
From ARTEM: Enhancing Large Language Model Agents with Spatial-Temporal Episodic Memory (2026) · cites this paper as Liu et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“If not ranked at the top, the key information may be neglected by the generative module, resulting in incomplete or inaccurate responses to the user queries24.”
From Leveraging long context in retrieval augmented language models for medical question answering (2025) · cites this paper as [24]
Experimental — accuracy not yet measured
86 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 2×
“Recent improvements in hardware (e.g., faster GPUs with more memory) and algorithms (Dai et al., 2019; Dao et al., 2022; Poli et al., 2023; Rubin and Berant, 2023, inter alia ) have resulted in language models with larger context windows (e.g., 4096, 32K, and even 100K tokens), but it remains…”
BackgroundLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2020) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2007.01282 · Cited by 2,263
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 1×
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019; Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
MethodThe serial position effect of free recall.
Murdock, B. B. (1962) · Journal of Experimental Psychology · doi:10.1037/h0045106 · Cited by 1,717 · cited 1×
“The U-shaped curve we observe in this work has a connection in psychology known as the serial-position effect (Ebbinghaus, 1913; Murdock Jr, 1962), that states that in free-association recall of elements from a list, humans tend to best re-member the first and last elements of the list.”
MethodMemory: A contribution to experimental psychology.
Ebbinghaus, H. (1913) · doi:10.1037/10011-000 · Cited by 1,645 · cited 1×
“The U-shaped curve we observe in this work has a connection in psychology known as the serial-position effect (Ebbinghaus, 1913; Murdock Jr, 1962), that states that in free-association recall of elements from a list, humans tend to best re-member the first and last elements of the list.”
Method
Research works printed but not linked: 1
- Sharan, 2018, Prediction with a short memory, 10.18653/v1/K18-3013 · DOI
Cited by
Items 201–300 of 1,595
Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory
Bulatov, A., Kuratov, Y., Kapushev, Y., & Burtsev, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29722 · Cited by 12
“Recurrent-based approaches, on the other, hand may be useful in complex step-by-step reasoning tasks, with specialized memory-intensive tasks or in cases where current models are limited (Liu et al. 2023).”
BackgroundConfigtrans: Network Configuration Translation Based on Large Language Models and Constraint Solving
Zheng, N., Li, F., Li, Z., Yang, Y., Hao, Y., Liu, C., & Wang, X. (2024) · IEEE International Conference on Network Protocols (ICNP) · doi:10.1109/icnp61940.2024.10858572 · Cited by 16
“However, LLMs have limitations on input text length [13], and their performance noticeably declines as the input text becomes longer [14].”
Compacting AI: In Search of the Small Language Model
Jovanovic, M., & Campbell, M. (2024) · Computer · doi:10.1109/mc.2024.3404674 · Cited by 17
A RAG-Based Institutional Assistant
Kuratomi, G., Pirozelli, P., Cozman, F. G., & Peres, S. M. (2024) · Anais do XXI Encontro Nacional de Inteligência Artificial e Computacional (ENIAC 2024) · doi:10.5753/eniac.2024.245243 · Cited by 24
Dong, Q., Ai, Q., Wang, H., Liu, Y., Li, H., Su, W., Liu, Y., Chua, T., & Ma, S. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714608 · Cited by 25
Stretching AI's reach: Assessing an AI-driven feedback system for extended academic writing
Lo, J., Wong, C., Ng, A., Wong, P., Cheung, D., & Lai, P. P. Y. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100511 · Cited by 28
Extracting Formal Specifications From Documents Using LLMS for Test Automation
Li, H., Dong, Z., Wang, S., Hui, Z., Shen, L., Peng, X., & She, D. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00039 · Cited by 21
“We attribute these deficiencies to the inherent limitations of LLM, including their limited processing and expressive capabilities [19], [20], as well as their propensity to hallucination [21].”
Agentic RAG with Human-in-the-Retrieval
Xu, X., Zhang, D., Liu, Q., Lu, Q., & Zhu, L. (2025) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c65153.2025.00074 · Cited by 22
Jiang, K., Jiang, J., Liu, X., Yao, H., & Lin, C. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3638153 · Cited by 20
“This necessitates multi-directional scans to recover spatial dependencies and global representation, introducing redundancy and inaccurate degradation modeling due to interference from unrelated tokens [15], [16].”
Annotation of biological samples data to standard ontologies with support from large language models
Riquelme-García, A., Mulero-Hernández, J., & Fernández‐Breis, J. T. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.05.020 · Cited by 14
“This observation aligns with recent findings emphasizing that the quality and relevance of contextual information often outweigh its sheer volume in large language model performance [41].”
Ludwig, H., Schmidt, T., & Kühn, M. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104289 · Cited by 20
Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach
Jiang, Z., Sun, M., Liang, L., & Zhang, Z. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3716889 · Cited by 75
“Firstly, due to multiple rounds of retrieval, iterative RAG methods have to deal with a longer context in contrast to single-round RAG methods, which consequently introduces more noise from the documents and increases the risk of the model missing key information during response generation (Liu et…”
Accurate and Interpretable Log-Based Fault Diagnosis Using Large Language Models
Sun, Y., Ma, S., Xiao, T., Zhao, Y., Cai, X., Dong, W., Shen, Y., Zhao, Y., Zhang, S., Han, J., et al. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3599494 · Cited by 19
“Although recent researche has focused on extending the context length of LLMs [18], [19], challenges persist due to the exponential increase in computational demands and accuracy degradation [20].”
War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars
Hua, W., Fan, L., Li, L., Mei, K., Ji, J., Ge, Y., Hemphill, L., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2311.17227 · Cited by 15
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 12
“The dataset’s data is sourced from the NaturalQuestions dataset, which contains historical queries issued to the Google search engine and human-annotated answers extracted from Wikipedia.”
BackgroundMethodInfluentialEvoR: Evolving Retrieval for Code Generation
Su, H., Jiang, S., Lai, Y., Wu, H., Shi, B., Liu, C., Liu, Q., & Yu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.143 · Cited by 12
“LLMs have low attention in the middle of long contexts (Liu et al., 2023c).”
Li, L., Mathrani, A., & Sušnjak, T. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2025.10065 · Cited by 28
“…that fabricate results, unacceptable in high-stakes applications like healthcare; propagation of implicit biases from training corpora into synthesized outputs; and limitations with extensive context windows when processing journal articles, dissertations, and complex figures/tables [91].”
Neurosymbolic Repair of Test Flakiness
Chen, Y., & Jabbarvand, R. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680369 · Cited by 17
“This will improve the performance of FlakyDoctor , as recent research shows that LLMs provide the best results when given fewer, more relevant contexts rather than larger, unfiltered ones [35].”
A Cooperative Multi-Agent Framework for Zero-Shot Named Entity Recognition
Wang, Z., Zhao, Z., Lyu, Y., Chen, Z., de Rijke, M., & Ren, Z. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714923 · Cited by 43
“Note that existing LLMs suffer from severe performance degradation with long input contexts [13, 21] and complex instruction following [33, 34].”
Report on the Search Futures Workshop at ECIR 2024
Azzopardi, L. A., Clarke, C. L. A., Kantor, P. B., Mitra, B., Trippas, J. R., Ren, Z., Aliannejadi, M., Arabzadeh, N., Chandrasekar, R., de Rijke, M., et al. (2024) · ACM SIGIR Forum · doi:10.1145/3687273.3687288 · Cited by 18
“Inspired by the study done by Liu et al. [2024], where it is shown that the position of the retrieved relevant results impacts the overall quality of the result generated by the LLM of a RAG system, we propose in [Cuconasu et al., 2024] to investigate the role of “noise” or unstructured, irrelevant…”
GPT Models Can Perform Thematic Analysis in Public Health Studies, Akin to Qualitative Researchers
Yang, Y., Alba, C., Wang, C., Wang, X., Anderson, J., & An, R. (2024) · Journal of Social Computing · doi:10.23919/jsc.2024.0024 · Cited by 15
Lafourcade, C., Kérourédan, O., Ballester, B., & Richert, R. (2025) · Journal of Dentistry · doi:10.1016/j.jdent.2025.105764 · Cited by 20
Road of Large Language Model: Source, Challenge, and Future Perspectives
Zhao, W., Yang, X. S., Lv, Z., Xu, C. X., & Guan, Z. (2025) · Research · doi:10.34133/research.0655 · Cited by 20
“Moreover, long contexts may lead to the “middle loss” problem [ 29 ], where LLMs always focus on the beginning and the end of texts, neglecting the middle sections.”
Large Language Models for In-File Vulnerability Localization Can Be “Lost in the End”
Sovrano, F., Bauer, A., & Bacchelli, A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715758 · Cited by 21
“However, with our experiments we empirically show that with LLMs, the most frequent weaknesses (e.g., CWE-79) are actually those less likely to be identified by the model compared to less frequent ones (e.g., CWE-22).”
RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code
Chen, J., Zhong, Q., Wang, Y., Ning, K., Liu, Y., Xu, Z., Zhao, Z., Chen, T., & Zheng, Z. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695480 · Cited by 70
“One of the reasons is the input in code-to-code scenarios has a longer context, which can divert the attention of LLMs and decrease their performance [42].”
BackgroundXiong, Y., Tu, X., & Zhao, W. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104232 · Cited by 17
Is Relevance Propagated from Retriever to Generator in RAG?
Tian, F., Ganguly, D., & Macdonald, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_3 · Cited by 33
Using Large Language Models to Generate Process Knowledge from Enterprise Content
Franzoi, S., Delwaulle, M., Dyong, J., Schäffner, J., Burger, M., & Brocke, J. V. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-78666-2_19 · Cited by 12
From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
Uddin, M. N., Shubham, K., Blanco, E., Baral, C., & Wang, G. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1337 · Cited by 11
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Becker, J., Rush, N., Barnes, E., & Rein, D. (2025) · arXiv · doi:10.48550/arxiv.2507.09089 · Cited by 16
“This would be consistent with existing literature studying the effects of environment complexity on AI performance [3; 38; 39].”
Revealing the Barriers of Language Agents in Planning
Xie, J., Zhang, K., Chen, J., Yuan, S., Zhang, K., Zhang, Y., Li, L., & Xiao, Y. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.93 · Cited by 11
Agent-DA: Enhancing low-resource event extraction with collaborative multi-agent data augmentation
Tian, X., Guo, Y., Ge, B., Yuan, X., Zhang, H., Yang, Y., Ke, W., & Li, G. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112625 · Cited by 13
Lehmann, J., Bhandiwad, D., Gattogi, P., & Vahdati, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00671 · Cited by 24
“MPT-30B-Instruct (Liu et al., 2024) is an open instruction-tuned language model.”
BackgroundDynamic and Parametric Retrieval-Augmented Generation
Su, W., Ai, Q., Zhan, J., Dong, Q., & Liu, Y. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3731692 · Cited by 20
ClassEval-T: Evaluating Large Language Models in Class-Level Code Translation
Xue, P., Wu, L., Yang, Z., Wang, C., Li, X., Zhang, Y., Li, J., Jin, R., Pei, Y., Shen, Z., et al. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728940 · Cited by 49
Cost-Efficient Domain-Adaptive Pretraining of Language Models for Optoelectronics Applications
Huang, D., & Cole, J. M. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c02029 · Cited by 13
Large language models in digital forensics: capabilities, challenges and future directions
Chernyshev, M., Baig, Z., Syed, N., Doss, R., & Shore, M. (2026) · Forensic Science International: Digital Investigation · doi:10.1016/j.fsidi.2025.302043 · Cited by 10
Statically Contextualizing Large Language Models with Typed Holes
Blinn, A., Li, X., Kim, J., & Omar, C. (2024) · Proc. ACM Program. Lang. · doi:10.1145/3689728 · Cited by 25
“Moreover, contemporary LLMs struggle to attend to relevant information and ignore irrelevant information (such as the Model and Action types for other GUI components) in large prompts [35, 37, 41, 79].”
Exploring knowledge poisoning attacks to retrieval-augmented generation
Zhao, T., Chen, J., Ru, Y., Zhu, H., Hu, N., Liu, J., & Lin, Q. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103900 · Cited by 37
Li, F., Li, X., Wen, S., Huang, H., & Bao, J. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104336 · Cited by 15
LLMs Get Lost In Multi-Turn Conversation
Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025) · arXiv · Cited by 419
“This finding mirrors findings of a loss-in-the-middle phenomena of LLMs paying more attention to documents at the start or end of their provided context, at the cost of middle-context content [29, 50, 40].”
Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents
Khanal, A., Tao, Y., & Zhou, J. (2026) · arXiv · doi:10.48550/arxiv.2603.29231 · Cited by 10
Supporting Human Raters with the Detection of Harmful Content Using Large Language Models
Thomas, K., Kelley, P. G., Tao, D. D. J., Meiklejohn, S., Vallis, O., Tan, S., Bratanič, B., Ferreira, F. T., Eranti, V., & Bursztein, E. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00082 · Cited by 29
“This may be a consequence of LLMs not giving enough attention to short violative passages in comments that are otherwise innocuous [30].”
Large Language Models for Information Retrieval: Challenges and Chances
Breuer, T., Frihat, S., Fuhr, N., Lewandowski, D., Schaer, P., & Schenkel, R. (2025) · Datenbank-Spektrum · doi:10.1007/s13222-025-00503-x · Cited by 27
Data augmented large language models for medical record generation
Zhang, X., Zhao, G., Ren, Y., Wang, W., Cai, W., Zhao, Y., Zhang, X., & Liu, J. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05934-9 · Cited by 13
“Such hallucinations will frequently occur when LLMs misunderstand the context or cannot maintain consistency in the whole conversation, becausetherearesomeconstraints for LLMs inkeepinglong-term memory [52] or related context identification [53].”
Legal Requirements Translation from Law
Singhal, A., & Breaux, T. D. (2025) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re63999.2025.00028 · Cited by 13
“Despite emergent properties like Chain-of-Thought [7], LLMs struggle with long-range dependencies [8], [12] and logical consistency [5], including those affecting implications [6], negation [4] and transitivity [11], making them unreliable for tasks requiring structured and factual outputs, such as…”
Alhuzali, A. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3371 · Cited by 27
Zero-Shot Large Language Models for Long Clinical Text Summarization with Temporal ReasoningPreprint
Kruse, M., Hu, S., Derby, N., Wu, Y., Stonbraker, S., Yao, B., Wang, D., Goldberg, E., & Gao, Y. (2025) · medRxiv · doi:10.1101/2025.07.21.25331947 · Cited by 19
Ojemann, W. K. S., Xie, K., Liu, K., Chang, E., Roth, D., Litt, B., & Ellis, C. A. (2025) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-025-00198-5 · Cited by 13
“All Llama2 prompts significantly outperformed the Bio + ClinicalBERT zero-shot model (25% [23%, 29%]), and performed significantly worse than the fully fine-tuned Bio + ClinicalBERT model (84% [83%, 84”
Tolomei, G., Campagnano, C., Silvestri, F., & Trappolini, G. (2023) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi58952.2023.00027 · Cited by 14
Yang, L., Chen, H., Wang, X., Yang, J. F., Wang, F., & Liu, H. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103587 · Cited by 35
5G Specifications Formal Verification with Over-the-Air Validation: Prompting is All You Need
Wray, T., & Wang, Y. N. (2024) · IEEE Military Communications Conference (MILCOM) · doi:10.1109/milcom61039.2024.10773849 · Cited by 11
“[25] We found it important when working with GPT-3.”
Asking Clarification Questions to Handle Ambiguity in Open-Domain QA
Lee, D., Kim, S., Lee, M., Lee, H., Park, J., Lee, S., & Jung, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.772 · Cited by 58
Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL
Chung, Y., Kakkar, G. T., Gan, Y., Milne, B., & Özcan, F. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3742728.3742761 · Cited by 50
“This raises concerns that the model might get confused or “lost in the middle” of a large context window [9, 24].”
InfluentialImproving RAG Quality for Large Language Models with Topic-Enhanced Reranking
Ampazis, N. (2024) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-63215-0_6 · Cited by 13
Ziegler, I., Köksal, A., Elliott, D., & Schütze, H. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.56 · Cited by 33
“…challenges, including inaccurate assessment of the underlying data distribution (Song et al., 2023), biases related to small sample sizes (Song et al., 2023), and sensitivity to shot length (Liu et al., 2024), shot quality and noise (Perez et al., 2021; Chang et al., 2021; Chen et al., 2022).”
Fault Localization from the Semantic Code Search Perspective
Qin, Y., Wang, S., Lei, Y., Zhang, Z., Lin, B., Peng, X., Ma, J., Chen, L., & Mao, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3757915 · Cited by 15
“” With these multi-granularity queries, the subsequent CS process is able to pinpoint the bug-related methods from the codebase.”
In the beginning was the Word: LLM-VaR and LLM-ES
Pele, D. T., Bolovăneanu, V., Lin, M.-B., Ren, R., Ginavar, A. T., Spilak, B., Andrei, A.-V., Toma, F.-M., Lessmann, S., & Härdle, W. K. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128676 · Cited by 17
Enhanced Retrieval-Augmented Generation Using Low-Rank Adaptation
Choi, Y., Kim, S., Bassole, Y. C. F., & Sung, Y. (2025) · Applied Sciences · doi:10.3390/app15084425 · Cited by 16
Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
Pradeep, R., Thakur, N., Sharifymoghaddam, S., Zhang, E., Nguyen, R., Campos, D., Craswell, N., & Lin, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_9 · Cited by 63
Memory$^3$: Language Modeling with Explicit Memory
Yang, H., Lin, Z., Wang, W., Wu, H., Li, Z., Tang, B., Wei, W., Wang, J., Tang, Z., Song, S., et al. (2024) · Journal of Machine Learning · doi:10.4208/jml.240708 · Cited by 90
Lee, J., Choi, Y., Song, M., & Park, S. (2024) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3640794.3665572 · Cited by 9
“Analysis of the conversation logs revealed this stemmed from losing middle contextual elements in longer conversations - common in other LLMs [25, 33].”
BackgroundPromptPilot: Exploring User Experience of Prompting with AI-Enhanced Initiative in LLMs
Kim, S., Eun, J., Park, Y. E., Lee, K., Lee, G., & Lee, J. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2489030 · Cited by 12
“Prompting, the act of providing textual instructions to LLMs, serves as the primary interface between users and AI systems (Liu et al., 2023).”
Logical and Physical Optimizations for SQL Query Execution over Large Language Models
Satriani, D., Veltri, E., Santoro, D., Rosato, S., Varriale, S., & Papotti, P. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3725411 · Cited by 8
“While research in tabular language models indicates that such a scenario is not yet feasible, primarily due to context size limitations [4] and its issues with long inputs [39], recent advancements are overcoming this constraint [16, 28, 41].”
Lesson planning with ChatGPT for inquiry-based biology instruction – A(I) roll of the dice?
Großmann, L., Koberstein‐Schwarz, M., Krüger, D., & Krell, M. (2025) · International Journal of Science Education · doi:10.1080/09500693.2025.2567509 · Cited by 18
“…robustly make use of information in long input contexts and performance degrades significantly when relevant information must be accessed in the middle of a long prompt (Liu et al., 2024), the relatively low performance on some of our ten criteria might also be explained by our prompt structure.”
Palanisamy, B., Chakrabarti, A., Singh, A., Hassija, V., Chalapathi, G., & Singh, A. R. (2024) · IEEE Access · doi:10.1109/access.2024.3521596 · Cited by 11
Link-Context Learning for Multimodal LLMs
Tai, Y., Fan, W., Zhang, Z., & Liu, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02566 · Cited by 37
“Similarly, this phenomenon also exists in LLMs [15], where the language model tends to be “lost in the middle” when processing long contexts.”
Operationalizing machine-assisted translation in healthcare
López, I., Velasquez, D. E., Chen, J. H., & Rodriguez, J. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01944-0 · Cited by 9
“For example, LLMs tend to “ hallucinate ” (generate plausible but factually incorrect text) 30 , lose context (dif fi culty retrieving and using information located in the middle of long passages) 31 , anddegradehumanaccountability(whereusersbecomereliantontheLLM andfailtoevaluateitsoutputs) 32 .”
Zhang, Y., Ouh, E. L., Ho, A., Lo, S. L., Tan, K. W., & Lin, F. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729095 · Cited by 15
The Few-shot Dilemma: Over-prompting Large Language Models
Tang, Y., Tuncel, D., Körner, C., & Runkler, T. A. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391015 · Cited by 36
“A study finds that LLMs perform optimally when accessing relevant information at the beginning or end of the input context, but their performance degrades significantly for information in the middle [20].”
Explainable and Interactive LLMs-Augmented Depression Detection in Social Media
Qin, W., Chen, Z., Yang, X., Wang, L., Lan, Y., Ren, W., & Hong, R. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3591755 · Cited by 13
XiYan-SQL: A Novel Multi-Generator Framework for Text-to-SQL
Liu, Y., Zhu, Y., Gao, Y., Luo, Z., Li, X., Shi, X., Hong, Y., Gao, J., Li, Y., Ding, B., et al. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3657851 · Cited by 75
“(cid:2) For SQL selection, similar to previous studies [37], [46], we have also found that LLMs, whether fine-tuned or not, exhibit sensitivity to the order and organization of candidates.”
Unifying Bias and Unfairness in Information Retrieval: New Challenges in the LLM Era
Dai, S., Chen, X., Xu, S., Pang, L., Dong, Z., & Xu, J. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703478 · Cited by 14
SiReRAG: Indexing Similar and Related Information for Multihop ReasoningPreprint
Zhang, N., Choubey, P. K., Fabbri, A. R., Bernadett-Shapiro, G., Zhang, R., Mitra, P., Xiong, C., & Wu, C.-S. (2024) · Qeios · doi:10.32388/93o2s3 · Cited by 37
Wang, T., Du, P., Dang, P., Liu, T., & Li, P. (2025) · International Journal of Digital Earth · doi:10.1080/17538947.2025.2468407 · Cited by 8
“Liu developed an interactive mechanism for updating knowledge graphs, allowing LLMs to dynamically adjust and expand the graphs on the basis of real-time fire scenarios and user feedback (Liu et al. 2024).”
Liu, J., Nguyen, A. N., Capurro, D., & Verspoor, K. M. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3574254 · Cited by 12
“This could be related to the finding that long-context LLMs are not always effective in handling extended context [99], and the model performance could decrease as the input context increases [96].”
ProveRAG: Provenance-Driven Vulnerability Analysis With Automated Retrieval-Augmented LLMs
Fayyazi, R., Trueba, S. H., Zuzak, M., & Yang, S. J. (2025) · IEEE Access · doi:10.1109/access.2025.3638251 · Cited by 27
“This problem of inability of LLMs to process massive amounts of information and identify relevant information can lead to less effective responses, as discussed in literature [16].”
BackgroundA Novel RAG Framework with Knowledge-Enhancement for Biomedical Question Answering
Du, Y., Wang, Z., Wang, B., Jin, X., & Yu, P. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822837 · Cited by 11
“In this study, we leverage research findings from the "lost in middle" [9] effect, which demonstrate that models perform better when relevant information is placed at the beginning or the end of the input sequence.”
Exploring the use of LLMs in the Italian legal domain: A survey on recent applications
Siino, M. (2025) · Computer law & security review · doi:10.1016/j.clsr.2025.106164 · Cited by 12
Language model collaboration for relation extraction from classical Chinese historical documents
Tang, X., Wang, L., & Wang, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104286 · Cited by 14
LLM-Based Prompt Ensemble for Reliable Medical Entity Recognition from EHRs
Islam, K. M. S., Nipu, A. S., Wu, J., & Madiraju, P. (2025) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri66576.2025.00037 · Cited by 10
“We designed the template to be concise and modular to address the issue of models forgetting information in the middle of long prompts [18].”
GINGER: Grounded Information Nugget-Based Generation of Responses
Łajewska, W., & Balog, K. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730166 · Cited by 27
Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelines
Barrak, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00064 · Cited by 32
“A prominent issue is the ”lost in the middle” problem, where models exhibit degraded performance in utilizing information positioned in the center of long context windows [14].”
Zhang, F., Luo, Y., Gao, Z., & Han, A. (2025) · International Journal of Law and Psychiatry · doi:10.1016/j.ijlp.2025.102070 · Cited by 8
Chemist-Guided Human–AI Workflow for Covalent Organic Framework Synthesis
Chen, L., Lu, Z., Chen, L., Hou, L., & Zhang, D. (2026) · Journal of the American Chemical Society · doi:10.1021/jacs.5c20068 · Cited by 13
Do We Need a Detailed Rubric for Automated Essay Scoring Using Large Language Models?
Yoshida, L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98465-5_8 · Cited by 20
iDesignGPT enhances conceptual design via large language model agentic workflows
Liu, S., Shen, Y., Zhang, Y., Hou, Z., Wang, X., Luo, J., & Zhang, Z. (2026) · Nature Communications · doi:10.1038/s41467-026-68672-1 · Cited by 30
“This omission re fl ects not only attention-related limitations 62 , but alsothemodel ’ sabstraction oftask-speci fi c items.”
Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
Zhao, S., Shao, Y., Huang, Y., Song, J., Wan, C., Wan, C., & Ма, Л. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3802824 · Cited by 49
Can LLMs Generate User Stories and Assess Their Quality?
Quattrocchi, G., Pasquale, L., Spoletini, P., & Baresi, L. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3670612 · Cited by 20
“US rated with a score 3 provide a clear and explicit rationale, making the actor’s intent fully transparent, as illustrated in the first example.”
ReceiptQA: A Question-Answering Dataset for Receipt Understanding
Abdalla, M. I., Kasem, M. S., Mahmoud, M., Yagoub, B., Senussi, M. F., Abdallah, A., Kang, S. H., & Kang, H. (2025) · Mathematics · doi:10.3390/math13111760 · Cited by 12
Wang, X., & Luo, G. (2024) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/wn3cd · Cited by 12
A Comprehensive Evaluation of Embedding Models and LLMs for IR and QA Across English and Italian
Oro, E., Granata, F. M., & Ruffolo, M. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9050141 · Cited by 15
“Liu et al. [30] revealed the “Lost in the Middle” problem, where models struggle to maintain attention across long input contexts—a critical limitation for QA systems processing extensive documents or integrating information from multiple sources.”
Enhancing next activity prediction in process mining with Retrieval-Augmented Generation
Casciani, A., Bernardi, M. L., Cimitile, M., & Marrella, A. (2025) · Information Systems · doi:10.1016/j.is.2025.102642 · Cited by 12
RAGE Against the Machine: Retrieval-Augmented LLM Explanations
Rorseth, J., Godfrey, P., Golab, L., Srivastava, D., & Szlichta, J. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00430 · Cited by 14
“Specific concerns about RAG, such as the lack of provenance in LLM answers, or the “lost in the middle” context position bias observed in recent LLMs [2], warrant dedicated study under an explainability lens.”
BackgroundHolistic AI in medicine; improved performance and explainability
Petridis, P., Margaritis, G., Stoumpou, V., & Bertsimas, D. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02298-3 · Cited by 17
“…in their training or inference, they can be prompted to provide summaries or post-hoc explanations, and have been used in the context of medical question answering 23 , document summarization 24 , trial matching 25 , and reducing hallucinations through retrieval-augmented generation 26,27 .”
Benchmarking large language models for automated labeling: The case of issue report classification
Colavito, G., Lanubile, F., & Novielli, N. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107758 · Cited by 18
“This is probably due to the fact that the prompt becomes too long, and the model struggles to focus on the relevant information and follow the instructions [48].”
Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
Zhang, S., Ma, R., Ma, Y., Li, S., Xu, Y., Yi, X., & Li, H. (2025) · Proceedings of the 2025 Workshop on Human-Centered AI Privacy and Security · doi:10.1145/3733816.3760750 · Cited by 6
“However, as conversations lengthened, agent performance degraded due to distraction and information loss—often called “lost in the middle” [21, 31].”
LLM-Aided Automatic Modeling for Security Protocol Verification
Mao, Z., Wang, J., Sun, J., Qin, S., & Xiong, J. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00197 · Cited by 22
Getting it right: the limits of fine-tuning large language models
Browning, J. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09779-1 · Cited by 13
Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents
Liu, Z., Hao, Z., Chen, Y., Wang, H., Hou, J., Ding, R., Yang, Y., Ji, W., Xia, W., & Liu, F. (2026) · arXiv · Cited by 12
Metadata
- Authors
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Samaya AI, UK
- Corresponding author
- Liu, Nelson F.
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 157-173
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4391876619
- Semantic Scholar
- 1733eb7792f7a43dd21f51f4d1017a1bffd217b5
- Cited by
- 1,612
- References
- 59
- Citations by year
- 2026: 901 · 2025: 516 · 2024: 172
- FWCI
- 233.95
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- long-context language models, multi-document question answering, position bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-27 (prompt extract-v2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.