Lost in the Middle: How Language Models Use Long Contexts
Open accessThe DOI matches its Crossref record; 48 of the 59 references it cites were found in a registry or an index; 15 of the 23 sources its claims cite were checked for support.
Abstract
While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.
Overview
CoChat Summary
TL;DR
Language models’ performance on tasks requiring access to long input contexts is strongly affected by where relevant information appears, with a U-shaped curve: best at the beginning or end and worst in the middle. Extended-context models are not necessarily better at using context, and reader performance saturates before retriever recall in open-domain QA.
Key findings
- In multi-document QA, performance is highest when the answer-containing document is at the start or end of the context and degrades in the middle; GPT-3.5-Turbo can drop by more than 20%, and in 20- and 30-document settings middle-position performance is lower than its closed-book performance of 56.1% (oracle: 88.3%).
- Extended-context models are not necessarily better at using input context: when the input fits both windows, GPT-3.5-Turbo and GPT-3.5-Turbo(16K) perform nearly identically as a function of relevant-information position.
- In synthetic key-value retrieval, Claude-1.3 and Claude-1.3(100K) do nearly perfectly, while other models struggle especially with 140 or 300 key-value pairs; query-aware contextualization gives GPT-3.5-Turbo(16K) perfect performance at 300 pairs, versus worst-case 45.6% without it.
- In open-domain QA, reader performance saturates long before retriever recall: using 50 instead of 20 retrieved documents only marginally improves performance (∼1.5% for GPT-3.5-Turbo and ∼1% for Claude-1.3).
Methods
- Controlled multi-document QA on NaturalQuestions-Open: 2,655 queries whose annotated long answer is a paragraph; each context has exactly one answer-containing Wikipedia passage plus k−1 Contriever/MS-MARCO distractor passages; the paper varies answer position and context length (10, 20, 30 documents) and measures accuracy.
- Synthetic key-value retrieval: inputs are serialized JSON objects with unique random UUID key-value pairs; models must return the value for a specified key; experiments use 75, 140, and 300 key-value pairs (500 examples each), varying key position and number of pairs, with accuracy as the metric.
- Models include open MPT-30B-Instruct and LongChat-13B(16K), closed GPT-3.5-Turbo, GPT-3.5-Turbo(16K), Claude-1.3, and Claude-1.3(100K), plus additional architecture/instruction experiments with Flan-T5-XXL, Flan-UL2, MPT-30B, Llama-2, and GPT-4; an open-domain QA case study uses a Contriever retriever-reader setup.
Limitations
- Stated: the paper uses greedy decoding and leaves exploration of other decoding methods to future work; the investigations into why models are not robust are described as preliminary; GPT-4 is evaluated only on a subset of 500 multi-document QA examples because full evaluation would cost upwards of $6000.
- Stated: NaturalQuestions-Open has ambiguity, so the authors run an unambiguous subset and also test random distractors and randomized distractor order, finding similar trends.
- Evident: the controlled multi-document QA setting always contains exactly one document that answers the question, unlike open-domain QA; the synthetic key-value task removes natural language semantics; results may not generalize to all tasks, models, or decoding settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 0 Not valid · 5 Not checked · 13 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Language models have become an important and flexible building block in a variety of user-facing language technologies, including conversational interfaces, search and summarization, and collaborative writing (Shuster et al., 2022;Thoppilan et al., 2022;Lee et al., 2022, inter alia).”
§ Lost in the Middle: How Language Models Use Long Contexts > 1 Introduction
What the sources say
- Partly supportedcites Shuster et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.1.1 (p. 4) describes a conversational agent built on a language model that generates internet search queries, supporting conversational interfaces and search as part of dialogue. Section 2 (p. 3) mentions summarization only as related work, and the passages do not establish collaborative writing as an application of this paper’s model. (LLM, conf=0.85)
- Supportedcites Thoppilan et al., 2022 source
The Abstract (p. 1) identifies LaMDA as a family of language models specialized for dialog. In this sentence's list of applications, Thoppilan et al. supports the conversational-interface example; the other cited works need not each cover every example. (LLM, conf=0.92)
- Supportedcites Lee et al., 2022 source
Lee et al. is cited alongside other works, and its contribution to the sentence is the collaborative-writing example. Section 4.2.2 describes writers receiving and selecting model-generated suggestions in a writing interface; Section 2.2.1 also describes language models as applicable to dialogue and summarization. The paper need not establish every example in the jointly cited list. (LLM, conf=0.89)
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These observations complement prior work, which found that non-instruction fine-tuned language models are biased towards recent tokens (i.e., the end of the input context; Khandelwal et al., 2018;Press et al., 2021).”
§ Lost in the Middle: How Language Models Use Long Contexts > 4.3 Effect of Instruction Fine-Tuning
What the sources say
- Partly supportedcites Khandelwal et al., 2018 sourceExperimental — accuracy not yet measured
The Introduction and Section 5 report that the studied LSTM is much more sensitive to recent word order than to distant word order, supporting a recency effect. The paper studies a standard LSTM on two datasets, however, rather than establishing this tendency across non-instruction-fine-tuned language models generally; it also finds that distant words remain relevant. (LLM, conf=0.82)
- Partly supportedcites Press et al., 2021 sourceExperimental — accuracy not yet measured
Section 3.1 (page not available in our copy) finds that very short contexts hurt performance, while increasing subsequence length from 1,024 to 3,072 tokens gives negligible improvement. That is consistent with limited benefit from more distant context, but the paper does not establish that the models are biased toward recent tokens within an input context. (LLM, conf=0.78)
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
§ Lost in the Middle: How Language Models Use Long Contexts > 2.1 Experimental Setup
What the sources say
- Supportedcites Lee et al., 2019 source
Section 7.1 says the paper creates an open version of Natural Questions and uses it for training and evaluation. The Introduction describes the task as answering questions by retrieving evidence from an open corpus, supporting the cited use of NaturalQuestions-Open for this task. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites Kwiatkowski et al., 2019 source
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Following Kandpal et al. (2022) and Mallen et al. (2023), we use accuracy as our primary evaluation metric, judging whether any of the cor- rect answers (as taken from the NaturalQuestions annotations) appear in the predicted output.”
§ Lost in the Middle: How Language Models Use Long Contexts > 2.1 Experimental Setup
What the sources say
- Supportedcites Kandpal et al. (2022) source
Section 3 (p. 4) says the paper evaluates answers using exact match against a ground-truth answer set, and Section 3.1 (p. 4) reports QA accuracies as its evaluation results. This supports citing Kandpal et al. for using accuracy to evaluate question answering. (LLM, conf=0.90)
- Supportedcites Mallen et al. (2023) source
Section 3.1 (‘Focus and Task’) explicitly names accuracy as the metric and says a prediction is correct when it contains an exact match to any gold answer. This supports the evaluation approach attributed to Mallen et al.; the NaturalQuestions annotations describe the citing authors’ answer source. (LLM, conf=0.94)
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“or when language models are augmented with external information (relevant documents from a search engine, database query results, etc.; Petroni et al., 2020;Ram et al., 2023;Shi et al., 2023;Mallen et al., 2023;Schick et al., 2023, inter alia).”
§ Lost in the Middle: How Language Models Use Long Contexts > 1 Introduction
What the sources say
- Partly supportedcites Petroni et al., 2020 sourceExperimental — accuracy not yet measured
The Abstract (p. 1) says the paper integrates information from a retrieval system with a pre-trained language model, and §3.4.2 (p. 5) specifies a retrieved paragraph as context. This supports augmentation with retrieved documents, but our copy does not show language models being augmented with database query results. (LLM, conf=0.84)
- Partly supportedcites Ram et al., 2023 sourceExperimental — accuracy not yet measured
The Introduction describes augmenting language models with relevant documents retrieved from an external knowledge source, and Section 5 describes using a BM25 retriever to find those documents. The passages do not establish database query results as another type of external information used by this paper. (LLM, conf=0.82)
- Partly supportedcites Shi et al., 2023 sourceExperimental — accuracy not yet measured
Section 3.1 (page unavailable) describes retrieving relevant documents from a corpus, and the Conclusion says REPLUG uses those documents to augment a language model. The paper supports the document-retrieval example, but our copy does not describe augmenting models with database query results or documents from a search engine specifically. (LLM, conf=0.82)
- Partly supported
20 more claims · 2 with their cited sources checked
Create a free account to see all 25 claims and every source verdict.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In vanilla multi-document or NIAH-based benchmarks (Kamradt, 2023; Liu et al., 2024), models perform consistently well when the needle (or gold document) appears at the very beginning or end of the context window, with minimal impact from context length.”
From NoLiMa: Long-Context Evaluation Beyond Literal Matching (2025) · cites this paper as Liu et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Even within a single long sequence, attention effectiveness degrades with distance due to attention dilution , positional encoding limitations, and token interference, leading to the well-known 'lost-in-the-middle' and context-decay phenomena (Liu et al., 2024; Press et al., 2021a).”
From MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents (2026) · cites this paper as Liu et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current large language models (LLMs) exhibit significant deficiencies in episodic memory tasks including encoding, storing, and retrieving specific information from temporally dependent events over a long period of time.”
From ARTEM: Enhancing Large Language Model Agents with Spatial-Temporal Episodic Memory (2026) · cites this paper as Liu et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent approaches to handle memory tasks in LLMs, such as in-context learning, retrieval-augmented generation (RAG), and fine-tuning, may resolve the long-term retention issues, but are still inadequate to handle tasks requiring chronological awareness of the stored information.”
From ARTEM: Enhancing Large Language Model Agents with Spatial-Temporal Episodic Memory (2026) · cites this paper as Liu et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“If not ranked at the top, the key information may be neglected by the generative module, resulting in incomplete or inaccurate responses to the user queries24.”
From Leveraging long context in retrieval augmented language models for medical question answering (2025) · cites this paper as [24]
Experimental — accuracy not yet measured
86 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 2×
“Recent improvements in hardware (e.g., faster GPUs with more memory) and algorithms (Dai et al., 2019; Dao et al., 2022; Poli et al., 2023; Rubin and Berant, 2023, inter alia ) have resulted in language models with larger context windows (e.g., 4096, 32K, and even 100K tokens), but it remains…”
BackgroundLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2020) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2007.01282 · Cited by 2,263
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 1×
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019; Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
MethodThe serial position effect of free recall.
Murdock, B. B. (1962) · Journal of Experimental Psychology · doi:10.1037/h0045106 · Cited by 1,717 · cited 1×
“The U-shaped curve we observe in this work has a connection in psychology known as the serial-position effect (Ebbinghaus, 1913; Murdock Jr, 1962), that states that in free-association recall of elements from a list, humans tend to best re-member the first and last elements of the list.”
MethodMemory: A contribution to experimental psychology.
Ebbinghaus, H. (1913) · doi:10.1037/10011-000 · Cited by 1,645 · cited 1×
“The U-shaped curve we observe in this work has a connection in psychology known as the serial-position effect (Ebbinghaus, 1913; Murdock Jr, 1962), that states that in free-association recall of elements from a list, humans tend to best re-member the first and last elements of the list.”
Method
Research works printed but not linked: 1
- Sharan, 2018, Prediction with a short memory, 10.18653/v1/K18-3013 · DOI
Cited by
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Walter, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00448-3 · Cited by 1,113
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876
“A notable issue, termed the “lost in the middle” phenomenon, arises when a single retrieval yields redundant content that may dilute or contradict essential information, thereby degrading the generation quality [ Liu et al. , 2023a ] .”
BackgroundEvaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798
“The sensitivity of LLMs to the order of presented information has been well documented on multiple-choice questions 45,46 and information retrieval 47 .”
RAGAs: Automated Evaluation of Retrieval Augmented Generation
Es, S., James, J., Espinosa Anke, L., & Schockaert, S. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2024.eacl-demo.16 · Cited by 651
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534
“For instance, Li et al. (2023e) discuss the object hallucination of LVLMs, wherein LVLMs generate content containing objects that are inconsistent with or absent from the input image, such as the example in Figure 7.”
InfluentialCode Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531
“Liu et al. (2023b) showed that the inability to recall content placed in the middle of long prompts is a common failure mode in LLMs; our retrieval task is analogous to their setup, albeit tailored to code models which are not fine-tuned to follow instructions.”
BackgroundRAFT: Adapting Language Model to Domain Specific RAG
Zhang, T., Patil, S. G., Jain, N., Shen, S., Zaharia, M., Stoica, I., & Gonzalez, J. E. (2024) · arXiv · doi:10.48550/arxiv.2403.10131 · Cited by 286
“Instead of providing the correct name, it mistakenly cites one of the films written by the screenwriter.”
BackgroundEncouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.992 · Cited by 1,062
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1601 · Cited by 264
Query Rewriting in Retrieval-Augmented Large Language Models
Ma, X., Gong, Y., He, P., Zhao, H., & Duan, N. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.322 · Cited by 263
The power of noise: Redefining retrieval for rag systems
Cuconasu, F., Trappolini, G., Siciliano, F., Filice, S., Campagnano, C., Maarek, Y., Tonellotto, N., & Silvestri, F. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657834 · Cited by 236
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
“In retrieval-augmented settings, Liu et al. (2023e) highlighted a related challenge.”
Graph Retrieval-Augmented Generation: A Survey
Peng, B., Zhu, Y., Liu, Y., Bo, X., Shi, H., Hong, C., Zhang, Y., & Tang, S. (2026) · ACM Transactions on Information Systems · doi:10.1145/3777378 · Cited by 206
“The development of Large Language Models like GPT-4 [ 116 ],”
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732039 · Cited by 204
“We identify practical and foundational barriers to using LLMs-as-therapists.”
An evaluation framework for clinical use of large language models in patient interaction tasks
Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03328-5 · Cited by 201
Zhao, S., Yang, Y., Wang, Z., He, Z., Qiu, L. K., & Qiu, L. (2024) · arXiv · doi:10.48550/arxiv.2409.14924 · Cited by 180
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 172
“User-centric Reliable Response Search Query”
A survey on knowledge-oriented retrieval-augmented generation
Cheng, M., Luo, Y., Ouyang, J., Liu, Q., Liu, H., Li, L., Yu, S., Zhang, B., Cao, J., Ma, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.10677 · Cited by 140
Large language models for reticular chemistry
Zheng, Z., Rampal, N., Inizan, T. J., Borgs, C., Chayes, J. T., & Yaghi, O. M. (2025) · Nature Reviews Materials · doi:10.1038/s41578-025-00772-8 · Cited by 134
“For example, model performance tends to be highest when the relevant information is located at the beginning or end of the input prompt, whereas it substantially declines when the model needs to retrieve details embedded in the middle 56 .”
GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2023) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420231201154 · Cited by 120
GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2022) · doi:10.1101/2022.10.10.511571 · Cited by 120
Defining intelligence: Bridging the gap between human and artificial perspectives
Gignac, G. E., & Szodorai, E. T. (2024) · Intelligence · doi:10.1016/j.intell.2024.101832 · Cited by 122
“…focusing on the most recent or immediately relevant data for current tasks, and the mechanism of forgetting older information as new data comes in. Interestingly, LLM ’ s appear to manifest primacy and recency effects in a manner similar to that observed in humans (Liu, Lin, et al., 2023).”
Personality Traits in Large Language ModelsPreprint
Serapio‐García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Abdulhai, M., Faust, A., & Matarić, M. J. (2023) · Research Square · doi:10.21203/rs.3.rs-3296728/v1 · Cited by 137
A comprehensive evaluation of large Language models on benchmark biomedical text processing tasks
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. X. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108189 · Cited by 189
Exploring LLMs Applications in Law: A Literature Review on Current Legal NLP Approaches
Siino, M., Falco, M., Croce, D., & Rosso, P. (2025) · IEEE Access · doi:10.1109/access.2025.3533217 · Cited by 230
DRG-LLaMA : tuning LLaMA model to predict diagnosis-related group for hospitalized patients
Wang, H., Gao, C., Dantona, C., Hull, B., & Sun, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00989-3 · Cited by 99
“Interestingly, a recent study revealed that the optimal performance of LLMs is attained when pertinent information is positioned at either the beginning or the end of the input context, with a decline as the input context expands (Liu et al. 2023).”
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.91 · Cited by 400
Zhang, C., Liu, X., Ziska, K., Jeon, S., Yu, C.-L., & Xu, Y. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642647 · Cited by 96
Knowledge Graph-Guided Retrieval Augmented Generation
Zhu, X., Xie, Y., Liu, Y., Li, Y., & Hu, W. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.449 · Cited by 91
Ntinopoulos, V., Biefer, H. R. C., Tudorache, I., Papadopoulos, N., Odavic, D., Risteski, P. S., Haeussler, A., & Dzemali, O. (2025) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2024-101139 · Cited by 130
“OD: conceptualisation, resources, writing – review and editing, supervision, project administration, guarantor.”
Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?
Chen, Y., Arkin, J., Zhang, Y., Roy, N., & Fan, C. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610676 · Cited by 261
MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation
Qian, H., Liu, Z., Zhang, P., Mao, K., Lian, D., Dou, Z., & Huang, T. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714805 · Cited by 86
Large language models for scientific discovery in molecular property prediction
Zheng, Y., Koh, H. Y., Ju, J., Nguyen, A. T. N., May, L. T., Webb, G. I., & Pan, S. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00994-z · Cited by 253
“Even for models that support large input, they often cannot use the long context input effectively in practice 36,37 .”
Leveraging long context in retrieval augmented language models for medical question answering
Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01651-w · Cited by 81
“The quality of RAG completion also depends on the retrieval results, such as the density or positions of query-relevant information 14,22,24 – 26 .”
InfluentialMitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-Based Retrofitting
Guan, X., Liu, Y., Lin, H., Lu, Y., He, B., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29770 · Cited by 212
Language Model Crossover: Variation through Few-Shot Prompting
Meyerson, E., Nelson, M. J., Bradley, H., Gaier, A., Moradi, A., Hoover, A. K., & Lehman, J. (2024) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3694791 · Cited by 75
“Figure 4 shows the heritability of LMX in the binary strings domain, and Figure 5 shows how fitness evolves in OneMax using LMX.”
VideoAgent: Long-Form Video Understanding with Large Language Model as Agent
Wang, X., Zhang, Y., Zohar, O., & Yeung, S. Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72989-8_4 · Cited by 484
Language model crossover: Variation through few-shot prompting
Meyerson, E., Nelson, M. J., Bradley, H., Gaier, A., Moradi, A., Hoover, A. K., & Lehman, J. (2023) · arXiv · doi:10.48550/arxiv.2302.12170 · Cited by 75
Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Adlakha, V., BehnamGhader, P., Lu, X. H., Meade, N., & Reddy, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00667 · Cited by 74
“Finally, using metrics that correlate highly with human judgments, we evaluate instruction-following models for correctness (§4.3).”
MethodInfluentialDense X Retrieval: What Retrieval Granularity Should We Use?
Chen, T., Wang, H., Chen, S., Yu, W., Ma, K., Zhao, X., Zhang, H., & Yu, D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.845 · Cited by 74
Clinical entity augmented retrieval for clinical information extraction
López, I., Swaminathan, A., Vedula, K. S., Narayanan, S., Haredasht, F. N., Ma, S. P., Liang, A. S., Tate, S., Maddali, M. V., Gallo, R. J., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01377-1 · Cited by 148
LLM in the Shell: Generative Honeypots
Sladić, M., Valeros, V., Catania, C., & Garcia, S. (2024) · 2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) · doi:10.1109/eurospw61312.2024.00054 · Cited by 70
“False Discovery Rate = 𝐹𝑃 + 𝑇𝑃 (4)”
BackgroundA Survey on Retrieval-Augmented Text Generation for Large Language Models
Huang, Y., & Huang, J. X. (2026) · ACM Computing Surveys · doi:10.1145/3805774 · Cited by 348
Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
Cao, C., Sang, J., Arora, R., Chen, D., Kloosterman, R., Cecere, M., Gorla, J., Saleh, R., Drennan, I., Teja, B., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-24-02189 · Cited by 91
“Finally, for lengthy documents (that is, full-text articles), our ISO-ScreenPrompt approach addresses the lost-in-the-middle phenomenon (42 – 44) by repeating and numbering instructions at the start and end of the prompt (43).”
Wen, C., Cao, J., Su, J., Xu, Z., Qin, S., He, M., Li, H., Cheung, S.-C., & Tian, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65630-9_16 · Cited by 187
An advanced retrieval-augmented generation system for manufacturing quality control
Heredia, J. A., & Barreda, J. G. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.103007 · Cited by 93
Application of Large Language Models to DDoS Attack Detection
Guastalla, M., Li, Y., Hekmati, A., & Krishnamachari, B. (2024) · Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-51630-6_6 · Cited by 65
“Context Distribution Following the conclusions of [14] we theorized that the decrease in performance for our top-k method as k grows sufficiently large, could have been due to the context growing too large, causing the most relevant data to become “lost” in the context.”
BackgroundMethodLarge Language Model Applications for Health Information Extraction in Oncology: Scoping Review
Chen, D., Alnassar, S. A., Avison, K. E., Huang, R. S., & Raman, S. (2025) · JMIR Cancer · doi:10.2196/65984 · Cited by 66
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Jin, C., Zhang, Z., Jiang, X., Liu, F., Liu, S., Liu, X., & Jin, X. (2026) · ACM Transactions on Computer Systems · doi:10.1145/3768628 · Cited by 61
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Jin, C., Zhang, Z., Jiang, X., Liu, F., Liu, X., Liu, X., & Jin, X. (2024) · arXiv · doi:10.48550/arxiv.2404.12457 · Cited by 61
Foundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 183
Hybrid SLM and LLM for Edge-Cloud Collaborative Inference
Hao, Z., Jiang, H., Jiang, S., Ren, J., & Cao, T. (2024) · doi:10.1145/3662006.3662067 · Cited by 78
MemGPT: Towards LLMs as Operating Systems
Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2310.08560 · Cited by 64
“Its ability to satisfy these criteria will demonstrate whether unbounded context provides meaningful benefits for conversational agents.”
BackgroundSelf-citationSWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models
Wang, Q., Fu, Y., Cao, Y., Wang, S., Tian, Z., & Ding, L. (2023) · arXiv · doi:10.48550/arxiv.2308.15022 · Cited by 55
GPTCoach: Towards LLM-Based Physical Activity Coaching
Jörke, M., Sapkota, S., Warkenthien, L., Vainio, N., Schmiedmayer, P., Brunskill, E., & Landay, J. A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713819 · Cited by 141
A Survey of Personalization: From RAG to Agent
Li, X., Jia, P., Xu, D., Wen, Y., Zhang, Y., Zhang, W., Wang, W., Wang, Y., Du, Z., Li, X., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.10147 · Cited by 52
Technological folie à deux: feedback loops between AI chatbots and mental health
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00595-8 · Cited by 51
“In this Perspective, we argue that the psychological risks of chatbot use cannot be explained by a narrow consideration of chatbot limitations alone 31–37 .”
Contextual feature extraction hierarchies converge in large language models and the brain
Mischler, G., Li, Y. A., Bickel, S., Mehta, A. D., & Mesgarani, N. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00925-4 · Cited by 110
“…LLMs have been shown to suffer from unequal contextual information extraction when the prior context occurs at different distances from the target [55], supporting the notion that improving the robustness of modern LLMs to varying context lengths may lead to performance improvements.”
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang, Y., Lin, C.-Y., et al. (2024) · Findings of the Association for Computational Linguistics: ACL 2024 · doi:10.18653/v1/2024.findings-acl.57 · Cited by 58
Sushil, M., Zack, T. I., Mandair, D., Zheng, Z., Wali, A., Yu, Y.-N., Quan, Y., Lituiev, D., & Butte, A. J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae146 · Cited by 76
“However, it has been demonstrated earlier that the GPT-4 model cannot process long input contexts efficiently, 40 and we leave this Although we found promising performance of the GPT-4 model compared to task-specific supervised models, several design choices may have impacted the findings.”
ResultChain of Agents: Large Language Models Collaborating on Long-Context Tasks
Zhang, Y., Sun, R., Chen, Y., Pfister, T., Zhang, R., & Arik, S. Ö. (2024) · arXiv · doi:10.48550/arxiv.2406.02818 · Cited by 47
“To assess the “lost-in-the-middle” [42] effect on Vanilla and CoA models, we replicated the original study by randomly selecting 500 samples from their dataset to create a QA dataset.”
Automatic Programming: Large Language Models and Beyond
Lyu, M. R.-T., Ray, B., Roychoudhury, A., Tan, S. H., & Thongtanunam, P. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708519 · Cited by 131
VisEval: A Benchmark for Data Visualization in the Era of Large Language Models
Chen, N., Zhang, Y., Xu, J., Ren, K., & Yang, Y. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456320 · Cited by 154
Language agents achieve superhuman synthesis of scientific knowledge
Skarlinski, M. D., Cox, S., Laurent, J. M., Braza, J. D., Hinks, M., Hammerling, M. J., Ponnapati, M., Rodriques, S. G., & White, A. D. (2024) · arXiv · doi:10.48550/arxiv.2409.13740 · Cited by 49
“This is a balance because increasing the count improves the chance of including the key context needed to answer a question (improving accuracy), but also increases the amount of distracting irrelevant context that reduces precision 24 .”
BackgroundMLVU: Benchmarking Multi-task Long Video Understanding
Zhou, J., Shu, Y., Zhao, B., Wu, B., Liang, Z., Xiao, S., Qin, M., Yang, X., Xiong, Y., Zhang, B., et al. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.01278 · Cited by 44
Kwon, T., Ong, K. T.-I., Kang, D.-J., Moon, S., Lee, J. R., Hwang, D., Sohn, B., Sim, Y., Lee, D. H., & Yeo, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29802 · Cited by 110
Can Large Language Models Understand Real-World Complex Instructions?
He, Q., Zeng, J., Huang, W., Chen, L., Xiao, J., He, Q., Zhou, X., Chen, L., Wang, X., Huang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.09150 · Cited by 43
Generative AI for Requirements Engineering: A Systematic Literature Review
Cheng, H., Husen, J. H., Lu, Y., Racharak, T., Yoshioka, N., Ubayashi, N., & Washizaki, H. (2025) · Software Practice and Experience · doi:10.1002/spe.70029 · Cited by 164
Park, J., Fang, Y., Ta, C., Zhang, G., Idnay, B., Chen, F., Feng, D. D., Shyu, R., Gordon, E. R., Spotnitz, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104649 · Cited by 66
Liu, S., Wright, A. P., McCoy, A. B., Huang, S. S., Steitz, B., & Wright, A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf059 · Cited by 44
“27 Incorporating a knowledge graph as part of the RAG approach provided a structured way to represent relationships between medical concepts, which in turn enhanced the LLM’s inference capabilities.”
Large language models in urban planning
Fu, X., Li, C., Quan, S. J., Yiğitcanlar, T., & Wasserman, D. (2025) · Nature Cities · doi:10.1038/s44284-025-00261-7 · Cited by 55
“Although existing LLM systems have demonstrated efficiency in processing and extracting lengthy, complex and unstructured text data, they generally struggle with long-context understanding 53 and precise, factual information retrieval and generation 54 .”
Yan, L., Zhao, L., Echeverria, V., Jin, Y., Alfredo, R., Li, X., Gaševi’c, D., & Martinez-Maldonado, R. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-64299-9_13 · Cited by 40
Recursively summarizing enables long-term dialogue memory in large language models
Wang, Q., Fu, Y., Cao, Y., Wang, S., Tian, Z., & Ding, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130193 · Cited by 55
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
Du, Y., Tian, M., Ronanki, S., Rongali, S., Bodapati, S. B., Galstyan, A., Wells, A., Schwartz, R., Huerta, E. A., & Peng, H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1264 · Cited by 39
Building AI Agents for Autonomous Clouds: Challenges and Design Principles
Shetty, M., Chen, Y., Somashekar, G., Ma, M., Simmhan, Y. L., Zhang, X., Mace, J., Vandevoorde, D., Las-Casas, P., Gupta, S. M., et al. (2024) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3698038.3698525 · Cited by 104
Large language model for knowledge synthesis and AI-enhanced biomanufacturing
Li, W., Mao, Z., Xiao, Z., Liao, X., Koffas, M. A. G., Chen, Y., Ma, H., & Tang, Y. (2025) · Trends in biotechnology · doi:10.1016/j.tibtech.2025.02.008 · Cited by 55
Bjelobaba, S., Waddington, L., Perkins, M., Foltýnek, T., Bhattacharyya, S., & Weber-Wulff, D. (2025) · International Journal for Educational Integrity · doi:10.1007/s40979-025-00191-w · Cited by 120
Breitung, C., & Müller, S. (2025) · Journal of Financial Economics · doi:10.1016/j.jfineco.2025.104007 · Cited by 52
Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking
Loukas, L., Stogiannidis, I., Diamantopoulos, O., Malakasiotis, P., & Vassos, S. (2023) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3604237.3626891 · Cited by 93
“5 getting lost in the middle of long contexts [22].”
BackgroundIntegrating large language models with internet of things: applications
Zong, M., Hekmati, A., Guastalla, M., Li, Y., & Krishnamachari, B. (2025) · Discover Internet of Things · doi:10.1007/s43926-024-00083-4 · Cited by 63
“Specifically, if the model is able to dive into a complete project and read through all the documents, we will be able to test its limit on complex tasks such as test case generation and module integration.”
Interactive Debugging and Steering of Multi-Agent AI Systems
Epperson, W., Bansal, G., Dibia, V. C., Fourney, A., Gerrits, J., Zhu, E. (., & Amershi, S. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713581 · Cited by 35
Hassan, A. E., Lin, D., Rajbahadur, G. K., Gallaba, K., Cogo, F. R., Chen, B., Zhang, H., Thangarajah, K., Oliva, G. A., Lin, J., et al. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663849 · Cited by 81
Baek, J., Jauhar, S. K., Cucerzan, S., & Hwang, S. J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.342 · Cited by 46
Retrieval-Augmented Generation with Graphs (GraphRAG)
Han, H., Wang, Y., Shomer, H., Guo, K., Ding, J., Lei, Y., Halappanavar, M., Rossi, R. A., Mukherjee, S., Tang, X., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.00309 · Cited by 34
Croxford, E., Gao, Y., Pellegrino, N., Wong, K., Wills, G. J., First, E., Schnier, M., Burton, K. A., Ebby, C. G., Gorski, J. K., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf068 · Cited by 68
The mystery of In-context learning: a comprehensive survey on interpretation and analysis
Zhou, Y., Li, J., Xiang, Y., Yan, H., Gui, L., & He, Y. (2023) · arXiv · doi:10.48550/arxiv.2311.00237 · Cited by 33
Exploring the Effectiveness of LLMs in Automated Logging Statement Generation: An Empirical Study
Li, Y., Huo, Y., Jiang, Z., Zhong, R., He, P., Su, Y., Briand, L., & Lyu, M. R.-T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3475375 · Cited by 90
“Additionally, LLM performance tends to decline with longer inputs, even when within the specified context length [90], [91].”
Zhao, Q., Wang, R., Cen, Y., Zha, D., Tan, S., Dong, Y., & Tang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1259 · Cited by 32
Applications of Generative AI in Healthcare: algorithmic, ethical, legal and societal considerationsPreprint
Okonji, O. R., Yunusov, K., & Gordon, B. (2024) · doi:10.36227/techrxiv.171527587.75649430/v1 · Cited by 58
“…related to this is scenarios where the LLM fails to take cognizance of salient information providing relevant context in the input data, latest research [18] has shown the fallibility in LLMs whereby they are unable to keep track and take note of information placed deep within a user query.”
BackgroundGPT-5 and open-weight large language models: Advances in reasoning, transparency, and control
León, M. (2025) · Information Systems · doi:10.1016/j.is.2025.102620 · Cited by 88
Synthesizing scientific literature with retrieval-augmented language models
Asai, A., He, J., Shao, R., Shi, W., Singh, A., Chang, J. C., Lo, K., Soldaini, L., Feldman, S., D’Arcy, M., et al. (2026) · Nature · doi:10.1038/s41586-025-10072-4 · Cited by 38
“Feeding a large number of documents that might include irrelevant content to LLMs can cause efficiency and performance issues, even with state-of-the-art models 31,32 .”
SoapFL: A Standard Operating Procedure for LLM-Based Method-Level Fault Localization
Qin, Y., Wang, S., Lou, Y., Dong, J., Wang, K., Li, X., & Mao, X. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3543187 · Cited by 151
“However, due to the context limitation as well as the decreasing performance on longer input contexts [15], [17], existing LLM-based FL techniques primarily concentrate on localizing buggy lines within a code snippet, while neglecting the identi-fication of bugs in an entire software project.”
InfluentialSWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · arXiv · doi:10.48550/arxiv.2405.15793 · Cited by 31
Application of large language models to intelligently analyze long construction contract texts
Gao, Y., Gan, Y., Chen, Y., & Chen, Y. (2024) · Construction Management and Economics · doi:10.1080/01446193.2024.2415676 · Cited by 41
“Secondly, the intrinsic mechanisms by which LLMs use given context information are not fully clear, and a model’s memory capacity does not directly correlate with its semantic understanding abilities (Liu et al .”
Ray, M., Kats, D. J., Moorkens, J., Rai, D., Shaar, N., Quinones, D., Vermeulen, A., Mateo, C. M., Brewster, R., Khan, A., et al. (2025) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2025.1729 · Cited by 40
A psychometric framework for evaluating and shaping personality traits in large language models
Serapio-García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Romero, P., Abdulhai, M., Faust, A., & Matarić, M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01115-6 · Cited by 30
Metadata
- Authors
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Samaya AI, UK
- Corresponding author
- Liu, Nelson F.
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 157-173
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4391876619
- Semantic Scholar
- 1733eb7792f7a43dd21f51f4d1017a1bffd217b5
- Cited by
- 1,612
- References
- 59
- Citations by year
- 2026: 901 · 2025: 516 · 2024: 172
- FWCI
- 233.95
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- long-context language models, multi-document question answering, position bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-27 (prompt extract-v2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.