The DOI matches its DataCite record; 52 of the 63 references it cites were found in a registry or an index, and 6 are not checked yet.
Large Language Models Can Be Easily Distracted by Irrelevant Context
Open accessAbstract
Large language models have achieved impressive performance on various natural language processing tasks. However, so far they have been evaluated primarily on benchmarks where all information in the input context is relevant for solving the task. In this work, we investigate the distractibility of large language models, i.e., how the model problem-solving accuracy can be influenced by irrelevant context. In particular, we introduce Grade-School Math with Irrelevant Context (GSM-IC), an arithmetic reasoning dataset with irrelevant information in the problem description. We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included. We also identify several approaches for mitigating this deficiency, such as decoding with self-consistency and adding to the prompt an instruction that tells the language model to ignore the irrelevant information.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By disclosing task intent at the input side and relying on surface-form matching at the output side, the evaluation increasingly reflects models' adaptation to task prompts and generation styles rather than their ability to retain and apply conversational context.”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Shi et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Longer prompts introduce irrelevant and redundant information, which can weaken LLMs' performance (Shi et al., 2023), as illustrated in Figure 1a.”
From LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2024) · cites this paper as Shi et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, indiscriminately retrieving and incorporating a fixed number of retrieved passages, regardless of whether retrieval is necessary, or passages are relevant, diminishes LM versatility or can lead to unhelpful response generation.”
From CareerX: A Retrieval-Augmented Generation Framework for Personalized AI-Driven Career Guidance (2023) · cites this paper as Shi et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prompting approaches can reduce the sensitivity of CoT explanations to input perturbations and stereotypes (Shaikh et al., 2022;Ganguli et al., 2023;Shi et al., 2023), which our findings on prompting for debiasing corroborate.”
From Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023) · cites this paper as Shi et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing agents struggle with the capability to perform self-reflection [69, 41]. That is to say they tend to take all information/feedback and do not know how to filter out or correct irrelevant, incorrect, or misleading information [84, 107].”
From Agentless: Demystifying LLM-based Software Engineering Agents (2024) · cites this paper as [84]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“most LLM evaluation work selects a prompt x, run the model once, and interprets the resulting output y as the model's behavior under that prompt (Shi et al., 2023;Zhu et al., 2024;Cao et al., 2024).”
From Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks (2026) · cites this paper as Shi et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Small changes in input tokens can drastically alter model outputs, indicating a strong token bias and suggesting that these models are highly sensitive and fragile (Jiang et al., 2024; Shi et al., 2023).”
From GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024) · cites this paper as Shi et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Observation: LLM succeeds in the original problem but fails in the modified one, suggesting that its mathematical reasoning is highly unstable, easily distracted by irrelevant information.”
From Large Language Model Reasoning Failures (2026) · cites this paper as Shi et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Recent empirical experiments have shown that LLM performance decreases with less effective information in a prompt (Bai et al., 2023; Li et al., 2023a; Shi et al., 2023).”
From LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2024) · cites this paper as Shi et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“as incorporating irrelevant context can distract the LLM and degrade response quality (Liu et al., 2024b; Shi et al., 2023).”
From In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents (2025) · cites this paper as Shi et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“as incorporating irrelevant context can distract the LLM and degrade response quality (Liu et al., 2024b; Shi et al., 2023).”
From In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents (2025) · cites this paper as Shi et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Robustness benchmarks systematically perturb prompts via paraphrasing, rephrasing, or adversarial noise injection (Zhu et al., 2024;Shi et al., 2023;Cao et al., 2024), reporting dramatic performance swings despite the underlying task remaining unchanged.”
From Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks (2026) · cites this paper as Shi et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“GSM-IC (Shi et al., 2023) shows that irrelevant context can impair LLM performance, focusing on prompting techniques.”
From GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024) · cites this paper as Shi et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“We find that the alignment with human data is better without the full paper context (likely due to the phenomenon that LLMs are easily distracted by too much irrelevant context (Shi et al., 2023)), except for Verifiability , which could benefit from additional context.”
From The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors (2025) · cites this paper as Shi et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Prompting large language models performs decently well in a variety of domains (Brown et al., 2020; Chowdhery et al., 2022, inter alia ).”
BackgroundMethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 2×
“In particular, Li et al. (2022) evaluated T5 (Raffel et al., 2020) and PaLM (Chowd-hery et al., 2022) with few-shot prompts, and proposed knowledge-aware finetuning that finetunes the model on problems with counterfactual and irrelevant context, which strengthens the model robustness to noisy context.”
BackgroundChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 5×
“Chain-of-thought prompting (C O T; Wei et al., 2022) is a prompting technique that guides the language models to solve a problem in a step-by-step manner.”
BackgroundMethodInfluentialSelf-citationTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 3×
“In addition to demonstrating how to handle irrelevant information via exemplars, we also investigate the usage of task-specific instructions (Wei et al., 2021; Sanh et al., 2021; Ouyang et al., 2022; Suzgun et al., 2022; Chung et al., 2022), where we prepend an instruction sentence “feel free to ignore irrelevant info…”
BackgroundMethodPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 9×
“Prompting large language models performs decently well in a variety of domains (Brown et al., 2020; Chowdhery et al., 2022, inter alia ).”
BackgroundMethodResultInfluentialSelf-citationLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 6×
“C O T; Kojima et al., 2022) is a variation of C O T where the prompt does not contain any exemplar.”
BackgroundInfluentialScaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 1×
“In addition to demonstrating how to handle irrelevant information via exemplars, we also investigate the usage of task-specific instructions (Wei et al., 2021; Sanh et al., 2021; Ouyang et al., 2022; Suzgun et al., 2022; Chung et al., 2022), where we prepend an instruction sentence “feel free to ignore irrelevant info…”
BackgroundSelf-citationAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 2×
“…input perturbations for natural language tasks, including model-agnostic input transformations (Liang et al., 2022; Ravichan-der et al., 2022, inter alia ) and adversarial example generation against individual models (Jia & Liang, 2017; Shi et al., 2018; Morris et al., 2020; Wang et al., 2021).”
BackgroundMethodFinetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 1×
“In addition to demonstrating how to handle irrelevant information via exemplars, we also investigate the usage of task-specific instructions (Wei et al., 2021; Sanh et al., 2021; Ouyang et al., 2022; Suzgun et al., 2022; Chung et al., 2022), where we prepend an instruction sentence “feel free to ignore irrelevant info…”
Background
Cited by
Sorted by citations
Items 101–170 of 170
SUGAR: Leveraging Contextual Confidence for Smarter Retrieval
Zubkova, H., Park, J.-H., & Lee, S.-W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890064 · Cited by 9
“Moreover, the retrieved results sometimes contain documents that are irrelevant [11], factually incorrect [12] or even contain harmful information [13].”
Effective Context Selection in LLM-Based Leaderboard Generation: An Empirical Study
Kabongo, S., D’Souza, J., & Auer, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70242-6_15 · Cited by 6
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024) · arXiv · doi:10.48550/arxiv.2410.10813 · Cited by 493
“Although the retrieval accuracy can be enhanced through the preceding designs, it does not guarantee that the LLM can effectively reason over the extensive context (Shi et al., 2023; Liu et al., 2024).”
Design Process for Retrieval Augmented Generation Systems
Xu, X., Zhang, D., Zhang, W., Lu, Q., & Zhu, L. (2025) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c65153.2025.00072 · Cited by 3
“Coarse-grained retrieval units have the potential to provide more relevant information for queries but may also contain more redundant information that could distract retrievers and GenAI models [10].”
Park, S.-I., & Lee, J.-Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00724 · Cited by 9
“Moreover, various studies have shown that LLMs can be easily distracted by irrelevant information (Jia and Liang, 2017; Petroni et al., 2020; Creswell et al., 2022; Cao et al., 2022; Shi et al., 2023a; Yoran et al., 2023).”
mFollowIR: A Multilingual Benchmark for Instruction Following in Retrieval
Weller, O., Chang, B., Yang, E., Yarmohammadi, M., Barham, S., MacAvaney, S., Cohan, A., Soldaini, L., Durme, B. V., & Lawrie, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88711-6_19 · Cited by 8
Cho, S., Seo, J., Jeong, S., & Park, J. C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.207 · Cited by 7
Tian, Q., Zhu, H., Wang, L., Li, Y., & Lan, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.114 · Cited by 4
Relationalizing Tables with Large Language Models: The Promise and Challenges
Huang, Z., & Wu, E. (2024) · IEEE International Conference on Data Engineering Workshops (ICDEW) · doi:10.1109/icdew61823.2024.00045 · Cited by 3
“This complexity could be too overwhelming for LLMs [5], [6].”
BackgroundYakura, H. (2024) · Journal of Intelligence · doi:10.3390/jintelligence12070070 · Cited by 7
“Nevertheless, as a limitation of this study, it is imperative to consider that the behavior of LLMs may not directly mirror human neurological functions, even though high-level commonalities have been observed [10,17].”
BackgroundSRSA: A Cost-Efficient Strategy-Router Search Agent for Real-world Human-Machine Interactions
Wang, Y., & Xu, H. (2024) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw65004.2024.00046 · Cited by 5
“However, this often results in too many unrelated search results, which negatively impacts the final answer generation [8] and makes the process more expensive due to the need for multiple LLM inferences.”
Cyber Shadows: Neutralizing Security Threats with AI and Targeted Policy MeasuresPreprint
Schmitt, M., & Koutroumpis, P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5081091 · Cited by 3
Large Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3
Liao, H., Tian, J., Hu, S., Zhu, Z., He, H., & Jin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887828 · Cited by 6
“Thus, PEP can easily be combined with previous prompting methods.”
InfluentialInReAcTable: LLM-powered Interactive Visual Data Story Construction from Tabular Data
Aodeng, G., Li, G., Feng, Y., Chen, Q., Zhang, Y., & Liu, C. H. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747719 · Cited by 2
“Specifically, removal of the Acting module will introduce substantial noise into the LLMs input, significantly degrading the quality of recommendation re-sults, as demonstrated by existing works [49, 75].”
POSQA: Probe the World Models of LLMs with Size Comparisons
Shu, C., Han, J., Liu, F., Shareghi, E., & Collier, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.504 · Cited by 3
Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention
Foroutan, N., Banaei, M., Aberer, K., & Bosselut, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.632 · Cited by 5
Fang, K., Chai, Y., & Xu, C. K. (2024) · Asia Conference on Algorithms, Computing and Machine Learning (CACML) · doi:10.1145/3654823.3654825 · Cited by 4
“Without sufficient contemplation, their performance will decline obviously[19].”
BackgroundProbing the Dual Logic Ability of Privatized Medical-Domain LLMs
Du, Y., Zhao, S., Cai, M., Ma, M., Zhao, D., Cao, J., & Qin, B. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822782 · Cited by 2
“• Prompt Focus : Recent study [18] indicates that LLMs can become distracted when processing overly lengthy texts.”
LateSplit: Lightweight Post-Retrieval Chunking for Query-Aligned Text Segmentation in RAG Systems
Peng, Z., Liu, X., & Yang, G. (2025) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit65432.2025.11036011 · Cited by 2
Towards Building a Robust Knowledge Intensive Question Answering Model with Large Language Models
Hong, X., Shao, Y., Wang, Z., Duan, M., & Jin, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9431-7_18 · Cited by 1
Cost, Complexity, and Efficacy of Prompt Engineering Techniques for Large Language Models
Cherukuri, M. (2025) · International Journal on Science and Technology · doi:10.71097/ijsat.v16.i2.2584 · Cited by 1
THINK: Tackling API Hallucinations in LLMs via Injecting Knowledge
Liu, J., Zhang, Y., Wang, D., Li, Y., & Dong, W. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00029 · Cited by 1
“Research shows that the results generated by LLMs are easily influenced by irrelevant context [18].”
Toward Enhancing Reasoning Capabilities of LLMs: An Approach via Synthetic Logic Corpus
Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.520 · Cited by 1
Jiang, M., Huang, T., Guo, B., Lu, Y., & Zhang, F. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-7030-7_15 · Cited by 11
MisstepMath: A Diverse Student Mistake Dataset for AI in Mathematics Teacher Training
Ansari, S. M. A., Bywater, J. P., Lilly, S., Brown, D. E., & Chiu, J. L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98414-3_27 · Cited by 2
Elliott, T. J., Levitt, S. P., Nixon, K. J., & Bekker, M. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-96262-2_15 · Cited by 1
“Whilst sentence-level chunking improves accuracy but loses context [32,33].”
Can You Follow Me? Testing Situational Understanding for ChatGPT
Yang, C., & Ettinger, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.394 · Cited by 3
Retrieval Augmented Tree of Thoughts
Liu, X., & Liu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_36 · Cited by 1
Lambert, M., Döring, N., Schuster, T., & Schmitz, N. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-87217-4_8 · Cited by 1
Wieczorek, O., Steinhardt, I., Schneijderberg, C., Schmidt, R., & Mauermeister, S. (2023) · doi:10.31235/osf.io/493qg · Cited by 1
Challenging the Notion of Trust Around Chatgpt in the High-Stakes Use Case of InsurancePreprint
Ressel, J., Völler, M., Murphy, F., & Mullins, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4569683 · Cited by 1
Predicting typeScript type annotations and definitions with machine learning
Yee, M. (2024) · doi:10.17760/d20653005 · Cited by 1
Enhancing Reliability in Llm-Integrated Robotic Systems: A Unified Approach to Security and SafetyPreprint
Zhang, W., Kong, X., Dewitt, C., Bräunl, T., & Hong, J. B. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5202517 · Cited by 1
song, Z., Wu, H. D., Zhu, Y., Xv, Z., & Fang, H. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5370687 · Cited by 1
Multilingual Rag Agents For Localized Knowledge: Adaptive Indexing For Under-Represented LanguagesPreprint
Ugwumba, N. K., & Okechukwu, K. E. (2026) · Research Square · doi:10.21203/rs.3.rs-8714867/v1 · Cited by 1
In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents
Tan, Z., Yan, J., Hsu, I.-H., Han, R., Wang, Z., Le, L. T., Song, Y., Chen, Y., Palangi, H., Lee, G., et al. (2025) · arXiv · Cited by 54
“Second, the agent must accurately retrieve relevant past information from the memory, as incorporating irrelevant context can distract the LLM and degrade response quality (Liu et al., 2024b; Shi et al., 2023).”
LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
Li, Y., Guo, W., Zhang, L., Xu, R., Huang, M., Liu, H., Xu, L., Xu, Y., & Liu, J. (2026) · arXiv · doi:10.48550/arxiv.2602.10715
“(Perez et al., 2021; Min et al., 2022; Shi et al., 2023) As a result, the reported performance can be misleading, undermining both cross-task and cross-model comparability.”
Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation
Caruzzo, C., Yoo, D., & Kim, T. S. (2026) · arXiv · doi:10.48550/arxiv.2607.09349
“A productive line of work studies model behavior when parametric knowledge contradicts retrieved context [Longpre et al., 2021, Mallen et al., 2023, Shi et al., 2023, Xie et al., 2024].”
SEMA-RAG: A Self-Evolving Multi-Agent Retrieval-Augmented Generation Framework for Medical Reasoning
Huang, Y., Chen, R., & Cheng, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.917
“…explicitly (Soldaini et al., 2017); and (ii) the retrieval process lacks mechanisms for sufficiency assessment and feed-back , hindering self-evolving iterative convergence under insufficient evidence and thus weakening reliable evidence-chain formation (Mallen et al., 2023; Shi et al., 2023).”
Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16008
“The strong correlation between the model’s output and the first document could suggest that they are highly relevant, distracting the model (Shi et al., 2023a).”
Helwig, M. (2026) · arXiv · doi:10.48550/arxiv.2609.05510
“Model-side evidence that irrelevant context degrades output [Shi 2023; Liu 2024; Chroma 2025] argues for injecting less; the reader-side skim reflex argues for silence as the default .”
Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks
Haase, J., Gonnermann-Müller, J., Hanel, P. H. P., Leins, N., Kosch, T., Mendling, J., & Pokutta, S. (2026) · arXiv · doi:10.48550/arxiv.2601.21339
“Consequently, most LLM evaluation work selects a prompt x , run the model once, and interprets the resulting output y as the model’s behavior under that prompt (Shi et al., 2023; Zhu et al., 2024; Cao et al., 2024).”
Fine-tuning human for LLM projectsPreprint
Guha, R. (2023) · doi:10.31219/osf.io/9js3b
Fine-Tuning Human for LLM ProjectsPreprint
Guha, R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4574477
Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension
Kawabata, A., & Sugawara, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.9 · Cited by 3
Improving Financial QA via Enhancing Argument Understanding with an Argument Predictor
Lim, J., Hwang, Y., Lee, Y.-J., & Choi, H.-J. (2024) · International Conference on Big Data and Smart Computing · doi:10.1109/bigcomp60711.2024.00065
“Due to this noise, irrelevant information can easily mislead the model and drop performance [6].”
BackgroundCan Large Language Models Understand Uncommon Meanings of Common Words?Preprint
Wu, J., Che, F., Zheng, X., Zhang, S., Nie, S., Jin, R., Shao, P., & Tao, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4822247 · Cited by 7
No Consciousness? No Meaning (and no AGI!)Preprint
Masi, M. (2024) · Qeios · doi:10.32388/dn232y
No Consciousness? No Meaning (and no AGI)!Preprint
Masi, M. (2025) · Qeios · doi:10.32388/dn232y.2
Exploring performance and consistency in mathematical reasoning across problem types
Hwang, Y., Lim, J., Lee, Y., & Choi, H. (2025) · IEEE International Conference on Big Data and Smart Computing (BigComp) · doi:10.1109/bigcomp64353.2025.00031
“⚫ Options for MC-short answer : For the MC-short answer type, drawing inspiration from [11], which observed that LLMs tend to disregard answers that include unreasonable numbers easily such as out-of-range values, we designed options using numbers that are not too far from the original answer while…”
No Consciousness? No Meaning (and no AGI)!Preprint
Masi, M. (2025) · Qeios · doi:10.32388/dn232y.3
Bourguet, M.-L. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016372
“The most effective prompts include pertinent details and constraints, such as background information, specific facts, or user preferences [28], helping the LLM understand the context and scenario more accurately, leading to more relevant responses.”
Language-Guided Adaptive Vision Token Pruning for Efficient Multimodal Large Language Models
Deniz, O. F., Arici, T., Sheikholeslami, F., Gozluklu, B., Trabelsi, A., Khan, S. A., Tian, Y., & Khan, L. R. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8186-0_9
Pirenne, L., Mokeddem, S., Ernst, D., & Louppe, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_1
Research on Optimization of Hint Words of Large Model in Multimodal Learning
Cheng, C., Lu, F., & Wu, Z. (2025) · Learning and analytics in intelligent systems · doi:10.1007/978-3-031-98607-9_3
Towards an AI-Driven Threat Intelligence and Incident Response
Khadiri, R. E., Nasri, M., & Boukhalfa, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-95330-9_35
Task-Classifying Retrieval Augmented Generation
Ju, J., Zang, P., Zhong, H., & Wang, L. (2025) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd64986.2025.11082008 · Cited by 1
“If the retrieved documents contain irrelevant or misleading information, it will not only affect the quality of the generation results but may also exacerbate the hallucination phenomenon [11].”
No Consciousness? No Meaning (and no AGI)!Preprint
Masi, M. (2025) · Qeios · doi:10.32388/dn232y.4
Probing the reasoning abilities of LLMs in blocks world
Zhao, K. Z., & Macbeth, J. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101421
Richardson, E., Lischwe, P., Bennett, J., Blazeska, N., Greenbaum, J., Harlan, B., Lallmamode, N. A. J., Marrama, D., Talbott, M. D., Vita, R., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.08.698331
“…length; furthermore, it has been shown that supplying the LLM with irrelevant context may significantly reduce response accuracy, and that response accuracy declines when the relevant information is in the middle of large context window, as is likely to be the case for scientific papers 27,28 .”
Research on Formal Specification Generation Methods forAirborne Software Testing
Wen, J., Wu, L., Chen, X., Zhang, X., & Ma, X. (2026) · International Conference on Computer Network Security and Software Engineering (CNSSE) · doi:10.1145/3803633.3803662
“However, the code logic of the RAG library itself may deviate significantly from actual software requirements, thus requiring feedback modifications to the library itself [7].”
Self-citationIn Line with Context: Repository-Level Code Generation via Context Inlining
Hu, C., Zeng, W., Shi, Y., Shen, B., & Gu, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797094 · Cited by 2
“While it contains the crucial information needed for accurate generation, directly feeding the entire repository into an LLM is infeasible due to context window limitations and the overwhelming amount of irrelevant or redundant code [49].”
MACA-PA: A Modular Agent-based Clinical Assistant for Perioperative Anesthesia
Jiang, Y., Ma, C., Xu, Z.-Z., et al. (2026) · Proceedings of the 2026 2nd International Conference on Health Informatization and Data Analysis · doi:10.1145/3819113.3819136
“With RAG, by directly connecting retrieved knowledge to patient-specific reasoning in a monolithic manner, lots of noises may be brought into the model, which may break controllability and even cause retrieval-induced hallucinations [18, 22].”
SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
Pham, T., Nguyen, N., Zunjare, P., Chen, W., Tseng, Y.-M., & Vu, T. (2025) · arXiv · doi:10.48550/arxiv.2506.01062
“Reasoning under knowledge conflict: Prior work shows that LLM S can be vulnerable to misinfor-mation [Pan et al., 2023], irrelevant context [Shi et al., 2023], and conflicting sources [Kazemi et al., 2023].”
FOCAL: Filtered On-device Continuous Activity Logging for Efficient Personal Desktop Summarization
Yin, H., Wen, Z., Cao, J., Yuan, B., & Yang, R. (2026) · arXiv · doi:10.48550/arxiv.2604.19541
Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
Li, Y., Guo, W., Zhang, L., Xu, R., Huang, M., Liu, H., Xu, L., Xu, Y., & Liu, J. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1150
Tsui, K. (2025) · arXiv · doi:10.48550/arxiv.2507.02778 · Cited by 29
The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · arXiv · doi:10.48550/arxiv.2509.04484
“We find that the alignment with human data is better without the full paper context (likely due to the phenomenon that LLMs are easily distracted by too much irrelevant context (Shi et al., 2023)), except for Verifiability , which could benefit from additional context.”
Reducing Token Usage of State-in-Context Agents using Minification
Hrubec, N., & Cito, J. (2026) · arXiv · doi:10.48550/arxiv.2606.01326
“Additionally, studies have shown that LLMs can struggle to fully utilize long contexts, leading to performance degradation [3, 6, 8, 15, 16].”
Metadata
- Authors
- Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D.
- Affiliations
- Toyota Technological Institute at Chicago (US)
- Purdue University West Lafayette (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.00093
- OpenAlex
- W4319049323
- Semantic Scholar
- 3d68522abfadfc8ee6b7ec9edaaf91f1b2f38e5e
- Cited by
- 134
- References
- 63
- Citations by year
- 2026: 7 · 2025: 43 · 2024: 55 · 2023: 29
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, distractibility, arithmetic reasoning, prompting techniques, instruction following, problem-solving accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
