Large Language Models Can Be Easily Distracted by Irrelevant Context
Open accessAbstract
Large language models have achieved impressive performance on various natural language processing tasks. However, so far they have been evaluated primarily on benchmarks where all information in the input context is relevant for solving the task. In this work, we investigate the distractibility of large language models, i.e., how the model problem-solving accuracy can be influenced by irrelevant context. In particular, we introduce Grade-School Math with Irrelevant Context (GSM-IC), an arithmetic reasoning dataset with irrelevant information in the problem description. We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included. We also identify several approaches for mitigating this deficiency, such as decoding with self-consistency and adding to the prompt an instruction that tells the language model to ignore the irrelevant information.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
By disclosing task intent at the input side and relying on surface-form matching at the output side, the evaluation increasingly reflects models' adaptation to task prompts and generation styles rather than their ability to retain and apply conversational context.
“By disclosing task intent at the input side and relying on surface-form matching at the output side, the evaluation increasingly reflects models' adaptation to task prompts and generation styles rather than their ability to retain and apply conversational context.”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Shi et al., 2023
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredLonger prompts introduce irrelevant and redundant information, which can weaken LLMs' performance.
“Longer prompts introduce irrelevant and redundant information, which can weaken LLMs' performance (Shi et al., 2023), as illustrated in Figure 1a.”
From LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2024) · cites this paper as Shi et al., 2023
The Introduction reports that adding irrelevant information to a problem description can disrupt LLM predictions. Section 5.4 finds that a longer, four-exemplar prompt was less robust to irrelevant context than a one-exemplar prompt. The paper does not establish that longer prompts inherently introduce irrelevant or redundant information; it also reports that the four-exemplar prompt performed better on the original benchmark. (LLM, conf=0.85)
Experimental — accuracy not yet measuredIndiscriminately retrieving and incorporating a fixed number of retrieved passages, regardless of whether retrieval is necessary or passages are relevant, diminishes LM versatility or can lead to unhelpful response generation.
“However, indiscriminately retrieving and incorporating a fixed number of retrieved passages, regardless of whether retrieval is necessary, or passages are relevant, diminishes LM versatility or can lead to unhelpful response generation.”
From CareerX: A Retrieval-Augmented Generation Framework for Personalized AI-Driven Career Guidance (2023) · cites this paper as Shi et al., 2023
The Introduction (page unavailable) reports that adding irrelevant information to a problem description distracts language models and produces inconsistent predictions. This supports the concern about unhelpful responses from irrelevant context, but the paper does not examine fixed-number passage retrieval, whether retrieval is necessary, or diminished LM versatility. (LLM, conf=0.86)
Experimental — accuracy not yet measuredPrompting approaches can reduce the sensitivity of CoT explanations to input perturbations and stereotypes, which our findings on prompting for debiasing corroborate.
“Prompting approaches can reduce the sensitivity of CoT explanations to input perturbations and stereotypes (Shaikh et al., 2022;Ganguli et al., 2023;Shi et al., 2023), which our findings on prompting for debiasing corroborate.”
From Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023) · cites this paper as Shi et al., 2023
Section 5.3 reports that instructed prompting and exemplars containing distractors improve the robustness of CoT-based problem solving to irrelevant context. That supports the input-perturbation part in this specific setting, but the paper does not establish reduced sensitivity of CoT explanations to stereotypes or evaluate debiasing. (LLM, conf=0.86)
Experimental — accuracy not yet measuredExisting agents struggle with the capability to perform self-reflection, tending to take all information/feedback and not filter out or correct irrelevant, incorrect, or misleading information.
“Existing agents struggle with the capability to perform self-reflection [69, 41]. That is to say they tend to take all information/feedback and do not know how to filter out or correct irrelevant, incorrect, or misleading information [84, 107].”
From Agentless: Demystifying LLM-based Software Engineering Agents (2024) · cites this paper as [84]
The Introduction and §5.1 show that large language models can be distracted by irrelevant information in problem descriptions and sometimes use it to produce wrong answers. The paper does not establish that agents generally accept all feedback or fail to correct incorrect or misleading information. Section 5.3 also finds that instructions can improve models’ ability to ignore irrelevant context. The self-reflection sentence is attached to other citations, not this source. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Most LLM evaluation work selects a prompt, runs the model once, and interprets the resulting output as the model's behavior under that prompt.
“most LLM evaluation work selects a prompt x, run the model once, and interprets the resulting output y as the model's behavior under that prompt (Shi et al., 2023;Zhu et al., 2024;Cao et al., 2024).”
From Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks (2026) · cites this paper as Shi et al., 2023
Section 5 (Experiments; page not provided) says the paper uses greedy decoding for evaluations without self-consistency, which supports the single-response description for part of its own evaluation. It also reports sampling 20 responses for self-consistency experiments. The paper does not establish the claim that most LLM evaluation work runs a model only once per prompt. (LLM, conf=0.82)
Experimental — accuracy not yet measuredSmall changes in input tokens can drastically alter model outputs, indicating strong token bias and high sensitivity and fragility.
“Small changes in input tokens can drastically alter model outputs, indicating a strong token bias and suggesting that these models are highly sensitive and fragile (Jiang et al., 2024; Shi et al., 2023).”
From GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024) · cites this paper as Shi et al., 2023
The Introduction reports that adding a small amount of irrelevant information produced inconsistent predictions, supporting the claim that the models are sensitive to input changes. But the paper tests added irrelevant sentences, not small changes to individual tokens, and it does not establish a general 'strong token bias.' (LLM, conf=0.84)
Experimental — accuracy not yet measuredLLM's mathematical reasoning is highly unstable, easily distracted by irrelevant information, as evidenced by succeeding in the original problem but failing in the modified one.
“Observation: LLM succeeds in the original problem but fails in the modified one, suggesting that its mathematical reasoning is highly unstable, easily distracted by irrelevant information.”
From Large Language Model Reasoning Failures (2026) · cites this paper as Shi et al., 2023
Recent empirical experiments have shown that LLM performance decreases with less effective information in a prompt.
“Recent empirical experiments have shown that LLM performance decreases with less effective information in a prompt (Bai et al., 2023; Li et al., 2023a; Shi et al., 2023).”
From LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2024) · cites this paper as Shi et al., 2023
Incorporating irrelevant context can distract the LLM and degrade response quality.
“as incorporating irrelevant context can distract the LLM and degrade response quality (Liu et al., 2024b; Shi et al., 2023).”
From In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents (2025) · cites this paper as Shi et al., 2023
Irrelevant context can distract the LLM and degrade response quality.
“as incorporating irrelevant context can distract the LLM and degrade response quality (Liu et al., 2024b; Shi et al., 2023).”
From In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents (2025) · cites this paper as Shi et al., 2023
Robustness benchmarks systematically perturb prompts via paraphrasing, rephrasing, or adversarial noise injection, reporting dramatic performance swings despite the underlying task remaining unchanged.
“Robustness benchmarks systematically perturb prompts via paraphrasing, rephrasing, or adversarial noise injection (Zhu et al., 2024;Shi et al., 2023;Cao et al., 2024), reporting dramatic performance swings despite the underlying task remaining unchanged.”
From Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks (2026) · cites this paper as Shi et al., 2023
GSM-IC shows that irrelevant context can impair LLM performance, focusing on prompting techniques.
“GSM-IC (Shi et al., 2023) shows that irrelevant context can impair LLM performance, focusing on prompting techniques.”
From GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024) · cites this paper as Shi et al., 2023
Alignment with human data is better without the full paper context, except for Verifiability which could benefit from additional context.
“We find that the alignment with human data is better without the full paper context (likely due to the phenomenon that LLMs are easily distracted by too much irrelevant context (Shi et al., 2023)), except for Verifiability , which could benefit from additional context.”
From The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors (2025) · cites this paper as Shi et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2× · see citing works
“Prompting large language models performs decently well in a variety of domains (Brown et al., 2020; Chowdhery et al., 2022, inter alia ).”
BackgroundMethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 2× · see citing works
“In particular, Li et al. (2022) evaluated T5 (Raffel et al., 2020) and PaLM (Chowd-hery et al., 2022) with few-shot prompts, and proposed knowledge-aware finetuning that finetunes the model on problems with counterfactual and irrelevant context, which strengthens the model robustness to noisy context.”
BackgroundChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 5× · see citing works
“Chain-of-thought prompting (C O T; Wei et al., 2022) is a prompting technique that guides the language models to solve a problem in a step-by-step manner.”
BackgroundMethodInfluentialSelf-citationTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 3× · see citing works
“In addition to demonstrating how to handle irrelevant information via exemplars, we also investigate the usage of task-specific instructions (Wei et al., 2021; Sanh et al., 2021; Ouyang et al., 2022; Suzgun et al., 2022; Chung et al., 2022), where we prepend an instruction sentence “feel free to ignore irrelevant info…”
BackgroundMethodPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 9× · see citing works
“Prompting large language models performs decently well in a variety of domains (Brown et al., 2020; Chowdhery et al., 2022, inter alia ).”
BackgroundMethodResultInfluentialSelf-citationLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 6× · see citing works
“C O T; Kojima et al., 2022) is a variation of C O T where the prompt does not contain any exemplar.”
BackgroundInfluentialScaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 1× · see citing works
“In addition to demonstrating how to handle irrelevant information via exemplars, we also investigate the usage of task-specific instructions (Wei et al., 2021; Sanh et al., 2021; Ouyang et al., 2022; Suzgun et al., 2022; Chung et al., 2022), where we prepend an instruction sentence “feel free to ignore irrelevant info…”
BackgroundSelf-citationAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 2× · see citing works
“…input perturbations for natural language tasks, including model-agnostic input transformations (Liang et al., 2022; Ravichan-der et al., 2022, inter alia ) and adversarial example generation against individual models (Jia & Liang, 2017; Shi et al., 2018; Morris et al., 2020; Wang et al., 2021).”
BackgroundMethodFinetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 1× · see citing works
“In addition to demonstrating how to handle irrelevant information via exemplars, we also investigate the usage of task-specific instructions (Wei et al., 2021; Sanh et al., 2021; Ouyang et al., 2022; Suzgun et al., 2022; Chung et al., 2022), where we prepend an instruction sentence “feel free to ignore irrelevant info…”
BackgroundDecomposed Prompting: A Modular Approach for Solving Complex TasksPreprint
Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., & Sabharwal, A. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.02406 · Cited by 839 · cited 2× · see citing works
“…al., 2022; Suzgun et al., 2022; Shi et al., 2022b, inter alia ), problem decomposition (Zhou et al., 2022; Drozdov et al., 2022; Dohan et al., 2022; Khot et al., 2022; Press et al., 2022, inter alia ), generating programs (Austin et al., 2021; Chowdhery et al., 2022; Gao et al., 2022; Chen et al.,…”
BackgroundSelf-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · cited 8× · see citing works
“This means that in addition to improving model performance on clean arithmetic reasoning tasks (Wang et al., 2022c), SC also substantially reduces the distractibility of large language models to irrelevant context.”
BackgroundInfluentialSelf-citationRethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · cited 2× · see citing works
“A line of work studies the model performance with incorrect prompting ex-emplars, i.e., the example problems are paired with wrong answers (Min et al., 2022; Kim et al., 2022).”
BackgroundTowards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670 · cited 1× · see citing works
“There exist some logical reasoning benchmarks that contain irrelevant content in task descriptions (Weston et al., 2015; Sinha et al., 2019; Clark et al., 2021; Han et al., 2022; Tafjord et al., 2020, inter alia ).”
BackgroundMultitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · cited 2× · see citing works
“In addition to demonstrating how to handle irrelevant information via exemplars, we also investigate the usage of task-specific instructions (Wei et al., 2021; Sanh et al., 2021; Ouyang et al., 2022; Suzgun et al., 2022; Chung et al., 2022), where we prepend an instruction sentence “feel free to ignore irrelevant info…”
BackgroundTraining Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 6× · see citing works
“, how is large language model prompting affected by irrelevant context, and what strategies can be used to improve performance? To measure distractibility, we construct the GSM-IC dataset, a grade-school math problem dataset derived from GSM8K (Cobbe et al., 2021) and introduce two different metrics.”
BackgroundMethodInfluentialTextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP
Morris, J. X., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., & Qi, Y. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.16 · Cited by 570 · cited 1× · see citing works
“…input perturbations for natural language tasks, including model-agnostic input transformations (Liang et al., 2022; Ravichan-der et al., 2022, inter alia ) and adversarial example generation against individual models (Jia & Liang, 2017; Shi et al., 2018; Morris et al., 2020; Wang et al., 2021).”
MethodHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 2× · see citing works
“Meanwhile, Liang et al. (2022) evaluate various large language models under several metrics, including accuracy, robustness, fairness, etc. Specifically, the input transformations in their robustness evaluation include semantics-preserving and semantics-altering perturbations, such as injecting…”
BackgroundTowards Understanding Chain-of-Thought Prompting: An Empirical Study of What MattersPreprint
Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., & Sun, H. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.10001 · Cited by 428 · cited 8× · see citing works
“This means that in addition to improving model performance on clean arithmetic reasoning tasks (Wang et al., 2022c), SC also substantially reduces the distractibility of large language models to irrelevant context.”
BackgroundInfluentialDROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393 · cited 2× · see citing works
“In addition to GSM-IC, we extend our evaluation to the DROP dataset (Dua et al., 2019), where the task is to answer a question according to a long passage that naturally contains irrelevant context.”
BackgroundMethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 3× · see citing works
“…decomposition (Zhou et al., 2022; Drozdov et al., 2022; Dohan et al., 2022; Khot et al., 2022; Press et al., 2022, inter alia ), generating programs (Austin et al., 2021; Chowdhery et al., 2022; Gao et al., 2022; Chen et al., 2022, inter alia ), marginalizing intermediate steps that share the same…”
BackgroundMethodSelf-citationLeast-to-Most Prompting Enables Complex Reasoning in Large Language Models
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10625 · Cited by 380 · see citing works
Self-citationProgram Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325 · cited 1× · see citing works
“…et al., 2020; Chowdhery et al., 2022, inter alia ) has been signifi-cantly boosted with various techniques, including generating intermediate steps (Ling et al., 2017; Cobbe et al., 2021; Nye et al., 2021; Wei et al., 2022; Suzgun et al., 2022; Shi et al., 2022b, inter alia ), problem…”
MethodAre NLP Models really able to Solve Simple Math Word Problems?
Patel, A., Bhattamishra, S., & Goyal, N. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.168 · Cited by 307 · cited 5× · see citing works
“The same holds for SVAMP (Patel et al., 2021), an arithmetic reasoning benchmark constructed by applying different types of variations to math problems from existing clean datasets, e.”
BackgroundResultInfluentialMeasuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294 · cited 2× · see citing works
“…al., 2022; Shi et al., 2022b, inter alia ), problem decomposition (Zhou et al., 2022; Drozdov et al., 2022; Dohan et al., 2022; Khot et al., 2022; Press et al., 2022, inter alia ), generating programs (Austin et al., 2021; Chowdhery et al., 2022; Gao et al., 2022; Chen et al., 2022, inter alia…”
BackgroundNegated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
Kassner, N., & Schütze, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.698 · Cited by 259 · cited 1× · see citing works
“Similar issues have been found for general-purpose pretrained language models, on the tasks of factual reasoning (Kassner & Sch¨utze, 2020; Pandia & Ettinger, 2021; Misra et al., 2023; Li et al., 2022), code generation (Jones & Steinhardt, 2022), and syntactic generalization (Chaves & Richter,…”
BackgroundWorking memory and intrusions of irrelevant information in a group of specific poor problem solvers
Passolunghi, M. C., Cornoldi, C., & Liberto, S. D. (1999) · Memory & Cognition · doi:10.3758/bf03198531 · Cited by 242 · cited 1× · see citing works
“Studies in psychology have shown that irrelevant information may significantly decrease some children and even adults problem-solving accuracy (Hoyer et al., 1979; Pasolunghi et al., 1999; Marzocchi et al., 2002, inter alia ).”
BackgroundLarge Language Models with Controllable Working MemoryPreprint
Li, D., Rawat, A., Zaheer, M., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2211.05110 · Cited by 236 · cited 5× · see citing works
“While prior work has shown that training and fine-tuning with different types of problems improves model robustness (Li et al., 2022), our results show that prompting with exem-plars that demonstrate how to ignore irrelevant context also results in significant robustness improvement.”
BackgroundResultInfluentialDo Prompt-Based Models Really Understand the Meaning of Their Prompts?
Webson, A., & Pavlick, E. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.167 · Cited by 203 · cited 1× · see citing works
“In addition, prior work has investigated the model sensitivity to other parts of the prompt, such as instruction tuning with misleading and irrelevant instructions (Webson & Pavlick, 2021) and wrong reasoning steps in the examples (Madaan & Yaz-danbakhsh, 2022; Wang et al., 2022a).”
BackgroundTransformers as Soft Reasoners over Language
Clark, P., Tafjord, O., & Richardson, K. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/537 · Cited by 176 · cited 1× · see citing works
“There exist some logical reasoning benchmarks that contain irrelevant content in task descriptions (Weston et al., 2015; Sinha et al., 2019; Clark et al., 2021; Han et al., 2022; Tafjord et al., 2020, inter alia ).”
BackgroundPAL: Program-aided Language Models
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2211.10435 · Cited by 155 · cited 3× · see citing works
“Following prior work on solving GSM8K problems with code (Chowdhery et al., 2022; Gao et al., 2022; Chen et al., 2022), we include a Python program as the problem solution in the prompt, and execute the generated Python code using an external Python interpreter to obtain the final answer.”
MethodResultShow Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144 · cited 1× · see citing works
“…inter alia ) has been signifi-cantly boosted with various techniques, including generating intermediate steps (Ling et al., 2017; Cobbe et al., 2021; Nye et al., 2021; Wei et al., 2022; Suzgun et al., 2022; Shi et al., 2022b, inter alia ), problem decomposition (Zhou et al., 2022; Drozdov et al.,…”
MethodSelf-citationChen, W., Ma, X., Wang, X., & Cohen, W. W. (2022) · arXiv · doi:10.48550/arxiv.2211.12588 · Cited by 125 · cited 3× · see citing works
“Following prior work on solving GSM8K problems with code (Chowdhery et al., 2022; Gao et al., 2022; Chen et al., 2022), we include a Python program as the problem solution in the prompt, and execute the generated Python code using an external Python interpreter to obtain the final answer.”
BackgroundMethodProofWriter: Generating implications, proofs, and abductive statements over natural language
Tafjord, O., Dalvi, B., & Clark, P. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.317 · Cited by 122 · cited 1× · see citing works
“There exist some logical reasoning benchmarks that contain irrelevant content in task descriptions (Weston et al., 2015; Sinha et al., 2019; Clark et al., 2021; Han et al., 2022; Tafjord et al., 2020, inter alia ).”
BackgroundSelection-Inference: Exploiting Large Language Models for Interpretable Logical ReasoningPreprint
Creswell, A., Shanahan, M., & Higgins, I. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.09712 · Cited by 122 · cited 2× · see citing works
“However, previous work largely focuses on designing models that require extra training, and prompting alone still hardly achieves the same level of performance as finetuned models for these tasks (Han et al., 2022; Creswell et al., 2022).”
BackgroundCLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text
Sinha, K., Sodhani, S., Dong, J., Pineau, J., & Hamilton, W. L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1458 · Cited by 110 · cited 1× · see citing works
“There exist some logical reasoning benchmarks that contain irrelevant content in task descriptions (Weston et al., 2015; Sinha et al., 2019; Clark et al., 2021; Han et al., 2022; Tafjord et al., 2020, inter alia ).”
BackgroundAdversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A. H., & Li, B. (2021) · arXiv · doi:10.48550/arxiv.2111.02840 · Cited by 88 · cited 2× · see citing works
“…input perturbations for natural language tasks, including model-agnostic input transformations (Liang et al., 2022; Ravichan-der et al., 2022, inter alia ) and adversarial example generation against individual models (Jia & Liang, 2017; Shi et al., 2018; Morris et al., 2020; Wang et al., 2021).”
BackgroundMethodMarzocchi, G. M., Lucangeli, D., De Meo, T., Fini, F., & Cornoldi, C. (2002) · Developmental Neuropsychology · doi:10.1207/s15326942dn2101_4 · Cited by 86 · cited 1× · see citing works
“Studies in psychology have shown that irrelevant information may significantly decrease some children and even adults problem-solving accuracy (Hoyer et al., 1979; Pasolunghi et al., 1999; Marzocchi et al., 2002, inter alia ).”
BackgroundEffects of varying irrelevant information on adult age differences in problem solving.
Hoyer, W., Rebok, G., & Sved, S. (1979) · Journal of Gerontology · doi:10.1093/geronj/34.4.553 · Cited by 78 · cited 1× · see citing works
“Studies in psychology have shown that irrelevant information may significantly decrease some children and even adults problem-solving accuracy (Hoyer et al., 1979; Pasolunghi et al., 1999; Marzocchi et al., 2002, inter alia ).”
BackgroundLanguage Models are Multilingual Chain-of-Thought Reasoners
Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W., Tay, Y., Ruder, S., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.03057 · Cited by 60 · cited 5× · see citing works
“…techniques, including generating intermediate steps (Ling et al., 2017; Cobbe et al., 2021; Nye et al., 2021; Wei et al., 2022; Suzgun et al., 2022; Shi et al., 2022b, inter alia ), problem decomposition (Zhou et al., 2022; Drozdov et al., 2022; Dohan et al., 2022; Khot et al., 2022; Press et…”
BackgroundMethodInfluentialSelf-citationCONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation
Ravichander, A., Gardner, M., & Marasović, A. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2211.00295 · Cited by 55
Natural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · arXiv · doi:10.48550/arxiv.2204.11454 · Cited by 49 · cited 4×
“Self-consistency (SC; Wang et al., 2022c; Shi et al., 2022a) may further boost the reasoning performance by marginal-izing over intermediate reasoning steps that share the same final result.”
BackgroundMethodInfluentialSelf-citationCapturing Failures of Large Language Models Via Human Cognitive Biases
Jones, E., & Steinhardt, J. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-0856 · Cited by 48 · cited 1× · see citing works
“…issues have been found for general-purpose pretrained language models, on the tasks of factual reasoning (Kassner & Sch¨utze, 2020; Pandia & Ettinger, 2021; Misra et al., 2023; Li et al., 2022), code generation (Jones & Steinhardt, 2022), and syntactic generalization (Chaves & Richter, 2021).”
BackgroundChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.09261 · Cited by 44 · cited 3× · see citing works
“…boosted with various techniques, including generating intermediate steps (Ling et al., 2017; Cobbe et al., 2021; Nye et al., 2021; Wei et al., 2022; Suzgun et al., 2022; Shi et al., 2022b, inter alia ), problem decomposition (Zhou et al., 2022; Drozdov et al., 2022; Dohan et al., 2022; Khot et…”
BackgroundMethodSelf-citationLearning What is Essential in Questions
Khashabi, D., Khot, T., Sabharwal, A., & Roth, D. (2017) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k17-1010 · Cited by 43 · cited 1× · see citing works
“Jia & Liang (2017) have shown that neural question answering systems are largely affected by adversarial distracting sentences, whereas follow up work (Khashabi et al., 2017; Ni et al., 2019) proposes learning strategies that mitigate the problem.”
BackgroundCompositional Semantic Parsing with Large Language ModelsPreprint
Drozdov, A., Nathanael Schärli, A., Akyürek, E., Scales, N., Song, X., Chen, X., Bousquet, O., & Zhou, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15003 · Cited by 43 · cited 3× · see citing works
“…et al., 2021; Nye et al., 2021; Wei et al., 2022; Suzgun et al., 2022; Shi et al., 2022b, inter alia ), problem decomposition (Zhou et al., 2022; Drozdov et al., 2022; Dohan et al., 2022; Khot et al., 2022; Press et al., 2022, inter alia ), generating programs (Austin et al., 2021; Chowdhery…”
BackgroundSelf-citationFOLIO: Natural Language Reasoning with First-Order Logic
Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Zhou, W., Coady, J., Peng, D., Qiao, Y., Benson, L., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.00840 · Cited by 35 · cited 3× · see citing works
“There exist some logical reasoning benchmarks that contain irrelevant content in task descriptions (Weston et al., 2015; Sinha et al., 2019; Clark et al., 2021; Han et al., 2022; Tafjord et al., 2020, inter alia ).”
BackgroundLearning Visually-Grounded Semantics from Contrastive Adversarial SamplesPreprint
Shi, H., Mao, J., Xiao, T., Jiang, Y., & Sun, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10348 · Cited by 32 · cited 2× · see citing works
“…input perturbations for natural language tasks, including model-agnostic input transformations (Liang et al., 2022; Ravichan-der et al., 2022, inter alia ) and adversarial example generation against individual models (Jia & Liang, 2017; Shi et al., 2018; Morris et al., 2020; Wang et al., 2021).”
BackgroundMethodSelf-citationNi, J., Zhu, C., Chen, W., & McAuley, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1030 · Cited by 30 · cited 1× · see citing works
“Jia & Liang (2017) have shown that neural question answering systems are largely affected by adversarial distracting sentences, whereas follow up work (Khashabi et al., 2017; Ni et al., 2019) proposes learning strategies that mitigate the problem.”
BackgroundRationale-Augmented Ensembles in Language ModelsPreprint
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Ed, H., & Zhou, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.00747 · Cited by 30 · cited 8× · see citing works
“This means that in addition to improving model performance on clean arithmetic reasoning tasks (Wang et al., 2022c), SC also substantially reduces the distractibility of large language models to irrelevant context.”
BackgroundInfluentialSelf-citationPandia, L., & Ettinger, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.119 · Cited by 25 · cited 1× · see citing works
“Similar issues have been found for general-purpose pretrained language models, on the tasks of factual reasoning (Kassner & Sch¨utze, 2020; Pandia & Ettinger, 2021; Misra et al., 2023; Li et al., 2022), code generation (Jones & Steinhardt, 2022), and syntactic generalization (Chaves & Richter,…”
BackgroundText and Patterns: For Effective Chain of Thought, It Takes Two to Tango
Madaan, A., & Yazdanbakhsh, A. (2022) · arXiv · doi:10.48550/arxiv.2209.07686 · Cited by 24 · cited 1× · see citing works
“In particular, Madaan & Yazdanbakhsh (2022) conclude that the correctness of numbers and equations in chain-of-thought prompts does not play a key role in model performance, but using wrong entities and removing either equations or text explanation in the reasoning steps drastically hamper the…”
BackgroundMisra, K., Rayz, J. T., & Ettinger, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.213 · Cited by 23 · cited 1× · see citing works
“…issues have been found for general-purpose pretrained language models, on the tasks of factual reasoning (Kassner & Sch¨utze, 2020; Pandia & Ettinger, 2021; Misra et al., 2023; Li et al., 2022), code generation (Jones & Steinhardt, 2022), and syntactic generalization (Chaves & Richter, 2021).”
BackgroundSelf-citationDohan, D., Xu, W., Lewkowycz, A., Austin, J., Bieber, D., Lopes, R. G., Wu, Y., Michalewski, H., Saurous, R. A., Sohl-dickstein, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.10342 · Cited by 22 · cited 2× · see citing works
“…al., 2021; Wei et al., 2022; Suzgun et al., 2022; Shi et al., 2022b, inter alia ), problem decomposition (Zhou et al., 2022; Drozdov et al., 2022; Dohan et al., 2022; Khot et al., 2022; Press et al., 2022, inter alia ), generating programs (Austin et al., 2021; Chowdhery et al., 2022; Gao et…”
BackgroundSelf-citationAdversarial Examples for Evaluating Math Word Problem Solvers
Kumar, V., Maheshwary, R., & Pudi, V. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.230 · Cited by 14 · cited 2× · see citing works
“In contrast to prior work that derives benchmark variations by substituting sentences of the base problems with variations (Patel et al., 2021; Kumar et al., 2021, inter alia ), we keep the base problem description and add to it one irrelevant sentence, while making sure that it does not affect the…”
BackgroundResultInfluentialMisra, K., Rayz, J., & Ettinger, A. (2022) · arXiv.org · doi:10.48550/arxiv.2210.01963 · Cited by 11 · cited 1× · see citing works
“…issues have been found for general-purpose pretrained language models, on the tasks of factual reasoning (Kassner & Sch¨utze, 2020; Pandia & Ettinger, 2021; Misra et al., 2023; Li et al., 2022), code generation (Jones & Steinhardt, 2022), and syntactic generalization (Chaves & Richter, 2021).”
BackgroundSelf-citationLook at that! BERT can be easily distracted from paying attention to morphosyntax
Chaves, R. P., & Richter, S. N. (2021) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/b92s-qd21 · Cited by 7 · cited 1× · see citing works
“…issues have been found for general-purpose pretrained language models, on the tasks of factual reasoning (Kassner & Sch¨utze, 2020; Pandia & Ettinger, 2021; Misra et al., 2023; Li et al., 2022), code generation (Jones & Steinhardt, 2022), and syntactic generalization (Chaves & Richter, 2021).”
BackgroundGround-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations
Yoo, K. M., Kim, J., Kim, H. J., Cho, H., Jo, H., Lee, S.-W., Lee, S.-G., & Kim, T. (2022) · arXiv · doi:10.48550/arxiv.2205.12685 · Cited by 2 · cited 2×
“A line of work studies the model performance with incorrect prompting ex-emplars, i.e., the example problems are paired with wrong answers (Min et al., 2022; Kim et al., 2022).”
Background
Flagged first, then by citations
6 of the 63 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
“The question of “when to retrieve” has led to strategies ranging from single [ Wang et al. , 2023e, Shi et al. , 2023 ] to adaptive [Jiang et al. , 2023b, Huang et al. , 2023] and multiple retrieval [Izacard et al. , 2022] methods.”
BackgroundMethodInfluentialEvaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · see citing works
“All four of the hospitalists are internal medicine physicians with emergency department experience.”
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
“Irrelevant evidence can be propagated into the generation phase, possibly tainting the output (Li et al., 2023a; Shi et al., 2023a; Cho et al., 2023; Xu et al., 2023).”
Enabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216 · see citing works
From text to insight: large language models for chemical data extraction
Schilling-Wilhelmi, M., Ríos-García, M., Shabih, S., Gil, M. V., Miret, S., Koch, C. T., Marquez, J. A., & Jablonka, K. M. (2024) · Chemical Society Reviews · doi:10.1039/d4cs00913d · Cited by 155 · see citing works
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Tang, X., Zou, A., Zhang, Z., Li, Z., Zhao, Y., Zhang, X., Cohan, A., & Gerstein, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.33 · Cited by 133 · see citing works
How understanding large language models can inform the use of ChatGPT in physics education
Polverini, G., & Gregorcic, B. (2023) · European Journal of Physics · doi:10.1088/1361-6404/ad1420 · Cited by 131 · see citing works
“Shi et al. [62] show how superfluous details can distract an LLM and degrade the quality of its responses.”
BackgroundLanguage Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
“irrelevant context (Shi et al. 2023), and similar to people, reinforcement-learning-tuned”
BackgroundAsai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.11511 · Cited by 101 · see citing works
Exploring Human-Like Translation Strategy with Large Language Models
He, Z., Liang, T., Jiao, W., Zhang, Z., Yang, Y., Wang, R., Tu, Z., Shi, S., & Wang, X. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00642 · Cited by 124 · see citing works
“We conjecture that the model-generated knowledge may contain noise that could potentially mislead the translation process [53, 1].”
BackgroundLongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.91 · Cited by 96 · see citing works
Zhang, C., Liu, X., Ziska, K., Jeon, S., Yu, C.-L., & Xu, Y. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642647 · Cited by 96 · see citing works
Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262 · see citing works
VideoAgent: Long-Form Video Understanding with Large Language Model as Agent
Wang, X., Zhang, Y., Zohar, O., & Yeung, S. Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72989-8_4 · Cited by 97 · see citing works
Transforming literature screening: The emerging role of large language models in systematic reviews
Delgado-Chaves, F. M., Jennings, M. J., Atalaia, A., Wolff, J., Horvath, R., Mamdouh, Z. M., Baumbach, J., & Baumbach, L. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2411962122 · Cited by 86 · see citing works
“However, LLMs face a significant limitation in their context window, restricting the amount of text they can process simultaneously ( 7 ), which poses challenges when dealing with lengthy scientific articles, particularly in SRs.”
Evaluating the ChatGPT family of models for biomedical reasoning and classification
Chen, S., Li, Y., Lu, S., Van, H., Aerts, H. J. W. L., Savova, G. K., & Bitterman, D. S. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad256 · Cited by 91 · see citing works
Benchmarking Cognitive Biases in Large Language Models as Evaluators
Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2023) · arXiv · doi:10.48550/arxiv.2309.17012 · Cited by 72 · see citing works
“In addition to the original instruction, we follow a similar setup from (Shi et al., 2023) where we include irrelevant information about one of the comparand models to test the ability of evaluators.”
MethodWang, T., Vargas-Díaz, D., Brown, C., & Chen, Y. (2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00018 · Cited by 68 · see citing works
“Inspired by prior work [36], M1 adds distracting information to a given problem to mislead ChatGPT, rendering the problem unsolvable by ChatGPT to prevent potential misuse.”
MethodKim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58 · see citing works
“The specific question was randomly selected from a set of questions we created in advance based on examples of real user-LLM interactions [92] and prior work [107, 126].”
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., & Farajtabar, M. (2024) · arXiv · doi:10.48550/arxiv.2410.05229 · Cited by 57 · see citing works
“Small changes in input tokens can drastically alter model outputs, indicating a strong token bias and suggesting that these models are highly sensitive and fragile (Jiang et al., 2024; Shi et al., 2023).”
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Wang, B., Yue, X., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.795 · Cited by 71 · see citing works
“Shi et al. (2023) injects irrelevant sentences into the context of math questions and finds that LLMs could be easily distracted by them.”
BackgroundLanguage agents achieve superhuman synthesis of scientific knowledge
Skarlinski, M. D., Cox, S., Laurent, J. M., Braza, J. D., Hinks, M., Hammerling, M. J., Ponnapati, M., Rodriques, S. G., & White, A. D. (2024) · arXiv · doi:10.48550/arxiv.2409.13740 · Cited by 49 · see citing works
“An important RAG parameter is the number of contexts or text chunks to include in the final text generation step (“Generate Answer” tool in Figure 1A ).”
BackgroundInfluentialLeveraging AI to improve evidence synthesis in conservation
Berger‐Tal, O., Wong, B. B. M., Adams, C. A., Blumstein, D. T., Candolin, U., Gibson, M., Greggor, A. L., Lagisz, M., Macura, B., Price, C. J., et al. (2024) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2024.04.007 · Cited by 50 · see citing works
Yang, Y., Zhou, X., Mao, R., Xu, J., Yang, L., Zhang, Y., Shen, H., & Zhang, H. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112234 · Cited by 45 · see citing works
Collaborative, T. C. (2025) · BMJ · doi:10.1136/bmj-2024-083305 · Cited by 50 · see citing works
“4 Generative AI outputs vary considerably because of context, 33 requiring detailed documentation of the prompts used to elicit model output.”
Chorus: Foundation Models for Unified Data Discovery and Exploration
Kayali, M., Lykov, A., Fountalis, I., Vasiloglou, N., Olteanu, D., & Suciu, D. M. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3659437.3659461 · Cited by 52 · see citing works
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 36 · see citing works
“…of their generated content have drawn significant attention from the community (Bang et al., 2023; Alkaissi & McFarlane, 2023; Zheng et al., 2023; Shi et al., 2023; Carlini et al., 2021; Huang et al., 2022; Shao et al., 2023; Li et al., 2023; Wei et al., 2023; Zhou et al., 2023b; Zou et al.,…”
BackgroundSelf-citationThe Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., & Evans, O. (2023) · arXiv · doi:10.48550/arxiv.2309.12288 · Cited by 36 · see citing works
“Previous research has found similar inconsistencies in LLMs (Fluri et al., 2023; Elazar et al., 2021; Press et al., 2023; Hosseini et al., 2021; Lin et al., 2022; Shi et al., 2023) Forward vs backward recall in humans Does the Reversal Curse apply to humans?”
MethodLanguage Models Meet World Models: Embodied Experiences Enhance Language Models
Xiang, J., Tao, T., Gu, Y., Shu, T., Wang, Z., Yang, Z., & Hu, Z. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3295 · Cited by 34 · see citing works
Thanasi-Boçe, M., & Hoxha, J. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12775-z · Cited by 38 · see citing works
Exploring the Effectiveness of LLMs in Automated Logging Statement Generation: An Empirical Study
Li, Y., Huo, Y., Jiang, Z., Zhong, R., He, P., Su, Y., Briand, L., & Lyu, M. R.-T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3475375 · Cited by 32 · see citing works
“Additionally, LLM performance tends to decline with longer inputs, even when within the specified context length [90], [91].”
Project-specific code summarization with in-context learning
Yun, S. M., LIN, S.-J., Gu, X., & Shen, B. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112149 · Cited by 29 · see citing works
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
Du, Y., Tian, M., Ronanki, S., Rongali, S., Bodapati, S. B., Galstyan, A., Wells, A., Schwartz, R., Huerta, E. A., & Peng, H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1264 · Cited by 39 · see citing works
Šuster, S., Baldwin, T., & Verspoor, K. M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1749 · Cited by 30 · see citing works
“58,59 For example, Shi et al. 59 show that LLMs are sensitive to irrelevant information in arithmetic reasoning problems.”
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances
Zhang, Z., Fang, M., Chen, L., Namazi‐Rad, M., & Wang, Y.-X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.516 · Cited by 37 · see citing works
Gu, X., Chen, X., Lu, P., Li, Z., Du, Y., & Li, X. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.107907 · Cited by 29 · see citing works
Self-citationA survey on knowledge-enhanced multimodal learning
Lymperaiou, M., & Stamou, G. B. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10825-z · Cited by 35 · see citing works
ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models
Chen, Z., Zhou, K., Zhang, B., Gong, Z., Zhao, X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.985 · Cited by 30 · see citing works
METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities
Hyun, S., Guo, M., & Babar, M. A. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00019 · Cited by 26 · see citing works
“Recent studies have conducted testing with regards to the Quality Attributes (QAs), such as robustness, of LLMs by generating adversarial input texts and prompts [4]–[13].”
BackgroundInfluentialInvestigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following
Ye, S., Hwang, H., Yang, S., Yun, H., Kim, Y., & Seo, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29909 · Cited by 37 · see citing works
“This suggests the advantage of TAPP in real-time LLM serving scenarios: while the users can save the effort of manually constructing twice more task-specific demonstrations for each task, they can achieve similar performance by simply prepending TAPP to the input.”
Alam, M. F., Lentsch, A., Yu, N., Barmack, S., Kim, S., Acemoğlu, D., Hart, A. J., Johnson, S., & Ahmed, F. (2024) · doi:10.21428/e4baedd9.e39b392d · Cited by 26 · see citing works
“A major limitation of current Gen-AI technologies is the lack of reasoning capabilities (J. Huang et al. 2023; Dziri et al. 2023; Turpin et al. 2023; Shi et al. 2023).”
BackgroundInfluence of External Information on Large Language Models Mirrors Social Cognitive Patterns
Bian, N., Lin, H., Liu, P., Lu, Y., Zhang, C.-P., He, B., Han, X., & Sun, L. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3476030 · Cited by 36 · see citing works
“Other studies have examined LLMs’ vulnerability to adversarial attacks in input texts [19], [20], [21], [22], [23], [24], [25], [26].”
Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23 · see citing works
Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models
Hu, Z., Wang, C., Shu, Y., Paik, H.-Y., & Zhu, L. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671932 · Cited by 27 · see citing works
“Most of the time, the search ends up with a prefix that fails to move the embedding of the prompt close to that of the target passage.”
BackgroundLanguage Models Meet World Models: Embodied Experiences Enhance Language Models
Xiang, J., Tao, T., Gu, Y., Shu, T., Wang, Z., Yang, Z., & Hu, Z. (2023) · arXiv · Cited by 34 · see citing works
ERA-CoT: Improving Chain-of-Thought through Entity Relationship Analysis
Liu, Y., Peng, X., Du, T., Yin, J., Liu, W., & Zhang, X. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.476 · Cited by 22 · see citing works
Agentless: Demystifying LLM-based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2407.01489 · Cited by 24 · see citing works
“That is to say they tend to take all information/feedback and do not know how to filter out or correct irrelevant, incorrect, or misleading information [84, 107].”
BackgroundTheory of Mind Abilities of Large Language Models in Human-Robot Interaction: An Illusion?
Verma, M., Bhambri, S., & Subbarao Kambhampati, B. (2024) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3610978.3640767 · Cited by 21 · see citing works
“2, a robot can leverage the LLM to perform an internal critique while synthesizing behavior with a human-in-the-loop.”
BackgroundDescribing Differences in Image Sets with Natural Language
Dunlap, L., Zhang, Y., Wang, X., Zhong, R., Darrell, T., Steinhardt, J., Gonzalez, J. E., & Yeung, S. Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02284 · Cited by 19 · see citing works
The Fact Selection Problem in LLM-Based Program Repair
Parasaram, N., Yan, H., Yang, B., Flahy, Z., Qudsi, A., Ziaber, D., Barr, E. T., & Mechtaev, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00162 · Cited by 20 · see citing works
“We posit that this is because redundant or irrelevant information in the original prompt “distracted” the LLM from critical details in the GitHub issue (highlighted in Figure 1) necessary to repair the bug [12].”
Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generation
Li, Y., Liu, G., Wang, C., & Yang, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889013 · Cited by 31 · see citing works
Software Engineering Using Autonomous Agents: Are We There Yet?
Suri, S., Das, S. N., Singi, K., et al. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00174 · Cited by 17 · see citing works
“Avoiding irrelevant or insufficient information was crucial to prevent distractions that could degrade the model’s performance and hinder task comprehension [5].”
BackgroundMamalis, M. E., Kalampokis, E., Karamanou, A., et al. (2023) · doi:10.1145/3635059.3635068 · Cited by 17 · see citing works
“In the opposite scenario, where the context is not fitting, it may lead to worse performance [25], so the selection of the context is of critical importance.”
BackgroundGenerating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generation
Li, Y., Liu, G., Wang, C., & Yang, Y. (2024) · arXiv · doi:10.48550/arxiv.2406.19234 · Cited by 17 · see citing works
Misra, K., Rayz, J. T., & Ettinger, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.213 · Cited by 23 · see citing works
Self-citationThe Criticality of Implementing Principled Design When Using AI Technologies in Test Development
Bolender, B., Foster, C., & Vispoel, S. (2023) · Language Assessment Quarterly · doi:10.1080/15434303.2023.2288266 · Cited by 18 · see citing works
“…ability to recognize and mitigate features of text that could contribute to construct irrelevant variance in test outcomes, such as including concepts that are tangentially related to the topic, but not salient, given the context and purpose of the test (Pandia & Ettinger, 2021; Shi et al., 2023).”
BackgroundAdaptive Control of Retrieval-Augmented Generation for Large Language Models Through Reflective Tags
Yao, C., & Fujita, S. (2024) · Electronics · doi:10.3390/electronics13234643 · Cited by 17 · see citing works
Building the AI-Enabled Medical School of the Future
Succi, M. D., Chang, B. S., & Rao, A. S. (2025) · JAMA · doi:10.1001/jama.2025.2789 · Cited by 16 · see citing works
Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement
Xi, Z., Jin, S., Zhou, Y., Zheng, R., Gao, S., Liu, J., Gui, T., Zhang, Q., & Huang, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.762 · Cited by 23 · see citing works
In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents
Tan, Z., Yan, J., Hsu, I.-H., Han, R., Wang, Z., Le, L., Song, Y., Chen, Y., Palangi, H., Lee, G., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.413 · Cited by 16 · see citing works
Sun, S., Zhang, K., Li, J., Yu, M., Hou, K., Wang, Y., & Cheng, X. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104147 · Cited by 17 · see citing works
SAMamba: Structure-Aware Mamba for Ethereum Fraud Detection
Huang, T., Huang, J.-H., Dong, C., Duan, S.-S., & Pang, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3589015 · Cited by 14 · see citing works
“However, GT is easily distracted by irrelevant context when densely routing global information [46].”
Cyber Shadows: Neutralizing Security Threats With AI and Targeted Policy Measures
Schmitt, M., & Koutroumpis, P. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3527398 · Cited by 14 · see citing works
“First, Large language models can be easily distracted by irrelevant context which significantly reduces the consistency and performance [6].”
Is Semantic Chunking Worth the Computational Cost?Preprint
Qu, R., Bao, F. S., & Tu, R. (2024) · Qeios · doi:10.32388/m7ykdz · Cited by 13 · see citing works
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 12 · see citing works
“(2) Longer prompts introduce irrelevant and redundant information, which can weaken LLMs’ performance (Shi et al., 2023), as illustrated in Figure 1a. (3) LLMs exhibit position bias (Kamradt, 2023), also known as the "lost in the middle" issue (Liu et al., 2024), suggesting that the placement of…”
BackgroundOverloaded minds and machines: a cognitive load framework for human-AI symbiosis
Wang, P., Liu, H., Zou, L., & PAAS, F. G. W. C. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11510-z · Cited by 17 · see citing works
“Transformer architectures distribute a finite attention budget across every token in the prompt (Oymak et al. 2023 ); irrelevant text therefore competes directly with relevant instructions (Shi et al. 2023 ).”
MathAttack: Attacking Large Language Models towards Math Solving Ability
Zhou, Z., Wang, Q., Jin, M., Yao, J., Ye, J., Liu, W., Wang, W., Huang, X., & Huang, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29949 · Cited by 18 · see citing works
A comprehensive review of techniques for documenting artificial intelligence
Königstorfer, F. (2024) · Digital Policy Regulation and Governance · doi:10.1108/dprg-01-2024-0008 · Cited by 12 · see citing works
Fodor, J., Deyne, S. D., & Suzuki, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00536 · Cited by 14 · see citing works
Senath, T., Athukorala, K., Costa, R., Ranathunga, S., & Kaur, R. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100177 · Cited by 15 · see citing works
Ibrahim, M., Wan, Z., Li, H., Panda, P., Krishna, T., Kanerva, P., Chen, Y., & Raychowdhury, A. (2024) · International Conference on Hardware/Software Codesign and System Synthesis · doi:10.1109/codes-isss60120.2024.00012 · Cited by 13 · see citing works
A Study on Prompt Injection Attack Against LLM-Integrated Mobile Robotic Systems
Zhang, W., Kong, X., Dewitt, C., Bräunl, T., & Hong, J. B. (2024) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew63542.2024.00103 · Cited by 11 · see citing works
“LLMs possess advanced capabilities for reasoning and processing complex inputs but are highly susceptible to various input variations [3].”
BackgroundOn the Impact of Requirements Smells in Prompts: The Case of Automated Traceability
Vogelsang, A., Korn, A., Broccia, G., Ferrari, A., Fischbach, J., & Arora, C. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00016 · Cited by 12 · see citing works
“We hypothesize that the effect of smells on downstream tasks will be stronger in complex systems due to the increased difficulty LLMs face handling complex contextual information [17].”
GROVE: A Retrieval-augmented Complex Story Generation Framework with A Forest of Evidence
Wen, Z., Tian, Z., Wu, W., Yang, Y., Shi, Y., Huang, Z., & Li, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.262 · Cited by 15 · see citing works
Scientific evidence and specific context: leveraging large language models for health fact-checking
Ni, Z., Qian, Y., Chen, S., Jaulent, M., & Cédric Bousquet, B. (2024) · Online Information Review · doi:10.1108/oir-02-2024-0111 · Cited by 10 · see citing works
Smirnov, E. A. (2024) · Qualitative Research in Psychology · doi:10.1080/14780887.2024.2428255 · Cited by 13 · see citing works
“(4) Using only necessary data : LLMs are prone to irrelevant context (Shi et al. 2023).”
Enhancing risk and crisis communication with computational methods: A systematic literature review
Munro, M. H., Gore, R. J., Lynch, C. J., Hastings, Y. D., & Reinhold, A. M. (2024) · Risk Analysis · doi:10.1111/risa.17690 · Cited by 9 · see citing works
MARSHA: multi-agent RAG system for hazard adaptation
Xie, Y., Jiang, B., Mallick, T., Bergerson, J., Hutchison, J. K., Verner, D. R., Branham, J., Alexander, M. R., Ross, R. B., Feng, Y. W., et al. (2025) · npj Climate Action · doi:10.1038/s44168-025-00254-1 · Cited by 19 · see citing works
“Wenotethattheseissuesare notuniquetoWild fi reGPT 99,100 , and future improvements should focus on enhancing the language model to mitigate hallucinations.”
Huda, N. U., Sahito, S. F., Gilal, A. R., Abro, A., Alshanqiti, A. M., Alsughayyir, A., & Palli, A. S. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.0150442 · Cited by 11 · see citing works
“Sentence length [52][53], and 5.”
Sang, V. M., Khuong, B. D., Nhi, L. Y., et al. (2024) · Cogent Business & Management · doi:10.1080/23311975.2024.2425706 · Cited by 8 · see citing works
“…2020; chakraborty & Bhat, 2018; Fan et al., 2022; hammedi et al., 2015; hidalgo‑alcázar et al., 2014; Moody et al., 2022; Mudambi & schuff, 2010; shi, 2023; Wibisono et al., 2021; Yang et al., 2017); and (3) the quality of video needs to focus on high resolution and quality video, clear and…”
InfluentialOptimizing Human-AI Interaction: Innovations in Prompt Engineering
Deshmukh, R., Raut, R., Bhavsar, M., Gurav, S., & Patil, Y. (2025) · International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT) · doi:10.1109/idciot64235.2025.10914815 · Cited by 8 · see citing works
Chatter Generation through Language Models
Müller-Brockhausen, M., Barbero, G., & Preuß, M. (2023) · IEEE Conference on Games (CoG) · doi:10.1109/cog57401.2023.10333244 · Cited by 8 · see citing works
“This evolution will be necessary, as small changes in the context, which in this case would be the dialogue history, are enough to confuse an LM and make it give faulty answers [31], [32].”
BackgroundRakin, S., Shibly, M. A. R., Hossain, Z. M., Akbar, M. M., & Khan, Z. (2025) · Advances in intelligent systems and computing · doi:10.1007/978-3-031-89063-5_42 · Cited by 12 · see citing works
Harnessing generative AI for enhanced disaster management: a systematic review
Saengtabtim, K., Leelawat, N., Aumnoysombat, R., Adelifar, M., Saengwongwattana, N., Suktavornprasit, G., Suppasri, A., Imamura, F., & Tang, J. (2025) · Big Earth Data · doi:10.1080/20964471.2025.2521157 · Cited by 7 · see citing works
“However, the social media data fed to the LLM must be screened out for some irrelevant data (Shi et al., 2023).”
One Size Doesn’t Fit All: Predicting the Number of Examples for In-Context Learning
Chandra, M., Ganguly, D., & Ounis, I. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_5 · Cited by 10 · see citing works
“…for LLMs from two broad perspectives: a) prompt tuning in the embedding space (Li and Liang, 2021; Liu et al., 2022b; Qin and Eisner, 2021; Liu et al., 2023), and b) prompt searching in the text space (Lu et al., 2022; Zhang et al., 2023; Diao et al., 2023; Liu et al., 2022a; Shi et al., 2023).”
BackgroundUnderstanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
Zhang, S., Ma, R., Ma, Y., Li, S., Xu, Y., Yi, X., & Li, H. (2025) · Proceedings of the 2025 Workshop on Human-Centered AI Privacy and Security · doi:10.1145/3733816.3760750 · Cited by 6 · see citing works
Predicting Missing Values in Survey Data Using Prompt Engineering for Addressing Item Non-Response
Ji, J., Kim, J., & Kim, Y. (2024) · Future Internet · doi:10.3390/fi16100351 · Cited by 6 · see citing works
Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation
Gao, X., Xiong, Y., Wang, D., Guan, Z., Shi, Z., Wang, H., & Li, S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694987 · Cited by 6 · see citing works
Yan, Z., Zhang, R., & Jia, F. (2024) · Proceedings of the 2024 9th International Conference on Distance Education and Learning · doi:10.1145/3675812.3675837 · Cited by 7 · see citing works
Islam, C. M., Salman, S., Shams, M., Liu, X., & Kumar, P. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802618 · Cited by 6 · see citing works
“Several recent studies have also addressed adversarial attacks on multimodal models, which can potentially jailbreak aligned LLMs or VLMs ([24], [25], [26]).”
MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models
Nathani, D., Wang, D., Pan, L., & Wang, W. Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.407 · Cited by 6 · see citing works
Get an A in Math: Progressive Rectification Prompting
Wu, Z., Jiang, M., & Shen, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29898 · Cited by 6 · see citing works
Yang, Y., Zhou, X., Mao, R., Xu, J., Yang, L., Zhang, Y., Shen, H., & Zhang, H. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4831591 · Cited by 6 · see citing works
Context Is King: Large Language Models’ Interpretability in Divergent Knowledge Scenarios
Piñeiro-Martín, A., Santos-Criado, F.-J., García-Mateo, C., Docío-Fernández, L., & López-Pérez, M. D. C. (2025) · Applied Sciences · doi:10.3390/app15031192 · Cited by 4 · see citing works
ALCUNA: Large Language Models Meet New Knowledge
Yin, X., Huang, B., & Wan, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.87 · Cited by 10 · see citing works
Levy, M., Ravfogel, S., & Goldberg, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.569 · Cited by 4 · see citing works
Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis
Zheng, H., & Saparov, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.277 · Cited by 5 · see citing works
SUGAR: Leveraging Contextual Confidence for Smarter Retrieval
Zubkova, H., Park, J.-H., & Lee, S.-W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890064 · Cited by 7 · see citing works
“Moreover, the retrieved results sometimes contain documents that are irrelevant [11], factually incorrect [12] or even contain harmful information [13].”
Effective Context Selection in LLM-Based Leaderboard Generation: An Empirical Study
Kabongo, S., D’Souza, J., & Auer, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70242-6_15 · Cited by 4 · see citing works
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024) · arXiv · doi:10.48550/arxiv.2410.10813 · Cited by 3 · see citing works
“Although the retrieval accuracy can be enhanced through the preceding designs, it does not guarantee that the LLM can effectively reason over the extensive context (Shi et al., 2023; Liu et al., 2024).”
- Authors
- Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D.
- Affiliations
- Toyota Technological Institute at Chicago (US)
- Purdue University West Lafayette (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.00093
- OpenAlex
- W4319049323
- Semantic Scholar
- 3d68522abfadfc8ee6b7ec9edaaf91f1b2f38e5e
- Cited by
- 134
- References
- 63
- Citations by year
- 2026: 7 · 2025: 43 · 2024: 55 · 2023: 29
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, distractibility, arithmetic reasoning, prompting techniques, instruction following, problem-solving accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
