Limitations of large language models in clinical problem-solving arising from inflexible reasoning

LLMs often demonstrate lack of commonsense medical reasoning.

“often demonstrating lack of commonsense medical reasoning and a propensity to hallucinate.”

Medalpaca, Meditron3-7B, and Meditron3-70B models approached 0% performance due to tangential reasoning patterns.

“We observed that Medalpaca, Meditron3-7B, and Meditron3-70B models approached 0% performance due to tangential reasoning patterns (Supplementary Tables 2 and 3)”

GPT-4o's incorrect response and subsequent reasoning in Figure 5 reveal a deficiency in medical commonsense reasoning.

“GPT4o’s incorrect response and subsequent reasoning reveal a deficiency in medical commonsense reasoning.”

LLMs perform poorly compared to physicians on mARC-QA.

“We find that LLMs, including current state-of-the-art o1, Gemini, Claude, and DeepSeek models, perform poorly compared to physicians on mARC-QA”

Smaller models such as Mistral, GPT4o-mini, Gemini-1.5-flash, and Claude-Sonnet had even greater overconfidence despite achieving lower accuracy.

“We found that smaller models such as Mistral, GPT4o-mini, Gemini-1.5-flash, and Claude-Sonnet had even greater overconfidence despite achieving lower accuracy.”

Most LLMs perform poorly on mARC-QA tasks, with less than 50% accuracy.

“We observed that most LLMs perform poorly on mARC-QA tasks, with less than 50% accuracy (Figure 2).”

DeepSeek-R1 demonstrates a similar pattern of failure to o1.

“DeepSeek-R1 also demonstrate a similar pattern of failure (Supplementary Table 4 and 5).”

GPT-4o's incorrect response in Figure 4 arises from a deductive reasoning error in integrating key details about the patient's condition.

“GPT4o’s incorrect response arises from a deductive reasoning error in integrating key details about the patient’s condition”

Several models performed near or below the chance level (less than 20%).

“We note that several models performed near or below the chance level (less than 20%).”

LLama 3.1 405b performed worse than LLama 3.1 70b.

“We observed decreased performance with larger model size (405b vs 70b) for the LLama 3.1 model family”

DeepSeek-R1's mean accuracy on the human-miss subset was 51% (32.0-69.8%).

“It’s mean accuracy on this subset was 51% (32.0–69.8%).”

The best-performing models DeepSeek-R1, DeepSeek-V3, Gemini(1.5-pro), and o1 achieved accuracies of 52%, 50%, 50%, and 48%, respectively.

“The best-performing models, DeepSeek-R1, DeepSeek-V3, Gemini(1.5-pro), and o1, achieved accuracies of 52%, 50%, 50%, and 48%, respectively.”

The human-miss set comprised 20/100 items (20%).

“This comprised 20/100 items (20%).”

There is a general trend for improvement with larger model sizes in the Claude, Gemini, and OpenAI families.

“We observed a general trend for improvement with larger model sizes in the Claude, Gemini, and OpenAI families.”

Human average performance on mARC-QA was 66%, averaged across five physicians, with standard error ±5.3%.

“Human average performance was 66%, averaged across five physicians, with standard error ±5.3%.”

Inspection of the reasoning trace for DeepSeek-R1 confirms that the source of error in this example is hallucination that forehead blood pressure measurement is possible.

“Inspection of the reasoning trace for DeepSeek-R1, also confirms that the source of error in this example is hallucination that forehead blood pressure measurement is possible (Supplementary Table 4).”

Under more stringent criteria for incorrectness, the human-miss set comprised only 3 items.

“Under more stringent criteria for incorrectness defined as physician majority with () incorrect, the human-miss set comprised only 3 items (Fig. 6).”

Human performance was significantly higher than the performance of each respective LLM.

“Human performance was significantly higher than the performance of each respective LLM (paired bootstrap, for DeepSeek-V3, for DeepSeek-R1, for Gemini(1.5-pro), for o1, and for all others.”

mARC induced striking reductions in accuracy and Brier Score compared to MMLU-Pro Professional Clinical Knowledge subset.

“We assessed accuracy and Brier Score on MMLU-Pro Professional Clinical Knowledge subset (254 questions)29, and comparatively, mARC induced striking reductions in accuracy and Brier Score.”

• cites [29] Wang, Yubo, Ma, Xueguang, Zhang, Ge, Ni, Yuansheng, Chandra, Abhranil, Guo, Shiguang, Ren, Weiming, Arulraj, Aaran, He, Xuan, Jiang, Ziyan, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, (2024).Not checked yet

  Not checked in the free run.

Human performance on mARC-QA was limited.

“Lastly, we observe that human performance on mARC-QA was limited, consistent with long-standing findings that humans may be susceptible to the Einstellung effect69.”

• AssumptionRests on an assumption: mARC-QA validly measures Einstellung-driven reasoning or reasoning flexibility.

mARC-QA reveals limitations in LLM medical reasoning.

“mARC-QA reveals limitations in LLM medical reasoning, challenging the notion that human-level performance on medical QA benchmarks suggests robust medical reasoning capabilities.”

• AssumptionRests on an assumption: mARC-QA validly measures Einstellung-driven reasoning or reasoning flexibility.

The average human performance on mARC-QA was 66%.

“The average performance ( 66%)—comparable to typical accuracy on board examinations and in-training assessments70—reflects the inherent variability in effort and reasoning abilities among subjects.”

On the nine items in the human miss-set, the best model (DeepSeek-R1) outperformed humans.

“On the nine items in the human miss-set where the best model (DeepSeek-R1) outperformed humans (Figure 7), qualitatively, inspection of answer explanations (Supplementary Table 7) indicate that the model appeared to properly adhere to constraints in the question stem and follow logical negation, whereas the human majority did not.”

Each mARC-QA item instantiates at least one manipulation intended to preferentially trigger a familiar heuristic while making that heuristic counterproductive.

“Each item instantiates at least one of the following manipulations intended to preferentially trigger a familiar heuristic while making that heuristic counterproductive”

A recent study revealed a substantial discrepancy between LLMs' miscalibrated overconfidence in their outputs and their actual accuracy.

“a recent study revealed a substantial discrepancy between LLMs’ miscalibrated overconfidence in their outputs and their actual accuracy, underscoring the risks of overreliance on LLMs in the medical domain25.”

• cites [25] Griot, Maxime, Hemptinne, Coralie, Vanderdonckt, Jean & Yuksel, Demet. Large language models lack essential metacognition for reliable medical reasoning. Nat. Commun. 16(1), 642 (2025). sourceNot checked yet

  Not checked in the free run.

LLMs demonstrated limited performance in providing medical recommendations in real-world emergency room encounters.

“Notably, LLMs also demonstrated limited performance in providing medical recommendations in real-world emergency room encounters in a recent large-scale study13”

• cites [13] Williams, Christopher YK., Miao, Brenda Y., Kornblith, Aaron E. & Butte, Atul J. Evaluating the use of large language models to provide clinical recommendations in the emergency department. Nat. Commun. 15(1), 8236 (2024). sourceNot checked yet

  Not checked in the free run.

The mARC-QA dataset comprises 100 questions written by the authors to resist memorization, pattern matching, or interpolation from preexisting medical QA benchmarks and medical texts.

“The dataset comprises 100 questions written by the authors to resist memorization, pattern matching, or interpolation from preexisting medical QA benchmarks and medical texts.”

mARC-QA items were constrained so that the correct choice requires at most broadly taught, early-clinical knowledge.

“items were constrained so that the correct choice requires at most broadly taught, early-clinical knowledge.”

LLMs' capabilities in simulated real-world medical scenarios are more limited than their performance on medical QA benchmarks.

“While LLM performance on medical QA benchmarks has been demonstrated to rival human-level performance, their capabilities in simulated real-world medical scenarios have been more limited12.”

• cites [12] Hager, Paul et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. 30(9), 2613–2622 (2024). sourceNot checked yet

  Not checked in the free run.

LLMs have attained human-level accuracy on medical QA benchmarks.

“Large Language Models (LLMs) have attained human-level accuracy on medical question-answer (QA) benchmarks.”

• Also stated in: § Introduction · boosted

  “LLMs have demonstrated high accuracy on the United States Medical Licensing Exam (USMLE)3, USMLE-styled question banks4–6, subspecialty medical board examinations7,8, and clinical reasoning benchmarks validated for physicians9.”

• cites [3] Gilson, Aidan et al. How does chatgpt perform on the united states medical licensing examination (usmle)? the implications of large language models for medical education and knowledge assessment. JMIR Med. Educ. 9(1), e45312 (2023). sourceNot checked yet

  Not checked in the free run.

• cites [4] Jiang, Lavender Yao et al. Health system-scale language models are all-purpose prediction engines. Nature 619(7969), 357–362 (2023). sourceNot checked yet

  Not checked in the free run.

• cites [5] Peng, Cheng et al. A study of generative large language model for medical research and healthcare. NPJ Digit. Med. 6(1), 210 (2023). sourceNot checked yet

  Not checked in the free run.

• cites [6] Zhang, Kai, Zhou, Rong, Adhikarla, Eashan, Yan, Zhiling, Liu, Yixin, Yu, Jun, Liu, Zhengliang, Chen, Xun, Davison, Brian D, Ren, Hui, et al. A generalist vision–language foundation model for diverse biomedical tasks. Nature Medicine, 1–13, (2024).Not checked yet

  Not checked in the free run.

• cites [7] Longwell, Jack B. et al. Performance of large language models on medical oncology examination questions. JAMA Netw. Open 7(6), e2417641–e2417641 (2024). sourceNot checked yet

  Not checked in the free run.

• cites [8] Schubert, Marc Cicero, Wick, Wolfgang & Venkataramani, Varun. Performance of large language models on a neurology board-style examination. JAMA Netw. Open 6(12), e2346721–e2346721 (2023). sourceNot checked yet

  Not checked in the free run.

Striking failure modes of LLMs in seemingly trivial reasoning tasks have been identified.

“striking failure modes of LLMs in seemingly trivial reasoning tasks have been identified18,19.”

• cites [18] Nezhurina, Marianna, Cipolina-Kun, Lucia, Cherti, Mehdi, & Jitsev, Jenia. Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models. arXiv preprint arXiv:2406.02061, (2024).Not checked yet

  Not checked in the free run.

• cites [19] Li, Yinghui, Zhou, Qingyu, Luo, Yuanzhen, Ma, Shirong, Li, Yangning, Zheng, Hai-Tao, Hu, Xuming, & Philip, S Yu. When llms meet cunning texts: A fallacy understanding benchmark for large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, (2024).Not checked yet

  Not checked in the free run.

LLMs have demonstrated limited performance in clinical scenarios demanding flexible reasoning or information-seeking strategies.

“LLMs have demonstrated limited performance in clinical scenarios demanding flexible reasoning or information-seeking strategies23,24.”

• cites [23] Li, Shuyue Stella, Balachandran, Vidhisha, Feng, Shangbin, Ilgen, Jonathan S, Pierson, Emma, Koh, Pang Wei, & Tsvetkov, Yulia. Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, (2024). sourceNot checked yet

  Not checked in the free run.

• cites [24] Pfohl, Stephen R., Cole-Lewis, Heather, Sayres, Rory, Neal, Darlene, Asiedu, Mercy, Dieng, Awa, Tomasev, Nenad, Rashid, Qazi Mamunur, Azizi, Shekoofeh, Rostamzadeh, Negar, et al. A toolbox for surfacing health equity harms and biases in large language models. Nature Medicine, pages 1–11, (2024). sourceNot checked yet

  Not checked in the free run.

The Abstraction and Reasoning Corpus (ARC) reveals surprising deficiencies of LLMs' ability to reason in tasks that even children may solve.

“the Abstraction and Reasoning Corpus (ARC) introduced by Francois Chollet20 reveals surprising deficiencies of LLMs’ ability to reason in tasks that even children may solve”

• cites [20] Chollet, François. On the measure of intelligence. arXiv preprint arXiv:1911.01547, (2019).Not checked yet

  Not checked in the free run.

The limited reasoning capabilities of LLMs have been partially attributed to their reliance on memorization of tasks seen frequently during training, leading to a loss of generalization for novel tasks.

“The limited reasoning capabilities of LLMs have been partially attributed to their reliance on memorization of tasks seen frequently during training, leading to a loss of generalization for novel tasks22.”

• cites [22] Wu, Zhaofeng, Qiu, Linlu, Ross,Alexis, Akyürek, Ekin, Chen, Boyuan, Wang, Bailin, Kim, Najoung, Andreas, Jacob, & Kim, Yoon. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477, (2023). sourceNot checked yet

  Not checked in the free run.

Many of the observed errors likely reflect the absence of a robust world model-an internal causal representation that enforces physical and logical constraints, maintains latent state, and supports plausibility and counterfactual checks.

“More broadly, many of the observed errors likely reflect the absence of a robust world model—an internal causal representation that enforces physical and logical constraints, maintains latent state, and supports plausibility and counterfactual checks.”

The disparity between LLM performance on mARC-QA and conventional medical QA aligns with studies suggesting that benchmark successes may stem from overfitting to statistical patterns in training data rather than reflecting emergent reasoning abilities.

“The disparity between LLM performance on mARC-QA and conventional medical QA aligns with studies suggesting that benchmark successes may stem from overfitting to statistical patterns in training data rather than reflecting emergent reasoning abilities19,54–57.”

• cites [19] Li, Yinghui, Zhou, Qingyu, Luo, Yuanzhen, Ma, Shirong, Li, Yangning, Zheng, Hai-Tao, Hu, Xuming, & Philip, S Yu. When llms meet cunning texts: A fallacy understanding benchmark for large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, (2024).Not checked yet

  Not checked in the free run.

• cites [54] McCoy, R Thomas, Yao, Shunyu, Friedman, Dan, Hardy, Matthew, & Griffiths, Thomas L. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638, (2023). sourceNot checked yet

  Not checked in the free run.

• cites [55] Goetz, Lea, Seedat, Nabeel, Vandersluis, Robert & van der Schaar, Mihaela. Generalization—a key challenge for responsible ai in patient-facing clinical applications. npj Digital Medicine 7(1), 126 (2024). sourceNot checked yet

  Not checked in the free run.

• cites [56] Moskvichev, Arseny, Odouard, Victor Vikram, & Mitchell, Melanie. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain. arXiv preprint arXiv:2305.07141, (2023).Not checked yet

  Not checked in the free run.

• cites [57] Dong, Yihong, Jiang, Xue, Liu, Huanyu, Jin, Zhi, Gu, Bin, Yang, Mengfei, & Li, Ge. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. arXiv preprint arXiv:2402.15938, (2024). sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: mARC-QA validly measures Einstellung-driven reasoning or reasoning flexibility.

Long-tail concepts are difficult for LLMs to capture effectively.

“mARC-QA targets this inductive bias by disrupting the predictability of familiar medical problems through incorporation of long-tail concepts which are difficult for LLMs to capture effectively60.”

• cites [60] Kandpal, Nikhil, Deng, Haikang, Roberts, Adam, Wallace, Eric, & Raffel, Colin. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696–15707. PMLR, (2023).Not checked yet

  Not checked in the free run.

This pattern argues against presence of systematic benchmark flaws.

“This pattern argues against presence of systematic benchmark flaws.”

mARC-QA examples demonstrate the adversarial strategy of disrupting the predictability of familiar medical text patterns to exploit the Einstellung effect.

“mARC-QA examples (Figures 3, 4, 5) demonstrate the adversarial strategy of disrupting the predictability of familiar medical text patterns to exploit the Einstellung effect”

It is anticipated that LLMs may exhibit inductive biases that bear functional resemblance to cognitive biases in humans.

“it is anticipated that LLMs may exhibit inductive biases that bear functional resemblance to cognitive biases in humans51–53.”

• cites [51] Echterhoff, Jessica, Liu, Yao, Alessa, Abeer, McAuley, Julian & He, Zexue. Cognitive bias in decision-making with llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, 12640–12653 (2024).Not checked yet

  Not checked in the free run.

• cites [52] Liu, Xuan, Zhang, Jie, Guo, Song, Shang, Haoyang, Yang, Chengxu, & Zhu, Quanyan. Exploring prosocial irrationality for llm agents: A social cognition view. arXiv preprint arXiv:2405.14744, (2024).Not checked yet

  Not checked in the free run.

• cites [53] Naeini, Saeid, Saqur, Raeid, Saeidi, Mozhgan, Giorgi, John, & Taati, Babak. Large language models are fixated by red herrings: Exploring creative problem solving and einstellung effect using the only connect wall dataset. arXiv preprint arXiv:2306.11167, (2023).Not checked yet

  Not checked in the free run.

Varying the age of the subject in each question by up to 10 days between runs does not clinically alter the medical principle or reasoning being assessed for questions in this dataset.

“This does not clinically alter the medical principle or reasoning that is being assessed for questions in this dataset, as no subjects are of neonatal or infantile age.”

The mARC-QA benchmark utilizes an adversarial framework to probe failure modes potentially linked to inflexibility in LLM reasoning.

“Here, we introduce the Medical Abstraction and Reasoning Corpus Question and Answer (mARC-QA) benchmark, which utilizes an adversarial framework to probe failure modes potentially linked to inflexibility in LLM reasoning.”

LLMs are vulnerable to the Einstellung effect in medical QA tasks, where their inflexible adherence to matching learned statistical patterns impedes effective adaptation to medical scenarios that deviate from traditional medical texts and QA.

“Here, we demonstrate that LLMs are vulnerable to the Einstellung effect in medical QA tasks, where their inflexible adherence to matching learned statistical patterns impedes effective adaptation to medical scenarios that deviate from traditional medical texts and QA.”

• AssumptionRests on an assumption: The mARC-QA design manipulations actually induce the Einstellung effect in LLMs.

LLMs may be biased towards the Einstellung effect due to their training paradigm involving next-token prediction of textual patterns prevalent in their training data.

“which LLMs may be biased towards due to their training paradigm involving the next-token prediction of textual patterns prevalent in their training data.”

• AssumptionRests on an assumption: The mARC-QA design manipulations actually induce the Einstellung effect in LLMs.

mARC-QA alters predictable aspects of medical problems, emphasizing long-tail or low-probability reasoning patterns underrepresented in medical texts and QA benchmarks to induce the Einstellung effect.

“mARC-QA alters predictable aspects of medical problems, emphasizing ’long-tail’ or low-probability reasoning patterns underrepresented in medical texts and QA benchmarks (Figure 1) to induce this effect.”

LLMs have limited generalization-the ability to effectively apply reasoning to novel, out-of-distribution scenarios.

“Our findings align with recent evidence that LLMs have limited generalization—the ability to effectively apply reasoning to novel, out-of-distribution scenarios62,63.”

• cites [62] Stechly, Kaya, Valmeekam, Karthik, & Kambhampati, Subbarao. Chain of thoughtlessness: An analysis of cot in planning. arXiv preprint arXiv:2405.04776, (2024).Not checked yet

  Not checked in the free run.

• cites [63] Chollet, Francois, Knoop, Mike, Kamradt, Gregory, & Landers, Bryan. Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604, (2024).Not checked yet

  Not checked in the free run.

Lack of metacognition and common sense in LLMs can lead to adverse outcomes if they are overrelied upon in clinical contexts.

“Lack of metacognition and common sense in LLMs can lead to adverse outcomes if they are overrelied upon in clinical contexts44.”

• cites [44] Kim, Sunnie SY, Liao, Q Vera, Vorvoreanu, Mihaela, Ballard, Stephanie, & Vaughan, Jennifer Wortman. “ i’m not sure, but...”: Examining the impact of large language models’ uncertainty expression on user reliance and trust. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 822–835, (2024). sourceNot checked yet

  Not checked in the free run.

Human-level performance on medical QA benchmarks does not suggest robust medical reasoning capabilities.

“challenging the notion that human-level performance on medical QA benchmarks suggests robust medical reasoning capabilities.”

Reduced adaptability to novelty or distribution shifts could undermine healthcare systems' preparedness for future disease outbreaks-particularly those involving novel pathogens akin to COVID-19.

“Subsequently, reduced adaptability to novelty or distribution shifts could undermine healthcare systems’ preparedness for future disease outbreaks—particularly those involving novel pathogens akin to COVID-19.”

Average human performance on mARC-QA (66%) is comparable to typical accuracy on board examinations and in-training assessments.

“comparable to typical accuracy on board examinations and in-training assessments70”

• cites [70] McCrary, Hilary C., Colbert-Getz, Jorie M., Poss, W Bradley & Smith, Brigitte K. A systematic review of the relationship between in-training examination scores and specialty board examination scores. J. Grad. Med. Educ. 13(1), 43–57 (2021). sourceNot checked yet

  Not checked in the free run.

LLM limitations in reasoning may be exacerbated in long-tail or out-of-distribution contexts.

“Our findings suggest that LLM limitations in reasoning may be exacerbated in long-tail or out-of-distribution contexts.”

• AssumptionRests on an assumption: mARC-QA validly measures Einstellung-driven reasoning or reasoning flexibility.

Reasoning models can occasionally surpass human performance.

“These observations indicate that reasoning models can occasionally surpass human performance, and the potential of newer reasoning models to overcome Einstellung effect will be explored in future work.”

Cognitive flexibility is foundational for effective clinical reasoning.

“this cognitive flexibility is foundational for effective clinical reasoning66.”

• cites [66] Durning, Steven J. et al. Functional neuroimaging correlates of thinking flexibility and knowledge structure in memory: Exploring the relationships between clinical reasoning and diagnostic thinking. Med. Teach. 38(6), 570–577 (2016). sourceNot checked yet

  Not checked in the free run.