Evaluation and mitigation of cognitive biases in medical language models
Open accessAbstract
Increasing interest in applying large language models (LLMs) to medicine is due in part to their impressive performance on medical exam questions. However, these exams do not capture the complexity of real patient–doctor interactions because of factors like patient compliance, experience, and cognitive bias. We hypothesized that LLMs would produce less accurate responses when faced with clinically biased questions as compared to unbiased ones. To test this, we developed the BiasMedQA dataset, which consists of 1273 USMLE questions modified to replicate common clinically relevant cognitive biases. We assessed six LLMs on BiasMedQA and found that GPT-4 stood out for its resilience to bias, in contrast to Llama 2 70B-chat and PMC Llama 13B, which showed large drops in performance. Additionally, we introduced three bias mitigation strategies, which improved but did not fully restore accuracy. Our findings highlight the need to improve LLMs’ robustness to cognitive biases, in order to achieve more reliable applications of LLMs in healthcare.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This reduction is a much smaller drop than was observed in previous work (10.2% maximum reduction on BiasMedQA dataset15).
“it is a much smaller drop than was observed in previous work (10.2% maximum reduction on BiasMedQA dataset15)”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [15]
The Results (Model evaluation; page unavailable) report a 24.9% average accuracy decrease for false-consensus bias and a 14.0% worst-case drop for GPT-4 alone. Both exceed the claimed 10.2% maximum reduction on BiasMedQA. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredAnswer relevance mismatch was observed in approximately 25% of cases.
“This pattern, observed in approximately 25% of cases, reflects instruction-following limitations where RAG systems struggle to align response style with the specific demands of the question type [52,60].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [60]
The Methods (Automated evaluation; page unavailable) describe two medical language models that sometimes failed to follow the requested answer format, supporting a general instruction-following limitation. The Results (Model evaluation; page unavailable) report a 24.9% decrease in performance due to false-consensus bias—not answer-relevance mismatches in approximately 25% of cases. The cited passages do not establish that rate or the claim about RAG systems aligning response style with question type. (LLM, conf=0.82)
Experimental — accuracy not yet measuredAnswer relevance mismatch reflects instruction-following limitations where RAG systems struggle to align response style with the specific demands of the question type.
“This pattern, observed in approximately 25% of cases, reflects instruction-following limitations where RAG systems struggle to align response style with the specific demands of the question type [52,60].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [60]
Methods > Automated evaluation (page unavailable) reports that two medical language models frequently ignored an instruction to answer with only a single letter, supporting the general point about instruction-following and response format. It does not establish a roughly 25% rate or examine RAG systems matching response style to different question types. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe presence of prompts which induce cognitive biases can decrease the diagnostic accuracy of LLMs by as much as 26%.
“Additionally, it has been found that the presence of prompts which induce cognitive biases can decrease the diagnostic accuracy of LLMs by as much as 26%15.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [15]
GPT-4 is relatively robust to bias compared with other language models.
“Previous work studying cognitive bias in LLMs has shown that GPT-4 is relatively robust to bias compared with other language models15”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 23 · Cited by 68
Research works printed but not linked: 5
- [2]Hammond, M. E. H., Stehlik, J., Drakos, S. G. & Kfoury, A. G. Bias in medicine: lessons learned and mitigation strategies. Basic Transl. Sci. 6, 78–85 (2021). PMC7838049 33532668
- [13]Ziaei, R. & Schmidgall, S. Language models are susceptible to incorrect patient self-diagnosis in medical applications. In Deep Generative Models for Health Workshop NeurIPS 2023 (2023).
- [16]Barham, P. et al. Pathways: asynchronous distributed dataflow for ML. Proc. Mach. Learn. Syst. 4, 430–449 (2022).
- [21]Barnard, F., Van Sittert, M. & Rambhatla, S. Self-diagnosis and large language models: A new front for medical misinformation. Preprint at https://arxiv.org/abs/2307.04910 (2023). · arXiv
- [27]Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020).
- Authors
- Schmidgall, S., Harris, C., Essien, I., Olshvang, D., Rahman, T., Kim, J. W., Ziaei, R., Eshraghian, J., Abadir, P., & Chellappa, R.
- Affiliations
- Johns Hopkins University (US)
- University of Maryland, College Park (US)
- University of California, Santa Cruz (US)
- Corresponding author
- Schmidgall, Samuel
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2024-10-21
- Volume
- 7
- Issue
- 1
- Pages
- 295-295
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11494053
- OpenAlex
- W4403601954
- Cited by
- 68
- References
- 23
- Citations by year
- 2026: 34 · 2025: 33 · 2024: 1
- FWCI
- 6.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Science Foundation (2139757)
- U.S. Department of Health & Human Services | NIH | National Institute on Aging (P30AG073104)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- cognitive biases, large language models, clinical language models, bias mitigation, GPT-4, Llama 2, clinical reasoning, bias robustness, medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
