The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index; 22 of the 26 sources its claims cite were checked for support.
Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study
Open accessPublished version of a preprint
Abstract
Background: Cognitive biases in clinical decision-making significantly contribute to errors in diagnosis and suboptimal patient outcomes. Addressing these biases presents a formidable challenge in the medical field. Objective: This study aimed to explore the role of large language models (LLMs) in mitigating these biases through the use of the multi-agent framework. We simulate the clinical decision-making processes through multi-agent conversation and evaluate its efficacy in improving diagnostic accuracy compared with humans. Methods: A total of 16 published and unpublished case reports where cognitive biases have resulted in misdiagnoses were identified from the literature. In the multi-agent framework, we leveraged GPT-4 (OpenAI) to facilitate interactions among different simulated agents to replicate clinical team dynamics. Each agent was assigned a distinct role: (1) making the final diagnosis after considering the discussions, (2) acting as a devil’s advocate to correct confirmation and anchoring biases, (3) serving as a field expert in the required medical subspecialty, (4) facilitating discussions to mitigate premature closure bias, and (5) recording and summarizing findings. We tested varying combinations of these agents within the framework to determine which configuration yielded the highest rate of correct final diagnoses. Each scenario was repeated 5 times for consistency. The accuracy of the initial diagnoses and the final differential diagnoses were evaluated, and comparisons with human-generated answers were made using the Fisher exact test. Results: A total of 240 responses were evaluated (3 different multi-agent frameworks). The initial diagnosis had an accuracy of 0% (0/80). However, following multi-agent discussions, the accuracy for the top 2 differential diagnoses increased to 76% (61/80) for the best-performing multi-agent framework (Framework 4-C). This was significantly higher compared with the accuracy achieved by human evaluators (odds ratio 3.49; P=.002). Conclusions: The multi-agent framework demonstrated an ability to re-evaluate and correct misconceptions, even in scenarios with misleading initial investigations. In addition, the LLM-driven, multi-agent conversation framework shows promise in enhancing diagnostic accuracy in diagnostically challenging medical scenarios.
Overview
CoChat Summary
TL;DR
This simulation study used GPT-4 Turbo and an AutoGen multi-agent framework to re-evaluate 16 clinical case reports in which cognitive biases had caused misdiagnosis. The best-performing 4-agent framework, which explicitly discussed cognitive biases, improved final top-2 differential diagnostic accuracy to 76% (61/80), significantly better than human evaluators (odds ratio 3.49; P=.002).
Key findings
- Initial diagnoses by the first-responder agent were correct in 0% (0/80) of responses across all three multi-agent frameworks, whereas human-generated initial diagnoses were correct in 27% (13/48).
- After multi-agent discussion, final diagnosis accuracy was 64% (51/80) for Framework 3, 69% (55/80) for Framework 4, and 76% (61/80) for Framework 4-C; only Framework 4 (odds ratio 2.39; P=.03) and Framework 4-C (odds ratio 3.49; P=.002) were significantly better than human answers.
- Human final diagnoses were correct in 48% (23/48).
- Across all three frameworks, multi-agent discussions led to the correct final diagnosis in 13 of 16 scenarios (81%), and identified anchoring bias, confirmation bias, availability bias, and premature closure. In case 12, Framework 4-C reached the correct diagnosis of pneumothorax in 100% (5/5) of simulations after an initially misleading chest radiograph interpretation.
Methods
- PubMed was searched for case reports of misdiagnosis attributed to cognitive bias; 162 reports were found, 37 were eligible, and 15 published cases were selected, plus 1 unpublished personal critical care case, for 16 scenarios.
- GPT-4 Turbo was accessed via API and used within the AutoGen multi-agent conversation framework. Three configurations were tested: Framework 3 (Junior Resident I, Junior Resident II, Recorder), Framework 4 (plus Professional Expert), and Framework 4-C (plus Senior Doctor tasked with discussing cognitive biases). Each scenario was repeated 5 times.
- Diagnostic accuracy was assessed for initial and final diagnoses. Two physicians graded answers, with consensus for discrepancies. Three doctors with at least 5 years of clinical experience provided human comparator differential diagnoses. Multi-agent and human answers were compared using the Fisher exact test.
Limitations
- The paper states that reliance on published case reports limits the breadth of clinical scenarios and may affect generalizability.
- The paper states that excluding visual data such as medical imaging confines the model to text-based information, omitting a critical component of clinical diagnosis.
- The paper states that inherent LLM biases from pre-training data and technical limitations in understanding complex medical terminology and nuances may limit applicability compared with experienced clinicians. Evident from the text: the study used only 16 simulated scenarios, did not test real-time clinical outcomes, and the human comparator involved only 3 doctors.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
15 Valid · 1 Not valid · 3 Not checked · 46 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite concerted efforts involving educational strategies, optimal work environments, and a culture promoting bias awareness and correction [5,6], the eradication of these biases remains an elusive goal.”
§ Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study > Introduction
What the sources say
- Partly supportedcites [5] sourceExperimental — accuracy not yet measured
The Metacognition and Clinical Decision Making section recommends education and awareness to reduce bias and describes working conditions that make analytical thinking difficult. The Conclusion likewise advocates sustained training. The paper does not establish that concerted efforts have already provided optimal work environments or a culture of bias correction; the sentence's claim about elusive eradication follows the citation rather than being clearly attributed to it. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [6] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Human cognitive biases in clinical decision-making are increasingly recognized as a crucial factor in health care errors and suboptimal patient outcomes [1].”
§ Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study > Introduction
What the sources say
- Partly supportedcites [1] sourceExperimental — accuracy not yet measured
The Results and Discussion report associations between cognitive biases and diagnostic inaccuracies or management errors, supporting the claim about health care errors. But the Abstract Conclusions say more research is needed to determine their impact on patient outcomes, so the claim presents that link more firmly than the review does. (LLM, conf=0.86)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Cognitive biases in clinical decision-making significantly contribute to errors in diagnosis and suboptimal patient outcomes.”
- Also stated in: § Introduction · boosted
“Human cognitive biases in clinical decision-making are increasingly recognized as a crucial factor in health care errors and suboptimal patient outcomes [1].”
§ Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study > Abstract > Background
What the sources say
- Partly supportedcites [1] sourceExperimental — accuracy not yet measured
The Results section reports associations between cognitive biases and diagnostic inaccuracies, as well as potential effects on management errors. But the claim’s stronger wording, “significantly contribute,” exceeds the review’s cautious conclusions. The Abstract Results section says only two studies evaluated patient outcomes, with one finding an association involving tolerance to ambiguity; it does not establish a general effect of cognitive biases on patient outcomes. Page numbers are not provided in our copy. (LLM, conf=0.87)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This was significantly higher compared with the accuracy achieved by human evaluators (odds ratio 3.49; P=.002).”
- Also stated in: § Overall Performance of Multi-Agent Conversation Framework · plain
“which was significantly better than the human answers (odds ratio 3.49; P=.002)”
§ Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study > Abstract > Results
plainstates an associationn = 80 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Overall Performance of Multi-Agent Conversation Framework · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, following multi-agent discussions, the accuracy for the top 2 differential diagnoses increased to 76% (61/80) for the best-performing multi-agent framework (Framework 4-C).”
- Also stated in: § Overall Performance of Multi-Agent Conversation Framework · plain
“After the multi-agent discussions, the final diagnosis was correct in 76% (61/80) of cases in Framework 4-C”
§ Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study > Abstract > Results
plainstates a causen = 80 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Overall Performance of Multi-Agent Conversation Framework · plain
24 more claims
Create a free account to see all 65 claims and every source verdict.
- Probably not a substantial claim. 29
- It may state two separate claims as one. 13
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“As agents negotiate strategies to achieve clinical goals, they may exhibit emergent behaviors that were neither programmed nor anticipated during validation (38).”
From Ethical issues in multi-agent AI systems for healthcare: a narrative review (2026) · cites this paper as [38]
Experimental — accuracy not yet measured
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“Notably, these models have demonstrated their proficiency by successfully passing advanced medical examinations [10] and scoring clinical risk gradings on par with experienced physicians [11].”
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“…while LLMs have demonstrated potential as a valuable clinical aid in correcting cognitive biases, the implementation of such technology in health care necessitates rigorous ethical and regulatory oversight [52] and should continue to augment rather than replace the human clinician’s expertise [14].”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Scientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823
Teamwork in healthcare: Key discoveries enabling safer, high-quality care.
Rosen, M. A., DiazGranados, D., Dietz, A. S., Benishek, L. E., Thompson, D., Pronovost, P. J., & Weaver, S. J. (2018) · American Psychologist · doi:10.1037/amp0000298 · Cited by 1,283 · cited 1×
“In the absence of established literature recommending an optimal team size for mitigating cognitive biases in medical settings, we constructed a simulation using 3 to 4 different agents, representing a typical clinical team composition [31].”
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047
The future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045 · cited 1×
“While LLMs can process and analyze vast quantities of data far beyond human capacity [18], they lack the nuanced understanding, empathy, and ethical reasoning inherent to human practitioners [19].”
Cognitive biases associated with medical decisions: a systematic review
Saposnik, G., Redelmeier, D., Ruff, C. C., & Tobler, P. N. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0377-1 · Cited by 1,001 · cited 1×
“Human cognitive biases in clinical decision-making are increasingly recognized as a crucial factor in health care errors and suboptimal patient outcomes [1].”
Understanding how discrimination can affect health
Williams, D. R., Lawrence, J. A., Davis, B. A., & Vu, C. (2019) · Health Services Research · doi:10.1111/1475-6773.13222 · Cited by 853
Cited by
Enhancing diagnostic capability with multi-agents conversational large language models
Chen, X., Yi, H., You, M., Liu, W., Wang, L., Li, H., Zhang, X., Guo, Y., Fan, L., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01550-0 · Cited by 147
Retrieval-augmented generation for generative artificial intelligence in health care
Yang, R., Ning, Y., Keppo, E., Liu, M., Hong, C., Bitterman, D. S., Ong, J. C. L., Ting, D. S. W., & Liu, N. (2025) · npj Health Systems · doi:10.1038/s44401-024-00004-1 · Cited by 269
Self-citationLarge Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis
Su, H., Sun, Y., Li, R., Zhang, A., Yang, Y., Xiao, F., Duan, Z., Chen, J., Hu, Q., Yang, T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72062 · Cited by 79
Generative AI for decision-making: A multidisciplinary perspective
Albashrawi, M. A. (2025) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2025.100751 · Cited by 101
AI Agents in Clinical Medicine: A Systematic ReviewPreprint
Gorenshtein, A., Omar, M., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2025) · medRxiv · doi:10.1101/2025.08.22.25334232 · Cited by 94
Metadata
- Authors
- Ke, Y., Yang, R., Lie, S. A., Lim, T. X. Y., Ning, Y., Li, I., Abdullah, H. R., Ting, D. S. W., & Liu, N.
- Affiliations
- Singapore General Hospital (SG)
- Duke-NUS Medical School (SG)
- The University of Tokyo (JP)
- Singapore National Eye Center (SG)
- National University of Singapore (SG)
- Corresponding author
- Liu, Nan
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-11-19
- Volume
- 26
- Pages
- e59439
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11615553
- OpenAlex
- W4404518047
- Semantic Scholar
- dc185e9974367145c0f37d196f2c8ffdd23da4d5
- Cited by
- 64
- References
- 52
- Citations by year
- 2026: 35 · 2025: 27 · 2024: 2
- FWCI
- 5.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Duke-NUS Medical School
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- cognitive biases, clinical decision-making, large language models, multi-agent systems, GPT-4, diagnostic accuracy, misdiagnosis, confirmation bias, anchoring effect, devil's advocate, differential diagnosis, AI-assisted diagnosis
- MeSH terms
- Clinical Decision-Making, Cognition, Communication, Computer Simulation, Diagnostic Errors, Humans, Language, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study. https://app.cochat.ai/papers/10.2196/59439
- The paper
- Ke, Y., Yang, R., Lie, S. A., Lim, T. X. Y., Ning, Y., Li, I., Abdullah, H. R., Ting, D. S. W., & Liu, N. (2024). Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study. Journal of Medical Internet Research. https://doi.org/10.2196/59439
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.