Impact of AI misinformation on diagnostic accuracy and confidence calibration in novice medical students
Open accessAbstract
For novice medical learners, do the benefits of correct AI explanations outweigh the risks of plausible misinformation? In a randomized trial with 111 students, we found they do not. Our results reveal a significant and problematic asymmetry: misleading AI explanations significantly degraded diagnostic accuracy, while correct explanations offered no significant improvement over a no-explanation control. Misleading explanations reduced diagnostic accuracy and showed no evidence of confidence calibration, such that confidence did not reliably distinguish correct from incorrect responses. This study provides crucial empirical evidence that, without proper safeguards, the harm caused by AI-generated falsehoods in this population and task is more potent and robust than the benefit derived from correct guidance. This finding highlights a fundamental safety challenge for AI in medical education, demanding a strategic pivot towards building learners' critical appraisal skills. Trial registration: Chinese Clinical Trial Registry (ChiCTR), ChiCTR2500111932, registered on 7 November 2025.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Plausible AI-generated misinformation can reduce diagnostic accuracy in novice medical learners while leaving them poorly calibrated in their confidence.
“More recent evidence from novice medical learners similarly indicates that plausible AI-generated misinformation can reduce diagnostic accuracy while leaving learners poorly calibrated in their confidence (Teng et al., 2026).”
From Between Co-Intelligence and Cognitive Offloading: Educators’ Perspectives from Dutch Academic Medical Settings on Generative AI, Critical Thinking, and Adult Professional Learning (2026) · cites this paper as Teng et al., 2026
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Fitting Linear Mixed-Effects Models Using <b>lme4</b>
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835 · see citing works
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971 · see citing works
<b>lmerTest</b> Package: Tests in Linear Mixed Effects Models
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017) · Journal of Statistical Software · doi:10.18637/jss.v082.i13 · Cited by 24,346 · see citing works
Nakagawa, S., & Schielzeth, H. (2013) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2012.00261.x · Cited by 10,356 · see citing works
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 1× · see citing works
“While previous studies have shown this bias leads to commission errors primarily in monitoring tasks 20 – 23 , our results highlight a speci fi c danger in knowledge-intensive contexts.”
Comparing Individual Means in the Analysis of Variance
Tukey, J. W. (1949) · Biometrics · doi:10.2307/3001913 · Cited by 4,302 · cited 1× · see citing works
“…conducted using emmeans 40 . p -values were adjusted for multiple comparisons using the Tukey method; when probing group differences within each level of response correctness, the Tukey adjustment was applied separately within each correctness stratum (three pairwise comparisons per stratum) 41 .”
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · cited 1× · see citing works
“These conclusions must be contextualized within the study ’ s limitations 29 .”
Hyperglycemic crises in adult patients with diabetes
Kitabchi, A. E., Umpierrez, G. E., Miles, J. M., & Fisher, J. N. (2009) · Diabetes Care · doi:10.2337/dc09-9032 · Cited by 2,062 · see citing works
Eysenbach, G., & CONSORT-EHEALTH Group (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1923 · Cited by 1,863 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Complacency and Bias in Human Use of Automation: An Attentional Integration
Parasuraman, R., & Manzey, D. H. (2010) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720810376055 · Cited by 1,710 · see citing works
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 1× · see citing works
“This results in a fi rmly held misconception — a state arguably more harmful to a novice learner than simple ignorance 24 .”
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · see citing works
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · see citing works
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124 · see citing works
The misunderstood limits of folk science: An illusion of explanatory depth
Rozenblit, L., & Keil, F. (2002) · Cognitive Science · doi:10.1207/s15516709cog2605_1 · Cited by 1,032 · see citing works
Effects of Perceptual Fluency on Judgments of Truth
Reber, R., & Schwarz, N. (1999) · Consciousness and Cognition · doi:10.1006/ccog.1999.0386 · Cited by 843 · see citing works
Chan, K. S., & Zary, N. (2019) · JMIR Medical Education · doi:10.2196/13930 · Cited by 765 · see citing works
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707 · see citing works
Does automation bias decision-making?
SKITKA, L. J., MOSIER, K. L., & BURDICK, M. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0252 · Cited by 694 · cited 1× · see citing works
“Theoretically, this vulnerability aligns with established research on automation bias — de fi ned by Mosier and Skitka as the tendency to use automatedcuesasaheuristicreplacementforvigilantinformationseeking 19 .”
Knowledge does not protect against illusory truth.
Fazio, L. K., Brashier, N. M., Payne, B. K., & Marsh, E. J. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000098 · Cited by 598 · cited 1× · see citing works
“1), con-fi dence ratings became invariant to correctness; the fl awed explanations provided a coherent, albeit fallacious, rationale that led students to report high con fi dence for incorrect answers 14 – 17 .”
CONSORT 2025 statement: updated guideline for reporting randomised trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · The Lancet · doi:10.1016/s0140-6736(25)00672-5 · Cited by 510 · cited 1× · see citing works
“Thisstudywasconductedandreportedinaccordance withtheConsolidatedStandardsofReportingTrials(CONSORT) guidelines 30,31 .”
Artificial intelligence in medical education
Masters, K. (2019) · Medical Teacher · doi:10.1080/0142159x.2019.1595557 · Cited by 469 · see citing works
Automation bias and verification complexity: a systematic review
Lyell, D., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw105 · Cited by 451 · see citing works
ChatGPT: these are not hallucinations – they’re fabrications and falsifications
Emsley, R. (2023) · Schizophrenia · doi:10.1038/s41537-023-00379-4 · Cited by 258 · see citing works
How Effective are Instructional Explanations in Example-Based Learning? A Meta-Analytic Review
Wittwer, J., & Renkl, A. (2010) · Educational Psychology Review · doi:10.1007/s10648-010-9136-5 · Cited by 140 · cited 1× · see citing works
“Targeting a medium effect size (Cohen ’ s f = 0.30), informed by meta-analyses of instructional explanations 35 , this estimation indicated a minimum required total sample size of 111 participants.”
InfluentialThe fundamentals of Artificial Intelligence in medical education research: AMEE Guide No. 156.
Tolsgaard, M. G., Pusic, M. V., Sebok-Syer, S. S., Gin, B., Svendsen, M. B., Syer, M. D., Brydges, R., Cuddy, M. M., & Boscardin, C. K. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2180340 · Cited by 137 · cited 1× · see citing works
“These capabilities suggest that AI can revolutionize traditional teaching models and enhance learning ef fi ciency 4,5 .”
Artificial intelligence and clinical anatomical education: Promises and perils
Lazarus, M. D., Truong, M., Douglas, P., & Selwyn, N. (2024) · Anatomical Sciences Education · doi:10.1002/ase.2221 · Cited by 116 · see citing works
The Impact of an Online Evidence System on Confidence in Decision Making in a Controlled Setting
Westbrook, J. I., Gosling, A. S., & Coiera, E. W. (2005) · Medical Decision Making · doi:10.1177/0272989x05275155 · Cited by 62 · cited 1× · see citing works
“This decoupling mirrors earlier fi ndingswherecliniciansmaintainedhighcon fi denceinincorrectdiagnoses even after consulting online evidence systems 18 , suggesting a pervasive vulnerability in technology-assisted reasoning.”
Quantifying the reasoning abilities of LLMs on clinical cases
Qiu, P., Wu, C., Liu, S., Fan, Y., Zhao, W., Chen, Z., Gu, H., Peng, C., Zhang, Y., Wang, Y., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64769-1 · Cited by 53 · see citing works
Sampson, M., Jones, C., & Care, T. J. B. D. S. (. F. I. (2018) · Diabetic Medicine · doi:10.1111/dme.13672 · Cited by 32 · cited 1× · see citing works
“A fundamental, life-saving principle of DKA management is that the patient ’ s total body potassium is always severely depleted due to massive urinary losses, even when initial serum (blood) potassiumlevelsparadoxicallyappearnormalorevenelevated 8 – 10 .”
Profound hypokalemia associated with severe diabetic ketoacidosis
Davis, S., Maddux, A. B., Alonso, G. T., Okada, C., Mourani, P. M., & Maahs, D. M. (2014) · Pediatric Diabetes · doi:10.1111/pedi.12246 · Cited by 31 · see citing works
Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions
Roy, S., Khatua, A., Ghoochani, F., Hadler, U., Nejdl, W., & Ganguly, N. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657882 · Cited by 17 · see citing works
Generative artificial intelligence in medical education: way to solve the problems
Li, Y., & Li, J. (2023) · Postgraduate Medical Journal · doi:10.1093/postmj/qgad116 · Cited by 7 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Tezcan, H., Tunçez, A., Gürses, K. M., Özen, Y., & Yalçın, M. U. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09141-7 · Cited by 4 · see citing works
“This distinction is especially important in medical education, because novice learners may interpret confident and readable prose as evidence of correctness [22–24].”
Malkoç, M. (2026) · JMIR Infodemiology · doi:10.2196/96664 · Cited by 4 · see citing works
“A related randomized controlled experiment involving medical students [25] found that misleading AI explanations significantly degraded diagnostic accuracy, while accurate AI explanations produced no statistically significant improvement.”
Reconciling how clinical reasoning is learned in the age of artificial intelligence
Ong, A. Y., Sui, M., Rosen, K. L., & Kvedar, J. C. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02873-2 · Cited by 2 · see citing works
Fu, C., Li, J., Fan, H., Ren, R., Kefei, C., Zhang, Y., Yu, J., & Li, J. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1871524 · Cited by 3 · see citing works
Self-citationAI-Empowered Nuclear Medicine Education, Part 3: Practical AI Applications for Learners
Jacobs, K. M., & Peacock, J. G. (2026) · Journal of Nuclear Medicine Technology · doi:10.2967/jnmt.126.273333 · Cited by 1 · see citing works
AI-Empowered Nuclear Medicine Education, Part 1: Theoretical and Ethical Foundation
Jacobs, K. M., & Peacock, J. G. (2026) · Journal of Nuclear Medicine Technology · doi:10.2967/jnmt.126.273207 · Cited by 1 · see citing works
Joo, E., Ma, O., Woolley, T., & Haleagrahara, N. (2026) · International Medical Education · doi:10.3390/ime5030089 · Cited by 1 · see citing works
“Experimental studies show that incorrect AI-generated information can reduce diagnostic accuracy in both medical students [23] and clinicians [24], and performance in healthcare-related decisions remains inconsistent [25].”
Toma, M., & YUSUPOV, D. (2026) · Bioengineering · doi:10.3390/bioengineering13091034 · Cited by 1 · see citing works
Bruijn, K., & Vlachopoulos, D. (2026) · Education Sciences · doi:10.3390/educsci16091470
“More recent evidence from novice medical learners similarly indicates that plausible AI-generated misinformation can reduce diagnostic accuracy while leaving learners poorly calibrated in their confidence (Teng et al., 2026).”
Alqaderi, H. E., Kapadia, U., Brahmbhatt, Y., Papathanasiou, A., Rodgers, D., Arsenault, P., Cardarelli, J., Zavras, A. I., & Li, H. (2026) · medRxiv · doi:10.64898/2026.04.12.26350726
Reflection: AI as a catalyst for change in postgraduate medical education
Adam, S., & Burch, V. C. (2026) · Journal of the Colleges of Medicine of South Africa · doi:10.4102/jcmsa.v4i1.413
Beyond the algorithm: nursing as a core foundation for ethical and equitable AI in healthcare
Sun, Y., Li, Y. D., Liu, Y., & Yu, S. (2026) · Cognition Technology & Work · doi:10.1007/s10111-026-00886-1
TGSFL: Trustworthy group-signature-based federated learning for LLMs in healthcare
Liu, X., Jin, C., Wu, Y., Wu, Y., Zhai, Z., Chen, G., Zhang, H., & Jian, W. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01237-w
“Such outputs may contribute to misdiagnosis, inappropriate treatment, delayed intervention, and diminished trust in AI-assisted systems (Wang and Zhang 2024; Teng et al. 2026).”
Kremer, P., Schlicker, N., Hasnaj, R., Bamberger, J., Witte, T., Haase, I., Mayr, A., Schmidt, C., Østerås, N., Baraliakos, X., et al. (2026) · medRxiv · doi:10.64898/2026.08.29.26361715
Ding, C., Zhang, B., & Jia, S. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1916897
“Existing concerns surrounding AI in healthcare frequently emphasize depersonalization, automation bias, and diminished emotional connection in clinical practice (3, 40, 41).”
Gualda-Gea, J. J., Barón-Miras, L. E., Bertran, M. J., García‐Basteiro, A. L., Mena, G., Aldea, M., Santana, G., Grau, J., Marín, A. P., Rocamora, I. T., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03390-y
Source-Aware Belief Cues and Evidence Weighting in Clinical Large Language Models
Luo, S., Zou, K., Huang, L., Liu, M., Zhou, N., Zhong, J., Jiang, L., Zhang, D., Wang, L., Zhang, Y., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41960-6_28
When AI Becomes the First Consultant: Implications for Medical Training
Ortiz‐Prado, E., & Vásconez-González, J. (2026) · Healthcare · doi:10.3390/healthcare14213606
- Authors
- Teng, D., Tan, L., Cao, Q., Xia, Y., Zhang, N., Li, J., & Zhao, D.
- Affiliations
- Beijing Institute of Petrochemical Technology (CN)
- Chinese Academy of Medical Sciences & Peking Union Medical College (CN)
- Chinese PLA General Hospital (CN)
- Corresponding author
- Zhang, Na
- Li, Jiantao
- Zhao, Dan
- ORCID
- Teng, Da
- Venue
- npj Digital Medicine
- Published
- 2026-03-17
- Volume
- 9
- Issue
- 1
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC13144481
- OpenAlex
- W7138105710
- Cited by
- 18
- References
- 36
- Citations by year
- 2026: 18
- FWCI
- 35.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- npj Digital Medicine (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Zhiyuan Scientific Research Fund Project (2026005)
- National Natural Science Foundation of China (82302788, 62176267)
- Capital’s Funds for Health Improvement and Research (No. 2022-2-4026)
- Natural Science Foundation of Beijing Municipality (7252116)
- CAMS Inno- vation Fund for Medical Sciences (o. 2024-I2M-C&T-A-003)
- Qinghai Province Key Research and Development and Transformation Plan (2025-QY-220)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- AI-generated misinformation, diagnostic accuracy, confidence calibration, medical education, AI explanations, critical appraisal skills, clinical reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
