Black Box Warning: Large Language Models and the Future of Infectious Diseases Consultation
Open accessAbstract
Large language models (LLMs) are artificial intelligence systems trained by deep learning algorithms to process natural language and generate text responses to user prompts. Some approach physician performance on a range of medical challenges, leading some proponents to advocate for their potential use in clinical consultation and prompting some consternation about the future of cognitive specialties. However, LLMs currently have limitations that preclude safe clinical deployment in performing specialist consultations, including frequent confabulations, lack of contextual awareness crucial for nuanced diagnostic and treatment plans, inscrutable and unexplainable training data and methods, and propensity to recapitulate biases. Nonetheless, considering the rapid improvement in this technology, growing calls for clinical integration, and healthcare systems that chronically undervalue cognitive specialties, it is critical that infectious diseases clinicians engage with LLMs to enable informed advocacy for how they should-and shouldn't-be used to augment specialist care.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large language models such as ChatGPT, GPT-4 API and Llama2 can interrogate and summarise large volumes of medical notes to create succinct summaries of radiology reports, internal medical progress notes and patient dialogue.
“Large language models (LLM) such as ChatGPT, GPT-4 API and Llama2 can interrogate and summarise large volumes of medical notes to create succinct summaries of radiology reports, internal medical progress notes and patient dialogue [5–8].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [7]
The ‘Use Cases for Large Language Models in Healthcare’ section says LLMs might synthesize text from electronic medical records into discharge summaries. It does not establish that ChatGPT, GPT-4 API, and Llama2 can each summarize large volumes of notes, or specifically radiology reports, progress notes, and patient dialogue. (LLM, conf=0.85)
Experimental — accuracy not yet measuredMany studies on large language models have been conducted using large anonymised US datasets.
“Many of the studies on LLM have been conducted using large anonymised US datasets [5–7].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [7]
The 'How Large Language Models Work' section says LLMs are trained on large quantities of text, but does not say that many LLM studies used large anonymised US datasets. Searches for that phrase and for anonymized US patient datasets found no passage addressing the claim. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1× · see citing works
“For example, an AI algo-rithm widely used by hospitals and insurers in the US systematically prioritized White patients over Black patients for healthcare resources [33].”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Explainability for artificial intelligence in healthcare: a multidisciplinary perspective
the Precise4Q consortium, Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01332-6 · Cited by 2,203 · see citing works
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 1× · see citing works
“Automated bias is well documented in healthcare [22].”
BackgroundAdversarial attacks on medical machine learning
Finlayson, S. G., Bowers, J. D., Ito, J., Zittrain, J. L., Beam, A. L., & Kohane, I. S. (2019) · Science · doi:10.1126/science.aaw4399 · Cited by 1,088 · see citing works
Does automation bias decision-making?
SKITKA, L. J., MOSIER, K. L., & BURDICK, M. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0252 · Cited by 694 · cited 1× · see citing works
“…eliminate crashes partly because of human complacency and impaired decision making fostered by imperfect automated guidance, leading Skitka et al [21] to argue, “The presence of automated cues appears to diminish the likelihood that decision makers will either put forth the cognitive effort to…”
BackgroundPotential liability for physicians using artificial intelligence
Price, W. N., Gerke, S., & Cohen, I. G. (2019) · JAMA · doi:10.1001/jama.2019.15064 · Cited by 579 · see citing works
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · cited 1× · see citing works
“General-purpose LLMs like OpenAI’s GPT-3 and GPT-4, Google’s BERT, and Meta’s LLaMA—sometimes called foundation models—have been trained on massive quantities of text without specific domain (eg, biomedical) training [3].”
BackgroundEthics of large language models in medicine and medical research
Li, H., Moon, J. T., Purkayastha, S., Celi, L. A., Trivedi, H., & Gichoya, J. W. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00083-3 · Cited by 286 · see citing works
AI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176 · see citing works
Automation bias in electronic prescribing
Lyell, D., Magrabi, F., Raban, M. Z., Pont, L. G., Baysari, M. T., Day, R. O., & Coiera, E. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0425-5 · Cited by 103 · cited 1× · see citing works
“…incorrect diagnoses of atrial fibrillation in one-quarter of cases, leading to unnecessary antiarrhythmics and anticoagulation in 10% [24]; in yet another, erroneous alerts from clinical decision-support systems during electronic prescribing led to an overall increase in prescribing errors [25].”
Background
Flagged first, then by citations
Research works printed but not linked: 25
- [1]Schwartz WB. Medicine and the computer. N Engl J Med 1970; 283: 1257– 64. 4920342
- [4]Luo R, Sun L, Xia Y, et al BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings Bioinformatics 2022; 23: bbac409. doi: 10.1093/bib/bbac409. 36156661 · DOI · Paper page
- [6]Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on medical challenge problems. arXiv 2023, preprint: not peer-reviewed. Available at: 10.48550/arXiv.2303.13375. · DOI · Paper page
- [8]Boulware DR, Meya DB, Muzoora C, et al Timing of antiretroviral therapy after diagnosis of cryptococcal meningitis. N Engl J Med 2014; 370: 2487– 98. 24963568 PMC4127879
- [10]Wagner MW, Ertl-Wagner BB. Accuracy of information and references using ChatGPT-3 for retrieval of clinical radiological information [manuscript published online ahead of print 2023]. Can Assoc Radiol J 2023. doi: 10.1177/08465371231171125. 37078489 · DOI · Paper page
- [11]Johnson D, Goodman R, Patrinely J, et al Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the chat-GPT model. Res Sq, 2023, preprint: not peer-reviewed. Available at: 10.21203/rs.3.rs-2566942/v1. · DOI · Paper page
- [12]Dash D, Thapa R, Banda J, et al Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery. arXiv, 2023, preprint: not peer-reviewed. Available at: 10.48550/arXiv.2304.13714. · DOI
- [13]Howard A, Hope W, Gerada A. ChatGPT and antimicrobial advice: the end of the consulting infection doctor? Lancet Infect Dis 2023; 23: 405– 6. 36822213
- [14]Sarink M, Bakker IL, Anas AA, Yusuf E. A study on the performance of ChatGPT in infectious diseases clinical consultation. Clin Microbiol Infect. 2023; 29: 1088– 9. 37207982
- [18]Tshitoyan V, Dagdelen J, Weston L, et al Unsupervised word embeddings capture latent knowledge from materials science literature. Nature 2019; 571: 95– 8. 31270483
- [19]Mosier KL, Skitka LJ. Human decision makers and automated decision aids: made for each other? In: Parasuraman R, Mouloua M, eds. Automation and human performance. Boca Raton, FL: CRC Press, 1996: 201– 20.
- [23]Dratsch T, Chen X, Rezazade Mehrizi M, et al Automation bias in mammography: the impact of artificial intelligence BI-RADS suggestions on reader performance. Radiology 2023; 307: e222176. 37129490
- [24]Bogun F, Anh D, Kalahasty G, et al Misdiagnosis of atrial fibrillation and its clinical consequences. Am J Med 2004; 117: 636– 42. 15501200
- [26]Reddy S. Explainability and artificial intelligence in medicine. Lancet Digit Health 2022; 4: e214– e5. 35337639
- [27]Harrer S. Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine. EBioMedicine 2023; 90: 104512. 36924620 PMC10025985
- [28]Touvron H, Lavril T, Izacard G, et al Llama: open and efficient foundation language models. arXiv, 2023, preprint: not peer-reviewed. Available at: 10.48550/arXiv.2302.13971. · DOI · Paper page
- [29]Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 2019; 1: 206– 15. 35603010 PMC9122117
- [32]Weidinger L, Mellor J, Rauh M, et al Ethical and social risks of harm from language models. arXiv, 2021, preprint: not peer-reviewed. Available at: 10.48550/arXiv.2112.04359. · DOI · Paper page
- [34]Califf RM, editor. Speech by Robert M. Califf, M.D., to the National Health Council’s 2023 Science for Patient Engagement Symposium—patient empowerment in the Digital Health Era. Presented at: National Health Council's 2023 Science for Patient Engagement Symposium—Patient Empowerment in the Digital Health Era; Washington, DC; 8 May 2023.
- [37]Reece R, Beckwith CG. The infectious diseases specialist, at risk of extinction. J Infect Dis 2023:jiad160. doi: 10.1093/infdis/jiad160. 37192330 · DOI
- [38]Walensky R, McQuillen DP, Shahbazi S, Goodson JD. Where is the ID in COVID-19? Ann Intern Med 2020; 173: 587– 9. 32491920 PMC7277486
- [39]National Academies of Sciences, Engineering, and Medicine. Taking action against clinician burnout: a systems approach to professional well-being. Washington, DC: National Academies Press, 2019. 31940160
- [40]Lenert LA, Lane S, Wehbe R. Could an artificial intelligence approach to prior authorization be more human? J Am Med Inform Assoc 2023; 30: 989– 94. 36809561 PMC10114030
- [41]Patel SB, Lam K. ChatGPT: the future of discharge summaries? Lancet Digit Health 2023; 5: e107– e8. 36754724
- [45]Fitzsimmons-Craft EE, Chan WW, Smith AC, et al Effectiveness of a chatbot for eating disorders prevention: a randomized clinical trial. Int J Eat Disord 2022; 55: 343– 53. 35274362
Cited by
Sorted by citations
Abbas, A., Rehman, M. S., & Rehman, S. S. (2024) · Cureus · doi:10.7759/cureus.55991 · Cited by 58 · see citing works
Omar, M., Nassar, S., Hijazi, K., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109545 · Cited by 43 · see citing works
Urquhart, E., Ryan, J., Hartigan, S., Nita, C., Hanley, C., Moran, P., Bates, J., Jooste, R., Judge, C., Laffey, J. G., et al. (2024) · Intensive Care Medicine Experimental · doi:10.1186/s40635-024-00656-1 · Cited by 18 · see citing works
Benchmarking the Confidence of Large Language Models in Clinical Questions
Omar, M., Agbareia, R., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2024) · doi:10.1101/2024.08.11.24311810 · Cited by 10 · see citing works
- Authors
- Schwartz, I. S., Link, K. E., Daneshjou, R., & Cortés-Penfield, N.
- Affiliations
- Duke University (US)
- Hugging Face (US)
- Stanford Medicine (US)
- Division of Infectious Diseases, University of Nebraska Medical Center , Omaha, Nebraska , USA
- Corresponding author
- Schwartz, Ilan S
- Venue
- Clinical Infectious Diseases
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-11-16
- Volume
- 78
- Issue
- 4
- Pages
- 860-866
- ISSN
- 1058-4838 (ISSN-L)
- 1537-6591
- PMC
- PMC11006107
- OpenAlex
- W4388722291
- Cited by
- 131
- References
- 14
- Citations by year
- 2026: 27 · 2025: 64 · 2024: 39 · 2023: 1
- FWCI
- 5.23
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Clinical Infectious Diseases (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- large language models, artificial intelligence in healthcare, infectious disease consultation, clinical implementation of AI, confabulation, context awareness, algorithmic bias, explainability
- MeSH terms
- Artificial Intelligence, Communicable Diseases, Drug Labeling, Humans, Language, Referral and Consultation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
