Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
It is likely that individuals are turning to Large Language Models (LLMs) to seek health advice, much like searching for diagnoses on Google. We evaluate clinical accuracy of GPT-3·5 and GPT-4 for suggesting initial diagnosis, examination steps and treatment of 110 medical cases across diverse clinical disciplines. Moreover, two model configurations of the Llama 2 open source LLMs are assessed in a sub-study. For benchmarking the diagnostic task, we conduct a naïve Google search for comparison. Overall, GPT-4 performed best with superior performances over GPT-3·5 considering diagnosis and examination and superior performance over Google for diagnosis. Except for treatment, better performance on frequent vs rare diseases is evident for all three approaches. The sub-study indicates slightly lower performances for Llama models. In conclusion, the commercial LLMs show growing potential for medical question answering in two successive major releases. However, some weaknesses underscore the need for robust and regulated AI models in health care. Open source LLMs can be a viable option to address specific needs regarding data privacy and transparency of training.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“Research has shown that ChatGPT can pass the United States Medical Licensing Exam (USMLE) or Advanced Cardio-vascularLifeSupport(ACLS)exam 12,13 .”
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Pheochromocytoma and paraganglioma: an endocrine society clinical practice guideline
Lenders, J. W. M., Duh, Q.-Y., Eisenhofer, G., Gimenez-Roqueplo, A.-P., Grebe, S. K. G., Murad, M. H., Naruse, M., Pacak, K., & Young, W. F. (2014) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2014-1498 · Cited by 2,827
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838
The benefits and challenges of chatgpt: An overview
Deng, J., & Lin, Y. (2023) · Frontiers in Computing and Intelligent Systems · doi:10.54097/fcis.v2i2.4465 · Cited by 611
ChatGPT and the future of medical writing
Biswas, S. (2023) · Radiology · doi:10.1148/radiol.223312 · Cited by 571 · cited 1×
“In additiontoexamsimulations, previousworkshavealsoillustratedthepotentialbene fi tofChatGPTin everyday medicine, for instance, in medical writing by extracting information from electronic health records, assisting with literature searches or providing guidance on writing style and formatting 14 .”
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549
Clinical research for rare disease: Opportunities, challenges, and solutions
Griggs, R. C., Batshaw, M., Dunkle, M., Gopal-Srivastava, R., Kaye, E., Krischer, J., Nguyen, T., Paulus, K., & Merkel, P. A. (2009) · Molecular Genetics and Metabolism · doi:10.1016/j.ymgme.2008.10.003 · Cited by 401 · cited 1×
“In particular, patients with rare diseases run a higher risk of being undiagnosed, yet collectively, they constitute a signi fi cant portion of the population 18 .”
Cited by
Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies
Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2025) · Computers & Education · doi:10.1016/j.compedu.2024.105224 · Cited by 531
Benchmark evaluation of DeepSeek large language models in clinical decision-making
Sandmann, S., Hegselmann, S., Fujarski, M., Bickmann, L., Wild, B., Eils, R., & Varghese, J. (2025) · Nature Medicine · doi:10.1038/s41591-025-03727-2 · Cited by 233
“Although LLMs excel on widely used benchmarks such as multiple-choice tests, their evaluation for clinical decision support tasks remains underexplored 15–17 .”
Self-citationFFA-GPT: an automated pipeline for fundus fluorescein angiography interpretation and question-answer
Chen, X., Zhang, W., Xu, P., Zhao, Z., Zheng, Y., Shi, D., & He, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01101-z · Cited by 72
Testing AI on language comprehension tasks reveals insensitivity to underlying meaning
Dentella, V., Günther, F., Murphy, E., Marcus, G., & Leivada, E. (2024) · Scientific Reports · doi:10.1038/s41598-024-79531-8 · Cited by 54
Quantifying the reasoning abilities of LLMs on clinical cases
Qiu, P., Wu, C., Liu, S., Fan, Y., Zhao, W., Chen, Z., Gu, H., Peng, C., Zhang, Y., Wang, Y., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64769-1 · Cited by 53
Metadata
- Authors
- Sandmann, S., Riepenhausen, S., Plagwitz, L., & Varghese, J.
- Venue
- Nature Communications
- Published
- 2024-03-06
- Volume
- 15
- Issue
- 1
- Pages
- 2050-2050
- ISSN
- 2041-1723 (ISSN-L)
- OpenAlex
- W4392504747
- Cited by
- 212
- References
- 14
- FWCI
- 17.00
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks. https://app.cochat.ai/papers/10.1038/s41467-024-46411-8
- The paper
- Sandmann, S., Riepenhausen, S., Plagwitz, L., & Varghese, J. (2024). Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks. Nature Communications. https://doi.org/10.1038/s41467-024-46411-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.