Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2)
Open accessPublished version of a preprint
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Background It is thought that ChatGPT, an advanced language model developed by OpenAI, may in the future serve as an AI-assisted decision support tool in medicine. Objective To evaluate the accuracy of ChatGPT’s recommendations on medical questions related to common cardiac symptoms or conditions. Methods We tested ChatGPT’s ability to address medical questions in two ways. First, we assessed its accuracy in correctly answering cardiovascular trivia questions (n=50), based on quizzes for medical professionals. Second, we entered 20 clinical case vignettes on the ChatGPT platform and evaluated its accuracy compared to expert opinion and clinical course. Results We found that ChatGPT correctly answered 74% of the trivia questions, with slight variation in accuracy in the domains coronary artery disease (80%), pulmonary and venous thrombotic embolism (80%), atrial fibrillation (70%), heart failure (80%) and cardiovascular risk management (60%). In the case vignettes, ChatGPT’s response matched in 90% of the cases with the actual advice given. In more complex cases, where physicians (general practitioners) asked other physicians (cardiologists) for assistance or decision support, ChatGPT was correct in 50% of cases, and often provided incomplete or inappropriate recommendations when compared with expert consultation. Conclusions Our study suggests that ChatGPT has potential as an AI-assisted decision support tool in medicine, particularly for straightforward, low-complex medical questions, but further research is needed to fully evaluate its potential.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
2018 ESC/ESH Guidelines for the management of arterial hypertension
Williams, B., Mancia, G., Spiering, W., Agabiti Rosei, E., Azizi, M., Burnier, M., Clement, D. L., Coca, A., de Simone, G., Dominiczak, A., et al. (2018) · European Heart Journal · doi:10.1093/eurheartj/ehy339 · Cited by 10,286 · cited 1×
“Development of these models has been limited to a handful of large companies with extensive resources, many of which do not publish their full models or details regarding their training, including OpenAI [19].”
BackgroundTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“Similar models have shown a capacity for storing and reasoning with clinical information contained in their training data [7,8].”
BackgroundTools such as ChatGPT threaten transparent science; here are our ground rules for their use
(2023) · Nature · doi:10.1038/d41586-023-00191-1 · Cited by 605
ChatGPT and the future of medical writing
Biswas, S. (2023) · Radiology · doi:10.1148/radiol.223312 · Cited by 571 · cited 1×
“It is though that ChatGPT will shape the future of medical writing, with human judgement and oversight for quality assurances [15,16].”
BackgroundLearning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
Aung, Y. Y. M., Wong, D. C. S., & Ting, D. S. W. (2021) · British Medical Bulletin · doi:10.1093/bmb/ldab016 · Cited by 511 · cited 4×
“In certain areas, such as radiology, AI is already used to make predictions and to assist in the interpretation of imaging or other diagnostic tests [1,2].”
BackgroundInfluentialBlease, C., Kaptchuk, T. J., Bernstein, M. H., Mandl, K. D., Halamka, J. D., & DesRoches, C. M. (2019) · Journal of Medical Internet Research · doi:10.2196/12802 · Cited by 346
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2022) · arXiv · doi:10.48550/arxiv.2207.08143 · Cited by 343
Research works printed but not linked: 4
- 10.1101/2022.12.19.22283643v1.full · doi:10.1101/2022.12.19.22283643v1.full
- 10.1038/d41586-023-00056-7 · DOI · Paper page
- Zhuo TY Huang Y Chen C et al. Exploring AI ethics of ChatGPT: a diagnostic analysis. https://arxiv.org/pdf/2301.12867.pdf. · arXiv
- Goldstein JA Sastry G Musser M et al. Generative language models and automated influence operations: emerging threats and potential mitigations.https://arxiv.org/pdf/2301.04246.pdf.
Cited by
Large language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417
Haltaufderheide, J., & Ranisch, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01157-x · Cited by 396
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283
Ullah, E., Parwani, A., Baig, M. M., & Singh, R. (2024) · Diagnostic Pathology · doi:10.1186/s13000-024-01464-7 · Cited by 281
Sohail, S. S., Farhat, F., Himeur, Y., Nadeem, M., Madsen, D. Ø., Singh, Y., Atalla, S., & Mansoor, W. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101675 · Cited by 192
Metadata
- Authors
- Harskamp, R. E., & De Clercq, L.
- Venue
- Acta Cardiologica
- Publisher
- Informa UK Limited
- Published
- 2024-02-13
- Volume
- 79
- Issue
- 3
- Pages
- 358-366
- ISSN
- 0373-7934 (ISSN-L)
- 0001-5385
- OpenAlex
- W4391779256
- Cited by
- 91
- References
- 11
- FWCI
- 5.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2). https://app.cochat.ai/papers/10.1080/00015385.2024.2303528
- The paper
- Harskamp, R. E., & De Clercq, L. (2024). Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2). Acta Cardiologica. https://doi.org/10.1080/00015385.2024.2303528
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.