PreprintCited by 92Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareArtificial Intelligence Applications

Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2)

Open access
Harskamp, Ralf E., De Clercq, Lukas·2023·Acta cardiologica. Supplementum

Preprint — published as 10.1080/00015385.2024.2303528

Abstract

Background: It is thought that ChatGPT, an advanced language model developed by OpenAI, may in the future serve as an AI-assisted decision support tool in medicine. Objective: To evaluate the accuracy of ChatGPT’s recommendations on medical questions related to common cardiac symptoms or conditions. Methods: We tested ChatGPT’s ability to address medical questions in two ways. First, we assessed its accuracy in correctly answering cardiovascular trivia questions (n=50), based on quizzes for medical professionals. Second, we entered 20 clinical case vignettes on the ChatGPT platform and evaluated its accuracy compared to expert opinion and clinical course. Results: We found that ChatGPT correctly answered 74% of the trivia questions, with slight variation in accuracy in the domains coronary artery disease (80%), pulmonary and venous thrombotic embolism (80%), atrial fibrillation (70%), heart failure (80%) and cardiovascular risk management (60%). In the case vignettes, ChatGPT’s response matched in 90% of the cases with the actual advice given. In more complex cases, where physicians (general practitioners) asked other physicians (cardiologists) for assistance or decision support, ChatGPT was correct in 50% of cases, and often provided incomplete or inappropriate recommendations when compared with expert consultation. Conclusions: Our study suggests that ChatGPT has potential as an AI-assisted decision support tool in medicine, particularly for straightforward, low-complex medical questions, but further research is needed to fully evaluate its potential.

Integrity signals

Registry

Crossref · 10.1101/2023.03.25.23285475

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Harskamp, R. E., & De Clercq, L.
Affiliations
Amsterdam UMC Location University of Amsterdam (NL)
Corresponding author
Harskamp, Ralf E.
Venue
Acta cardiologica. Supplementum
Publisher
openRxiv
Published
2023-03-26
Volume
79
Issue
3
Pages
358-366
ISSN
0373-7934 (ISSN-L)
OpenAlex
W4391779256
Cited by
92
References
7
Citations by year
2026: 14 · 2025: 26 · 2024: 35 · 2023: 17
FWCI
7.03
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Acta cardiologica. Supplementum (publishedVersion, cc-by-nc-nd)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Pure Amsterdam UMC (submittedVersion, cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
ChatGPT, large language models, AI-assisted decision support, cardiovascular symptoms, cardiac conditions, clinical case vignettes, model accuracy, coronary artery disease, atrial fibrillation, heart failure, thromboembolism, cardiovascular risk management
MeSH terms
Proof of Concept Study, Artificial Intelligence, Cardiovascular Diseases, Decision Support Techniques, Heart Diseases, Humans, Decision Support Systems, Clinical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review