CoChat

Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2)

Open access
Harskamp, Ralf E., De Clercq, Lukas · 2024 · Acta Cardiologica

Published version of a preprint

Cited by 91

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

Abstract

Background It is thought that ChatGPT, an advanced language model developed by OpenAI, may in the future serve as an AI-assisted decision support tool in medicine. Objective To evaluate the accuracy of ChatGPT’s recommendations on medical questions related to common cardiac symptoms or conditions. Methods We tested ChatGPT’s ability to address medical questions in two ways. First, we assessed its accuracy in correctly answering cardiovascular trivia questions (n=50), based on quizzes for medical professionals. Second, we entered 20 clinical case vignettes on the ChatGPT platform and evaluated its accuracy compared to expert opinion and clinical course. Results We found that ChatGPT correctly answered 74% of the trivia questions, with slight variation in accuracy in the domains coronary artery disease (80%), pulmonary and venous thrombotic embolism (80%), atrial fibrillation (70%), heart failure (80%) and cardiovascular risk management (60%). In the case vignettes, ChatGPT’s response matched in 90% of the cases with the actual advice given. In more complex cases, where physicians (general practitioners) asked other physicians (cardiologists) for assistance or decision support, ChatGPT was correct in 50% of cases, and often provided incomplete or inappropriate recommendations when compared with expert consultation. Conclusions Our study suggests that ChatGPT has potential as an AI-assisted decision support tool in medicine, particularly for straightforward, low-complex medical questions, but further research is needed to fully evaluate its potential.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1080/00015385.2024.2303528

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 11 references

Research works printed but not linked: 4

  1. 10.1101/2022.12.19.22283643v1.full · doi:10.1101/2022.12.19.22283643v1.full
  2. 10.1038/d41586-023-00056-7 · DOI · Paper page
  3. Zhuo TY Huang Y Chen C et al. Exploring AI ethics of ChatGPT: a diagnostic analysis. https://arxiv.org/pdf/2301.12867.pdf. · arXiv
  4. Goldstein JA Sastry G Musser M et al. Generative language models and automated influence operations: emerging threats and potential mitigations.https://arxiv.org/pdf/2301.04246.pdf.

Cited by

See all 91

Metadata

Authors
Harskamp, R. E., & De Clercq, L.
Venue
Acta Cardiologica
Published
2024-02-13
Volume
79
Issue
3
Pages
358-366
ISSN
  • 0373-7934 (ISSN-L)
  • 0001-5385
OpenAlex
W4391779256
Cited by
91
References
11
FWCI
5.93
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2). https://app.cochat.ai/papers/10.1080/00015385.2024.2303528
The paper
Harskamp, R. E., & De Clercq, L. (2024). Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2). Acta Cardiologica. https://doi.org/10.1080/00015385.2024.2303528

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.