Appropriateness of ChatGPT in Answering Heart Failure Related Questions
Open accessPublished version of a preprint
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Heart failure requires complex management, and increased patient knowledge has been shown to improve outcomes. This study assessed the knowledge of Chat Generative Pre-trained Transformer (ChatGPT) and its appropriateness as a supplemental resource of information for patients with heart failure. METHOD: A total of 107 frequently asked heart failure-related questions were included in 3 categories: "basic knowledge" (49), "management" (41) and "other" (17). Two responses per question were generated using both GPT-3.5 and GPT-4 (i.e., two responses per question per model). The accuracy and reproducibility of responses were graded by two reviewers, board-certified in cardiology, with differences resolved by a third reviewer, board-certified in cardiology and advanced heart failure. Accuracy was graded using a four-point scale: (1) comprehensive, (2) correct but inadequate, (3) some correct and some incorrect, and (4) completely incorrect. RESULTS: GPT-4 provided 107/107 (100%) responses with correct information. Further, GPT-4 displayed a greater proportion of comprehensive knowledge for the categories of "basic knowledge" and "management" (89.8% and 82.9%, respectively). For GPT-3, there were two total responses (1.9%) graded as "some correct and incorrect" for GPT-3.5, while no "completely incorrect" responses were produced. With respect to comprehensive knowledge, GPT-3.5 performed best in the "management" category and "other" category (prognosis, procedures, and support) (78.1%, 94.1%). The models also provided highly reproducible responses, with GPT-3.5 scoring above 94% in every category and GPT-4 with 100% for all answers. CONCLUSIONS: GPT-3.5 and GPT-4 answered the majority of heart failure-related questions accurately and reliably. If validated in future studies, ChatGPT may serve as a useful tool in the future by providing accessible health-related information and education to patients living with heart failure. In its current state, ChatGPT necessitates further rigorous testing and validation to ensure patient safety and equity across all patient demographics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., et al. (2023) · doi:10.1101/2023.02.06.23285449 · Cited by 666
Self-citationSamaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., et al. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06603-5 · Cited by 282
Self-citationGPT-4 outperforms ChatGPT in answering non-English questions related to cirrhosis
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ma, X., Ting, P.-S., Kwak, M.-S., Panduro, A., Lizaola-Mayo, B., Trivedi, H., Vipani, A., et al. (2023) · doi:10.1101/2023.05.04.23289482 · Cited by 35
Self-citation
Research works printed but not linked: 8
- Savarese, 2023, Global burden of heart failure: a comprehensive and updated review of epidemiology [published correction appears in Cardiovasc Res 2023;119:1453], Cardiovasc Res, 118, 3272, 10.1093/cvr/cvac013 · DOI · Paper page
- Ditewig, 2010, Effectiveness of self-management interventions on mortality, hospital readmissions, chronic heart failure hospitalization rate and quality of life in patients with chronic heart failure: a systematic review, Patient Educ Couns, 78, 297, 10.1016/j.pec.2010.01.016 · DOI · Paper page
- Sarraju, 2023, Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model, JAMA, 329, 842, 10.1001/jama.2023.1044 · DOI · Paper page
- Bart, 2023, Emerging roles of artificial intelligence (AI) in cardiology: benefits and barriers in a ‘brave New World’, Heart Lung Circ, 32, 883, 10.1016/j.hlc.2023.07.005 · DOI
- Cascella, 2023, Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios, J Med Syst, 47, 33
- Lolic, 2021, Racial and ethnic representation in US clinical trials of new drugs and biologics, 2015–2019, JAMA, 326, 2201, 10.1001/jama.2021.16680 · doi:10.1001/jama.2021.16680
- Ong, 2023, GPT technology to help address longstanding barriers to care in free medical clinics, Ann Biomed Eng, 51, 1906, 10.1007/s10439-023-03256-4 · DOI
- Samaan, 2023, ChatGPT's ability to comprehend and answer cirrhosis related questions in Arabic, Arab J Gastroenterol, 24, 145, 10.1016/j.ajg.2023.08.001 · DOI · Paper page
Cited by
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358
Large language models in patient education: a scoping review of applications in medicine
Aydin, S., Karabacak, M., Vlachos, V., & Margetis, K. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1477898 · Cited by 288
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283
Analyzing evaluation methods for large language models in the medical field: a scoping review
Lee, J., Park, S., Shin, J., & Cho, B. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02709-7 · Cited by 52
Metadata
- Authors
- King, R. C., Samaan, J. S., Yeo, Y. H., Mody, B., Lombardo, D. M., & Ghashghaei, R.
- Venue
- Heart, Lung and Circulation
- Publisher
- Elsevier BV
- Published
- 2024-09
- Volume
- 33
- Issue
- 9
- Pages
- 1314-1318
- ISSN
- 1443-9506 (ISSN-L)
- OpenAlex
- W4399238566
- Cited by
- 44
- References
- 3
- FWCI
- 3.45
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.