CoChat
Sign Up
Cited by 16Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareArtificial Intelligence in Education

Evaluating the accuracy and reliability of AI chatbots in patient education on cardiovascular imaging: a comparative study of ChatGPT, gemini, and copilot

Open access
Marey, Ahmed, Saad, Abdelrahman M., Tanas, Yousef, Ghorab, Hossam, Niemierko, Julia, Backer, Hazif, Umair, Muhammad·2025·Egyptian Journal of Radiology and Nuclear Medicine
DOI 10.1186/s43055-025-01452-x
Open this paper in CoChat — free account

Abstract

Background: The integration of artificial intelligence (AI) chatbots in medicine is expanding rapidly, with notable models like ChatGPT by OpenAI, Gemini by Google, and Copilot by Microsoft. These chatbots are increasingly used to provide medical information, yet their reliability in specific areas such as cardiovascular imaging remains underexplored. This study aims to evaluate the accuracy and reliability of ChatGPT (versions 3.5 and 4), Gemini, and Copilot in responding to patient inquiries about cardiovascular imaging. Methods: We sourced 30 patient-oriented questions on cardiovascular imaging. The questions were submitted to ChatGPT-4, ChatGPT-3.5, Copilot Balanced Mode, Copilot Precise Mode, and Gemini. Responses were evaluated by two cardiovascular radiologists based on accuracy, clarity, completeness, neutrality, and appropriateness using a structured rubric. Inter-rater reliability was assessed using Cohen’s Kappa. Results: ChatGPT-4 achieved the highest performance with 78.3% accuracy, 86.87% clarity and appropriateness, 81.7% completeness, and 100% neutrality. Gemini showed balanced performance, while Copilot Balanced Mode excelled in clarity and accuracy but lagged in completeness. Copilot Precise Mode had the lowest scores in completeness and accuracy. Penalty assessments revealed that ChatGPT-4 had the lowest incidence of missing or misleading information. Conclusion: ChatGPT-4 emerged as the most reliable AI model for providing accurate, clear, and comprehensive patient information on cardiovascular imaging. While other models showed potential, they require further refinement. This study underscores the value of integrating AI chatbots into clinical practice to enhance patient education and engagement.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s43055-025-01452-x

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A comparative study of ChatGPT, Gemini, and Copilot for cardiovascular imaging patient education found ChatGPT to perform best on accuracy and completeness.

    “A comparative study of ChatGPT, Gemini, and Copilot for cardiovascular imaging patient education found ChatGPT to perform best on accuracy and completeness (18).”

    From ChatGPT for diabetes education: potential, accuracy, and accessibility in patient support (2026) · cites this paper as [18]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al (2017) Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. Curran Associates Inc.: Red Hook, NY, USA
  2. Radford A, Narasimhan K (2018) Improving Language Understanding by Generative Pre-Training. Available from: https://www.semanticscholar.org/paper/Improving-Language-Understanding-by-Generative-Radford-Narasimhan/cd18800a0fe0b668a1cc19f2ec95b5003d0a5035
  3. Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al (2020) Language models are few-shot learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. Curran Associates Inc.: Red Hook, NY, USA
  4. Patil NS, Huang RS, van der Pol CB, Larocque N (2024) Comparative performance of ChatGPT and bard in a text-based radiology knowledge assessment. Can Assoc Radiol J 75(2):344–350
  5. Elkassem AA, Smith AD (2023) Potential use cases for ChatGPT in radiology reporting. Am J Roentgenol 221(3):373–376
  6. Rao A, Kim J, Kamineni M, Pang M, Lie W, Succi MD (2023) Evaluating ChatGPT as an Adjunct for Radiologic Decision-Making [Internet]. medRxiv [cited 2024 Jun 23]. Available from: https://doi.org/10.1101/2023.02.02.23285399v1 · DOI
  7. Bhayana R, Bleakney RR, Krishna S (2023) GPT-4 in radiology: improvements in advanced reasoning. Radiology 307(5):e230987
  8. Bhayana R, Krishna S, Bleakney RR (2023) Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations. Radiology 307(5):e230582
  9. Roth GA, Mensah GA, Johnson CO, Addolorato G, Ammirati E, Baddour LM et al (2020) Global burden of cardiovascular diseases and risk factors, 1990–2019: update from the GBD 2019 study. J Am Coll Cardiol 76(25):2982–3021
  10. Martin-Isla C, Campello VM, Izquierdo C, Raisi-Estabragh Z, Baeßler B, Petersen SE, et al (2020) Image-Based Cardiac Diagnosis With Machine Learning: A Review. Front Cardiovasc Med [Internet]. [cited 2024 Jun 23] Available from: https://doi.org/10.3389/fcvm.2020.00001 · DOI
  11. Perone F, Bernardi M, Redheuil A, Mafrica D, Conte E, Spadafora L et al (2023) Role of cardiovascular imaging in risk assessment: recent advances, gaps in evidence, and future directions. J Clin Med 12(17):5563
  12. van Nuland M, Lobbezoo AFH, van de Garde EMW, Herbrink M, van Heijl I, Bognàr T et al (2024) Assessing accuracy of ChatGPT in response to questions from day to day pharmaceutical care in hospitals. Explor Res Clin Soc Pharm 1(15):100464
  13. Scheschenja M, Viniol S, Bastian MB, Wessendorf J, König AM, Mahnken AH (2024) Feasibility of GPT-3 and GPT-4 for in-Depth patient education prior to interventional radiological procedures: a comparative analysis. Cardiovasc Intervent Radiol 47(2):245–250
  14. Currie G, Robbie S, Tually P (2023) ChatGPT and patient information in nuclear medicine: GPT-3.5 Versus GPT-4. J Nucl Med Technol 51(4):307–13
  15. OpenAI, Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, et al (2024) GPT-4 Technical Report [Internet]. arXiv [cited 2024 Jun 23]. Available from: http://arxiv.org/abs/2303.08774 · arXiv
  16. Tepe M, Emekli E (2024) Assessing the responses of large language models (ChatGPT-4, Gemini, and Microsoft Copilot) to frequently asked questions in breast imaging: a study on readability and accuracy. Cureus 16(5):e59960
  17. Shah YB, Ghosh A, Hochberg AR, Rapoport E, Lallas CD, Shah MS et al (2024) Comparison of ChatGPT and Traditional Patient Education Materials for Men’s Health. Urol Pract 11(1):87–94
  18. Schulhoff S, Ilie M, Balepur N, Kahadze K, Liu A, Si C, et al (2024) The Prompt Report: A Systematic Survey of Prompting Techniques [Internet]. arXiv [cited 2025 Jan 21]. Available from: http://arxiv.org/abs/2406.06608 · arXiv
  19. Generative Adversarial Networks-based Face Hallucination with Identity-Preserving [Internet]. [cited 2024 Jun 23]. Available from: https://ieeexplore.ieee.org/document/9603171
  20. Hallucinations in Large Language Models (LLMs) [Internet]. [cited 2024 Jun 23]. Available from: https://ieeexplore.ieee.org/document/10542617
  21. González Corbelle J, Bugarín-Diz A, Alonso-Moral J, Taboada J. (2022) Dealing with hallucination and omission in neural Natural Language Generation: A use case on meteorology. In: Shaikh S, Ferreira T, Stent A, editors. Proceedings of the 15th International Conference on Natural Language Generation [Internet]. Waterville, Maine, USA and virtual meeting: Association for Computational Linguistics
  22. Yao JY, Ning KP, Liu ZH, Ning MN, Yuan L. LLM Lies (2023) Hallucinations are not Bugs, but Features as Adversarial Examples [Internet]. arXiv [cited 2024 Jun 23]. Available from: http://arxiv.org/abs/2310.01469 · arXiv
  23. Pelrine K, Imouza A, Thibault C, Reksoprodjo M, Gupta C, Christoph J, et al (2023) Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4 [Internet]. arXiv [cited 2024 Jun 23]. Available from: http://arxiv.org/abs/2305.14928 · arXiv
  24. Tonmoy SMTI, Zaman SMM, Jain V, Rani A, Rawte V, Chadha A, et al (2024) A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models [Internet]. arXiv [cited 2024 Jun 23]. Available from: http://arxiv.org/abs/2401.01313 · arXiv
  25. Ng KKY, Matsuba I, Zhang PC (2025) RAG in health care: a novel framework for improving communication and decision-making by addressing LLM limitations. NEJM AI. 2(1):2400380
  26. Njeh C, Nakouri H, Jaafar F (2025) Enhancing RAG-Retrieval to Improve LLMs Robustness and Resilience to Hallucinations. In: Quintián H, Corchado E, Troncoso Lora A, Pérez García H, Jove Pérez E, Calvo Rolle JL et al (eds) Hybrid Artificial Intelligent Systems. Springer Nature Switzerland, Cham, pp 201–213
  27. Schmidgall S, Harris C, Essien I, Olshvang D, Rahman T, Kim JW et al (2024) Evaluation and mitigation of cognitive biases in medical language models. Npj Digit Med 7(1):1–9

Cited by

Sorted by citations

Authors
Marey, A., Saad, A. M., Tanas, Y., Ghorab, H., Niemierko, J., Backer, H., & Umair, M.
Affiliations
  • Alexandria University (EG)
  • Houston Methodist (US)
  • Papworth Hospital NHS Foundation Trust (GB)
  • Johns Hopkins University (US)
Corresponding author
Marey, Ahmed
ORCID
Venue
Egyptian Journal of Radiology and Nuclear Medicine
Publisher
Springer Science and Business Media LLC
Published
2025-03-27
Volume
56
Issue
1
ISSN
  • 0378-603X (ISSN-L)
  • 2090-4762
DOI
10.1186/s43055-025-01452-x
OpenAlex
W4408905696
Cited by
16
References
2
Citations by year
2026: 11 · 2025: 5
FWCI
3.89
Type
journal article
Language
en
Open access
Licence
Locations
  • The Egyptian Journal of Radiology and Nuclear Medicine (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
AI chatbots, patient education, cardiovascular imaging, ChatGPT-4, Gemini, Copilot, large language models, accuracy and reliability, inter-rater reliability, health information accuracy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review