Assessing AI-Powered Patient Education: A Case Study in Radiology
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
RATIONALE AND OBJECTIVES With recent advancements in the power and accessibility of artificial intelligence (AI) Large Language Models (LLMs) patients might increasingly turn to these platforms to answer questions regarding radiologic examinations and procedures, despite valid concerns about the accuracy of information provided. This study aimed to assess the accuracy and completeness of information provided by the Bing Chatbot-a LLM powered by ChatGPT-on patient education for common radiologic exams. MATERIALS AND METHODS We selected three common radiologic examinations and procedures: computed tomography (CT) abdomen, magnetic resonance imaging (MRI) spine, and bone biopsy. For each, ten questions were tested on the chatbot in two trials using three different chatbot settings. Two reviewers independently assessed the chatbot's responses for accuracy and completeness compared to an accepted online resource, radiologyinfo.org. RESULTS Of the 360 reviews performed, 336 (93%) were rated "entirely correct" and 24 (7%) were "mostly correct," indicating a high level of reliability. Completeness ratings showed that 65% were "complete" and 35% were "mostly complete." The "More Creative" chatbot setting produced a higher proportion of responses rated "entirely correct" but there were otherwise no significant difference in ratings based on chatbot settings or exam types. The readability level was rated eighth-grade level. CONCLUSION The Bing Chatbot provided accurate responses answering all or most aspects of the question asked of it, with responses tending to err on the side of caution for nuanced questions. Importantly, no responses were inaccurate or had potential to cause harm or confusion for the user. Thus, LLM chatbots demonstrate potential to enhance patient education in radiology and could be integrated into patient portals for various purposes, including exam preparation and results interpretation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Johnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., Chang, S., Berkowitz, S., Finn, A., Jahangir, E., et al. (2023) · doi:10.21203/rs.3.rs-2566942/v1 · Cited by 598
High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content
Bhattacharyya, M., Miller, V. M., Bhattacharyya, D., & Miller, L. E. (2023) · Cureus · doi:10.7759/cureus.39238 · Cited by 294
Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT
Haver, H. L., Ambinder, E. B., Bahl, M., Oluyemi, E. T., Jeudy, J., & Yi, P. H. (2023) · Radiology · doi:10.1148/radiol.230424 · Cited by 247
Self-citation
Cited by
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358
Large language models in patient education: a scoping review of applications in medicine
Aydin, S., Karabacak, M., Vlachos, V., & Margetis, K. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1477898 · Cited by 288
Large Language Models in Cybersecurity: State-of-the-Art
Motlagh, F. N., Hajizadeh, M., Majd, M., Najafi, P., Cheng, F., & Meinel, C. (2024) · arXiv · doi:10.48550/arxiv.2402.00891 · Cited by 92
“Meanwhile, the utilization of LLMs in healthcare enables professionals by providing real-time clinical decision support [24, 69], medical education [50, 81], and prediction of disease progression [70, 78].”
BackgroundTranslating musculoskeletal radiology reports into patient-friendly summaries using ChatGPT-4
Kuckelman, I. J., Wetley, K., Yi, P. H., & Ross, A. B. (2024) · Skeletal Radiology · doi:10.1007/s00256-024-04599-2 · Cited by 58
“This includes answering patient questions prior to radiology exams and procedures and providing appropriate responses to breast cancer screening recommendations [8, 9].”
Self-citationLarge Language Models in Spine Surgery: A Promising Technology
Shah, R., & Schwab, J. H. (2025) · HSS Journal®: The Musculoskeletal Journal of Hospital for Special Surgery · doi:10.1177/15563316251340696 · Cited by 5
Metadata
- Authors
- Kuckelman, I. J., Yi, P. H., Bui, M., Onuh, I., Anderson, J. A., & Ross, A. B.
- Venue
- Academic Radiology
- Publisher
- Elsevier BV
- Published
- 2024-01
- Volume
- 31
- Issue
- 1
- Pages
- 338-342
- ISSN
- 1076-6332 (ISSN-L)
- OpenAlex
- W4386753580
- Cited by
- 46
- References
- 3
- FWCI
- 1.59
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.