Evaluation of a retrieval-augmented generation-powered chatbot for pre-CT informed consent: a prospective comparative study
Open accessAbstract
This study aims to investigate the feasibility, usability, and effectiveness of a Retrieval-Augmented Generation (RAG)-powered Patient Information Assistant (PIA) chatbot for pre-CT information counseling compared to the standard physician consultation and informed consent process. This prospective comparative study included 86 patients scheduled for CT imaging between November and December 2024. Patients were randomly assigned to either the PIA group (n = 43), who received pre-CT information via the PIA chat app, or the control group (n = 43), with standard doctor-led consultation. Patient satisfaction, information clarity and comprehension, and concerns were assessed using six ten-point Likert-scale questions after information counseling with PIA or the doctor's consultation. Additionally, consultation duration was measured, and PIA group patients were asked about their preference for pre-CT consultation, while two radiologists rated each PIA chat in five categories. Both groups reported similarly high ratings for information clarity (PIA: 8.64 ± 1.69; control: 8.86 ± 1.28; p = 0.82) and overall comprehension (PIA: 8.81 ± 1.40; control: 8.93 ± 1.61; p = 0.35). However, the doctor consultation group showed greater effectiveness in alleviating patient concerns (8.30 ± 2.63 versus 6.46 ± 3.29; p = 0.003). The PIA group demonstrated significantly shorter subsequent consultation times (median: 120 s [interquartile range (IQR): 100-140] versus 195 s [IQR: 170-220]; p = 0.04). Both radiologists rated overall quality, scientific and clinical evidence, clinical usefulness and relevance, consistency, and up-to-dateness of PIA high. The RAG-powered PIA effectively provided pre-CT information while significantly reducing physician consultation time. While both methods achieved comparable patient satisfaction and comprehension, physicians were more effective at addressing worries or concerns regarding the examination.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The patient satisfaction scale was not previously validated but was specifically designed for this study based on similar AI-patient interaction studies and pilot-tested on 10 participants to ensure usability.
“While this was not a previously validated scale, it was specifically designed for this study based on similar AI–patient interaction studies and pilot-tested on 10 participants to ensure usability [11,12].”
From Comparative Efficacy of ChatGPT and DeepSeek in Addressing Patient Queries on Gonarthrosis and Total Knee Arthroplasty (2025) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
1 retracted
Singer, M., Fu, J. J., Chow, J., & Teng, C. C. (2023) · Journal of surgical education · doi:10.1016/j.jsurg.2023.11.019 · Cited by 35 · see citing works
RetractedSciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430 · see citing works
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826 · see citing works
Harris, C., Millman, K., van der Walt, S., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N., et al. (2020) · RePEc: Research Papers in Economics · Cited by 17,844 · see citing works
Data Structures for Statistical Computing in Python
McKinney, W. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-00a · Cited by 11,511 · see citing works
Statsmodels: Econometric and Statistical Modeling with Python
Seabold, S., & Perktold, J. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-011 · Cited by 6,673 · see citing works
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · cited 2× · see citing works
“This dynamic feedback loop ensures that patient concerns are addressed as they arise, thereby reducing uncertainty and fear [31].”
Author Correction: SciPy 1.0: fundamental algorithms for scientific computing in Python
Contributors, S. 1. 0., Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T. J., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., et al. (2020) · Nature Methods · doi:10.1038/s41592-020-0772-5 · Cited by 1,848 · see citing works
Effectiveness of empathy in general practice: a systematic review
Derksen, F., Bensing, J., & Lagro-Janssen, A. (2013) · British Journal of General Practice · doi:10.3399/bjgp13x660814 · Cited by 1,034 · see citing works
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
Teach-back: A systematic review of implementation and impacts
Talevski, J., Shee, A. W., Rasmussen, B., Kemp, G., & Beauchamp, A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231350 · Cited by 397 · see citing works
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358 · see citing works
Self-citationBenchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · arXiv · doi:10.48550/arxiv.2402.13178 · Cited by 312 · see citing works
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.372 · Cited by 310 · see citing works
Use and Effectiveness of the Teach-Back Method in Patient Education and Health Outcomes.
Yen, P. H., & Leasure, A. R. (2019) · PubMed · Cited by 274 · see citing works
Decker, H., Trang, K., Ramirez, J., Colley, A., Pierce, L., Coleman, M., Bongiovanni, T., Melton, G. B., & Wick, E. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.36997 · Cited by 178 · see citing works
AoE: Angle-optimized Embeddings for Semantic Textual Similarity
Li, X., & Li, J. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.101 · Cited by 169 · see citing works
Mohammad‐Rahimi, H., Ourang, S. A., Pourhoseingholi, M. A., Omid Dianat, A., Dummer, P. M. H., & Nosrat, A. (2023) · International Endodontic Journal · doi:10.1111/iej.14014 · Cited by 138 · see citing works
Large language models for structured reporting in radiology: past, present, and future
Busch, F., Hoffmann, L., dos Santos, D. P., Makowski, M. R., Saba, L., Prucker, P., Hadamitzky, M., Navab, N., Kather, J. N., Truhn, D., et al. (2024) · European Radiology · doi:10.1007/s00330-024-11107-6 · Cited by 134 · see citing works
Self-citationWu, J., Zhu, J., Qi, Y., Chen, J., Xu, M., Menolascina, F., & Grau, V. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.04187 · Cited by 115 · see citing works
Using ChatGPT to Facilitate Truly Informed Medical Consent
Mirza, F. N., Tang, O. Y., Connolly, I. D., Abdulrazeq, H. A., Lim, R. K., Roye, G. D., Priebe, C. J., Chandler, C., Libby, T. J., Groff, M. W., et al. (2024) · NEJM AI · doi:10.1056/aics2300145 · Cited by 90 · see citing works
Truhn, D., Weber, C. D., Braun, B. J., Bressem, K., Kather, J. N., Kuhl, C., & Nebelung, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-47500-2 · Cited by 71 · see citing works
Self-citationChatGPT: Can You Prepare My Patients for [18F]FDG PET/CT and Explain My Reports?
Rogasch, J. M. M., Metzger, G., Preisler, M., Galler, M., Thiele, F., Brenner, W., FELDHAUS, F. W., Wetz, C., Amthauer, H., Furth, C., et al. (2023) · Journal of Nuclear Medicine · doi:10.2967/jnumed.123.266114 · Cited by 59 · see citing works
ChatGPT and Patient Information in Nuclear Medicine: GPT-3.5 Versus GPT-4
Currie, G. M., Robbie, S., & Tually, P. (2023) · Journal of Nuclear Medicine Technology · doi:10.2967/jnmt.123.266151 · Cited by 54 · see citing works
Mahmood, U. A., Shukla‐Dave, A., Chan, H., Drukker, K., Samala, R. K., Chen, Q., Vergara, D., Greenspan, H. K., Petrick, N., Sahiner, B., et al. (2024) · BJR|Artificial Intelligence · doi:10.1093/bjrai/ubae003 · Cited by 42 · see citing works
Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions
Luo, M., Warren, C. J., Cheng, L., Abdul-Muhsin, H. M., & Banerjee, I. (2024) · 2024 IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825307 · Cited by 36 · see citing works
Development and Testing of Retrieval Augmented Generation in Large Language ModelsPreprint
Ke, Y. H., Jin, L., Elangovan, K., Abdullah, H. R., Liu, N., Sia, A. T. H., Soh, C. R., Tung, J. Y. M., Ong, J. C. L., & Ting, D. S. W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4719185 · Cited by 30 · see citing works
Floyd, W., Kleber, T. J., Carpenter, D. J., Pasli, M., Qazi, J. J., Huang, C. C., Leng, J. X., Ackerson, B. G., Pierpoint, M., Salama, J. K., et al. (2023) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2023.10.020 · Cited by 24 · see citing works
Use of ChatGPT in Urology and its Relevance in Clinical Practice: Is it useful?
Braga, A. V. N. M., Nunes, N. C., Santos, E. N., Veiga, M. L., Braga, A. A. N. M., de Abreu, G. E., de Bessa, J., Braga, L. H., Kirsch, A. J., & Barroso, U. (2024) · International braz j urol · doi:10.1590/s1677-5538.ibju.2023.0570 · Cited by 24 · see citing works
Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generation
Agrawal, G., Kumarage, T., Alghamdi, Z., & Liu, H. (2024) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852457 · Cited by 23 · see citing works
Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generation
Agrawal, G., Kumarage, T., Alghamdi, Z., & Liu, H. (2024) · arXiv · doi:10.48550/arxiv.2407.12216 · Cited by 23 · see citing works
Campos, C. F. C., Olivo, C. R., Martins, M. A., & Tempski, P. Z. (2024) · Clinics · doi:10.1016/j.clinsp.2024.100377 · Cited by 21 · see citing works
Ke, Y., Jin, L., Elangovan, K., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01733 · Cited by 20 · see citing works
Informed consent for intravascular administration of contrast material: how much is enough?
Hopper, K. D., & Tyler, H. N. (1989) · Radiology · doi:10.1148/radiology.171.2.2704817 · Cited by 17 · see citing works
Can Large Language Models Safely Address Patient Questions Following Cataract Surgery?
Chowdhury, M., Lim, E., Higham, A., McKinnon, R., Ventoura, N., He, Y., & De Pennington, N. (2023) · Proceedings of the 5th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2023.clinicalnlp-1.17 · Cited by 17 · see citing works
A Large Language Model Agent Based Legal Assistant for Governance Applications
Mamalis, M. E., Kalampokis, E., Fitsilis, F., Theodorakopoulos, G., & Tarabanis, K. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70274-7_18 · Cited by 15 · see citing works
Moll, M., Heilemann, G., Georg, D., Kauer-Dorner, D., & Kuess, P. (2024) · Strahlentherapie und Onkologie · doi:10.1007/s00066-023-02190-7 · Cited by 12 · see citing works
Large Language Models in Healthcare: A Review
Zou, S., & He, J. (2023) · International Symposium on Computer Science and Intelligent Control (ISCSIC) · doi:10.1109/iscsic60498.2023.00038 · Cited by 10 · see citing works
Enhancing Medical History Collection using LLMs
Kumar, R., Gattani, D. R. K., & Singh, K. (2024) · Australasian Computer Science Week (ACSW) · doi:10.1145/3641142.3641174 · Cited by 8 · see citing works
Unity in Diversity: Collaborative Pre-training Across Multimodal Medical Sources
Wang, X., Luo, J., Wang, J., Zhong, Y., Zhang, X., Wang, Y., Bhatia, P., Xiao, C., & Ma, F. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.199 · Cited by 8 · see citing works
Healthcare Copilot: Eliciting the Power of General LLMs for Medical ConsultationPreprint
Ren, Z., Zhan, Y., Yu, B., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.13408 · Cited by 8 · see citing works
Behavioral Emotion Analysis Model for Large Language Models
Chang, E. Y. (2024) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr62202.2024.00094 · Cited by 6 · see citing works
MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical ReasoningPreprint
Li, S. S., Balachandran, V., Feng, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.00922 · Cited by 4 · see citing works
Assessing Empathy in Large Language Models with Real-World Physician-Patient InteractionsPreprint
Luo, M., Warren, C. J., Cheng, L., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.16402 · Cited by 3 · see citing works
A Methodology for Evaluating RAG Systems: A Case Study On Configuration Dependency ValidationPreprint
Simon, S., Mailach, A., Dorn, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.08801 · Cited by 3 · see citing works
Towards an LLM based approach for medical e-consent
Naji, M., Masmoudi, M., & Zghal, H. B. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.187 · Cited by 1 · see citing works
WundtGPT: Shaping Large Language Models To Be An Empathetic, Proactive PsychologistPreprint
Ren, C., Zhang, Y., He, D., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.15474 · Cited by 1 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Cited by
Sorted by citations
Ozmen, B. B., Singh, N., Shah, K., Berber, I., Singh, D., Pinsky, E., Rampazzo, A., & Schwarz, G. S. (2025) · Journal of Hand and Microsurgery · doi:10.1016/j.jham.2025.100293 · Cited by 11 · see citing works
Gurbuz, S., Bahar, H., Yavuz, U., Keskin, A., Karslioglu, B., & Solak, Y. (2025) · Arthroplasty Today · doi:10.1016/j.artd.2025.101730 · Cited by 9 · see citing works
“While this was not a previously validated scale, it was speci fi cally designed for this study based on similar AI e patient interaction studies and pilot-tested on 10 participants to ensure usability [11,12].”
- Authors
- Busch, F., Kaibel, L., Nguyen, H., Lemke, T., Ziegelmayer, S., Graf, M., Marka, A. W., Endrös, L., Prucker, P., Spitzl, D., Mergen, M., Makowski, M. R., Bressem, K. K., Petzoldt, S., Adams, L. C., & Landgraf, T.
- Affiliations
- TUM Klinikum (DE)
- Freie Universität Berlin (DE)
- DRK Kliniken Berlin (DE)
- Corresponding author
- Busch, Felix
- Landgraf, Tim
- ORCID
- Venue
- Journal of Imaging Informatics in Medicine
- Published
- 2025-03-21
- Volume
- 38
- Issue
- 6
- Pages
- 4312-4323
- ISSN
- 0897-1889 (ISSN-L)
- 2948-2933
- PMC
- PMC12701176
- OpenAlex
- W4408735159
- Cited by
- 10
- References
- 50
- Citations by year
- 2026: 7 · 2025: 3
- FWCI
- 2.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Journal of Imaging Informatics in Medicine (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Refubium (Universitätsbibliothek der Freien Universität Berlin) (submittedVersion, cc-by)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
- Universitätsbibliothek der FU Berlin Hochschulschriftenstelle u. Dokumentenserver (cc-by)
- Funders
- Technische Universität München
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Patient-Provider Communication in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- retrieval-augmented generation, chatbots, informed consent, computed tomography, patient satisfaction, information comprehension, patient concerns
- MeSH terms
- Generative Artificial Intelligence, Adult, Aged, Female, Humans, Informed Consent, Male, Middle Aged, Prospective Studies, Tomography, X-Ray Computed, Patient Satisfaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
