Comparing Patient’s Confidence in Clinical Capabilities in Urology: Large Language Models Versus Urologists
Open accessAbstract
Background and objective: Data on interaction of patients with artificial intelligence (AI) are limited, primarily derived from small-scale studies, cross-sectional surveys, and qualitative reviews. Most patients have not yet encountered AI in their clinical experience. This study explored patients' confidence in AI, specifically large language models, after a direct interaction with a chatbot in a clinical setting. Through hands-on experience, the study sought to reduce potential biases due to an anticipated lack of AI experience in a real-world urological patient sample. Methods: A total of 300 patients scheduled for counseling were enrolled from February to July 2024. Participants voluntarily conversed about their medical questions with a GPT-4 powered chatbot, followed by a survey assessing their confidence in clinical capabilities of AI compared with their counseling urologists. Clinical capabilities included history taking, diagnostics, treatment recommendation, anxiety reduction, and time allocation. Key findings and limitations: < 0.001). However, urologists were overwhelmingly favored for all other capabilities, especially treatment recommendations and anxiety reduction. Notably, age did not influence patients' confidence in AI. Limitations include a potential social desirability bias. Conclusions and clinical implications: Our study demonstrates that urological patients prefer AI as a powerful complement to-rather than a replacement for-human expertise in clinical care. Patients appreciated the additional consultation time provided by AI. Interestingly, age was not associated with confidence in AI, suggesting that large language models are user-friendly tools for patients of all age groups. Patient summary: In this report, we explored how patients feel about using an artificial intelligence (AI)-powered chatbot in a medical setting. Patients interacted with the AI for medical questions and compared its skills with those of doctors through a survey. They appreciated the AI for providing more time during consultations but preferred doctors for other tasks, for example, diagnostics, recommendation of treatments, and reduction of anxieties.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Participants' age ranged from 11 to 96 years.
“The participants’ age ranged from 11 [30] to 96 [49] years.”
From Generative AI Mental Health Chatbots as Therapeutic Tools: Systematic Review and Meta-Analysis of Their Role in Reducing Mental Health Issues (2025) · cites this paper as [49]
Citation [49] is attached to the upper age of 96 years. Table 1 in Results reports mean ages by group, and Patient enrollment specifies a minimum eligibility age of 18, but neither gives the oldest participant’s age. Searches for “96” and the maximum age found no passage establishing 96 as the upper limit. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
The future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045 · see citing works
Self-citationKatz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2023.0254 · Cited by 420 · see citing works
Haque, M. D. R., & Rubya, S. (2023) · JMIR mHealth and uHealth · doi:10.2196/44838 · Cited by 404 · see citing works
Development of a chatbot for depression: adolescent perceptions and recommendations
Dosovitsky, G., & Bunge, E. (2023) · Child and Adolescent Mental Health · doi:10.1111/camh.12627 · Cited by 65 · cited 1× · see citing works
“Moreover, data on patient interactions with LLMs are sparse, with most evidence stemming from cross-sectional surveys [18–20], one interventional study with very limited sample size ( n = 9) [21], and qualitative research evaluating online user reviews posted on AppS-tores [22].”
BackgroundRodler, S., Kopliku, R., Ulrich, D., Kaltenhauser, A., Casuscelli, J., Eismann, L., Waidelich, R., Buchner, A., Butz, A., Cacciamani, G. E., et al. (2024) · European Urology Focus · doi:10.1016/j.euf.2023.10.020 · Cited by 65 · see citing works
Flagged first, then by citations
Research works printed but not linked: 22
- [2]Jiang L.Y. Liu X.C. Nejatian N.P. Health system-scale language models are all-purpose prediction engines Nature 619 2023 357 362 37286606 PMC10338337
- [3]Perez-Lopez R. Ghaffari Laleh N. Mahmood F. Kather J.N. A guide to artificial intelligence for cancer researchers Nat Rev Cancer 24 2024 427 441 38755439
- [4]Singhal K. Azizi S. Tu T. Large language models encode clinical knowledge Nature 620 2023 172 180 37438534 PMC10396962
- [5]Brown TB, Mann B, Ryder N, et al. Language models are few-shot learners 2020. arXiv: 2005.14165. 10.48550/arXiv.2005.14165. · DOI · Paper page
- [6]Iannantuono G.M. Bracken-Clarke D. Floudas C.S. Roselli M. Gulley J.L. Karzai F. Applications of large language models in cancer care: current evidence and future perspectives Front Oncol 13 2023 1268915 PMC10507617 37731643
- [8]Rydzewski N.R. Dinakaran D. Zhao S.G. Comparative evaluation of LLMs in clinical oncology NEJM AI 1 2024 AIoa2300151 PMC11315428 39131700
- [10]Kollitsch L. Eredics K. Marszalek M. How does artificial intelligence master urological board examinations? A comparative analysis of different large language models’ accuracy and reliability in the 2022 In-Service Assessment of the European Board of Urology World J Urol 42 2024 20 38197996
- [11]Kung T.H. Cheatham M. Medenilla A. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models PLOS Digit Health 2 2023 e0000198 PMC9931230 36812645
- [12]Barbour A.B. Barbour T.A. A radiation oncology board exam of ChatGPT Cureus 15 2023 e44541 PMC10544698 37790062
- [14]Hetz MJ, Carl N, Haggenmüller S, et al. Superhuman performance in urology board questions by an explainable large language model enabled for context integration of the European Association of Urology guidelines: the UroBot study 2024. arXiv: 2406.01428. PMC12836625 41646097 · arXiv
- [15]Swoboda C.M. Van Hulle J.M. McAlearney A.S. Huerta T.R. Odds of talking to healthcare providers as the initial source of healthcare information: updated cross-sectional results from the Health Information National Trends Survey (HINTS) BMC Fam Pract 19 2018 146 30157770 PMC6116497
- [16]Cacciamani G.E. Dell’Oglio P. Cocci A. Asking “Dr. Google” for a second opinion: the devil is in the details Eur Urol Focus 7 2021 479 481 31690539
- [17]Davis R. Eppler M. Ayo-Ajibola O. Evaluating the effectiveness of artificial intelligence-powered large language models application in disseminating appropriate and readable health information in urology J Urol 210 2023 688 694 37428117
- [19]Lennartz S. Dratsch T. Zopfs D. Use and control of artificial intelligence in patients across the medical workflow: single-center questionnaire study of patient perspectives J Med Internet Res 23 2021 e24221 PMC7929746 33595451
- [20]Shahsavar Y. Choudhury A. User intentions to use ChatGPT for self-diagnosis and health-related purposes: cross-sectional survey study JMIR Hum Factors 10 2023 e47564 PMC10233444 37195756
- [24]Gallifant J, Afshar M, Ameen S, et al. The TRIPOD-LLM statement: a targeted guideline for reporting large language models use. medRxiv. In press. 10.1101/2024.07.24.24310930. · DOI
- [26]Haggenmüller S. Maron R.C. Hekler A. Patients’ and dermatologists’ preferences in artificial intelligence–driven skin cancer diagnostics: a prospective multicentric survey study J Am Acad Dermatol 91 2024 366 370 38670313
- [27]Nedbal C, Juliebø-Jones P, Rogers E, et al. Improving patient information and enhanced consent in urology: the impact of simulation and multimedia tools. A systematic literature review from the European Association of Urology Patient Office. Eur Urol. In press. 10.1016/j.eururo.2024.04.009. 38664166 · DOI
- [28]Hameed B.M.Z. S Dhavileswarapu A.V.L. Raza S.Z. Artificial intelligence and its impact on urological diseases and management: a comprehensive review of the literature J Clin Med 10 2021 1864 33925767 PMC8123407
- [29]Rodler S, Maruccia S, Abreu A, et al. Readability assessment of patient education materials on uro-oncological diseases using automated measures. Eur Urol Focus. In press. 10.1016/j.euf.2024.06.012. 39048402 · DOI
- [30]Hershenhouse JS, Mokhtar D, Eppler MB, et al. Accuracy, readability, and understandability of large language models for prostate cancer information to the public. Prostate Cancer Prostatic Dis. In press. 10.1038/s41391-024-00826-y. PMC12106072 38744934 · DOI
- [31]Becker G. Kempf D.E. Xander C.J. Momm F. Olschewski M. Blum H.E. Four minutes for a patient, twenty seconds for a relative - an observational study at a university hospital BMC Health Serv Res 10 2010 94 20380725 PMC2907756
Cited by
Sorted by citations
Zhang, Q., Zhang, R., Xiong, Y., Sui, Y., Tong, C., & Lin, F.-H. (2025) · Journal of Medical Internet Research · doi:10.2196/78238 · Cited by 56 · see citing works
“…nonclinical participants, while six recruited clinical populations, including patients attending urological counseling for elective surgery [49] (n=1), people experiencing symptoms of depression or anxiety [16,23,61] (n=3), people with high-risk eating disorders and other mental health…”
Influential
- Authors
- Carl, N., Nguyen, L., Haggenmüller, S., Joachim Hetz, M., Theres Winterstein, J., Otto Hartung, F., Gruene, B., Nikolas Kather, J., Holland-Letz, T., Stephan Michel, M., Wessels, F., & Josef Brinker, T.
- Affiliations
- German Cancer Research Center (DE)
- Heidelberg University (DE)
- Venue
- European Urology Open Science
- Publisher
- Elsevier BV
- Published
- 2024-12
- Volume
- 70
- Pages
- 91-98
- ISSN
- 2666-1683 (ISSN-L)
- PMC
- PMC11538625
- OpenAlex
- W4403687580
- Cited by
- 15
- References
- 6
- Citations by year
- 2026: 6 · 2025: 9
- FWCI
- 1.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- European Urology Open Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Patient-Provider Communication in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
- Keywords
- large language models, AI chatbots, patient trust in AI, urology, GPT-4, patient-physician interaction, treatment guidelines, anxiety reduction, consultation time, human-AI collaboration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
