Evaluating the effectiveness of artificial intelligence–powered large language models application in disseminating appropriate and readable health information in urology.
Abstract
PURPOSE: The Internet is a ubiquitous source of medical information, and natural language processors are gaining popularity as alternatives to traditional search engines. However, suitability of their generated content for patients is not well understood. We aimed to evaluate the appropriateness and readability of natural language processor-generated responses to urology-related medical inquiries. MATERIALS AND METHODS: Eighteen patient questions were developed based on Google Trends and were used as inputs in ChatGPT. Three categories were assessed: oncologic, benign, and emergency. Questions in each category were either treatment or sign/symptom-related questions. Three native English-speaking Board-Certified urologists independently assessed appropriateness of ChatGPT outputs for patient counseling using accuracy, comprehensiveness, and clarity as proxies for appropriateness. Readability was assessed using the Flesch Reading Ease and Flesh-Kincaid Reading Grade Level formulas. Additional measures were created based on validated tools and assessed by 3 independent reviewers. RESULTS: .01). There was no significant difference in appropriateness of the responses between treatments and symptoms or between different categories of conditions. The most common reason from urologists for low scores was responses lacking information-sometimes vital information. The mean (SD) Flesch Reading Ease score was 35.5 (SD=10.2) and the mean Flesh-Kincaid Reading Grade Level score was 13.5 (1.74). Additional quality assessment scores showed no significant differences between different categories of conditions. CONCLUSIONS: Despite impressive capabilities, natural language processors have limitations as sources of medical information. Refinement is crucial before adoption for this purpose.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CHAS report varying levels of chatbot performance.
“Thus, it is unsurprising that CHAS report varying levels of chatbot performance, raising concerns about their limitations and risks.1,12,13”
From Large Language Models for Chatbot Health Advice Studies: A Systematic Review (2025) · cites this paper as [13]
Cites
Flesch, R. (1948) · Journal of Applied Psychology · doi:10.1037/h0057532 · Cited by 5,391 · see citing works
Charnock, D., Shepperd, S., Needham, G., & Gann, R. (1999) · Journal of Epidemiology and Community Health · doi:10.1136/jech.53.2.105 · Cited by 3,329 · see citing works
Analyzing and Interpreting Data From Likert-Type Scales
Sullivan, G. M., & Artino, A. R. (2013) · Journal of Graduate Medical Education · doi:10.4300/jgme-5-4-18 · Cited by 3,044 · see citing works
Assessing, Controlling, and Assuring the Quality of Medical Information on the Internet
Silberg, W. M. (1997) · JAMA · doi:10.1001/jama.277.15.1244 · Cited by 2,399 · see citing works
Xu, L., Sanders, L., Li, K., & Chow, J. C. L. (2021) · JMIR Cancer · doi:10.2196/27850 · Cited by 571 · see citing works
Sarraju, A., Bruemmer, D., Iterson, E. H. V., Cho, L., Rodríguez, F., & Laffin, L. J. (2023) · JAMA · doi:10.1001/jama.2023.1044 · Cited by 455 · see citing works
Improving comprehension for cancer patients with low literacy skills: strategies for clinicians
Doak, C. C., Doak, L. G., Friedell, G. H., & Meade, C. D. (1998) · CA A Cancer Journal for Clinicians · doi:10.3322/canjclin.48.3.151 · Cited by 309 · see citing works
UROLOGIC DISEASES IN AMERICA PROJECT: ANALYTICAL METHODS AND PRINCIPAL FINDINGS
Litwin, M. S., Saigal, C. S., Yano, E. M., Avila, C. M. F., Geschwind, S. A., Hanley, J. M., Joyce, G., Madison, R. A., Pace, J., Polich, S. M., et al. (2005) · The Journal of Urology · doi:10.1097/01.ju.0000152365.43125.3b · Cited by 257 · see citing works
Readability of patient education materials: implications for clinical practice
Albright, J. C., de Guzman, C., Acebo, P., et al. (1996) · Applied Nursing Research · doi:10.1016/s0897-1897(96)80254-0 · Cited by 219 · see citing works
Gilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040269 · Cited by 183 · see citing works
Guidelines for Reporting of Statistics for Clinical Research in Urology
Assel, M. J., Sjoberg, D. D., Elders, A., et al. (2019) · The Journal of Urology · doi:10.1097/ju.0000000000000001 · Cited by 166 · see citing works
The QUEST for quality online health information: validation of a short quantitative tool
Robillard, J. M., Jun, J. H., Lai, J., & Feng, T. L. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0668-9 · Cited by 143 · see citing works
Swoboda, C. M., Hulle, J. M. V., McAlearney, A. S., & Huerta, T. R. (2018) · BMC Family Practice · doi:10.1186/s12875-018-0805-7 · Cited by 125 · see citing works
Hellawell, G. O., Turner, K. J., Monnier, K. J. L., & BREWSTER, S. F. (2000) · British Journal of Urology · doi:10.1046/j.1464-410x.2000.00831.x · Cited by 104 · see citing works
Consulting “Dr Google” for sexual dysfunction: a contemporary worldwide trend analysis
On behalf of EAU-YAU Men’s Health Working Group, Russo, G. I., Mauro, M. D., Cocci, A., Cacciamani, G. E., Cimino, S., Şerefoğlu, E. C., Albersen, M., Capogrosso, P., Fode, M. M., et al. (2019) · International Journal of Impotence Research · doi:10.1038/s41443-019-0203-2 · Cited by 48 · see citing works
Self-citationCacciamani, G. E., Bassi, S., Sebben, M. A., Marcer, A., Russo, G. I., Cocci, A., Dell’Oglio, P., Medina, L. G., Nassiri, N., Tafuri, A., et al. (2019) · European Urology Oncology · doi:10.1016/j.euo.2019.07.002 · Cited by 46 · see citing works
Self-citationAsking “Dr. Google” for a Second Opinion: The Devil Is in the Details
Cacciamani, G. E., Dell’Oglio, P., Cocci, A., Russo, G. I., Abreu, A. D. C., Gill, I. S., Briganti, A., & Artibani, W. (2019) · European Urology Focus · doi:10.1016/j.euf.2019.10.011 · Cited by 30 · see citing works
Self-citationGuidelines for Reporting of Figures and Tables for Clinical Research in Urology
Vickers, A. J., Assel, M. J., Sjoberg, D. D., et al. (2020) · The Journal of Urology · doi:10.1097/ju.0000000000001096 · Cited by 30 · see citing works
Yoon, H. Y., You, K. H., Kwon, J. H., Kim, J. S., Rha, S. Y., Chang, Y. J., & Lee, S. (2022) · Journal of Medical Internet Research · doi:10.2196/39571 · Cited by 21 · see citing works
Cacciamani, G. E., Sebben, M. A., Tafuri, A., Nassiri, N., Cocci, A., Russo, G. I., Hung, A. J., Abreu, A. L., Gill, I. S., & Artibani, W. (2021) · International Journal of Medical Robotics and Computer Assisted Surgery · doi:10.1002/rcs.2250 · Cited by 14 · see citing works
Self-citationCancer awareness crusades—pink ribbons and growing moustaches
Cacciamani, G. E., Stern, M. C., Medina, L. G., et al. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30639-4 · Cited by 11 · see citing works
Self-citationWeb search queries and prostate cancer
Cacciamani, G. E., Gill, K. S., & Gill, I. S. (2020) · The Lancet Oncology · doi:10.1016/s1470-2045(20)30138-8 · Cited by 11 · see citing works
Self-citationGonzález‐Padilla, D. A., Navarro, R. E., Subiela, J. D., Kumar, R., Medina, L. G., Aumatell, J., la Morena-Gallego, J. M. D., & Cacciamani, G. E. (2021) · Société Internationale d’Urologie Journal · doi:10.48083/uhsi5324 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Ayo-Ajibola, O., Davis, R. J., Lin, M. E., Riddell, J., & Kravitz, R. L. (2024) · Journal of Medical Internet Research · doi:10.2196/55138 · Cited by 95 · see citing works
“ChatGPT has demonstrated proficiency in performing tasks on par with, and sometimes surpassing, physicians [10,11].”
Self-citationGanjavi, C., Eppler, M., O’Brien, D., Ramacciotti, L. S., Ghauri, M. S., Anderson, I., Choi, J., Dwyer, D., Stephens, C., Shi, V., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000596 · Cited by 73 · see citing works
Self-citationRodler, S., Kopliku, R., Ulrich, D., Kaltenhauser, A., Casuscelli, J., Eismann, L., Waidelich, R., Buchner, A., Butz, A., Cacciamani, G. E., et al. (2024) · European Urology Focus · doi:10.1016/j.euf.2023.10.020 · Cited by 65 · see citing works
Self-citationYan, Z., Fan, K.-Q., Zhang, Q., Wu, X., Chen, Y., Wu, X., Yu, T., Su, N., Zou, Y., Chi, H., et al. (2025) · World Journal of Urology · doi:10.1007/s00345-025-05757-4 · Cited by 24 · see citing works
Deng, J., Li, L., Oosterhof, J. J., Malliaras, P., Silbernagel, K. G., Breda, S. J., Eygendaal, D., Oei, E. H., & de Vos, R.-J. (2025) · Musculoskeletal Science and Practice · doi:10.1016/j.msksp.2025.103275 · Cited by 12 · see citing works
- Authors
- Davis, R., Eppler, M., Ayo-Ajibola, O., Loh-Doyle, J. C., Nabhani, J., Samplaski, M., Gill, I., & Cacciamani, G. E.
- Affiliations
- University of Southern California (US)
- USC Institute of Urology, and Catherine and Joseph Aresty Department of Urology
- ORCID
- Venue
- Journal of Urology
- Published
- 2023-10
- Volume
- 210
- Issue
- 4
- Pages
- 688-694
- ISSN
- 0022-5347 (ISSN-L)
- 1527-3792
- OpenAlex
- W4383710066
- Cited by
- 119
- References
- 23
- Citations by year
- 2026: 24 · 2025: 36 · 2024: 48 · 2023: 11
- FWCI
- 4.75
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The Journal of Urology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, ChatGPT, urology, patient education, readability, natural language processing, patient counseling, health information dissemination, urologic oncology, urological emergencies, Flesch Reading Ease
- MeSH terms
- Artificial Intelligence, Humans, Language, Urology, Internet, Comprehension, Health Literacy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
