Comparison of artificial intelligence-generated and physician-generated patient education materials on early diabetic kidney disease
Open accessAbstract
Background: Diabetic kidney disease (DKD) is a common and serious complication of diabetes mellitus and has become the most important cause of end-stage renal disease (ESRD). In light of the rising prevalence of diabetes, there is a growing imperative for the early detection and intervention of DKD. With the rapid development of artificial intelligence (AI) technologies, its potential applications in patient education are receiving increasing attention, especially large language models (LLMs). The aim of this study was to evaluate the quality of LLMs-generated patient education materials (PEMs) for early DKD and to explore its feasibility in patient education. Methods: Four LLMs (ERNIE Bot 4.0, GPT-4o, ChatGLM4, and ChatGPT-o1) were selected for this study to generate PEMs. Among them, ERNIE Bot 4.0, GPT-4o, and ChatGLM4 generated 2 versions of PEMs based on American Diabetes Association(ADA) guidelines and without ADA guidelines, respectively. ChatGPT-o1 only generated a PEM without ADA guidelines. An experienced physician wrote a PEM based on ADA guidelines. All materials were assessed using a Likert scale which covered the dimensions of accuracy, completeness, safety, and patient comprehensibility. A total of 7 medical experts (including nephrologists and endocrinologists) and 50 diabetic patients were invited to evaluate the study. We recorded basic information on the patient evaluators. Results: Experts evaluated PEMs from ERNIE Bot 4.0, GPT-4o, ChatGLM4, and ChatGPT-o1, plus physician-sourced PEM. Results showed ERNIE Bot 4.0’s non-guideline PEM and physician-sourced PEM were the top two. Patient assessments of the 2 top-scoring PEMs found that the ERNIE Bot 4.0’s non-guideline PEM performed as well as, if not slightly better than, the physician-sourced PEM in terms of patient comprehensibility, completeness, and safety. In addition, the non-guideline-based PEM was preferred for patients with a history of diabetes longer than 5 years and for patients with proteinuria. Surprisingly, GPT-4o and ChatGLM4’s non-guideline PEMs outperformed guideline-based ones. Conclusion: The LLMs-sourced PEMs, especially the ERNIE Bot 4.0’s non-guideline PEM for early DKD, performed comparably to the physician-sourced PEM in terms of accuracy, completeness, safety, and patient comprehensibility, and exerted a high degree of feasibility. AI may show the potential for broader applications in patient education in the near future.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have potential in patient education and health management for diabetic complications.
“Cheng et al. [137] used ERNIE Bot 4.0, GPT-4o, and ChatGLM4 to generate patient education materials for early DN, and comparisons with physician-authored texts showed that LLM outputs were comparable or superior in accuracy, completeness, and readability, highlighting their potential in patient education and health management.”
From Artificial intelligence-powered prediction of diabetic complications: from clinical data to molecular omics (2026) · cites this paper as [137]
The Methods and Results report that ERNIE Bot 4.0, GPT-4o, and ChatGLM4 generated materials for early diabetic kidney disease, and the Discussion identifies potential for patient education. But the Results say that ERNIE Bot 4.0’s materials were comparable to the physician’s, while materials from other LLMs generally scored lower; patient comprehensibility was compared only for ERNIE Bot 4.0 and the physician. Thus, the sentence overgeneralizes the comparable-or-superior finding across the named models and readability. (LLM, conf=0.88)
Experimental — accuracy not yet measuredLLM outputs were comparable or superior to physician-authored texts in accuracy, completeness, and readability for early diabetic nephropathy patient education materials.
“Cheng et al. [137] used ERNIE Bot 4.0, GPT-4o, and ChatGLM4 to generate patient education materials for early DN, and comparisons with physician-authored texts showed that LLM outputs were comparable or superior in accuracy, completeness, and readability, highlighting their potential in patient education and health management.”
From Artificial intelligence-powered prediction of diabetic complications: from clinical data to molecular omics (2026) · cites this paper as [137]
The Results (Expert evaluations of all PEMs; page unavailable) report that ERNIE Bot 4.0 materials were comparable to the physician-authored material, and its non-guideline version had the highest accuracy. Other LLMs generally scored lower, so the finding does not apply to LLM outputs as a group. The study assessed patient comprehensibility, but the Discussion says it lacked a tool for an objective readability evaluation. (LLM, conf=0.88)
Experimental — accuracy not yet measuredCheng et al. used ERNIE Bot 4.0, GPT-4o, and ChatGLM4 to generate patient education materials for early diabetic nephropathy.
“Cheng et al. [137] used ERNIE Bot 4.0, GPT-4o, and ChatGLM4 to generate patient education materials for early DN, and comparisons with physician-authored texts showed that LLM outputs were comparable or superior in accuracy, completeness, and readability, highlighting their potential in patient education and health management.”
From Artificial intelligence-powered prediction of diabetic complications: from clinical data to molecular omics (2026) · cites this paper as [137]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Self-citationPre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · see citing works
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · see citing works
Zhou, B., Rayner, A. W., Gregg, E. W., Sheffer, K. E., Carrillo-Larco, R. M., Bennett, J. E., Shaw, J. E., Paciorek, C. J., Singleton, R. K., Barradas Pires, A., et al. (2024) · The Lancet · doi:10.1016/s0140-6736(24)02317-1 · Cited by 1,013 · see citing works
Diabetic Nephropathy: Challenges in Pathogenesis, Diagnosis, and Treatment
Samsu, N. (2021) · BioMed Research International · doi:10.1155/2021/1497449 · Cited by 1,001 · see citing works
Structured information extraction from scientific text with large language models
Dagdelen, J., Dunn, A., Lee, S., Walker, N., Rosen, A. S., Ceder, G., Persson, K. A., & Jain, A. (2024) · Nature Communications · doi:10.1038/s41467-024-45563-x · Cited by 771 · see citing works
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408 · see citing works
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 394 · see citing works
Diagnosis and Management of Type 2 Diabetic Kidney Disease
Doshi, S., & Friedman, A. N. (2017) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.11111016 · Cited by 348 · see citing works
Large language models in patient education: a scoping review of applications in medicine
Aydin, S., Karabacak, M., Vlachos, V., & Margetis, K. D. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1477898 · Cited by 333 · see citing works
Integrated image-based deep learning and language models for primary diabetes care
Li, J., Guan, Z., Wang, J., Cheung, C. Y., Zheng, Y., Lim, L.-L., Lim, C. C., Ruamviboonsuk, P., Raman, R., Corsino, L., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03139-8 · Cited by 250 · see citing works
Self-citationMarcus, C. (2014) · Health Psychology and Behavioral Medicine · doi:10.1080/21642850.2014.900450 · Cited by 197 · see citing works
Self-management: One size does not fit all
Trappenburg, J. C. A., Jonkman, N. H., Jaarsma, T., van Os‐Medendorp, H., Kort, H. S. M., de Wit, N. J., Hoes, A. W., & Schuurmans, M. J. (2013) · Patient Education and Counseling · doi:10.1016/j.pec.2013.02.009 · Cited by 158 · see citing works
Special Issue “Diabetic Nephropathy: Diagnosis, Prevention and Treatment”
Ruiz‐Ortega, M., Rodrigues-Díez, R. R., Lavoz, C., & Rayego‐Mateos, S. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9030813 · Cited by 140 · see citing works
Schouten, B. C., Meeuwesen, L., Tromp, F., & Harmsen, H. A. M. (2007) · Patient Education and Counseling · doi:10.1016/j.pec.2007.03.018 · Cited by 127 · see citing works
Timmers, T., Janssen, L., Kool, R. B., & Kremer, J. A. M. (2020) · Journal of Medical Internet Research · doi:10.2196/17342 · Cited by 127 · see citing works
The Role of Artificial Intelligence in Medical Education: A Systematic Review
Tozsin, A., Uçmak, H., Soytürk, S., Aydın, A., Gözen, A. S., Fahim, M. M. A., Güven, S., & Ahmed, K. (2024) · Surgical Innovation · doi:10.1177/15533506241248239 · Cited by 96 · see citing works
Cheong, R. C. T., Unadkat, S. N., Mcneillis, V., Williamson, A., Joseph, J., Randhawa, P. S., Andrews, P. J., & Paleri, V. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08319-9 · Cited by 94 · see citing works
Training data influence analysis and estimation: a survey
Hammoudeh, Z., & Lowd, D. (2024) · Machine Learning · doi:10.1007/s10994-023-06495-7 · Cited by 80 · see citing works
Azzopardi, M., Ng, B., Logeswaran, A., Loizou, C. L., Cheong, R. C. T., Gireesh, P., Ting, D. S. J., & Chong, Y. J. (2024) · BMJ Open Ophthalmology · doi:10.1136/bmjophth-2024-001824 · Cited by 33 · see citing works
EYE-Llama, an in-domain large language model for ophthalmology
Haghighi, T., Gholami, S., Sokol, J. T., Kishnani, E., Ahsaniyan, A., Rahmanian, H., Hedayati, F., Leng, T., & Alam, M. N. (2025) · iScience · doi:10.1016/j.isci.2025.112984 · Cited by 31 · see citing works
Halawani, A., Mitchell, A., Saffarzadeh, M., Wong, V. K. F., Chew, B. H., & Forbes, C. M. (2024) · Urology · doi:10.1016/j.urology.2023.11.042 · Cited by 18 · see citing works
Insights into LLM Long-Context Failures: When Transformers Know but Don’t Tell
Gao, M., Lu, T., Yu, K., Byerly, A., & Khashabi, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.447 · Cited by 10 · see citing works
Lindstrøm, L., Clausen, M., Jensen, N. A., Nielsen, M. H., Nikontovic, A., & Cichosz, S. L. (2024) · Journal of Diabetes Science and Technology · doi:10.1177/19322968241265882 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Karaçaylı, C., Tahir, E., & Altuntaş, E. E. (2025) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-025-09932-6 · Cited by 6 · see citing works
Li, H., Lin, Y., & Lv, L. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1805848 · Cited by 5 · see citing works
Xie, X., Wu, C., Huang, Z., Zhou, Y., Huang, J., Dao, F., Yan, D., Deng, K., Lyu, H., Ma, C., et al. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag083 · Cited by 4 · see citing works
“Cheng et al. [137] used ERNIE Bot 4.0, GPT-4o, and ChatGLM4 to generate patient education materials for early DN, and comparisons with physician-authored texts showed that LLM outputs were comparable or superior in accuracy, completeness, and readability, highlighting their potential in patient…”
InfluentialHallock, R., Griebeler, M. L., Burguera, B. G., & Cabandugama, P. K. (2026) · Obesity and Endocrinology · doi:10.1093/obendo/wjaf018 · Cited by 4 · see citing works
Cha, R., Liu, J., Wang, L., Li, S., Mei, L., Gong, X., Weng, Y., Jiang, X., & He, X. (2026) · Journal of Diabetes Science and Technology · doi:10.1177/19322968261422628 · Cited by 3 · see citing works
Yang, Y. T., Edwards, B. I., & Sanmugam, M. (2025) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2025.101729 · Cited by 6 · see citing works
Wu, Z., Hu, B.-B., Mao, Q.-X., Han, Y., & Zhao, Q. (2026) · Digital Health · doi:10.1177/20552076261444223 · Cited by 2 · see citing works
Zhao, Y., Miao, Y., Luo, Y., Guo, R., Wang, H., & Wu, Y. N. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106513 · Cited by 2 · see citing works
Zheng, C., Zhang, Y., Luo, T., Li, F., Qin, Z., Xiong, S., Zhang, J., Lei, E., Lu, L., Zhang, L., et al. (2026) · BMC Medical Education · doi:10.1186/s12909-025-08499-4 · Cited by 2 · see citing works
Zhang, Q., Yu, Q., Wang, G., Zeng, Z., Liu, S., Rao, X., Cheng, C., Wang, X., Shi, L., Wang, Y. L., et al. (2026) · Journal of Cancer Education · doi:10.1007/s13187-026-02918-w · Cited by 2 · see citing works
Self-citationDeveloping AI-generated Storybooks for Patient Education
Boy, Y., & Çam, F. (2026) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000001476
Hata, N., Oami, T., Kawakami, E., Hanai, A., & Nakada, T. (2026) · JMIR Formative Research · doi:10.2196/81606
Hu, S., 林邠芬, & 林丰兰 (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1850486
The Role of Artificial Intelligence in Nephrology Education
Miao, J., Thongprayoon, C., & Cheungpasitporn, W. (2026) · Advances in Kidney Disease and Health · doi:10.1053/j.akdh.2026.06.003
Song, J., Makhdzir, N., Haron, Z., Qin, C., & Wang, L. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1902424 · Cited by 1 · see citing works
Cook, N., Roy, A., Foster, M. J., Holland, K. M., Shaikh, A., Jungbauer, R., & Yudkin, J. (2026) · Childhood Obesity · doi:10.1177/21532176261498254
Yu, S., Zhou, X., Wang, Y., & Xu, L. (2026) · Digital Health · doi:10.1177/20552076261502905
Self-citation
- Authors
- Cheng, M., Zhang, Q., Liang, H., Wang, Y., Qin, J., Gong, L., Wang, S., Li, L., & Xiao, X.
- Affiliations
- Qilu Hospital of Shandong University (CN)
- Shandong University (CN)
- Corresponding author
- Xiao, Xiaoyan
- Venue
- Frontiers in Endocrinology
- Publisher
- Frontiers Media SA
- Published
- 2025-04-22
- Volume
- 16
- Pages
- 1559265-1559265
- ISSN
- 1664-2392 (ISSN-L)
- PMC
- PMC12052532
- OpenAlex
- W4409692207
- Cited by
- 17
- References
- 25
- Citations by year
- 2026: 15 · 2025: 2
- FWCI
- 4.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Endocrinology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Shandong University
- Qilu Hospital of Shandong University
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Keywords
- diabetic nephropathy, patient education materials, large language models, AI-generated content, GPT-4o, guideline adherence, proteinuria, diabetes mellitus, end-stage renal disease prevention
- MeSH terms
- Adult, Artificial Intelligence, Diabetic Nephropathies, Female, Humans, Male, Middle Aged, Patient Education as Topic, Physicians
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
