Investigating the capabilities of advanced large language models in generating patient instructions and patient educational material
Abstract
OBJECTIVES: Large language models (LLMs) with advanced language generation capabilities have the potential to enhance patient interactions. This study evaluates the effectiveness of ChatGPT 4.0 and Gemini 1.0 Pro in providing patient instructions and creating patient educational material (PEM). METHODS: A cross-sectional study employed ChatGPT 4.0 and Gemini 1.0 Pro across six medical scenarios using simple and detailed prompts. The Patient Education Materials Assessment Tool for Print materials (PEMAT-P) evaluated the understandability, actionability, and readability of the outputs. RESULTS: LLMs provided consistent responses, especially regarding drug information, therapeutic goals, administration, common side effects, and interactions. However, they lacked guidance on expiration dates and proper medication disposal. Detailed prompts yielded comprehensible outputs for the average adult. ChatGPT 4.0 had mean understandability and actionability scores of 80% and 60%, respectively, compared with 67% and 60% for Gemini 1.0 Pro. ChatGPT 4.0 produced longer outputs, achieving 85% readability with detailed prompts, while Gemini 1.0 Pro maintained consistent readability. Simple prompts resulted in ChatGPT 4.0 outputs at a 10th-grade reading level, while Gemini 1.0 Pro outputs were at a 7th-grade level. Both LLMs produced outputs at a 6th-grade level with detailed prompts. CONCLUSION: LLMs show promise in generating patient instructions and PEM. However, healthcare professional oversight and patient education on LLM use are essential for effective implementation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT-4 reportedly provided advice that was concordant with guidelines that were publicly available.
“Similarly, ChatGPT-4 reportedly provided advice that was concordant with guidelines that were publicly available, 79”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [79]
Studies highlighted the potential risks of data privacy breaches associated with the use of proprietary LLMs in clinical practice.
“Studies highlighted the potential risks of data privacy breaches associated with the use of proprietary LLMs in clinical practice.74,79,83”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [79]
Cites
Medical error—the third leading cause of death in the US
Makary, M. A., & Daniel, M. (2016) · BMJ · doi:10.1136/bmj.i2139 · Cited by 3,264 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · see citing works
Shoemaker, S. J., Wolf, M. S., & Brach, C. (2014) · Patient Education and Counseling · doi:10.1016/j.pec.2014.05.027 · Cited by 1,366 · see citing works
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047 · see citing works
The future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045 · see citing works
Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/50638 · Cited by 709 · see citing works
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 394 · see citing works
Improving Health Outcomes Through Patient Education and Partnerships with Patients
Paterick, T. E., Patel, N. J., Tajik, A. J., & Chandrasekaran, K. (2017) · Baylor University Medical Center Proceedings · doi:10.1080/08998280.2017.11929552 · Cited by 368 · see citing works
Preventable medication harm across health care settings: a systematic review and meta-analysis
Hodkinson, A., Tyler, N., Ashcroft, D. M., Keers, R. N., Khan, K., Phipps, D. L., Abuzour, A. S., Bower, P., Avery, A. J., Campbell, S., et al. (2020) · BMC Medicine · doi:10.1186/s12916-020-01774-9 · Cited by 295 · see citing works
Hopkins, A. M., Logan, J. M., Kichenadasse, G., & Sorich, M. J. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad010 · Cited by 259 · see citing works
The value of inpatient pharmaceutical counselling to elderly patients prior to discharge
Alrashed, S. A., Wright, D. J., Roebuck, N., Sunter, W., & Chrystyn, H. (2002) · British Journal of Clinical Pharmacology · doi:10.1046/j.1365-2125.2002.01707.x · Cited by 206 · see citing works
Marcus, C. (2014) · Health Psychology and Behavioral Medicine · doi:10.1080/21642850.2014.900450 · Cited by 197 · see citing works
Chen, J., Mullins, C. D., Novak, P. J., & Thomas, S. B. (2015) · Health Education & Behavior · doi:10.1177/1090198115579415 · Cited by 192 · see citing works
Moons, P., & Bulck, L. V. (2023) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvad087 · Cited by 115 · see citing works
Yeh, M., Wu, S., & Tung, T. (2017) · Applied Nursing Research · doi:10.1016/j.apnr.2017.10.008 · Cited by 113 · see citing works
A Comparative Analysis of the Quality of Patient Education Materials From Medical Specialties
Agarwal, N., Hansberry, D. R., Sabourin, V. M., et al. (2013) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2013.6060 · Cited by 103 · see citing works
Patient-centred care in general dental practice—a systematic review of the literature
Mills, I., Frost, J., Cooper, C., Moles, D. R., & Kay, E. (2014) · BMC Oral Health · doi:10.1186/1472-6831-14-64 · Cited by 100 · see citing works
Reddy, S. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101304 · Cited by 99 · see citing works
The significance of patient participation in nursing care – a concept analysis
Nilsson, M. M., From, I., & Lindwall, L. (2018) · Scandinavian Journal of Caring Sciences · doi:10.1111/scs.12609 · Cited by 89 · see citing works
Murphy, P. W., Chesson, A. L., WALKER, L., et al. (2000) · Southern Medical Journal · doi:10.1097/00007611-200093030-00010 · Cited by 74 · see citing works
Utilization of ChatGPT for Plastic Surgery Research: Friend or Foe?
Gupta, R., Herzog, I., Weisberger, J. S., Chao, J. A. K., Chaiyasate, K., & Lee, E. S. (2023) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2023.03.004 · Cited by 72 · see citing works
Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4Preprint
Bsharat, S. M., Myrzakhan, A., & Shen, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.16171 · Cited by 51 · see citing works
Heart-to-heart with ChatGPT: the impact of patients consulting AI for cardiovascular health advice
Lautrup, A. D., Hyrup, T., Schneider‐Kamp, A., Dahl, M., Lindholt, J. S., & Schneider–Kamp, P. (2023) · Open Heart · doi:10.1136/openhrt-2023-002455 · Cited by 50 · see citing works
Preventable adverse drug events: Descriptive epidemiology
Woo, S. A., Cragg, A., Wickham, M. E., Villanyi, D., Scheuermeyer, F. X., Hau, J. P., & Hohl, C. M. (2019) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.14139 · Cited by 46 · see citing works
Patino, G. A., Amiel, J. M., Brown, M., Lypson, M. L., & Chan, T. M. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005636 · Cited by 43 · see citing works
Sridharan, K., & Sivaramakrishnan, G. (2024) · Exploratory Research in Clinical and Social Pharmacy · doi:10.1016/j.rcsop.2024.100492 · Cited by 18 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations
Yakob, R. A., Bamgboje-Ayodele, A., Collins, J. C., & Aslani, P. (2026) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2026.09.007
- Authors
- Sridharan, K., & Sivaramakrishnan, G.
- Affiliations
- Arabian Gulf University (BH)
- Bahrain Defence Force Royal Medical Services, Riffa, Bahrain
- Corresponding author
- Sridharan, Kannan
- Venue
- European Journal of Hospital Pharmacy
- Publisher
- BMJ
- Published
- 2024-10-11
- Volume
- 32
- Issue
- 6
- Pages
- 501-507
- ISSN
- 2047-9956 (ISSN-L)
- 2047-9964
- OpenAlex
- W4403331297
- Cited by
- 15
- References
- 27
- Citations by year
- 2026: 9 · 2025: 6
- FWCI
- 1.41
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- European Journal of Hospital Pharmacy (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmacy and Medical Practices
- Keywords
- large language models, ChatGPT, Gemini, patient instructions, patient education materials, drug information, readability, comprehensibility, actionability, prompt engineering, PEMAT-P, drug disposal, adverse drug reactions, patient education
- MeSH terms
- Large Language Models, Adult, Cross-Sectional Studies, Female, Humans, Language, Male, Patient Education as Topic, Comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
