Evaluating a generative artificial intelligence accuracy in providing medication instructions from smartphone images
Abstract
Background The Food and Drug Administration mandates patient labeling materials like the Medication Guide (MG) and Instructions for Use (IFU) to support appropriate medication use. However, challenges such as low health literacy and difficulties navigating these materials may lead to incorrect medication usage, resulting in therapy failure or adverse outcomes. The rise of generative AI, presents an opportunity to provide scalable, personalized patient education through image recognition and text generation.Objective This study aimed to evaluate the accuracy and safety of medication instructions generated by ChatGPT based on user-provided drug images, compared to the manufacturer's standard instructions.Methods Images of 12 medications requiring multiple steps for administration were uploaded to ChatGPT's image recognition function. ChatGPT's responses were compared to the official IFU and MG using text classifiers, Count Vectorization (CountVec), and Term Frequency-Inverse Document Frequency (TF-IDF). The clinical accuracy was further evaluated by independent pharmacists to determine if ChatGPT responses were valid for patient instruction.Results ChatGPT correctly identified all medications and generated patient instructions. CountVec outperformed TF-IDF in text similarity analysis, with an average similarity score of 76%. However, clinical evaluation revealed significant gaps in the instructions, particularly for complex administration routes, where ChatGPT's guidance lacked essential details, leading to lower clinical accuracy scores.Conclusion While ChatGPT shows promise in generating patient-friendly medication instructions, its effectiveness varies based on the complexity of the medication. The findings underscore the need for further refinement and clinical oversight to ensure the safety and accuracy of AI-generated medical guidance, particularly for medications with complex administration processes.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Although ChatGPT was the most commonly evaluated LLM, some studies did not find a performance advantage over other LLMs.
“Although ChatGPT was the most commonly evaluated LLM, some studies did not find a performance advantage over other LLMs.31,83”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [83]
The Discussion (page unavailable) says the authors did not know how other generative AI tools performed and could not complete their planned comparison with Google Bard. This paper therefore did not find that ChatGPT lacked a performance advantage over other LLMs; it made no such comparison. It also does not establish that ChatGPT was the most commonly evaluated LLM. (LLM, conf=0.78)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredIn a qualitative assessment of 4 LLMs on hallucination mitigation, reported ratings differed across individual queries within domain categories.
“The reported hallucination mitigation ratings for the LLMs differed across individual queries within the domain categories.83”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [83]
The Discussion (page unavailable) addresses ChatGPT’s medication-image responses and mentions hallucination as a general concern, but does not report hallucination-mitigation ratings. Searches for ratings across individual queries, domain categories, and four LLMs found no passage addressing the claimed result. (LLM, conf=0.75)
Experimental — accuracy not yet measuredStudies highlighted the potential risks of data privacy breaches associated with the use of proprietary LLMs in clinical practice.
“Studies highlighted the potential risks of data privacy breaches associated with the use of proprietary LLMs in clinical practice.74,79,83”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [83]
The Discussion describes risks of inaccurate information, hallucinations, and insufficient clinical oversight, but does not address data privacy breaches. Searches for privacy breaches and patient-data confidentiality in proprietary LLMs found no relevant passage in our copy. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 23 · Cited by 4
Research works printed but not linked: 6
- Sheikh MS, Barreto EF, Miao J, et al. Evaluating ChatGPT's efficacy in assessing the safety of non-prescription medications and supplements in patients with kidney disease. Digit Health. 2024;10:20552076241248082.
- Sarlis S, Maglogiannis I. On the reusability of sentiment analysis datasets in applications with dissimilar contexts. Art Intell Appl Innovat. 2020;583: 409e418.
- Salton G, Yang CS. On the specification of term values in automatic indexing. J Doc. 1973;29(4):351e372.
- Abraham A, Dutta P, Mandal JK, et al. Emerging Technologies in Data Mining and Information Security e Proceedings of IEMIS 2018. 3. Springer; 2019:475e486.
- Deza MM, Deza E. Distances in systems and mathematical engineering. In: Deza MM, Deza E, eds. Encyclopedia of Distances. Berlin, Heidelberg: Springer; 2012:307e310.
- Keller I, Lohan KS. On the illumination influence for object learning on robot companions. Front Robot AI. 2020;6(154):1e10.
- Authors
- Yassin, Y., Nguyen, T., Panchal, K., Getchell, K., & Aungst, T.
- Affiliations
- Massachusetts College of Pharmacy and Health Sciences (US)
- Corresponding author
- Aungst, Timothy
- ORCID
- Aungst, Timothy
- Venue
- Journal of the American Pharmacists Association
- Publisher
- Elsevier BV
- Published
- 2025-01
- Volume
- 65
- Issue
- 1
- Pages
- 102284
- ISSN
- 1544-3191 (ISSN-L)
- OpenAlex
- W4404095264
- Cited by
- 4
- References
- 23
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 1
- FWCI
- 0.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Pharmacists Association (publishedVersion)
- PubMed (publishedVersion)
- Funders
- University of Sussex
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmacy and Medical Practices
- Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
- Keywords
- generative AI, ChatGPT, medication instructions, medication administration, smartphone images, image recognition, text similarity, medication guides, instructions for use, patient education, health literacy, clinical accuracy
- MeSH terms
- Smartphone, Generative Artificial Intelligence, Artificial Intelligence, Drug Labeling, Humans, Patient Education as Topic, Pharmacists, United States, United States Food and Drug Administration, Health Literacy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
