Assessing the accuracy and consistency of ChatGPT in clinical pharmacy management: a preliminary analysis with clinical pharmacy experts worldwide
Abstract
Background ChatGPT conversation system has ushered in a revolutionary new era of information retrieval and stands as one of the fastest-growing platforms. Clinical pharmacy, as a dynamic discipline, necessitates an advanced comprehension of drugs and diseases. The process of decision-making in clinical pharmacy demands accuracy and consistency in medical information, as it directly affects patient safety.Objective The objective was to evaluate ChatGPT's accuracy and consistency in managing pharmacotherapy cases across multiple time points. Additionally, input was gathered from global clinical pharmacy experts, and the agreement between ChatGPT's responses and those of clinical pharmacy experts worldwide was assessed.Methods A set of 20 cases of pharmacotherapy was entered into ChatGPT at three different time points. Reliability analysis was performed using inter-rater reliability to measure the accuracy of the output generated by ChatGPT at each time point. Test-retest reliability was performed to measure the consistency of the output generated by ChatGPT across the three time points. Pharmacy expert performance was evaluated, and the overall results were compared.Results ChatGPT achieved a hit rate of 70.83% at week 1, 79.2% at week 3, and 75% at week 5. The percent agreement between weeks 1 and 3 was 79.2%, whereas it was 87.5% between weeks 3 and 5, and 83.3% between weeks 1 and 5. In contrast, accuracy rates among clinical pharmacy experts showed considerable variation according to their geographic location. The highest agreement between clinical pharmacist responses and ChatGPT responses was observed at the last time point examined.Conclusions Overall, the analysis suggested that ChatGPT is capable of generating clinically relevant pharmaceutical information, albeit with some variation in accuracy and consistency. It should be noted that clinical pharmacy experts worldwide may provide varying degrees of accuracy depending on their expertise. This study highlights the potential of AI chatbots in clinical pharmacy.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
One study tested GPT-3.5 on 3 occasions over 5 weeks and reported reproducibility rates of 79.2%, 87.5%, and 83.3%, with statistically significant agreement (P < 0.05).
“One study tested GPT-3.5 on 3 occasions over 5 weeks and reported reproducibility rates of 79.2%, 87.5%, and 83.3%, with statistically significant agreement (P < 0.05).32”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [32]
ChatGPT-3.5 performed adequately in providing medication information but had poor performance in recognizing adverse drug reactions and on high-complexity tasks.
“In other studies, ChatGPT-3.5 was reported to have performed adequately in providing medication information, 32,55 but had poor performance in J o u r n a l P r e -p r o o f recognizing adverse drug reactions and similar poor performance on high-complexity tasks.55,91,92”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [32]
The citation lists haven’t been fetched yet
References 22 · Cited by 56
- Authors
- Al-Dujaili, Z., Omari, S., Pillai, J., & Al Faraj, A.
- Affiliations
- American University of Iraq Baghdad (IQ)
- American University of Beirut (LB)
- Venue
- Research in Social and Administrative Pharmacy
- Publisher
- Elsevier BV
- Published
- 2023-12
- Volume
- 19
- Issue
- 12
- Pages
- 1590-1594
- ISSN
- 1551-7411 (ISSN-L)
- OpenAlex
- W4386516233
- Cited by
- 56
- References
- 22
- Citations by year
- 2026: 13 · 2025: 18 · 2024: 23 · 2023: 2
- FWCI
- 2.23
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Research in Social and Administrative Pharmacy (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmacy and Medical Practices
- Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
- Keywords
- ChatGPT, large language models, clinical pharmacy, pharmacological management, AI accuracy, inter-rater reliability, test-retest reliability, patient safety
- MeSH terms
- Generative Artificial Intelligence, Humans, Pharmacists, Pharmacy Service, Hospital, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
