Performance of ChatGPT on factual knowledge questions regarding clinical pharmacy
Open accessAbstract
ChatGPT is a language model that was trained on a large dataset including medical literature. Several studies have described the performance of ChatGPT on medical exams. In this study, we examine its performance in answering factual knowledge questions regarding clinical pharmacy. Questions were obtained from a Dutch application that features multiple-choice questions to maintain a basic knowledge level for clinical pharmacists. In total, 264 clinical pharmacy-related questions were presented to ChatGPT and responses were evaluated for accuracy, concordance, quality of the substantiation, and reproducibility. Accuracy was defined as the correctness of the answer, and results were compared to the overall score by pharmacists over 2022. Responses were marked concordant if no contradictions were present. The quality of the substantiation was graded by two independent pharmacists using a 4-point scale. Reproducibility was established by presenting questions multiple times and on various days. ChatGPT yielded accurate responses for 79% of the questions, surpassing pharmacists' accuracy of 66%. Concordance was 95%, and the quality of the substantiation was deemed good or excellent for 73% of the questions. Reproducibility was consistently high, both within day and between days (>92%), as well as across different users. ChatGPT demonstrated a higher accuracy and reproducibility to factual knowledge questions related to clinical pharmacy practice than pharmacists. Consequently, we posit that ChatGPT could serve as a valuable resource to pharmacists. We hope the technology will further improve, which may lead to enhanced future performance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLM response reproducibility refers to the consistency of its outputs to the same query and has been shown to differ on repeat assessment.
“LLM response reproducibility refers to the consistency of its outputs to the same query and has been shown to differ on repeat assessment.58,59,66”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [59]
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Roosan, D., Padua, P., Khan, R. K. R., Khan, H., Verzosa, C., & Wu, Y. (2023) · Journal of the American Pharmacists Association · doi:10.1016/j.japh.2023.11.023 · Cited by 124 · see citing works
Huang, X., Estau, D., Liu, X., Yu, Y., Qin, J., & Li, Z. (2023) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.15896 · Cited by 117 · see citing works
Performance of ChatGPT on the pharmacist licensing examination in Taiwan
Wang, Y.-M., Shen, H.-W., & Chen, T.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000942 · Cited by 109 · see citing works
Performance and risks of ChatGPT used in drug information: an exploratory real-world analysis
Morath, B., Chiriac, U., Jaszkowski, E., Deiß, C., Nürnberg, H., Hörth, K., Hoppe‐Tichy, T., & Green, K. (2023) · European Journal of Hospital Pharmacy · doi:10.1136/ejhpharm-2023-003750 · Cited by 79 · see citing works
Al-Dujaili, Z., Omari, S., Pillai, J., & Al Faraj, A. (2023) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2023.08.012 · Cited by 56 · see citing works
Tallon, M., Barragry, J., Allen, A., Breslin, N. P., Deasy, E., Moloney, E. D., Delaney, T., Wall, C. A. M., O'Byrne, J., & Grimes, T. (2015) · European Journal of Hospital Pharmacy · doi:10.1136/ejhpharm-2014-000511 · Cited by 52 · see citing works
Assessing the applicability and appropriateness of ChatGPT in answering clinical pharmacy questions
Fournier, A., Fallet, C., Sadeghipour, F., & Perrottet, N. (2023) · Annales Pharmaceutiques Françaises · doi:10.1016/j.pharma.2023.11.001 · Cited by 29 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Domián, B. M., Ashraf, A. R., Fittler, A. T., Káplár, M., & Vida, R. G. (2025) · Exploratory Research in Clinical and Social Pharmacy · doi:10.1016/j.rcsop.2025.100655 · Cited by 6 · see citing works
Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations
Yakob, R. A., Bamgboje-Ayodele, A., Collins, J. C., & Aslani, P. (2026) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2026.09.007
- Authors
- van Nuland, M., Erdogan, A., Aςar, C., Contrucci, R., Hilbrants, S., Maanach, L., Egberts, T., & van der Linden, P. D.
- Affiliations
- Department of Clinical Pharmacy Tergooi Medical Center Hilversum The Netherlands
- Tergooi (NL)
- Amphia Ziekenhuis (NL)
- Medisch Centrum Leeuwarden (NL)
- Apotheek Haagse Ziekenhuizen (NL)
- Utrecht University (NL)
- Venue
- The Journal of Clinical Pharmacology
- Publisher
- Wiley
- Published
- 2024-04-16
- Volume
- 64
- Issue
- 9
- Pages
- 1095-1100
- ISSN
- 0021-9754 (ISSN-L)
- 0091-2700
- 1552-4604
- OpenAlex
- W4394853549
- Cited by
- 27
- References
- 8
- Citations by year
- 2026: 6 · 2025: 14 · 2024: 7
- FWCI
- 2.51
- Type
- journal article
- Language
- en
- Open access
- green
- dspace.library.uu.nl
- Licence
- Locations
- The Journal of Clinical Pharmacology (publishedVersion)
- PubMed (publishedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmacy and Medical Practices
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- ChatGPT, large language models, clinical pharmacy, multiple-choice questions, accuracy, reproducibility, concordance, pharmacist knowledge, AI-assisted learning
- MeSH terms
- Educational Measurement, Humans, Pharmacists, Pharmacy Service, Hospital, Surveys and Questionnaires, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
