Assessing the applicability and appropriateness of ChatGPT in answering clinical pharmacy questions
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
OBJECTIVES Clinical pharmacists rely on different scientific references to ensure appropriate, safe, and cost-effective drug use. Tools based on artificial intelligence (AI) such as ChatGPT (Generative Pre-trained Transformer) could offer valuable support. The objective of this study was to assess ChatGPT's capacity to correctly respond to clinical pharmacy questions asked by healthcare professionals in our university hospital. MATERIAL AND METHODS ChatGPT's capacity to respond correctly to the last 100 consecutive questions recorded in our clinical pharmacy database was assessed. Questions were copied from our FileMaker Pro database and pasted into ChatGPT March 14 version online platform. The generated answers were then copied verbatim into an Excel file. Two blinded clinical pharmacists reviewed all the questions and the answers given by the software. In case of disagreements, a third blinded pharmacist intervened to decide. RESULTS Documentation-related issues (n=36) and drug administration mode (n=30) were preponderantly recorded. Among 69 applicable questions, the rate of correct answers varied from 30% to 57.1% depending on questions type with a global rate of 44.9%. Regarding inappropriate answers (n=38), 20 were incorrect, 18 gave no answers and 8 were incomplete with 8 answers belonging to 2 different categories. No better answers than the pharmacists were observed. CONCLUSIONS ChatGPT demonstrated a mitigated performance in answering clinical pharmacy questions. It should not replace human expertise as a high rate of inappropriate answers was highlighted. Future studies should focus on the optimization of ChatGPT for specific clinical pharmacy questions and explore the potential benefits and limitations of integrating this technology into clinical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Eysenbach, G. (2023) · JMIR Medical Education · doi:10.2196/46885 · Cited by 840
Can artificial intelligence help for scientific writing?
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04380-2 · Cited by 827
Using ChatGPT to write patient clinic letters
Ali, S. R., Dobbs, T. D., Hutchings, H. A., & Whitaker, I. S. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00048-1 · Cited by 375
ChatGPT passing USMLE shines a spotlight on the flaws of medical education
Mbakwe, A. B., Lourentzou, I., Celi, L. A., Mechanic, O. J., & Dagan, A. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000205 · Cited by 273
The exciting potential for ChatGPT in obstetrics and gynecology
Grünebaum, A., Chervenak, J., Pollet, S. L., Katz, A., & Chervenak, F. A. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.03.009 · Cited by 234
Homolak, J. (2023) · Croatian Medical Journal · doi:10.3325/cmj.2023.64.1 · Cited by 208
ChatGPT takes on the European Exam in Core Cardiology: an artificial intelligence success story?
Skalidis, I., Cagnina, A., Luangphiphat, W., Mahendiran, T., Muller, O., Abbe, E., & Fournier, S. (2023) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztad029 · Cited by 128
Cited by
Morreel, S., Verhoeven, V., & Mathysen, D. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000349 · Cited by 37
Performance of ChatGPT on factual knowledge questions regarding clinical pharmacy
van Nuland, M., Erdogan, A., Aςar, C., Contrucci, R., Hilbrants, S., Maanach, L., Egberts, T., & van der Linden, P. D. (2024) · The Journal of Clinical Pharmacology · doi:10.1002/jcph.2443 · Cited by 27
van Nuland, M., Lobbezoo, A.-F. H., van de Garde, E. M. W., Herbrink, M., van Heijl, I., Bognàr, T., Houwen, J. P. A., Dekens, M., Wannet, D., Egberts, T., et al. (2024) · Exploratory Research in Clinical and Social Pharmacy · doi:10.1016/j.rcsop.2024.100464 · Cited by 17
van Nuland, M., Snoep, J. D., Egberts, T., Erdogan, A., Wassink, R., & van der Linden, P. D. (2024) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-024-03687-5 · Cited by 15
Chase, A., Most, A., Sikora, A., Smith, S. E., Devlin, J. W., Xu, S., Liu, T., & Murray, B. (2026) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxaf168 · Cited by 11
Metadata
- Authors
- Fournier, A., Fallet, C., Sadeghipour, F., & Perrottet, N.
- Venue
- Annales Pharmaceutiques Françaises
- Publisher
- Elsevier BV
- Published
- 2024-05
- Volume
- 82
- Issue
- 3
- Pages
- 507-513
- ISSN
- 0003-4509 (ISSN-L)
- OpenAlex
- W4388821733
- Cited by
- 28
- References
- 13
- FWCI
- 0.97
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessing the applicability and appropriateness of ChatGPT in answering clinical pharmacy questions. https://app.cochat.ai/papers/10.1016/j.pharma.2023.11.001
- The paper
- Fournier, A., Fallet, C., Sadeghipour, F., & Perrottet, N. (2024). Assessing the applicability and appropriateness of ChatGPT in answering clinical pharmacy questions. Annales Pharmaceutiques Françaises. https://doi.org/10.1016/j.pharma.2023.11.001
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.