The Potential of GPT-4 as a Support Tool for Pharmacists: Analytical Study Using the Japanese National Examination for Pharmacists
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Background: The advancement of artificial intelligence (AI), as well as machine learning, has led to its application in various industries, including health care. AI chatbots, such as GPT-4, developed by OpenAI, have demonstrated potential in supporting health care professionals by providing medical information, answering examination questions, and assisting in medical education. However, the applicability of GPT-4 in the field of pharmacy remains unexplored. Objective: This study aimed to evaluate GPT-4’s ability to answer questions from the Japanese National Examination for Pharmacists (JNEP) and assess its potential as a support tool for pharmacists in their daily practice. Methods: The question texts and answer choices from the 107th and 108th JNEP, held in February 2022 and February 2023, were input into GPT-4. As GPT-4 cannot process diagrams, questions that included diagram interpretation were not analyzed and were initially given a score of 0. The correct answer rates were calculated and compared with the passing criteria of each examination to evaluate GPT-4’s performance. Results: For the 107th and 108th JNEP, GPT-4 achieved an accuracy rate of 64.5% (222/344) and 62.9% (217/345), respectively, for all questions. When considering only the questions that GPT-4 could answer, the accuracy rates increased to 78.2% (222/284) and 75.3% (217/287), respectively. The accuracy rates tended to be lower for physics, chemistry, and calculation questions. Conclusions: Although GPT-4 demonstrated the potential to answer questions from the JNEP and support pharmacists’ capabilities, it also showed limitations in handling highly specialized questions, calculation questions, and questions requiring diagram recognition. Further evaluation is necessary to explore its applicability in real-world clinical settings, considering the complexities of patient scenarios and collaboration with health care professionals. By addressing these limitations, GPT-4 could become a more reliable tool for pharmacists in their daily practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 2×
“Furthermore, it has been reported that ChatGPT and GPT-4 could correctly answer questions on the United States Medical Licensing Examination (USMLE) [7,8] and the Japanese National Medical Licensing Examination [9] at a passing level, despite not having specialized in a particular field of study.”
BackgroundKung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1×
“Therefore, it is expected to be applied to health care education and to support physicians regarding diagnosis and treatment decisions [10-12].”
BackgroundScalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Conversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388 · cited 1×
“More recently, AI chatbots (also known as interactive AI) have been invented and are being used in medicine to automatically converse with human text and voice input [4-6].”
BackgroundArtificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review.
Aggarwal, A., Tam, C. C., Wu, D., Li, X., & Qiao, S. (2023) · Journal of medical Internet research · doi:10.2196/40789 · Cited by 683 · cited 1×
“More recently, AI chatbots (also known as interactive AI) have been invented and are being used in medicine to automatically converse with human text and voice input [4-6].”
BackgroundChatGPT - Reshaping medical education and clinical management
Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.39.2.7653 · Cited by 628
Cited by
Jin, H. K., Lee, H. E., & Kim, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05944-8 · Cited by 85
“Five of the included studies were conducted in the United States [11, 13, 14, 28, 37], one in the UK [36], six in Japan [33, 35, 39–41, 45], four in The performance of ChatGPT-3.”
BackgroundHigher education assessment practice in the era of generative AI tools
(2024) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2024.7.1.28 · Cited by 37
Large language models management of complex medication regimens: a case-based evaluation
Chase, A., Most, A., Xu, S., Barreto, E., Murray, B., Henry, K., Smith, S., Hedrick, T., Chen, X., Li, S., et al. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1514445 · Cited by 14
Chase, A., Most, A., Sikora, A., Smith, S. E., Devlin, J. W., Xu, S., Liu, T., & Murray, B. (2026) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxaf168 · Cited by 11
Mohammed, M., Khan, M. A., Chaudhary, V., Alsaleh, H., Fatima, S. B., & Pal, B. (2026) · Exploratory Research in Clinical and Social Pharmacy · doi:10.1016/j.rcsop.2026.100733 · Cited by 2
“3,9 This study specifically evaluates ChatGPT-4, which is the leading, freely available LLM, DeepSeek-V3, which represents an emerging high-performance model, and Drugs.com, a free database with the highest accuracy rate for detecting DDIs among free resources.”
Metadata
- Authors
- Kunitsu, Y.
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-10-30
- Volume
- 9
- Pages
- e48452
- ISSN
- 2369-3762 (ISSN-L)
- OpenAlex
- W4387645063
- Cited by
- 43
- References
- 14
- FWCI
- 1.45
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.