Performance validation of large language models for drug-drug interaction consultations: potential pharmacist assistant in outpatient pharmaceutical services
Abstract
Background The growing prevalence of polypharmacy and time constraints pose significant challenges for pharmacists in managing drug-drug interactions during outpatient consultations. While large language models (LLMs) show potential in various medical applications, their practical utility in real-world pharmaceutical services particularly in providing accurate, context-aware drug-drug interaction advice, remains underexplored.Objectives This study aims to evaluate the performance of LLMs in responding to real-world drug-drug interaction consultation queries. The goal is to assess their feasibility as assistive tools for pharmacists in delivering evidence-based, patient-specific medication guidance.Methods A total of 165 nonredundant drug-drug interaction queries were extracted from outpatient pharmacy consultations at Beijing Chao-Yang Hospital from January to May 2023. Responses from 4 LLMs were compared against answers provided by pharmacists. A scoring framework was applied to evaluate accuracy, depth of analysis, and practicality of administration advice.Results The LLMs demonstrated high consistency with pharmacists' responses, with rates of 84.85%-89.09%. Discrepancies arose primarily from incomplete pharmacist documentation, LLM drug misidentification, and differences in practical medication use advice. In scored evaluations, DeepSeek R1 achieved the highest score, achieving 3-4-point scores in 137 queries by integrating multi-perspective drug analysis and actionable administration strategies.Conclusion LLMs, particularly those with advanced reasoning and internet retrieval capabilities, have the potential to support pharmacists in drug-drug interaction management by delivering accurate, context-specific recommendations. However, limitations such as potential hallucination and operational delays remain. LLMs should serve as complementary tools under pharmacist supervision to enhance consultation quality and efficiency, not as replacements for professional judgment. Further validation using clinical metrics is warranted before widespread adoption.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prompt-session handling and parameter settings can influence the quality and consistency of LLM responses.
“as prompt-session handling and parameter settings can influence the quality and consistency of LLM responses.57,82,98”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [98]
Cites
Large language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417 · see citing works
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408 · see citing works
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 394 · see citing works
Ethical and regulatory challenges of large language models in medicine
Ong, J. C. L., Chang, S. Y.-H., William, W., Butte, A. J., Shah, N. H., Chew, L. S. T., Liu, N., Doshi-Velez, F., Lu, W., Savulescu, J., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00061-x · Cited by 342 · see citing works
Leveraging large language models for decision support in personalized oncology
Benary, M., Wang, X. D., Schmidt, M., Soll, D., Hilfenhaus, G., Nassir, M., Sigler, C., Knödler, M., Keller, U., Beule, D., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.43689 · Cited by 306 · see citing works
Kresevic, S., Giuffrè, M., Ajcevic, M., Accardo, A., Crocè, L. S., & Shung, D. L. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01091-y · Cited by 207 · see citing works
Ge, J., Sun, S., Owens, J., Galvez, V., Gologorskaya, O., Lai, J. C., Pletcher, M. J., & Lai, K. (2024) · Hepatology · doi:10.1097/hep.0000000000000834 · Cited by 182 · see citing works
Freyer, O., Wiest, I. C., Kather, J. N., & Gilbert, S. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00124-9 · Cited by 162 · see citing works
Large language models for preventing medication direction errors in online pharmacies
Pais, C., Liu, J., Voigt, R., Gupta, V., Wade, E., & Bayati, M. (2024) · Nature Medicine · doi:10.1038/s41591-024-02933-8 · Cited by 94 · see citing works
Language models for data extraction and risk of bias assessment in complementary medicine
Lai, H., Liu, J., Bai, C., Liu, H., Pan, B., Luo, X., Hou, L., Zhao, W., Xia, D., Tian, J., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01457-w · Cited by 59 · see citing works
Self-citationLafeber, M., Spiering, W., van der Graaf, Y., Nathoe, H. M., Bots, M. L., Grobbee, D. E., & Visseren, F. L. J. (2013) · American Heart Journal · doi:10.1016/j.ahj.2013.04.011 · Cited by 58 · see citing works
Wei, L., Fahey, T. P., & MacDonald, T. M. (2008) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.2008.03212.x · Cited by 52 · see citing works
Probabilistic medical predictions of large language models
Gu, B., Desai, R. J., Lin, K. J., & Yang, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01366-4 · Cited by 50 · see citing works
Vito, A. D., Geremia, N., Bavaro, D. F., Seo, S. K., Laracy, J. C., Mazzitelli, M., Marıno, A., Maraolo, A. E., Russo, A., Colpani, A., et al. (2025) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2025.03.002 · Cited by 49 · see citing works
Tian, Y., Yi, J., Wang, N., Wu, C., Peng, J., Shao, L., Yang, G., & Cao, D. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae726 · Cited by 27 · see citing works
Zhang, Y., Ren, S., Wang, J., Lu, J., Wu, C., He, M., Liu, X., Wu, R., Zhao, J., Zhan, C., et al. (2024) · Drugs · doi:10.1007/s40265-024-02124-2 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations
Yakob, R. A., Bamgboje-Ayodele, A., Collins, J. C., & Aslani, P. (2026) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2026.09.007
LLM Hallucinations in Clinical Documentation: Scoping Review with ☸SAIMSARA
SAIMSARA (2026) · SAIMSARA Journal · doi:10.62487/saimsarab885dc08
You, J. H. S., Lim, W.-L., Wong, A. C. L., Ng, M. Y. T., Yuen, J. S. H., Tai, B. B. W., Chow, K., & Lam, T. T. N. (2026) · Digital Health · doi:10.1177/20552076261492976
- Authors
- Zhang, H., Wang, A., He, C., Qiu, S., & An, Z.
- Affiliations
- Beijing Chao-Yang Hospital, Capital Medical University (CN)
- Corresponding author
- Zhang, Haoran
- Venue
- Journal of the American Pharmacists Association
- Publisher
- Elsevier BV
- Published
- 2026-05
- Volume
- 66
- Issue
- 3
- Pages
- 103077
- ISSN
- 1544-3191 (ISSN-L)
- OpenAlex
- W7139135991
- Cited by
- 3
- References
- 16
- Citations by year
- 2026: 3
- FWCI
- 5.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Pharmacists Association (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmacy and Medical Practices
- Keywords
- large language models, drug-drug interactions, pharmacist counseling, outpatient pharmacy services, polypharmacy, DeepSeek-R1, hallucinations
- MeSH terms
- Large Language Models, Ambulatory Care, Drug Interactions, Humans, Outpatients, Pharmaceutical Services, Pharmacists, Pharmacy Service, Hospital, Referral and Consultation, Professional Role
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
