Evaluating the capability of ChatGPT in predicting drug-drug interactions: Real-world evidence using hospitalized patient data
Open accessAbstract
Drug–drug interactions (DDIs) present a significant health burden, compounded by clinician time constraints and poor patient health literacy. We assessed the ability of ChatGPT (generative artificial intelligence‐based large language model) to predict DDIs in a real‐world setting. Demographics, diagnoses and prescribed medicines for 120 hospitalized patients were input through three standardized prompts to ChatGPT version 3.5 and compared against pharmacist DDI evaluation to estimate diagnostic accuracy. Area under receiver operating characteristic and inter‐rater reliability (Cohen's and Fleiss' kappa coefficients) were calculated. ChatGPT's responses differed based on prompt wording style, with higher sensitivity for prompts mentioning ‘drug interaction’. Confusion matrices displayed low true positive and high true negative rates, and there was minimal agreement between ChatGPT and pharmacists (Cohen's kappa values 0.077–0.143). Low sensitivity values suggest a lack of success in identifying DDIs by ChatGPT, and further development is required before it can reliably assess potential DDIs in real‐world scenarios.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large language models such as ChatGPT, Gemini and DeepSeek could rapidly analyze medication lists and cross-reference known interactions, providing comprehensive and standardized DDI screening at high speed and potentially streamlining preoperative prescribing workflows.
“Large language models (LLMs) such as Chat-GPT, Gemini and specialized systems like DeepSeek could rapidly analyze medication lists and cross-reference known interactions, providing comprehensive and standardized DDI screening at high speed and potentially streamlining preoperative prescribing workflows [29][30][31][32].”
From Comparing Artificial Intelligence (ChatGPT, Gemini, DeepSeek) and Oral Surgeons in Detecting Clinically Relevant Drug–Drug Interactions in Dental Therapy (2025) · cites this paper as [31]
Split into 2 atoms — 0 confirmed, 1 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 25 · Cited by 29
Research works printed but not linked: 21
- [1]Snyder BD, Polasek TM, Doogue MP. Drug interactions: principles and practice. Aust Prescr. 2012; 35(3): 85‐ 88. doi: 10.18773/austprescr.2012.037 · DOI
- [2]Ajayi FO, Sun H, Perry J. Adverse drug reactions: a review of relevant factors. J Clin Pharmacol. 2000; 40(10): 1093‐ 1101. doi: 10.1177/009127000004001003 11028248 · DOI
- [3]Delara M, Murray L, Jafari B, et al. Prevalence and factors associated with polypharmacy: a systematic review and meta‐analysis. BMC Geriatr. 2022; 22(1): 601. doi: 10.1186/s12877-022-03279-x 35854209 PMC9297624 · DOI
- [5]Pazan F, Wehling M. Polypharmacy in older adults: a narrative review of definitions, epidemiology and consequences. Eur Geriatr Med. 2021; 12(3): 443‐ 452. doi: 10.1007/s41999-021-00479-3 33694123 PMC8149355 · DOI · Paper page
- [6]Wylie CE, Daniels B, Brett J, Pearson SA, Buckley NA. A national study on prescribed medicine use in Australia on a typical day. Pharmacoepidemiol Drug Saf. 2020; 29(9): 1046‐ 1053. doi: 10.1002/pds.5093 32779806 · DOI
- [7]Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011; 155(2): 97‐ 107. doi: 10.7326/0003-4819-155-2-201107190-00005 21768583 · DOI · Paper page
- [9]Baysari MT, Zheng WY, Li L, et al. Optimising computerised decision support to transform medication safety and reduce prescriber burden: study protocol for a mixed‐methods evaluation of drug–drug interaction alerts. BMJ Open. 2019; 9(8): e026034. doi: 10.1136/bmjopen-2018-026034 PMC6701635 31427312 · DOI
- [10]Beeler PE, Bates DW, Hug BL. Clinical decision support systems. Swiss Med Wkly. 2014; 144(5152): w14073. doi: 10.4414/smw.2014.14073 25668157 · DOI
- [11]Ayre J, Mac O, McCaffery K, et al. New frontiers in health literacy: using ChatGPT to simplify health information for people in the community. J Gen Intern Med. 2024; 39(4): 573‐ 577. doi: 10.1007/s11606-023-08469-w 37940756 PMC10973278 · DOI · Paper page
- [12]Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell. 2023; 6: 1169595. doi: 10.3389/frai.2023.1169595 37215063 PMC10192861 · DOI · Paper page
- [15]Huang X, Estau D, Liu X, Yu Y, Qin J, Li Z. Evaluating the performance of ChatGPT in clinical pharmacy: a comparative study of ChatGPT and clinical pharmacists. Br J Clin Pharmacol. 2024; 90(1): 232‐ 238. doi: 10.1111/bcp.15896 37626010 · DOI
- [16]Johnson D, Goodman R, Patrinely J, et al. Assessing the accuracy and reliability of AI‐generated medical responses: an evaluation of the Chat‐GPT model. Res Sq. 2023; rs.3.rs‐2566942. doi: 10.21203/rs.3.rs-2566942/v1 · DOI · Paper page
- [18]Al‐Ashwal FY, Zawiah M, Gharaibeh L, Abu‐Farha R, Bitar AN. Evaluating the sensitivity, specificity, and accuracy of ChatGPT‐3.5, ChatGPT‐4, Bing AI, and Bard against conventional drug‐drug interactions clinical tools. Drug Healthc Patient Saf. 2023; 15: 137‐ 147. doi: 10.2147/DHPS.S425858 37750052 PMC10518176 · DOI · Paper page
- [19]Juhi A, Pipil N, Santra S, Mondal S, Behera JK, Mondal H. The capability of ChatGPT in predicting and explaining common drug‐drug interactions. Cureus. 2023; 15(3): e36272. doi: 10.7759/cureus.36272 37073184 PMC10105894 · DOI
- [20]Von Elm E, Altman D, Egger M, Pocock S, Gøtzsche P, Vandenbroucke J. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008; 61(4): 344‐ 349. doi: 10.1016/j.jclinepi.2007.11.008 18313558 · DOI · Paper page
- [22]He N, Yan Y, Wu Z, et al. Chat GPT‐4 significantly surpasses GPT‐3.5 in drug information queries. J Telemed Telecare. 2023; 1357633X231181922. doi: 10.1177/1357633X231181922 37350055 · DOI
- [23]McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012; 22(3): 276‐ 282. doi: 10.11613/BM.2012.031 23092060 PMC3900052 · DOI · Paper page
- [24]Montastruc F, Storck W, de Canecaude C, et al. Will artificial intelligence chatbots replace clinical pharmacologists? An exploratory study in clinical practice. Eur J Clin Pharmacol. 2023; 79(10): 1375‐ 1384. doi: 10.1007/s00228-023-03547-8 37566133 · DOI
- [25]Jang ME, Lukasiewicz T. Consistency Analysis of ChatGPT. Published online November 13, 2023. Accessed February 27, 2024. http://arxiv.org/abs/2303.06273 · arXiv
- [26]Wang H, Ding YJ, Luo Y. Future of ChatGPT in pharmacovigilance. Drug Saf. 2023; 46(8): 711‐ 713. doi: 10.1007/s40264-023-01315-2 37306853 PMC11168724 · DOI
- [27]Mehrabi N, Morstatter F, Saxena N, Lerman K, Galstyan A. A survey on bias and fairness in machine learning. ACM Comput Surv. 2022; 54(6): 1‐ 35. doi: 10.1145/3457607 · DOI · Paper page
- Authors
- Radha Krishnan, R. P., Hung, E. H., Ashford, M., Edillo, C. E., Gardner, C., Hatrick, H. B., Kim, B., Lai, A. W. Y., Li, X., Zhao, Y. X., & Raubenheimer, J. E.
- Affiliations
- Faculty of Medicine and Health University of Sydney Sydney New South Wales Australia
- The University of Sydney (AU)
- Venue
- British Journal of Clinical Pharmacology
- Publisher
- Wiley
- Published
- 2024-10-02
- Volume
- 90
- Issue
- 12
- Pages
- 3361-3366
- ISSN
- 0306-5251 (ISSN-L)
- 1365-2125
- PMC
- PMC11602951
- OpenAlex
- W4403109979
- Cited by
- 29
- References
- 25
- Citations by year
- 2026: 12 · 2025: 17
- FWCI
- 2.71
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- British Journal of Clinical Pharmacology (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- University of Sydney
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- drug-drug interactions, ChatGPT, large language models, generative AI, hospitalized patients, diagnostic accuracy, inter-rater reliability, real-world evidence
- MeSH terms
- Adult, Aged, Aged, 80 and over, Artificial Intelligence, Drug Interactions, Female, Hospitalization, Humans, Male, Middle Aged, Pharmacists, ROC Curve, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
