Evaluating the sensitivity, specificity, and accuracy of ChatGPT-3.5, ChatGPT-4, bing AI, and bard against conventional drug-drug interactions clinical tools
Open accessAbstract
Background: AI platforms are equipped with advanced algorithms that have the potential to offer a wide range of applications in healthcare services. However, information about the accuracy of AI chatbots against conventional drug-drug interaction tools is limited. This study aimed to assess the sensitivity, specificity, and accuracy of ChatGPT-3.5, ChatGPT-4, Bing AI, and Bard in predicting drug-drug interactions. Methods: AI-based chatbots (ie, ChatGPT-3.5, ChatGPT-4, Microsoft Bing AI, and Google Bard) were compared for their abilities to detect clinically relevant DDIs for 255 drug pairs. Descriptive statistics, such as specificity, sensitivity, accuracy, negative predictive value (NPV), and positive predictive value (PPV), were calculated for each tool. Results: When a subscription tool was used as a reference, the specificity ranged from a low of 0.372 (ChatGPT-3.5) to a high of 0.769 (Microsoft Bing AI). Also, Microsoft Bing AI had the highest performance with an accuracy score of 0.788, with ChatGPT-3.5 having the lowest accuracy rate of 0.469. There was an overall improvement in performance for all the programs when the reference tool switched to a free DDI source, but still, ChatGPT-3.5 had the lowest specificity (0.392) and accuracy (0.525), and Microsoft Bing AI demonstrated the highest specificity (0.892) and accuracy (0.890). When assessing the consistency of accuracy across two different drug classes, ChatGPT-3.5 and ChatGPT-4 showed the highest variability in accuracy. In addition, ChatGPT-3.5, ChatGPT-4, and Bard exhibited the highest fluctuations in specificity when analyzing two medications belonging to the same drug class. Conclusion: Bing AI had the highest accuracy and specificity, outperforming Google's Bard, ChatGPT-3.5, and ChatGPT-4. The findings highlight the significant potential these AI tools hold in transforming patient care. While the current AI platforms evaluated are not without limitations, their ability to quickly analyze potentially significant interactions with good sensitivity suggests a promising step towards improved patient safety.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Although ChatGPT was the most commonly evaluated LLM, some studies did not find a performance advantage over other LLMs.
“Although ChatGPT was the most commonly evaluated LLM, some studies did not find a performance advantage over other LLMs.31,83”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [31]
The Discussion reports that ChatGPT models performed worse than the other AI tools on most evaluated metrics, supporting the claim that ChatGPT did not have a performance advantage in this study. The paper does not establish that ChatGPT was the most commonly evaluated LLM across studies. (LLM, conf=0.84)
Experimental — accuracy not yet measuredChatGPT has shown high sensitivity, correctly identifying approximately 99% of DDIs in a recent evaluation.
“Indeed, ChatGPT has shown high sensitivity, correctly identifying approximately 99% of DDIs in a recent evaluation [33].”
From Comparing Artificial Intelligence (ChatGPT, Gemini, DeepSeek) and Oral Surgeons in Detecting Clinically Relevant Drug–Drug Interactions in Dental Therapy (2025) · cites this paper as [33]
The Results tables report relatively high sensitivity for ChatGPT-3.5 and ChatGPT-4, but not approximately 99%. Table 3 gives 89.4%–91.5% against Micromedex, while Table 4 gives 74.7%–82.3% against Drugs.com; neither supports the claimed figure. Page numbers are unavailable in our copy. (LLM, conf=0.91)
Experimental — accuracy not yet measuredIn comparing drug-drug interaction prediction, Bing AI had the highest accuracy (0.788), followed by Bard (0.613), ChatGPT-4 (0.592), and ChatGPT-3.5 (0.469).
“When comparing different LLMs' ability to predict drug-drug interactions, one study found that Bing AI had the highest accuracy (0.788), followed by Bard (0.613), ChatGPT-4 (0.592), and ChatGPT-3.5 (0.469).31”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 28 · Cited by 125
Research works printed but not linked: 1
- [9]Liu L, Duffy VG. Exploring the future development of Artificial Intelligence (AI) applications in chatbots: a bibliometric analysis. Int J Soc Robot. 2023; 2023: 1– 14.
- Authors
- Al-Ashwal, F. Y., Zawiah, M., Gharaibeh, L., Abu-Farha, R., & Bitar, A. N.
- Affiliations
- Sana'a University (YE)
- Hodeidah University (YE)
- Al-Ahliyya Amman University (JO)
- Corresponding author
- Al-Ashwal, Fahmi Y
- Venue
- Drug Healthcare and Patient Safety
- Publisher
- Informa UK Limited
- Published
- 2023-09
- Volume
- Volume 15
- Pages
- 137-147
- ISSN
- 1179-1365 (ISSN-L)
- PMC
- PMC10518176
- OpenAlex
- W4386894187
- Cited by
- 125
- References
- 28
- Citations by year
- 2026: 27 · 2025: 51 · 2024: 40 · 2023: 7
- FWCI
- 4.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Drug Healthcare and Patient Safety (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Dove Medical Press (Taylor and Francis Group) (submittedVersion, cc-by-nc)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- drug-drug interactions, large language models, ChatGPT-3.5, ChatGPT-4, Google Bard, AI chatbots, diagnostic accuracy, sensitivity and specificity, clinical decision support systems, patient safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
