Accuracy and teachability of artificial intelligence chatbots in solving pharmaceutical calculations: a descriptive study

Abstract

Background Pharmaceutical calculations are required elements of the Doctor of Pharmacy curriculum in the United States. With the growth of artificial intelligence chatbots, pharmacists and educators are exploring their application. The accuracy of artificial intelligence chatbots in performing pharmaceutical calculations remains unknown.Aim To evaluate the accuracy of artificial intelligence chatbots for pharmaceutical calculations.Method Eleven free-access chatbots were tested using 7 faculty-generated questions: 1 control, 2 creatinine clearance, 1 oral to intravenous dose conversion, 2 antibiotic pharmacokinetic dosing, and 1 number needed to harm. Descriptive statistics were used to evaluate the primary outcome, which was proportion of correct responses. Secondary outcomes included types of errors and teachability.Results Ten (90.9%) chatbots answered the control question correctly, and all answered the dose conversion question correctly. Eight (72.7%) chatbots correctly calculated number needed to harm. Only 1 (9.1%) provided the correct antibiotic dosing, and none correctly calculated creatinine clearance. Common errors included incorrect weight selection for creatinine clearance and use of incorrect formulas. Nine (81.8%) chatbots were teachable on at least 1 question.Conclusion Artificial intelligence chatbots demonstrated limited accuracy for multi-step pharmaceutical calculations and may be more reliable for low complexity calculations.