Prescribing Caution: A Critique of OpenEvidence to Answer Medication‐Related Questions
Open accessAbstract
OpenEvidence is a newer medical large language model (LLM) that is growing in use among health professionals and students for clinical care and research. This paper provides an overview of existing literature on the accuracy of using OpenEvidence to answer clinical questions. Given the limited research exploring OpenEvidence's answers to clinical or medication-related questions from a pharmacist's perspective, this paper presents two illustrative examples that highlight inaccuracies of OpenEvidence's responses and source summarization errors related to pharmacotherapeutic content. The authors also offer suggestions for the responsible use of this tool, acknowledging that its adoption by pharmacists and health care professionals is likely to expand.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Specialized proprietary LLMs (e.g., OpenEvidence and UpToDate Expert AI) are designed to provide responses grounded in medical literature and are seeing increasing adoption in medical practice among health professionals and medical students.
“Also, specialized proprietary LLMs (e.g., OpenEvidence and UpToDate Expert AI) are designed to provide responses grounded in medical literature, 115 and are seeing increasing adoption in medical practice, 9 among health professionals and medical students.115,116”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [115]
A recent study published after this review's search date evaluated OpenEvidence on pharmacotherapy cases and highlighted dose-conversion errors and inconsistent advice on medication efficacy.
“For instance, a recent study published after this review's search date evaluated OpenEvidence on pharmacotherapy cases and highlighted dose-conversion errors and inconsistent advice on medication efficacy.115”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [115]
Cites
Perioperative Bridging Anticoagulation in Patients with Atrial Fibrillation
Douketis, J. D., Spyropoulos, A. C., Kaatz, S., Becker, R. C., Caprini, J. A., Dunn, A. S., Garcia, D. A., Jacobson, A., Jaffer, A. K., Kong, D. F., et al. (2015) · New England Journal of Medicine · doi:10.1056/nejmoa1501035 · Cited by 1,184 · see citing works
Hurt, R. T., Stephenson, C. R., Gilman, E. A., Aakre, C. A., Croghan, I. T., Mundi, M. S., Ghosh, K., & Edakkanambeth Varayil, J. (2025) · Journal of Primary Care & Community Health · doi:10.1177/21501319251332215 · Cited by 78 · cited 1× · see citing works
“…OpenEvidence in clinical settings and offer suggestions for responsible use of the tool. to different study methods, such as questions derived from clinical guidelines [6, 12, 13, 17, 18], or questions/scenarios based on clinician experiences among disparate subject areas [4–11, 13, 14, 16, 19–21].”
Low, Y. S., Jackson, M. L., Hyde, R. J., Brown, R. E., Sanghavi, N. M., Baldwin, J. D., Pike, C. W., Muralidharan, J., Hui, G., Alexander, N., et al. (2025) · DIGITAL HEALTH · doi:10.1177/20552076251348850 · Cited by 51 · cited 2× · see citing works
“…OpenEvidence in clinical settings and offer suggestions for responsible use of the tool. to different study methods, such as questions derived from clinical guidelines [6, 12, 13, 17, 18], or questions/scenarios based on clinician experiences among disparate subject areas [4–11, 13, 14, 16, 19–21].”
Flagged first, then by citations
Cited by
Sorted by citations
Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations
Yakob, R. A., Bamgboje-Ayodele, A., Collins, J. C., & Aslani, P. (2026) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2026.09.007
- Authors
- Bergsbaken, J. M., Wilson, P., Vandagriff, S., Christensen, L., & Vellardita, L.
- Affiliations
- University of Wisconsin–Madison (US)
- Venue
- JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY
- Publisher
- Wiley
- Published
- 2026-06-12
- Volume
- 9
- Issue
- 7
- Pages
- e70237-e70237
- ISSN
- 2574-9870 (ISSN-L)
- OpenAlex
- W7164674744
- Cited by
- 2
- References
- 3
- Citations by year
- 2026: 2
- FWCI
- 1.22
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- JACCP JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmacy and Medical Practices
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- OpenEvidence, large language models, medical question answering, AI response accuracy, responsible AI use, pharmacist perceptions, clinical decision support
- MeSH terms
- Large Language Models, Humans, Pharmacists
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
