Improving medication error classification using a reasoning large language model
Open accessAbstract
Objectives: To assess the performance of a reasoning large language model (LLM) in identifying medication errors in medical incident reports. Materials and Methods: OpenAI's O4-mini LLM was adapted using prompt engineering on 75 000 anonymized incident reports from the Västmanland region of Sweden (2019-2024). To guide the prompt design, we used a subset of 2434 reports, which were manually reclassified by pharmacists as medication-related or not. For validation, 200 reports (January 2024-March 2024) were independently classified by 2 pharmacists to establish a reference classification. Moreover, the LLM performed binary classification, with concordance rates measured against the expert consensus. Results: The LLM achieved a concordance rate of 96.0% (192/200; 95% CI, 92.3-98.3) with expert classification. Eight cases (4.0%) showed disagreements, primarily due to linguistic ambiguity or context-dependent interpretation. Five cases involved pharmacists classifying reports as non-medication-related, while the LLM classified them as medication-related, with the reverse in 3 cases. Subcategorization accuracy was 76.5%. Discussion: The LLM showed expert-level performance, outperforming existing automated methods. Thus, its integration into incident reporting systems might improve the efficiency, accuracy, and consistency of patient safety monitoring. Conclusion: This validated AI-driven method can be integrated directly into clinical informatics workflows, enabling healthcare organizations to rapidly and consistently identify medication errors, ultimately enhancing patient safety outcomes.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Two studies reported improved GPT-4 accuracy when answers did not require direct access to patient data.
“For factual knowledge questions or inquiries, two studies reported improved GPT-4 accuracy when answers did not require direct access to patient data.30,90”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [90]
The Methods describe O4-mini classifying anonymized medication-incident reports, and the Results report 96% concordance with pharmacist classifications. Searches for GPT-4, factual knowledge questions, and accuracy without direct access to patient data found no passage addressing the claimed comparison. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Panagioti, M., Khan, K., Keers, R. N., Abuzour, A., Phipps, D., Kontopantelis, E., Bower, P., Campbell, S., Haneef, R., Avery, A. J., et al. (2019) · BMJ · doi:10.1136/bmj.l4185 · Cited by 874 · see citing works
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · cited 1× · see citing works
“Recent research has shown that self-consistency approaches—wherein multiple reasoning paths are generated and aggregated—can significantly improve response quality (10).”
Keers, R. N., Williams, S. D., Cooke, J., & Ashcroft, D. M. (2013) · Drug Safety · doi:10.1007/s40264-013-0090-2 · Cited by 516 · see citing works
Toward a responsible future: recommendations for AI-enabled clinical decision support
Labkoff, S., Oladimeji, B., Kannry, J., Solomonides, A., Leftwich, R., Koski, E., Joseph, A. L., Lopez-Gonzalez, M., Fleisher, L. A., Nolen, K., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae209 · Cited by 252 · see citing works
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 219 · see citing works
How are medication errors defined? A systematic literature review of definitions and characteristics
Lisby, M., Nielsen, L. P., Brock, B., & Mainz, J. (2010) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzq059 · Cited by 170 · see citing works
Westbrook, J. I., Li, L., Lehnbom, E. C., Baysari, M. T., Braithwaite, J., Burke, R., Conn, C., & Day, R. O. (2015) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzu098 · Cited by 155 · see citing works
Evaluation of drug administration errors in a teaching hospital
Berdot, S., Sabatier, B., Gillaizeau, F., Caruba, T., Prognon, P., & Durieux, P. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-60 · Cited by 151 · see citing works
Systematic review of predictive risk models for adverse drug events in hospitalized patients
Falconer, N., Barras, M. A., & Cottrell, N. (2018) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.13514 · Cited by 78 · see citing works
Fong, A., Harriott, N., Walters, D. M., Foley, H., Morrissey, R., & Ratwani, R. M. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.05.005 · Cited by 53 · see citing works
Louët, H. L., & Pitts, P. J. (2022) · Therapeutic Innovation & Regulatory Science · doi:10.1007/s43441-022-00443-8 · Cited by 32 · see citing works
Practical Applications of Large Language Models for Health Care Professionals and Scientists
Reis, F., Lenz, C., Gossen, M., Volk, H., & Drzeniek, N. M. (2024) · JMIR Medical Informatics · doi:10.2196/58478 · Cited by 20 · see citing works
A text mining approach to categorize patient safety event reports by medication error type
Boxley, C. L., Fujimoto, M., Ratwani, R. M., & Fong, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-45152-w · Cited by 19 · see citing works
Fong, A., Bonk, C., Vasilchenko, V., De, S., Kovich, D., & Wyeth, J. (2022) · Frontiers in Drug Safety and Regulation · doi:10.3389/fdsfr.2022.1021068 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [14]Cohan A, Fong A, Goharian N, Ratwani R. A neural attention model for categorizing patient safety events. In: Jose J, et al., eds. Advances in Information Retrieval. Vol 10193. Cham: Springer International Publishing; 2017.
Cited by
Sorted by citations
Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations
Yakob, R. A., Bamgboje-Ayodele, A., Collins, J. C., & Aslani, P. (2026) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2026.09.007
AI Medical Device Safety: Scoping Review with ☸SAIMSARA
SAIMSARA (2026) · SAIMSARA Journal · doi:10.62487/saimsaraa33bf709
Hsu, H., Gazetta, G., Lu, C., & Jacobs, D. M. (2026) · Journal of Managed Care & Specialty Pharmacy · doi:10.18553/jmcp.2026.32.9.1052
Sirago, G., Sabato, S., Sebastiani, N., Zotti, F., Dell'Erba, A., & Ferorelli, D. (2026) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/j.jcjq.2026.09.002
Gobran, Y. (2026) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2026.09.009
- Authors
- Krifors, A., Beskow, T., Jonsson, M., Lindner, K.-J., Calås, J., Arwsbo, V., Sandström, K., & Norström, C.
- Affiliations
- Uppsala University (SE)
- Studi (SE)
- Region Västmanland (SE)
- Mälardalen University (SE)
- Corresponding author
- Krifors, Anders
- Venue
- JAMIA Open
- Publisher
- Oxford University Press (OUP)
- Published
- 2026-01-06
- Volume
- 9
- Issue
- 1
- Pages
- ooag004-ooag004
- ISSN
- 2574-2531 (ISSN-L)
- PMC
- PMC12832951
- OpenAlex
- W7125696833
- Cited by
- 5
- References
- 12
- Citations by year
- 2026: 5
- FWCI
- 47.62
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- JAMIA Open (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Diva portal (Dalarna University Library) (submittedVersion, other-oa)
- Funders
- Region Västmanland and Mälardalen University
- Topics
- Health Sciences › Health Professions › Emergency Medical Services › Patient Safety and Medication Errors
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Toxicology › Pharmacovigilance and Adverse Drug Reactions
- Keywords
- medication errors, large language models, prompt engineering, patient safety monitoring, incident reporting systems, pharmacists
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
