Analysis of overridden alerts in a drug-drug interaction detection system
Abstract
OBJECTIVE: The aim of this study was to evaluate the relevance of the signals generated by a computerized drug-drug interaction detection system and to design a classification of overridden drug-drug interaction alerts. STUDY DESIGN: Prospective study over two months. SETTING: Five hundred and ten-bed university paediatric hospital. MAIN OUTCOME MEASURES: In Robert Debré Hospital physicians generate drug orders online using a computerized physician order entry system that also detects drug-drug interactions in real time. We analysed the relevance of a sample of alerts overridden by physicians. RESULTS: We analysed a sample of 613 overridden alerts. We defined three categories of overridden alerts: informational errors (35); system errors (244) and accurate alerts (334). Two reasons account for 40% of false-positive alerts: an inability of the system to recognize real conflicts between drug treatments and guidelines stating that the two drugs can be used together, because the benefit outweighs the risk of side effects due to the drug-drug interaction. CONCLUSIONS: We created a classification of overridden alerts, in the context of computerized physician order entry system coupled with a drug-drug interaction detection system. There is clearly room for improvement in the development of drug-drug interaction software. This classification should make it possible to break this work down into smaller tasks, making it possible to decrease the sensitivity to background noise of drug-drug interaction detection systems.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Similarly to the results of Mille et al., the lack of DDI-specific screening intervals and lack of incorporation of context factors were identified, in both our quantitative evaluation and end-user survey, as the main barriers of our current DDI CDSS system.
“Similarly to the results of Mille et al. [30], the lack of DDI-specific screening intervals and lack of incorporation of context factors were identified, in both our quantitative evaluation and end-user survey, as the main barriers of our current DDI CDSS system.”
From Overall performance of a drug–drug interaction clinical decision support system: quantitative evaluation and end-user survey (2022) · cites this paper as [30]
Our study results are in agreement with previous reports.
“Our study results are in agreement with previous reports.”
From Overall performance of a drug–drug interaction clinical decision support system: quantitative evaluation and end-user survey (2022) · cites this paper as [30]
High override rates of 56.3%-95.7% have already been described for DDI alerts.
“High override rates of 56.3%-95.7% have already been described for DDI alerts [13, 21, 27–32].”
From Overall performance of a drug–drug interaction clinical decision support system: quantitative evaluation and end-user survey (2022) · cites this paper as [30]
Cites
Institute of Medicine (2000) · National Academies Press eBooks · doi:10.17226/9728 · Cited by 14,174 · see citing works
Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors
Koppel, R. (2005) · JAMA · doi:10.1001/jama.293.10.1197 · Cited by 2,126 · see citing works
Bates, D. W. (1998) · JAMA · doi:10.1001/jama.280.15.1311 · Cited by 2,107 · see citing works
The Impact of Computerized Physician Order Entry on Medication Error Prevention
Bates, D. W., Teich, J. M., Lee, J., Seger, D., Kuperman, G. J., Ma'Luf, N., Boyle, D., & Leape, L. (1999) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1999.00660313 · Cited by 1,260 · see citing works
Overriding of Drug Safety Alerts in Computerized Physician Order Entry
van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1809 · Cited by 1,117 · see citing works
Physicians' Decisions to Override Computerized Drug Alerts in Primary Care
Weingart, S. N., Toth, M., Sands, D. Z., Aronson, M. D., Davis, R. B., & Phillips, R. S. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.21.2625 · Cited by 487 · see citing works
Effects of Computerized Physician Order Entry on Prescribing Practices
Teich, J. M., Merchia, P., Schmiz, J. L., Kuperman, G. J., Spurr, C. D., & Bates, D. W. (2000) · Archives of Internal Medicine · doi:10.1001/archinte.160.18.2741 · Cited by 475 · see citing works
Improving Recognition of Drug Interactions
Glassman, P. A., Simon, B. F., Belperio, P. S., & Lanto, A. B. (2002) · Medical Care · doi:10.1097/00005650-200212000-00004 · Cited by 279 · see citing works
Exploring Barriers and Facilitators to the Use of Computerized Clinical Reminders
Saleem, J. J., Patterson, E. S., Militello, L. G., Render, M. L., Orshansky, G., & Asch, S. M. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1777 · Cited by 250 · see citing works
Hsieh, T. C., Kuperman, G. J., Jaggi, T., Hojnowski-Diaz, P., Fiskio, J., Williams, D. H., Bates, D. W., & Gandhi, T. K. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1556 · Cited by 232 · see citing works
Comprehensive Analysis of a Medication Dosing Error Related to CPOE: Table 1
Horsky, J., Kuperman, G. J., & Patel, V. L. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1740 · Cited by 197 · see citing works
Impact of emerging technologies on medication errors and adverse drug events
Oren, E., Shaffer, E. R., & Guglielmo, B. J. (2003) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/60.14.1447 · Cited by 190 · see citing works
GPs' views on computerized drug interaction alerts: questionnaire survey
Magnus, D., Rodgers, S., & Avery, A. J. (2002) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1046/j.1365-2710.2002.00434.x · Cited by 157 · see citing works
To err is not entirely human: Complex technology and user cognition
Horsky, J., Zhang, J., & Patel, V. L. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.05.002 · Cited by 138 · see citing works
Characteristics and override rates of order checks in a practitioner order entry system.
Payne, T. H., Nichol, W. P., Hoey, P., & Savarino, J. E. (2002) · PubMed Central · Cited by 129 · see citing works
Improving allergy alerting in a computerized physician order entry system.
Abookire, S. A., Teich, J. M., Sandige, H. L., Paterno, M. D., Martin, M. T., Kuperman, G. J., & Bates, D. W. (2000) · PubMed Central · Cited by 120 · see citing works
Ahearn, M. D., & Kerr, S. J. (2003) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.2003.tb05415.x · Cited by 89 · see citing works
Computerized physician order entry and medication errors: Finding a balance
Bates, D. W. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.05.003 · Cited by 59 · see citing works
Computerized Physician Order Entry and Online Decision Support
Handler, J. A., Feied, C. F., Coonan, K., Vozenilek, J., Gillam, M., Peacock, P. R., Sinert, R., & Smith, M. S. (2004) · Academic Emergency Medicine · doi:10.1197/j.aem.2004.08.007 · Cited by 58 · cited 2× · see citing works
“Alerts with a lower level of clinical impact could be shown in a less intrusive manner [12].”
BackgroundExposure to Automated Drug Alerts Over Time
Glassman, P. A., Belperio, P. S., Simon, B. F., Lanto, A. B., & Lee, M. (2006) · Medical Care · doi:10.1097/01.mlr.0000199849.08389.91 · Cited by 55 · see citing works
Bergk, V., Gasse, C., Schnell, R., & Haefeli, W. E. (2004) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-004-0812-6 · Cited by 41 · see citing works
CPOE and the facilitation of medication errors
Shortliffe, E. H. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.05.007 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Clinical Decision Support Systems
Helmons, P. J. (2009) · CRC Press eBooks · doi:10.1201/9781420071764-c15 · Cited by 295 · see citing works
Vilar, S., Friedman, C., & Hripcsak, G. M. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx010 · Cited by 155 · see citing works
Appropriateness of Overridden Alerts in Computerized Physician Order Entry: Systematic Review
Poly, T. N., Islam, M. M., Yang, H.-C., & Li, Y.-C. (. (2020) · JMIR Medical Informatics · doi:10.2196/15653 · Cited by 118 · see citing works
Design of decision support interventions for medication prescribing
Horsky, J., Phansalkar, S., Desai, A. A., Bell, D. S., & Blackford Middleton, A. (2013) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2013.02.003 · Cited by 108 · see citing works
Identifying Adverse Drug Reactions Associated with Drug-Drug Interactions
Leone, R., Magro, L., Moretti, U., Cutroneo, P. M., Moschini, M., Motola, D., Tuccori, M., & Conforti, A. (2010) · Drug Safety · doi:10.2165/11534400-000000000-00000 · Cited by 96 · see citing works
Improving medication-related clinical decision support
Tolley, C. L., Slight, S. P., Husband, A. K., Watson, N., & Bates, D. W. (2018) · American Journal of Health-System Pharmacy · doi:10.2146/ajhp160830 · Cited by 105 · see citing works
Santos, T. N. G. D. A., Macieira, G. M. D. C., Alves, B. M. C. S., Onozato, T., Cardoso, G. C., Nascimento, M. T. F., Martins‐Filho, P. R., de Lyra, D. P., & de Oliveira‐Filho, A. D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235353 · Cited by 55 · see citing works
Van De Sijpe, G., Quintens, C., Walgraeve, K., Van Laer, E., Penny, J., De Vlieger, G., Schrijvers, R., De Munter, P., Foulon, V., Casteels, M., et al. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01783-z · Cited by 54 · see citing works
“[30], the lack of DDI-specific screening intervals and lack of incorporation of context factors were identified, in both our quantitative evaluation and end-user survey, as the main barriers of our current DDI CDSS system.”
BackgroundTolley, C. L., Forde, N. E., Coffey, K. L., Sittig, D. F., Ash, J. S., Husband, A., Bates, D. W., & Slight, S. P. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx124 · Cited by 51 · see citing works
Phansalkar, S., Desai, A. A., Choksi, A. R., Yoshida, E., Doole, J., Czochanski, M., Tucker, A. D., Blackford Middleton, A., Bell, D. S., & Bates, D. W. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-65 · Cited by 47 · see citing works
Drug Interactions in Dying Patients
Frechen, S., Zoeller, A., Ruberg, K., Voltz, R., & Gaertner, J. (2012) · Drug Safety · doi:10.1007/bf03261971 · Cited by 43 · see citing works
Enabling Medication Management through Health Information Technology
(2011) · SciVee · doi:10.4016/37397.01 · Cited by 35 · see citing works
Overriding Drug-Drug Interaction Alerts in Clinical Decision Support Systems: A Scoping Review
Zapata, L. V., Subbian, V., Boyce, R. D., Hansten, P. D., Horn, J. R., Gephart, S. M., Romero, A. V., & Malone, D. C. (2022) · Studies in health technology and informatics · doi:10.3233/shti220101 · Cited by 28 · see citing works
Ahn, E. K., Cho, S.-Y., Shin, D., Jang, C., & Park, R. W. (2014) · Healthcare Informatics Research · doi:10.4258/hir.2014.20.4.280 · Cited by 26 · see citing works
Mannheimer, B., Wettermark, B., Lundberg, M., Pettersson, H., von Bahr, C., & Eliasson, E. (2009) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.2009.03598.x · Cited by 24 · see citing works
A meta‐analysis assessing the prevalence of drug–drug interactions among hospitalized patients
Aksoy, N., & Ozturk, N. (2023) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.5691 · Cited by 21 · see citing works
Differences among admitting departments in alerts and alert overrides for drug–drug interaction
Ahn, E. K., Kam, H. J., Park, D. K., Jung, E. Y., Lee, Y., & Park, R. W. (2014) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.3591 · Cited by 21 · see citing works
Taegtmeyer, A. B., Curkovic, I., Corti, N., Rosen, C., Egbring, M., Russmann, S., Gantenbein, A. R., Weller, M., & Kullak-Ublick, G. (2012) · Swiss Medical Weekly · doi:10.4414/smw.2012.13615 · Cited by 19 · see citing works
Taegtmeyer, A. B., Curkovic, I., Rufibach, K., Corti, N., Battegay, E., & Kullak-Ublick, G. A. (2011) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.2011.04032.x · Cited by 16 · see citing works
Physicians’ responses to computerized drug interaction alerts with password overrides
Nasuhara, Y., Sakushima, K., Endoh, A., Umeki, R., Oki, H., Yamada, T., Iseki, K., & Ishikawa, M. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0194-y · Cited by 15 · see citing works
Lau, A. Y. S., Tsafnat, G., Sintchenko, V., Magrabi, F., & Coiera, E. W. (2009) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638644 · Cited by 14 · see citing works
Validierung von Interaktionsdatenbanken in der Psychopharmakotherapie
Hahn, M., & Roll, S. (2017) · Der Nervenarzt · doi:10.1007/s00115-017-0385-0 · Cited by 11 · see citing works
Karajizadeh, M., Zand, F., Vazin, A., Saeidnia, H. R., Lund, B. D., Tummuru, S. P., & Sharifian, R. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105135 · Cited by 10 · see citing works
A scoping review of rule-based clinical decision support malfunctions
Thayer, J. G., Franklin, A., Miller, J. M., Grundmeier, R. W., Rogith, D., & Wright, A. T. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae187 · Cited by 9 · see citing works
Lee, S.-M., Lee, S.-O., & Kim, D. (2017) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzx085 · Cited by 9 · see citing works
An evaluation of the completeness of drug-drug interaction-related information in package inserts
Ng, G. Q., Sklar, G. E., & Chng, H. T. (2016) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-016-2151-9 · Cited by 7 · see citing works
Vélez‐Díaz‐Pallarés, M., Ruano, M., Villamañán, E., Larrubia, Y., Wagner, E., & Herrero, A. (2013) · European Journal of Hospital Pharmacy · doi:10.1136/ejhpharm-2013-000308 · Cited by 7 · see citing works
Gestion des interactions médicamenteuses en routine clinique : avons-nous le bon outil ?
Charpiat, B., Bornet, C., Bourdon, O., Calop, J., Dode, X., Grassin, J., Juste, M., & Lagrange, F. (2011) · Le Pharmacien Hospitalier et Clinicien · doi:10.1016/j.phclin.2011.06.004 · Cited by 6 · see citing works
Self-citationKam, H. J., Kim, J. A., Cho, I., Kim, Y. H., & Park, R. W. (2011) · PubMed Central · Cited by 5 · see citing works
Sijpe, G. V. D., Walgraeve, K., Laer, E. V., Quintens, C., Machiels, C., Foulon, V., Casteels, M., der Linden, L. R. V., & Spriet, I. (2024) · Journal of medical systems · doi:10.1007/s10916-024-02113-8 · Cited by 3 · see citing works
Tohic, S. L., Darbon, F., Paysant, C., & Fougereau, E. (2022) · Annales Pharmaceutiques Françaises · doi:10.1016/j.pharma.2022.09.003 · Cited by 4 · see citing works
Alshehri, N., & Alanazi, A. (2023) · Pharmacy · doi:10.3390/pharmacy11040119 · Cited by 3 · see citing works
Bertoliatti, G., Bertho, M., Tiret, I., Chenailler, C., Varin, R., & Doucet, J. (2020) · Le Pharmacien Hospitalier et Clinicien · doi:10.1016/j.phclin.2019.09.002 · Cited by 3 · see citing works
DRUG-RELATED PROBLEMS WITH SPECIAL EMPHASIS ON DRUG-DRUG INTERACTIONS
Mannheimer, B. (2009) · Figshare · Cited by 2 · see citing works
Zhang, T., Le, M.-H., Zlotnik, N., Yee, A. V., Patodia, A., & Austin, Z. (2026) · Pharmacy · doi:10.3390/pharmacy14050113 · Cited by 1 · see citing works
Support tools for paediatric inpatient prescribers: a review: Table 1
Burridge, A. M., Wilson, K., & Terry, D. R. P. (2013) · European Journal of Hospital Pharmacy · doi:10.1136/ejhpharm-2013-000330 · Cited by 1 · see citing works
Clinical Decision Support Systems
Helmons, P. J. (2009) · CRC Press eBooks · doi:10.1201/9781420071764-24 · Cited by 1 · see citing works
Paediatric prescribing practice and opinions of paediatric prescribers on ePrescribing†
Burridge, A. M., Terry, D. R. P., & Wilson, K. (2014) · International Journal of Pharmacy Practice · doi:10.1111/ijpp.12150 · Cited by 1 · see citing works
AVOIDING MEDICATION ERRORS-WHAT IS THE BEST EVIDENCED BASED PRACTICE
Samaranayake, N., & Cheung, B. M. Y. (2011) · The HKU Scholars Hub (University of Hong Kong) · Cited by 1 · see citing works
Sistem Entri Resep Dokter Terkomputerisasi dalam Sistem Informasi Rumah Sakit: Literature Review
Ilhamsyah, R., & Setyonugroho, W. (2020) · JATISI (Jurnal Teknik Informatika dan Sistem Informasi) · doi:10.35957/jatisi.v7i2.300
Negishi, A., Oshima, S., Horii, N., Mutoh, M., Inoue, N., Numajiri, S., Ohshima, S., & Kobayashi, D. (2021) · Biological and Pharmaceutical Bulletin · doi:10.1248/bpb.b20-00986
O’Sullivan, C., Starkey, T., & Pirmohamed, M. (2025) · doi:10.2196/preprints.88578 · Cited by 1 · see citing works
Sijpe, G. V. D., Quintens, C., Walgraeve, K., laer, E. V., Penny, J., Vlieger, G. D., Schrijvers, R., Munter, P. D., Foulon, V., Casteels, M., et al. (2022) · Figshare · doi:10.6084/m9.figshare.19218954
Sijpe, G. V. D., Quintens, C., Walgraeve, K., laer, E. V., Penny, J., Vlieger, G. D., Schrijvers, R., Munter, P. D., Foulon, V., Casteels, M., et al. (2022) · Figshare · doi:10.6084/m9.figshare.19218936
Sijpe, G. V. D., Quintens, C., Walgraeve, K., laer, E. V., Penny, J., Vlieger, G. D., Schrijvers, R., Munter, P. D., Foulon, V., Casteels, M., et al. (2022) · Figshare · doi:10.6084/m9.figshare.19218957
Sijpe, G. V. D., Quintens, C., Walgraeve, K., laer, E. V., Penny, J., Vlieger, G. D., Schrijvers, R., Munter, P. D., Foulon, V., Casteels, M., et al. (2022) · Figshare · doi:10.6084/m9.figshare.19218948
Sijpe, G. V. D., Quintens, C., Walgraeve, K., laer, E. V., Penny, J., Vlieger, G. D., Schrijvers, R., Munter, P. D., Foulon, V., Casteels, M., et al. (2022) · Figshare · doi:10.6084/m9.figshare.19218945
Sijpe, G. V. D., Quintens, C., Walgraeve, K., laer, E. V., Penny, J., Vlieger, G. D., Schrijvers, R., Munter, P. D., Foulon, V., Casteels, M., et al. (2022) · Figshare · doi:10.6084/m9.figshare.19218939
- Authors
- Mille, F., Schwartz, C., Brion, F., Fontan, J.-E., Bourdon, O., Degoulet, P., & Jaulent, M.-C.
- Affiliations
- Inserm (FR)
- Assistance Publique – Hôpitaux de Paris (FR)
- Venue
- International Journal for Quality in Health Care
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-07-24
- Volume
- 20
- Issue
- 6
- Pages
- 400-405
- ISSN
- 1353-4505 (ISSN-L)
- 1464-3677
- OpenAlex
- W2104850833
- Cited by
- 47
- References
- 22
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 2 · 2023: 3 · 2022: 9 · 2021: 1 · 2020: 4 · 2018: 1 · 2017: 4 · 2016: 1 · 2015: 1 · 2014: 3 · 2013: 4 · 2012: 2
- FWCI
- 8.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal for Quality in Health Care (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Toxicology › Pharmacovigilance and Adverse Drug Reactions
- Keywords
- computerized physician order entry, clinical decision support, false positive alerts, alert classification, children's hospitals, drug safety, clinical guidelines
- MeSH terms
- Drug Interactions, Hospitals, Pediatric, Hospitals, University, Humans, Medication Errors, Medication Systems, Prospective Studies, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
