Cranky comments: detecting clinical decision support malfunctions through free-text override reasons
Open accessAbstract
Background: Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited. Objective: Investigate whether user override comments can be used to discover malfunctions. Methods: We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: “broken,” “not broken, but could be improved,” and “not broken.” We used 3 methods (frequency of comments, cranky word list heuristic, and a Naïve Bayes classifier trained on a sample of comments) to automatically rank rules based on features of their override comments. We evaluated each ranking using the manual classification as truth. Results: Of the rules investigated, 62 were broken, 13 could be improved, and the remaining 45 were not broken. Frequency of comments performed worse than a random ranking, with precision at 20 of 8 and AUC = 0.487. The cranky comments heuristic performed better with precision at 20 of 16 and AUC = 0.723. The Naïve Bayes classifier had precision at 20 of 17 and AUC = 0.738. Discussion: Override comments uncovered malfunctions in 26% of all rules active in our system. This is a lower bound on total malfunctions and much higher than expected. Even for low-resource organizations, reviewing comments identified by the cranky word list heuristic may be an effective and feasible way of finding broken alerts. Conclusion: Override comments are a rich data source for finding alerts that are broken or could be improved. If possible, we recommend monitoring all override comments on a regular basis.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Additional efforts have focused on identifying opportunities for improvement by obtaining feedback from clinicians receiving alerts, such as sentiment analysis of comments entered when overriding alerts and feedback links within displayed alerts.
“Additional efforts have focused on identifying opportunities for improvement by obtaining feedback from clinicians receiving the alerts, such as sentiment analysis of comments entered by clinicians when overriding alerts52 and feedback links within displayed alerts.”
From Clinician collaboration to improve clinical decision support: the Clickbusters initiative (2022) · cites this paper as [52]
Such feedback approaches can successfully identify individual alerts with optimization opportunities, as well as alerts with potentially suboptimal design or that are no longer functioning as designed.
“Such feedback approaches can successfully identify individual alerts with optimization opportunities, as well as alerts with potentially suboptimal design or that are no longer functioning as designed.”
From Clinician collaboration to improve clinical decision support: the Clickbusters initiative (2022) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Improving Acceptance of Computerized Prescribing Alerts in Ambulatory Care
Shah, N. R., Seger, A. C., Seger, D. L., Fiskio, J. M., Kuperman, G. J., Blumenfeld, B., Recklet, E. G., Bates, D. W., & Gandhi, T. K. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1868 · Cited by 353 · see citing works
Effect of Computer-Based Alerts on the Treatment and Outcomes of Hospitalized Patients
Rind, D. M. (1994) · Archives of Internal Medicine · doi:10.1001/archinte.1994.00420130107014 · Cited by 323 · see citing works
Overrides of medication-related clinical decision support alerts in outpatients
Nanji, K. C., Slight, S. P., Seger, D. L., Cho, I., Fiskio, J. M., Redden, L. M., Volk, L. A., & Bates, D. W. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001813 · Cited by 219 · see citing works
Flagged first, then by citations
Research works printed but not linked: 22
- [1]Teich JM, Merchia PR, Schmiz JL et al Effects of computerized physician order entry on prescribing practices. Arch Intern Med 2000; 160 18: 2741– 7. 11025783
- [2]Evans RS, Pestotnik SL, Classen DC et al A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998; 338 4: 232– 8. 9435330
- [4]Krall M, Gerace A a metrics taxonomy and reporting strategy for rule-based alerts. Perm J 2015; 19 3: 11– 9. PMC4500476 26057684
- [5]McCoy AB, Thomas EJ, Krousel-Wood M et al Clinical decision support alert appropriateness: a review and proposal for improvement. Ochsner J 2014; 14 2: 195– 202. 24940129 PMC4052586
- [6]Charles D, Gabriel M, Searcy T Adoption of electronic health record systems among U.S. non-federal acute care hospitals: 2008–2014. ONC Data Brief 2015; 23: 1– 10.
- [7]Page N, Baysari MT, Westbrook JI A systematic review of the effectiveness of interruptive medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve patient safety. Int J Med Inform 2017; 105: 22– 30. 28750908
- [8]Pell JM, Cheung D, Jones MA et al Don’t fuel the fire: decreasing intravenous haloperidol use in high risk patients via a customized electronic alert. J Am Med Inform Assoc 2014; 21 6: 1109. 24928174 PMC4215057
- [10]Wright A, Hickman TT, McEvoy D et al Analysis of clinical decision support system malfunctions: a case series and survey. J Am Med Inform Assoc 2016; 23 6: 1068– 76. 27026616 PMC5070518
- [11]Kassakian SZ, Yackel TR, Gorman PN et al Clinical decisions support malfunctions in a commercial electronic health record. Appl Clin Inform 2017; 08 03: 910. PMC6220702 28880046
- [13]Wright A, Ai A, Ash J et al Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc 2018; 25: 496– 506. 29045651 PMC6019061
- [14]McCoy AB, Waitman LR, Lewis JB et al A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc 2012; 19 3: 346– 52. 21849334 PMC3341775
- [15]van der Sijs H, Aarts J, van Gelder T et al Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008; 15 4: 439– 48. 18436915 PMC2442265
- [16]van der Sijs H, Kowlesar R, Aarts J et al Unintended consequences of reducing QT-alert overload in a computerized physician order entry system. Eur J Clin Pharmacol 2009; 65 9: 919– 25. 19415251 PMC2729977
- [17]Chused AE, Kuperman GJ, Stetson PD Alert override reasons: a failure to communicate. AMIA Annu Symp Proc 2008 Nov 6; 111– 5. 18999082 PMC2655991
- [18]Ahn EK, Cho SY, Shin D et al Differences of reasons for alert overrides on contraindicated co-prescriptions by admitting department. Healthc Inform Res 2014; 20 4: 280– 7. 25405064 PMC4231178
- [19]Seidling HM, Paterno MD, Haefeli WE et al Coded entry versus free-text and alert overrides: what you get depends on how you ask. Int J Med Inform 2010; 79 11: 792– 6. 20869911
- [21]Kouloumpis E, Wilson T, Moore J Twitter sentiment analysis: the good the bad and the OMG! In: Fifth International AAAI Conference on Weblogs and Social Media; 2011; Barcelona, Spain.
- [22]Bollen J, Mao H, Pepe A Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. In: Fifth International AAAI Conference on Weblogs and Social Media; 2011; Barcelona, Catalonia, Spain: Association for the Advancement of Artificial Intelligence.
- [24]Wang H, Can D, Kazemzadeh A et al A system for real-time Twitter sentiment analysis of 2012 U.S. presidential election cycle. In: ACL 2012 System Demonstrations; 2012; Jeju Island, Korea: Association for Computational Linguistics.
- [25]Bird S, Loper E, Klein E Natural Language Processing with Python. O'Reilly Media, Inc 2009 http://nltk.org/book. Accessed November 12, 2018.
- [28]Robin X, Turck N, Hainard A et al pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 2011; 12: 77. 21414208 PMC3068975
- [29]Wickham H ggplot2: Elegant Graphics for Data Analysis. New York, NY: Springer-Verlag; 2009.
Cited by
Sorted by citations
Liaw, S.-T., Liyanage, H., Kuziemsky, C., Terry, A. L., Schreiber, R., Jonnagaddala, J., & de Lusignan, S. (2020) · Yearbook of Medical Informatics · doi:10.1055/s-0040-1701980 · Cited by 76 · see citing works
You, J. G., Hernandez-Boussard, T., Pfeffer, M. A., Landman, A., & Mishuris, R. G. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01506-4 · Cited by 59 · see citing works
Clinician collaboration to improve clinical decision support: the Clickbusters initiative
McCoy, A. B., Russo, E. M., Johnson, K. B., Addison, B., Patel, N., Wanderer, J. P., Mize, D. E., Jackson, J. G., Reese, T. J., Littlejohn, S., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac027 · Cited by 41 · see citing works
Self-citationFerro, D. F., Tobias, M., Carr, L. H., Wentz, P., Rodriguez, M., Pitts, C., Kane, E., & Shelov, E. (2026) · Applied Clinical Informatics · doi:10.1055/a-2793-0977
- Authors
- Aaron, S., McEvoy, D. S., Ray, S., Hickman, T.-T. T., & Wright, A.
- Affiliations
- Brigham and Women's Hospital (US)
- Partners eCare, Partners HealthCare, Boston, Massachusetts, USA
- Partners Community Health, Partners HealthCare, Boston, Massachusetts, USA
- Corresponding author
- Wright, Adam
- ORCID
- McEvoy, Dustin S
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-12-26
- Volume
- 26
- Issue
- 1
- Pages
- 37-43
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC6308015
- OpenAlex
- W2785336100
- Cited by
- 41
- References
- 3
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 7 · 2023: 2 · 2022: 7 · 2021: 1 · 2020: 10 · 2019: 3
- FWCI
- 4.27
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- AMIA
- Funders
- National Library of Medicine
- National Institutes of Health (R01LM011966)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Keywords
- clinical decision support alerts, Naive Bayes classifier
- MeSH terms
- Bayes Theorem, Documentation, Humans, Irritable Mood, Medication Errors, ROC Curve, Decision Support Systems, Clinical, Medical Order Entry Systems, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
