CoChat
Sign Up
Cited by 132Electronic Health Records SystemsArtificial Intelligence in Healthcare and EducationMedical Coding and Health Information

Analysis of clinical decision support system malfunctions: a case series and survey

Open access
Wright, Adam, Hickman, Thu-Trang T, McEvoy, Dustin, Aaron, Skye, Ai, Angela, Andersen, Jan Marie, Hussain, Salman, Ramoni, Rachel +3 more·2016·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocw005PMC PMC5070518
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions. Materials and Methods We identified and investigated several CDSS malfunctions at Brigham and Women’s Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions. Results We identified four CDSS malfunctions at Brigham and Women’s Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93% of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually. Discussion CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate. Conclusion CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocw005

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Another approach uses anomaly detection to identify alerts that are no longer performing as intended.

    “Another approach uses anomaly detection to identify alerts that are no longer performing as intended.55,56”

    From Clinician collaboration to improve clinical decision support: the Clickbusters initiative (2022) · cites this paper as [56]

    The Discussion (Future Work; page unavailable) says anomaly detection methods are likely to help find CDSS malfunctions in real time. But it proposes investigating those methods rather than reporting an approach already in use, as the claim implies. The Methods describe a retrospective model for estimating the size of known alert anomalies, not an anomaly-detection system. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Like utilization logs, anomaly detection can successfully identify alerts that are no longer functioning as designed, but with less personnel effort due to the machine learning approach; however, it does not identify specific opportunities for improving alerts that were suboptimally designed at their onset.

    “Like utilization logs, this approach can successfully identify alerts that are no longer functioning as designed, but with less personnel effort due to the machine learning approach; however, it does not identify specific opportunities for improving alerts that were suboptimally designed at their onset.”

    From Clinician collaboration to improve clinical decision support: the Clickbusters initiative (2022) · cites this paper as [56]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Provider impressions can differ from their actual use.

    “In order to better understand factors impacting the adoption of CDS, previous studies of deployed systems have used post-hoc surveys or interviews to gauge provider impressions 28,44,53 ; however, provider impressions can differ from their actual use 4,18,27 .”

    From Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing (2022) · cites this paper as [18]

    The Discussion (Key Findings; page unavailable) says user reports can miss CDS alert malfunctions, but it does not compare providers’ impressions of CDS with their actual use of it. A search for perceived versus measured CDS use found no passage addressing that comparison. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. [1]Buntin MB Burke MF Hoaglin MC Blumenthal D The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Affairs (Project Hope). 2011; 30(3): 464– 471. 21383365
  2. [2]Lyman JA Cohn WF Bloomrosen M Detmer DE Clinical decision support: progress and opportunities. J Am Med Inform Assoc. 2010; 17(5): 487– 492. 20819850 PMC2995690
  3. [3]Kawamoto K Lobach DF Proposal for fulfilling strategic objectives of the U.S. Roadmap for national action on clinical decision support through a service-oriented architecture leveraging HL7 services. J Am Med Inform Assoc. 2007; 14(2): 146– 155. 17213489 PMC2213469
  4. [4]Garg AX Adhikari NKJ McDonald H Effects of computerized clinical decision support systems on practitioner performance and patient outcomes. JAMA. 2005; 293(10): 1223– 1238. 15755945
  5. [5]Kawamoto K Houlihan CA Balas EA Lobach DF Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005; 330(7494): 765. 15767266 PMC555881
  6. [6]Chaudhry B Wang J Wu S Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Int Med. 2006; 144(10): 742– 752. 16702590
  7. [9]Vashitz G Meyer J Parmet Y Liebermann N Gilutz H Factors affecting physicians compliance with enrollment suggestions into a clinical reminders intervention. Stud Health Technol Informatics. 2010; 160(Pt 2): 796– 800. 20841795
  8. [11]Shackelton RJ Marceau LD Link CL McKinlay JB The intended and unintended consequences of clinical guidelines. J Eval Clin Pract. 2009; 15(6): 1035– 1042. 20367703 PMC3645281
  9. [12]McCoy AB Waitman LR Lewis JB A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc. 2012; 19(3): 346– 352. 21849334 PMC3341775
  10. [13]Landman AB Takhar SS Wang SL The hazard of software updates to clinical workstations: a natural experiment. J Am Med Inform Assoc. 2013; 20: 187– 190. PMC3715366 23492594
  11. [15]Bright TJ Wong A Dhurjati R Effect of clinical decision-support systems: a systematic review. Ann Int Med. 2012; 157(1): 29– 43. 22751758
  12. [16]Ash JS Sittig DF Poon EG Guappone K Campbell E Dykstra RH The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc. 2007; 14(4): 415– 423. 17460127 PMC2244906
  13. [17]Ash JS Sittig DF Campbell EM Guappone KP Dykstra RH Some unintended consequences of clinical decision support systems. AMIA Annual Symposium Proceedings/AMIA Symposium AMIA Symposium. 2007; 26– 30. 18693791 PMC2813668
  14. [25]McCoy AB Wright A Sittig DF Cross-vendor evaluation of key user-defined clinical decision support capabilities: a scenario-based assessment of certified electronic health records with guidelines for future development. J Am Med Inform Assoc. 2015; 22(5): 1081– 1088. 26104739 PMC5009930
  15. [26]Matheny ME Sequist TD Seger AC A randomized trial of electronic clinical reminders to improve medication laboratory monitoring. J Am Med Inform Assoc. 2008; 15(4): 424– 429. 18436905 PMC2442256
  16. [27]Hripcsak G Monitoring the monitor: automated statistical tracking of a clinical event monitor. Comp Biomed Res. 1993; 26(5): 449– 466. 8243069
  17. [28]Sittig DF Pace NL Gardner RM Beck E Morris AH Implementation of a computerized patient advice system using the HELP clinical information system. Comp Biomed Res. 1989; 22(5): 474– 487. 2776450
  18. [30]Institute of Medicine. Committee on Patient Safety and Health Information Technology. Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, DC: National Academies Press; 2012. 24600741
  19. [31]Garg AX Adhikari NK McDonald H Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005; 293(10): 1223– 1238. 15755945
  20. [32]Wright A Goldberg H Hongsermeier T Middleton B A description and functional taxonomy of rule-based decision support content at a large integrated delivery network. J Am Med Inform Assoc. 2007; 14(4): 489– 496. 17460131 PMC2244910
  21. [33]Collins SA Bavuso K Zuccotti G Rocha RA Lessons learned for collaborative clinical content development. Appl Clin Informatics. 2013; 4(2): 304– 316. PMC3716419 23874366
  22. [34]Sittig DF Wright A Meltzer S Comparison of clinical knowledge management capabilities of commercially-available and leading internally-developed electronic health records. BMC Med Informatics Decis Mak. 2011; 11: 13. PMC3063202 21329520
  23. [36]Hongsermeier T Glaser J Managing the investment in clinical decision support. In: Greenes RA, ed. Clinical Decision Support. The Road to Broad Adoption, 2nd edn New York: Elsevier; 2014: 665– 688.
  24. [37]Sittig DF Classen DC Singh H Patient safety goals for the proposed Federal Health Information Technology Safety Center. J Am Med Inform Assoc. 2015; 22(2): 472– 478. 25332353 PMC11749204
  25. [38]US Department of Health and Human Services. 45 CFR Part 170: 2015 Edition Health Information Technology (Health IT) Certification Criteria, 2015 Edition Base Electronic Health Record (EHR) Definition, and ONC Health IT Certification Program Modifications. 2015. https://s3.amazonaws.com/public-inspection.federalregister.gov/2015-06612.pdf. Accessed February 12, 2016. 26095483
  26. [39]Sittig DF Ash JS Singh H The SAFER guides: empowering organizations to improve the safety and effectiveness of electronic health records. Am J Managed Care. 2014; 20(5): 418– 423. 25181570
  27. [40]Sittig DF Classen DC Safe electronic health record use requires a comprehensive monitoring and evaluation framework. JAMA. 2010; 303(5): 450– 451. 20124542 PMC2965782

Cited by

Sorted by citations

Authors
Wright, A., Hickman, T.-T. T., McEvoy, D., Aaron, S., Ai, A., Andersen, J. M., Hussain, S., Ramoni, R., Fiskio, J., Sittig, D. F., & Bates, D. W.
Affiliations
  • Brigham and Women's Hospital (US)
  • Partners HealthCare, Boston, MA, USA
  • Dartmouth Institute for Health Policy and Clinical Practice (US)
  • Harvard University (US)
  • The University of Texas Health Science Center at Houston (US)
Corresponding author
Wright, Adam
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2016-03-28
Volume
23
Issue
6
Pages
1068-1076
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocw005
PMC
PMC5070518
OpenAlex
W2330482840
Cited by
132
References
3
Citations by year
2026: 10 · 2025: 14 · 2024: 8 · 2023: 10 · 2022: 12 · 2021: 10 · 2020: 12 · 2019: 18 · 2018: 11 · 2017: 18 · 2016: 9
FWCI
28.76
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
  • HighWire Press Open Archive (submittedVersion)
Funders
  • National Institutes of Health
  • U.S. National Library of Medicine (R01LM011966)
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
Keywords
clinical decision support systems, clinical alerts, electronic health records, software upgrade, amiodarone, lead screening
MeSH terms
Amiodarone, Boston, Child, Preschool, Equipment Failure, Hospitals, Special, Humans, Lead Poisoning, Monitoring, Physiologic, Software, Medical Errors, Organizational Case Studies, Decision Support Systems, Clinical, Medical Order Entry Systems, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review