CoChat
Sign Up
Cited by 44Electronic Health Records SystemsAnomaly Detection Techniques and ApplicationsMachine Learning in Healthcare

Using statistical anomaly detection models to find clinical decision support malfunctions

Open access
Ray, Soumi, McEvoy, Dustin S, Aaron, Skye, Hickman, Thu-Trang, Wright, Adam·2018·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocy041PMC PMC6016695
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: Malfunctions in Clinical Decision Support (CDS) systems occur due to a multitude of reasons, and often go unnoticed, leading to potentially poor outcomes. Our goal was to identify malfunctions within CDS systems. Methods: We evaluated 6 anomaly detection models: (1) Poisson Changepoint Model, (2) Autoregressive Integrated Moving Average (ARIMA) Model, (3) Hierarchical Divisive Changepoint (HDC) Model, (4) Bayesian Changepoint Model, (5) Seasonal Hybrid Extreme Studentized Deviate (SHESD) Model, and (6) E-Divisive with Median (EDM) Model and characterized their ability to find known anomalies. We analyzed 4 CDS alerts with known malfunctions from the Longitudinal Medical Record (LMR) and Epic® (Epic Systems Corporation, Madison, WI, USA) at Brigham and Women’s Hospital, Boston, MA. The 4 rules recommend lead testing in children, aspirin therapy in patients with coronary artery disease, pneumococcal vaccination in immunocompromised adults and thyroid testing in patients taking amiodarone. Results: Poisson changepoint, ARIMA, HDC, Bayesian changepoint and the SHESD model were able to detect anomalies in an alert for lead screening in children and in an alert for pneumococcal conjugate vaccine in immunocompromised adults. EDM was able to detect anomalies in an alert for monitoring thyroid function in patients on amiodarone. Conclusions: Malfunctions/anomalies occur frequently in CDS alert systems. It is important to be able to detect such anomalies promptly. Anomaly detection models are useful tools to aid such detections.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocy041

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Like utilization logs, anomaly detection can successfully identify alerts that are no longer functioning as designed, but with less personnel effort due to the machine learning approach; however, it does not identify specific opportunities for improving alerts that were suboptimally designed at their onset.

    “Like utilization logs, this approach can successfully identify alerts that are no longer functioning as designed, but with less personnel effort due to the machine learning approach; however, it does not identify specific opportunities for improving alerts that were suboptimally designed at their onset.”

    From Clinician collaboration to improve clinical decision support: the Clickbusters initiative (2022) · cites this paper as [55]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Another approach uses anomaly detection to identify alerts that are no longer performing as intended.

    “Another approach uses anomaly detection to identify alerts that are no longer performing as intended.55,56”

    From Clinician collaboration to improve clinical decision support: the Clickbusters initiative (2022) · cites this paper as [55]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. Blumenthal D, Tavenner M. The "meaningful use" regulation for elec- tronic health records. N Engl J Med 2010; 363 (6): 501-4.
  2. Coorevits P, Sundgren M, Klein GO, et al. Electronic health records: new opportunities for clinical research. J Intern Med 2013; 274 (6): 547-60.
  3. Osheroff JA, Teich JM, Middleton B, et al. A roadmap for national action on clinical decision support. J Am Med Inform Assoc 2007; 14 (2): 141-5.
  4. Zuccotti G, Maloney FL, Feblowitz J, et al. Reducing risk with clinical de- cision support. Appl Clin Inform 2014; 05 (03): 746-56.
  5. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of tri- als to identify features critical to success. BMJ 2005; 330 (7494): 765-73.
  6. Shackelton RJ, Marceau LD, Link CL, McKinlay JB. The intended and unintended consequences of clinical guidelines. J Eval Clin Pract 2009; 15 (6): 1035-42.
  7. Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (1): 29-43.
  8. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clini- cal decision support systems on practitioner performance and patient out- comes: a systematic review. JAMA 2005; 293 (10): 1223-38.
  9. Wright A, Hickman TT, McEvoy D, et al. Analysis of clinical decision support system malfunctions: a case series and survey. J Am Med Inform Assoc 2016; 23 (6): 1068-76.
  10. Kassakian SZ, Yackel TR, Gorman PN, et al. Clinical decisions support malfunctions in a commercial electronic health record. Appl Clin Inform 2017; 08 (03): 910-23.
  11. McCoy AB, Waitman LR, Lewis JB, et al. A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc 2012; 19 (3): 346-52.
  12. Wright A, Ai A, Ash J, et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc 2017; doi: 10.1093/jamia/ocx106. · DOI
  13. Chandola V, Banerjee A, Kumar V. Anomaly detection. ACM Comput Surv 2009; 41 (3): 1-58.
  14. Aggarwal CC. Outlier Analysis. New York, NY: Springer New York; 2013.
  15. Hodge V, Austin J. A survey of outlier detection methodologies. Artif Intell Rev 22 (2): 85-126.
  16. Sen A, Srivastava MS. On tests for detecting change in mean. Ann Stat 1975; 3 (1): 98-108.
  17. Hinkley DV, Hinkley EA. Inference about the change-point in a sequence of random variables. Biometrika 1970; 57 (3): 477-88.
  18. Gupta M, Gao J, Aggarwal CC, Han J. Outlier detection for temporal data: a survey. IEEE Trans Knowl Data Eng 2014; 26 (9): 2250-67.
  19. Pettitt AN. A non-parametric approach to the change-point problem. Appl Stat 1979; 28 (2): 126-35.
  20. Zeileis A, Shah A, Patnaik I. Testing, monitoring, and dating structural changes in exchange rate regimes. Comput Stat Data Anal 2010; 54 (6): 1696-706.
  21. Fan Z, Dror RO, Mildorf TJ, Piana S, Shaw DE. Identifying localized changes in large systems: change-point detection for biomolecular simula- tions. Proc Natl Acad Sci USA 2015; 112 (24): 7454-9.
  22. Adams RP, MacKay DJC. Bayesian online changepoint detection. Techni- cal report, University of Cambridge, Cambridge, UK, 2007. arXiv:0710.3742v1. · arXiv
  23. Western B, Kleykamp M. A Bayesian change point model for historical time series analysis. Polit Anal 2004; 12 (04): 354-75.
  24. Hill DJ, Minsker BS, Amir E. Real-time Bayesian anomaly detection in streaming environmental data. Water Resour Res 2016; 46 (4): W00D28.
  25. Akouemo HN, Povinelli RJ. Probabilistic anomaly detection in natural gas time series data. Int J Forecast 2016; 32 (3): 948-56.
  26. Hauskrecht M, Batal I, Hong C, et al. Outlier-based detection of unusual patient-management actions: an ICU study. J Biomed Inform 2016; 64: 211-21.
  27. Ray S, Wright A. Detecting Anomalies in Alert Firing within Clinical De- cision Support Systems Using Anomaly/Outlier Detection Techniques. Proceedings of the 7th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics USA 2016; 185-90.
  28. Vallis O, Hochenbaum J, Kejariwal A. A novel technique for long-term anomaly detection in the cloud. 17-18 June 2014 6th {USENIX} Work- shop on Hot Topics in Cloud Computing (HotCloud) Berkeley, CA, USA; 2014.
  29. James NA, Kejariwal A, Matteson DS. Leveraging cloud data to mitigate user experience from "breaking bad." IEEE International Conference on Big Data(Big Data) USA 2016: 3499-3508.
  30. Jen T, Gupta AK. On testing homogeneity of variances for gaussian mod- els. J Stat Comput Simul 1987; 27 (2): 155-73.
  31. Scott AJ, Knott M. A cluster analysis method for grouping means in the analysis of variance. Biometrics 1974; 30 (3): 507-12.
  32. Killick R, Eckley IA. changepoint: an R package for changepoint analysis. J Stat Soft 2014; 58 (3): 1-19.
  33. Pena EM, de Assis M, Proenc¸a ML. Anomaly detection using forecasting methods arima and hwds. In: 32nd International Conference of the Chil- ean Computer Science Society, IEEE Washington, DC, USA; 11-15 Nov, 2013: 63-6.
  34. Chen C, Liu L-M. Joint estimation of model parameters and outlier effects in time series. J Am Stat Assoc 1993; 88 (421): 284-97.
  35. Matteson DS, James NA. ecp: an r package for nonparametric multiple change point analysis of multivariate data. J. Stat. Softw 2015; 62 (7): 1-25.
  36. Szekely GJ, Rizzo ML. Hierarchical clustering via joint between-within distances: extending ward's minimum variance method. J Classif 2005; 22 (2): 151-83.
  37. Vostrikova LJ. Detecting disorder in multidimensional random process. Soviet Math Dokl 1981; 24: 55-9.
  38. Gandy A. Sequential implementation of Monte Carlo tests with uniformly bounded resampling risk. J Am Stati Assoc 2009; 104 (488): 1504-11.
  39. Erdman C, Emerson JW. bcp: an RPackage for performing a Bayesian analysis of change point problems. J Stat Soft 2007; 23 (3): 1-13.
  40. Barry D, Hartigan JA. A Bayesian analysis for change point problems. J Am Stat Assoc 1993; 88 (421): 309-19.
  41. Cleveland RB, Cleveland WS, McRae JE. STL: a seasonal-trend decompo- sition procedure based on loess. J Off Stat 1990; 6: 3-73.
  42. Rosner B. Percentage points for a generalized ESD many-outlier proce- dure. Technometrics 1983; 25 (2): 165-72.
  43. Rosner B. On the detection of many outliers. Technometrics 1975; 17 (2): 221-7.
  44. Donoho DL, Huber PJ. The Notion of Breakdown Point. A Festschrift for Erich L. Lehmann. Florida, USA: CRC Press; 1983: 157-84.
  45. Ruppert D. Robust statistics: the approach based on influence functions. Technometrics 1987; 29 (2): 240-1.
  46. Hampel FR. The influence curve and its role in robust estimation. J Am Stat Assoc 1974; 69 (346): 383-93.
  47. Good P. Permutation, Parametric and Bootstrap Tests of Hypotheses. New York: Springer-Verlag; 2005.

Cited by

Sorted by citations

Authors
Ray, S., McEvoy, D. S., Aaron, S., Hickman, T.-T., & Wright, A.
Affiliations
  • Brigham and Women's Hospital (US)
  • Partners Healthcare, Information Systems, Somerville, MA, USA
Corresponding author
Ray, Soumi
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2018-05-11
Volume
25
Issue
7
Pages
862-871
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocy041
PMC
PMC6016695
OpenAlex
W2801792225
Cited by
44
References
1
Citations by year
2026: 1 · 2025: 5 · 2024: 6 · 2023: 3 · 2022: 8 · 2021: 6 · 2020: 5 · 2019: 7 · 2018: 3
FWCI
6.41
Type
journal article
Language
en
Open access
Licence
academic.oup.com/journals/pages/about_us/legal/notices (vor)
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • National Library of Medicine
  • National Institutes of Health (R01LM011966)
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
clinical decision support systems, anomaly detection, change-point models, ARIMA
MeSH terms
Algorithms, Humans, Models, Statistical, Equipment Failure Analysis, Decision Support Systems, Clinical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review