CoChat
Sign Up
Cited by 160Machine Learning in HealthcareData Stream Mining TechniquesArtificial Intelligence in Healthcare

Detection of calibration drift in clinical prediction models to inform model updating

Open access
Davis, Sharon E., Greevy, Robert A., Lasko, Thomas A., Walsh, Colin G., Matheny, Michael E.·2020·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2020.103611PMC PMC8627243
Open this paper in CoChat — free accountOpen PDF

Abstract

Model calibration, critical to the success and safety of clinical prediction models, deteriorates over time in response to the dynamic nature of clinical environments. To support informed, data-driven model updating strategies, we present and evaluate a calibration drift detection system. Methods are developed for maintaining dynamic calibration curves with optimized online stochastic gradient descent and for detecting increasing miscalibration with adaptive sliding windows. These methods are generalizable to support diverse prediction models developed using a variety of learning algorithms and customizable to address the unique needs of clinical use cases. In both simulation and case studies, our system accurately detected calibration drift. When drift is detected, our system further provides actionable alerts by including information on a window of recent data that may be appropriate for model updating. Simulations showed these windows were primarily composed of data accruing after drift onset, supporting the potential utility of the windows for model updating. By promoting model updating as calibration deteriorates rather than on pre-determined schedules, implementations of our drift detection system may minimize interim periods of insufficient model accuracy and focus analytic resources on those models most in need of attention.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2020.103611

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 66 · Cited by 160

Research works printed but not linked: 54

  1. [1]Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implementing electronic health care predictive analytics: considerations and challenges. Health Affairs. 2014; 33(7): 1148– 1154. 25006140
  2. [4]Moons KG, Kengne AP, Woodward M, Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012; 98(9): 683– 690. 22397945
  3. [5]Moons KG, Kengne AP, Grobbee DE, Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012; 98(9): 691– 698. 22397946
  4. [6]Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009; 338: b605. 19477892
  5. [7]Moons KG, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009; 338: b606. 19502216
  6. [8]Shah ND, Steyerberg EW, Kent DM. Big Data and Predictive Analytics: Recalibrating Expectations. JAMA. 2018; 320(1): 27– 28. 29813156
  7. [9]Diamond GA. What price perfection? Calibration and discrimination of clinical prediction models. Journal of Clinical Epidemiology. 1992; 45(1): 85– 89. 1738016
  8. [10]Van Calster B, McLernon DJ, van Smeden M, Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019; 17(1): 230. 31842878 PMC6912996
  9. [11]Davis SE, Lasko TA, Chen G, Matheny ME. Calibration drift among regression and machine learning models for hospital mortality. Proceedings of the AMIA Annual Symposium. 2017. PMC5977580 29854127
  10. [14]Minne L, Eslami S, De Keizer N, De Jonge E, De Rooij SE, Abu-Hanna A. Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment. Intensive Care Medicine. 2012; 38(1): 40– 46. 22042520 PMC3233667
  11. [15]Cook DA, Joyce CJ, Barnett RJ, Prospective independent validation of APACHE III models in an Australian tertiary adult intensive care unit. Anaesthesia and Intensive Care. 2002; 30(3): 308– 315. 12075637
  12. [16]Paul E, Bailey M, Van Lint A, Pilcher V. Performance of APACHE III over time in Australia and New Zealand: a retrospective cohort study. Anaesthesia and Intensive Care. 2012; 40(6): 980– 994. 23194207
  13. [17]Mikkelsen MM, Johnsen SP, Nielsen PH, Jakobsen CJ. The EuroSCORE in western Denmark: a population-based study. Journal of Cardiothoracic and Vascular Anesthesia. 2012; 26(2): 258– 264. 22100858
  14. [18]Jenkins DA, Sperrin M, Martin GP, Peek N. Dynamic models to predict health outcomes: current status and methodological challenges. Diagnostic and Prognostic Research. 2018; 2(23). PMC6460710 31093570
  15. [19]Siregar S, Nieboer D, Vergouwe Y, Improved prediction by dynamic modelling: An exploratory study in the adult cardiac surgery database of the netherlands association for cardio-thoracic surgery. Circ Cardiovasc Qual Outcomes. 2016; 9(2): 171– 181. 26933048
  16. [20]Toll DB, Janssen KJ, Vergouwe Y, Moons KG. Validation, updating and impact of clinical prediction rules: a review. Journal of Clinical Epidemiology. 2008; 61(11): 1085– 1094. 19208371
  17. [21]Kappen TH, Vergouwe Y, van Klei WA, van Wolfswinkel L, Kalkman CJ, Moons KG. Adaptation of clinical prediction models for application in local settings. Medical Decision Making. 2012; 32(3): E1– 10. PMC3426927 22427369
  18. [22]Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y. Updating methods improved the performance of a clinical prediction model in new patients. Journal of Clinical Epidemiology. 2008; 61(1): 76– 86. 18083464
  19. [23]Debray TP, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, Moons KG. A new framework to enhance the interpretation of external validation studies of clinical prediction models. Journal of Clinical Epidemiology. 2015; 68(3): 279– 289. 25179855
  20. [24]Chen PM, Hemmen TM. Evolving Healthcare Delivery in Neurology During the Coronavirus Disease 2019 (COVID-19) Pandemic. Front Neurol. 2020; 11: 578. 32574251 PMC7273522
  21. [27]Castro VM, Perlis RH. Electronic Health Record Documentation of Psychiatric Assessments in Massachusetts Emergency Department and Outpatient Settings During the Coronavirus Disease 2019 (COVID-19) Pandemic. JAMA Network Open. 2020; 3(6):e2011346. PMC7280952 32511718
  22. [28]Hannan EL, Cozzens K, King SB, 3rd, Walford G, Shah NR. The New York State cardiac registries: history, contributions, limitations, and lessons for future efforts to assess and publicly report healthcare outcomes. Journal of the American College of Cardiology. 2012; 59(25): 2309– 2316. 22698487
  23. [29]Jin R, Furnary AP, Fine SC, Blackstone EH, Grunkemeier GL. Using Society of Thoracic Surgeons risk models for risk-adjusting cardiac surgery results. Annals of Thoracic Surgery. 2010; 89(3): 677– 682. 20172107
  24. [30]Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JD. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Statistics in Medicine. 2004; 23(16): 2567– 2586. 15287085
  25. [31]Vergouwe Y, Nieboer D, Oostenbrink R, A closed testing procedure to select an appropriate method for updating prediction models. Statistics in Medicine. 2017; 36(28): 4529– 4539. 27891652
  26. [32]Minne L, Eslami S, de Keizer N, de Jonge E, de Rooij SE, Abu-Hanna A. Statistical process control for monitoring standardized mortality ratios of a classification tree model. Methods of Information in Medicine. 2012; 51(4): 353– 358. 22773038
  27. [33]Davis SE, Greevy RA, Fonnesbeck C, Lasko TA, Walsh CG, Matheny ME. A nonparametric updating method to correct clinical prediction model drift. Journal of the American Medical Informatics Association. 2019; 26(12): 1448– 1457. 31397478 PMC6857513
  28. [34]Davis SE, Greevy RA, Lasko TA, Walsh CG, Matheny ME. Comparison of prediction model performance updating protocols: using a data-driven testing procedure to guide updating. Proceedings of the AMIA Annual Symposium. 2019: 1002– 1010. PMC7153129 32308897
  29. [35]Bifet A, Gavalda R. Learning from time-changing data with adaptive windowing. Paper presented at: Proceedings of the 2007 SIAM International Conference on Data Mining.
  30. [36]Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Statistics in Medicine. 2014; 33(3): 517– 535. 24002997 PMC4793659
  31. [37]Austin PC, Steyerberg EW. The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Statistics in Medicine. 2019; 38(21): 4051– 4065. 31270850 PMC6771733
  32. [38]Van Hoorde K, Van Huffel S, Timmerman D, Bourne T, Van Calster B. A spline-based tool to assess and visualize the calibration of multiclass risk predictions. Journal of Biomedical Informatics. 2015; 54: 283– 293. 25579635
  33. [39]Kingma DP, Ba J. Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015). 2015.
  34. [40]Ruder S. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:160904747. 2016.
  35. [41]Miyaguchi K, Kajino H. Cogra: Concept-Drift-Aware Stochastic Gradient Descent for Time-Series Forecasting. Paper presented at: Proceedings of the AAAI Conference on Artificial Intelligence2019.
  36. [42]Losing V, Hammer B, Wersing H. Incremental on-line learning: A review and comparison of state of the art algorithms. Neurocomputing. 2018; 275: 1261– 1274.
  37. [44]Nattino G, Finazzi S, Bertolini G. A new test and graphical tool to assess the goodness of fit of logistic regression models. Statistics in Medicine. 2016; 35(5): 709– 720. 26439593
  38. [47]Ross GJ, Adams NM, Tasoulis DK, Hand DJ. Exponentially weighted moving average charts for detecting concept drift. Pattern Recognition Letters. 2012; 33(2): 191– 198.
  39. [48]Chen K, Koh YS, Riddle P. Tracking Drift Severity in Data Streams. Australasian Joint Conference on Artificial Intelligence; 2015.
  40. [49]Baena-Garcıa M, del Campo-Ávila J, Fidalgo R, Bifet A, Gavalda R, Morales-Bueno R. Early drift detection method. Fourth International Workshop on Knowledge Discovery From Data Streams; 2006.
  41. [51]Jiang X, Osl M, Kim J, Ohno-Machado L. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association. 2012; 19(2): 263– 274. 21984587 PMC3277613
  42. [52]Grulich PM, Saitenmacher R, Traub J, Breß S, Rabl T, Markl V. Scalable Detection of Concept Drifts on Data Streams with Parallel Adaptive Windowing. Paper presented at: 21st International Conference on Extending Database Technology (EDBT) 2018.
  43. [53]Benneyan JC, Lloyd RC, Plsek PE. Statistical process control as a tool for research and healthcare improvement. BMJ Quality & Safety. 2003; 12(6): 458– 464. PMC1758030 14645763
  44. [54]Cook DA, Coory M, Webster RA. Exponentially weighted moving average charts to compare observed and expected values for monitoring risk-adjusted hospital indicators. BMJ Qual & Safety. 2011; 20(6): 469– 474. 21209145
  45. [55]Cronin RM, VanHouten JP, Siew ED, National Veterans Health Administration Inpatient Risk Stratification Models for Hospital-Acquired Acute Kidney Injury. Journal of the American Medical Informatics Association. 2015; 22(5): 1054– 1071. 26104740 PMC5009929
  46. [56]Thor J, Lundberg J, Ask J, Application of statistical process control in healthcare improvement: systematic review. BMJ Quality & Safety. 2007; 16(5): 387– 399. PMC2464970 17913782
  47. [57]Benneyan JC, Villapiano A, Katz N, Duffy M, Budman SH, Butler SF. Illustration of a statistical process control approach to regional prescription opioid abuse surveillance. Journal of Addiction Medicine. 2011; 5(2): 99– 109. 21769055
  48. [58]Matheny ME, Ohno-Machado L, Resnic FS. Risk-adjusted sequential probability ratio test control chart methods for monitoring operator and institutional mortality rates in interventional cardiology. American Heart Journal. 2008; 155(1): 114– 120. 18082501
  49. [59]Matheny ME, Normand SL, Gross TP, Evaluation of an automated safety surveillance system using risk adjusted sequential probability ratio testing. BMC Medical Informatics and Decision Making. 2011; 11: 75. 22168892 PMC3262755
  50. [60]Morton AP, Whitby M, McLaws ML, The application of statistical process control charts to the detection and monitoring of hospital-acquired infections. Journal of Quality in Clinical Practice. 2001; 21(4): 112– 117. 11856406
  51. [61]Baker AW, Haridy S, Salem J, Performance of statistical process control methods for regional surgical site infection surveillance: a 10-year multicentre pilot study. BMJ Quality & Safety. 2018; 27(8): 600– 610. PMC5967987 29175853
  52. [62]Seim A, Andersen B, Sandberg WS. Statistical process control as a tool for monitoring nonoperative time. Anesthesiology. 2006; 105(2): 370– 380. 16871072
  53. [63]Pimentel L, Barrueto F, Jr. Statistical process control: separating signal from noise in emergency department operations. Journal of Emergency Medicine. 2015; 48(5): 628– 638. 25726257
  54. [64]Minne L, Eslami S, de Keizer N, de Jonge E, de Rooij SE, Abu-Hanna A. Statistical process control for validating a classification tree model for predicting mortality--a novel approach towards temporal validation. Journal of Biomedical Informatics. 2012; 45(1): 37– 44. 21907826

Authors
Davis, S. E., Greevy, R. A., Lasko, T. A., Walsh, C. G., & Matheny, M. E.
Affiliations
  • Vanderbilt University Medical Center (US)
  • VA Tennessee Valley Healthcare System (US)
Corresponding author
Davis, Sharon E.
ORCID
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2020-12
Volume
112
Pages
103611
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2020.103611
PMC
PMC8627243
OpenAlex
W3096597179
Cited by
160
References
66
Citations by year
2026: 59 · 2025: 31 · 2024: 26 · 2023: 16 · 2022: 22 · 2021: 6
FWCI
4.32
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • Health Services Research and Development
  • National Institutes of Health (BCHI-R01-130828)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Data Stream Mining Techniques
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
clinical prediction model, model updating, adaptive sliding window, calibration error, model maintenance
MeSH terms
Algorithms, Calibration, Prognosis, Models, Statistical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review