Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment
Open accessAbstract
PURPOSE: The aim of our study was to explore, using an innovative method, the effect of temporal changes in the mortality prediction performance of an existing model on the quality of care assessment. The prognostic model (rSAPS-II) was a recalibrated Simplified Acute Physiology Score-II model developed for very elderly Intensive Care Unit (ICU) patients. METHODS: The study population comprised all 12,143 consecutive patients aged 80 years and older admitted between January 2004 and July 2009 to one of the ICUs of 21 Dutch hospitals. The prospective dataset was split into 30 equally sized consecutive subsets. Per subset, we measured the model's discrimination [area under the curve (AUC)], accuracy (Brier score), and standardized mortality ratio (SMR), both without and after repeated recalibration. All performance measures were considered to be stable if <2 consecutive points fell outside the green zone [mean ± 2 standard deviation (SD)] and none fell outside the yellow zone (mean ± 4SD) of pre-control charts. We compared proportions of hospitals with SMR>1 without and after repeated recalibration for the year 2009. RESULTS: For all subsets, the AUCs were stable, but the Brier scores and SMRs were not. The SMR was downtrending, achieving levels significantly below 1. Repeated recalibration rendered it stable again. The proportions of hospitals with SMR>1 and SMR<1 changed from 15 versus 85% to 35 versus 65%. CONCLUSIONS: Variability over time may markedly vary among different performance measures, and infrequent model recalibration can result in improper assessment of the quality of care in many hospitals. We stress the importance of the timely recalibration and repeated validation of prognostic models over time.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Data drift can cause models' performance to deteriorate or behave in unexpected ways.
“Data drift, defined as the systematic shift in the underlying distribution of input features 11 , can cause models' performance to deteriorate [11][12][13] or behave in unexpected ways, which can pose a threat to the safety of patients.”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [13]
The performance of a customized SAPS-II ICU mortality model changed over time, illustrating how temporal changes in clinical case mix and care processes can affect the reliability of severity-adjustment models.
“Minne et al. showed that the performance of a customized SAPS-II ICU mortality model changed over time, illustrating how temporal changes in clinical case mix and care processes can affect the reliability of severity-adjustment models [10].”
From Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems (2026) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 22 · Cited by 69
Research works printed but not linked: 1
- [14]Wheeler JW Advanced topics in statistical process control 2004 Knoxville SPC Press
- Authors
- Minne, L., Eslami, S., de Keizer, N., de Jonge, E., de Rooij, S. E., & Abu-Hanna, A.
- Affiliations
- Amsterdam UMC Location University of Amsterdam (NL)
- Corresponding author
- Minne, Lilian
- ORCID
- Eslami, Saeid
- Venue
- Intensive Care Medicine
- Published
- 2011-10-28
- Volume
- 38
- Issue
- 1
- Pages
- 40-46
- ISSN
- 0342-4642 (ISSN-L)
- 1432-1238
- PMC
- PMC3233667
- OpenAlex
- W2146297376
- Cited by
- 69
- References
- 22
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 6 · 2023: 3 · 2022: 7 · 2021: 6 · 2020: 4 · 2019: 11 · 2018: 1 · 2017: 4 · 2016: 4 · 2015: 5 · 2014: 4 · 2013: 5 · 2012: 6
- FWCI
- 3.84
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Intensive Care Medicine (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Topics
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Health Sciences › Health Professions › Health Information Management › Healthcare Quality and Management
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Palliative Care and End-of-Life Issues
- Keywords
- SAPS II, mortality prediction, quality of care evaluation, very elderly patients, intensive care unit, standardized mortality ratio, model discrimination, hospital performance comparison
- MeSH terms
- Aged, 80 and over, Female, Forecasting, Humans, Intensive Care Units, Male, Models, Theoretical, Netherlands, Prognosis, Prospective Studies, Quality Assurance, Health Care, Severity of Illness Index, Time Factors, Cohort Studies, Hospital Mortality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
