Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 86 of the 91 references it cites were found in a registry or an index; 23 of the 35 sources its claims cite were checked for support.

Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions

Open access

Abstract

The application of machine learning (ML) models in healthcare management offers high potential. In particular, resource allocation and operational decision-making in intensive care units (ICUs) can benefit from ML predictions, leading to improvements in patient outcomes and operational efficiency. However, the generalizability of these models across diverse hospital settings with potentially different patient populations remains a critical challenge. This study examines the generalizability of ML-based ICU outcome prediction models built using external data. We utilize data from two sources: a European University Hospital (EUH) dataset from Universitätsklinikum Carl Gustav Carus Dresden, Germany and the Medical Information Mart for Intensive Care (MIMIC)-IV database, representing different healthcare systems and patient populations. Our approach evaluates multiple models of varying architectures and complexity across three common prediction tasks in ICU settings (mortality, length of stay, and readmission), analyzes the impact of data availability on model performance, and applies interpretability techniques to identify features and scenarios where models succeed or fail in new environments. We found that locally trained models generally outperform those using external data when sufficient local data is available. Low and medium complexity models, such as generalized additive models, demonstrate significantly superior generalizability compared to high complexity models and require substantially less local data for high-quality predictions, offering evidence-based guidance for healthcare managers dealing with limited data resources. Our results demonstrate how interpretability techniques can identify dataset differences that hinder generalizability, providing valuable insights for healthcare practitioners in implementing ML solutions across diverse hospitals. This research contributes to the development of more generalizable and interpretable ML models in healthcare.

Overview

CoChat Summary

## TL;DR

This paper compares six ML models across three ICU prediction tasks using two hospital datasets and finds that simpler, interpretable models generalize better across hospitals than complex ones. It also shows that external data only helps when local data are very limited, and that interpretability can pinpoint features that undermine cross-hospital transfer.

## Key findings

  • Across mortality, length of stay, and readmission, local models outperformed external models, and the paper reports that model complexity was inversely related to cross-hospital generalizability.
  • For generalizability loss, logistic regression (LR) was the most stable model across all tasks, with ranks of 1.5, 1.2, and 1.4; multilayer perceptrons (MLP) ranked worst with 5.3, 5.5, and 4.6.
  • In comparative generalizability, only LR and EBM achieved best external performance (Δcomp = 0.0) across all tasks and directions.
  • Data-availability thresholds for when local-only models surpassed models supplemented with external MIMIC-IV data were reported as about 500 EUH samples for LR, 1250 for EBM, and 4250 for XGB.
  • Interpretability analysis found some features were consistent across hospitals (e.g., heart rate and partial pressure of CO2), while others differed substantially (e.g., fraction of inspired O2 and albumin), helping explain transfer failures.

## Methods

  • The study used a bidirectional cross-hospital design comparing a European University Hospital (EUH) dataset (25,062 patients) with MIMIC-IV (94,458 de-identified records).
  • It evaluated three binary ICU prediction tasks: mortality (first 24 hours), length of stay (≥7 days vs shorter stays, first 24 hours), and readmission within 72 hours after discharge (using the last 24 hours before discharge; deaths in the initial stay excluded).
  • Six models were tested across three complexity levels: low (LR, decision trees), medium (EBM, IGANN), and high (XGB, MLP), with AUROC as the main metric; statistical comparisons used Friedman and Nemenyi tests, and interpretability used direct model inspection or SHAP for complex models.

## Limitations

  • The paper states that the analysis is limited to two datasets representing European and North American healthcare systems, which may limit applicability to other regions and care settings.
  • It states that only structured tabular data were modeled; the text notes that real-world ICU decision-making is multimodal and temporal, so the findings may not extend to such settings.
  • Evidence made evident by the text: the interpretability analysis is based mainly on mortality examples, with the paper saying length of stay and readmission are comparable but less detailed in the main text.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10729-026-09760-y

Cites

86 of 86 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Low, medium, and high complexity models from left to right. An exemplary feature "Mean Body Temperature" has different effects on the model output for mortality prediction depending on the ML model complexity
Fig. 1 Low, medium, and high complexity models from left to right. An exemplary feature "Mean Body Temperature" has different effects on the model output for mortality prediction depending on the ML model complexityCC BY · PubMed Central
Fig. 2 Different evaluation and training settings are indicated by color: white for models trained and tested on local data and grey for models trained on local data and tested on external data
Fig. 2 Different evaluation and training settings are indicated by color: white for models trained and tested on local data and grey for models trained on local data and tested on external dataCC BY · PubMed Central
Fig. 3 Overview of the training and evaluation strategy. Hyperparameters are optimized using 5-fold cross-validation on the training data, selected based on the average validation AUROC. Using these hyperparameters, five models are created on each training fold and individually evaluated on all five test folds, resulting in 25 evaluations per model and scenario
Fig. 3 Overview of the training and evaluation strategy. Hyperparameters are optimized using 5-fold cross-validation on the training data, selected based on the average validation AUROC. Using these hyperparameters, five models are created on each training fold and individually evaluated on all five test folds, resulting in 25 evaluations per model and scenarioCC BY · PubMed Central
Fig. 4 Critical difference diagrams for generalizability assessment. Lower average ranks indicate better performance of externally trained model. Left column: Model-specific generalizability loss (\Delta _{\text {spec}}). Right column: Comparative generalizability loss (\Delta _{\text {comp}}). Non-significant differences are connected with horizontal lines (^*p> 0.05)
Fig. 4 Critical difference diagrams for generalizability assessment. Lower average ranks indicate better performance of externally trained model. Left column: Model-specific generalizability loss (\Delta _{\text {spec}}). Right column: Comparative generalizability loss (\Delta _{\text {comp}}). Non-significant differences are connected with horizontal lines (^*p> 0.05)CC BY · PubMed Central
Fig. 5 Model performance (AUROC) for mortality prediction over EUH sample size for LR, EBM, and XGB. Left: Models trained on EUH data only. Right: Models trained on same EUH samples supplemented with full MIMIC-IV dataset (additional 55832 samples). Shaded areas represent standard deviation
Fig. 5 Model performance (AUROC) for mortality prediction over EUH sample size for LR, EBM, and XGB. Left: Models trained on EUH data only. Right: Models trained on same EUH samples supplemented with full MIMIC-IV dataset (additional 55832 samples). Shaded areas represent standard deviationCC BY · PubMed Central
Fig. 6 Break-even points for LR, EBM, and XGB models. Solid lines represent models trained on EUH data only; dashed lines represent models trained on EUH data supplemented with MIMIC-IV data. Vertical lines indicate break-even points identified through systematic sampling from 250 to 6000 EUH samples
Fig. 6 Break-even points for LR, EBM, and XGB models. Solid lines represent models trained on EUH data only; dashed lines represent models trained on EUH data supplemented with MIMIC-IV data. Vertical lines indicate break-even points identified through systematic sampling from 250 to 6000 EUH samplesCC BY · PubMed Central
Fig. 7 Comparison of feature-mortality relationships learned by LR, EBM, and XGB on EUH (blue) and MIMIC-IV (red) datasets. Histograms show feature distributions in each dataset, with density shown on the y-axis. For LR and EBM, plots directly reveal model decision logic. For XGB, SHAP values for EUH test data are shown
Fig. 7 Comparison of feature-mortality relationships learned by LR, EBM, and XGB on EUH (blue) and MIMIC-IV (red) datasets. Histograms show feature distributions in each dataset, with density shown on the y-axis. For LR and EBM, plots directly reveal model decision logic. For XGB, SHAP values for EUH test data are shownCC BY · PubMed Central
Fig. 8 Convex hull for the EUH data using the full feature set. The analysis uses and shows the first two components from a PCA using only the training statistics from EUH to impute, encode, scale and decompose EUH (training / constructing convex hull) and MIMIC (test) into two components. MIMIC samples inside the hull of EUH data are marked with triangles and outside the hull marked with crosses
Fig. 8 Convex hull for the EUH data using the full feature set. The analysis uses and shows the first two components from a PCA using only the training statistics from EUH to impute, encode, scale and decompose EUH (training / constructing convex hull) and MIMIC (test) into two components. MIMIC samples inside the hull of EUH data are marked with triangles and outside the hull marked with crossesCC BY · PubMed Central
Fig. 9 Convex hull for the MIMIC data using the full feature set. The analysis uses and shows the first two components from a PCA using only the training statistics from MIMIC to impute, encode, scale and decompose MIMIC (training / constructing convex hull) and EUH (test) into two components. EUH samples inside the hull of MIMIC data are marked with triangles and outside the hull marked with crosses
Fig. 9 Convex hull for the MIMIC data using the full feature set. The analysis uses and shows the first two components from a PCA using only the training statistics from MIMIC to impute, encode, scale and decompose MIMIC (training / constructing convex hull) and EUH (test) into two components. EUH samples inside the hull of MIMIC data are marked with triangles and outside the hull marked with crossesCC BY · PubMed Central
Fig. 10 Convex hull for the EUH data using two features only: Fraction of inspired O2 (FiO2) and albumin. MIMIC samples inside the hull of EUH data are marked with triangles and outside the hull marked with crosses
Fig. 10 Convex hull for the EUH data using two features only: Fraction of inspired O2 (FiO2) and albumin. MIMIC samples inside the hull of EUH data are marked with triangles and outside the hull marked with crossesCC BY · PubMed Central
Fig. 11 Convex hull for the MIMIC data using two features only: Fraction of inspired O2 (FiO2) and albumin. EUH samples inside the hull of MIMIC are marked with triangles and outside the hull marked with crosses
Fig. 11 Convex hull for the MIMIC data using two features only: Fraction of inspired O2 (FiO2) and albumin. EUH samples inside the hull of MIMIC are marked with triangles and outside the hull marked with crossesCC BY · PubMed Central
Fig. 12 Model performance (AUROC) for mortality prediction over EUH sample size for LR, EBM, and XGB. Left: Models trained on EUH data only. Right: Models trained on same EUH samples supplemented with full MIMIC-IV dataset (additional 55832 samples). Shaded areas represent standard deviation
Fig. 12 Model performance (AUROC) for mortality prediction over EUH sample size for LR, EBM, and XGB. Left: Models trained on EUH data only. Right: Models trained on same EUH samples supplemented with full MIMIC-IV dataset (additional 55832 samples). Shaded areas represent standard deviationCC BY · PubMed Central
Fig. 13 Model performance (AUROC) for length of stay prediction over EUH sample size for LR, EBM, and XGB. Left: Models trained on EUH data only. Right: Models trained on same EUH samples supplemented with full MIMIC-IV dataset (additional 55832 samples). Shaded areas represent standard deviation
Fig. 13 Model performance (AUROC) for length of stay prediction over EUH sample size for LR, EBM, and XGB. Left: Models trained on EUH data only. Right: Models trained on same EUH samples supplemented with full MIMIC-IV dataset (additional 55832 samples). Shaded areas represent standard deviationCC BY · PubMed Central
Fig. 14 Model performance (AUROC) for readmission prediction over EUH sample size for LR, EBM, and XGB. Left: Models trained on EUH data only. Right: Models trained on same EUH samples supplemented with full MIMIC-IV dataset (additional 55832 samples). Shaded areas represent standard deviation
Fig. 14 Model performance (AUROC) for readmission prediction over EUH sample size for LR, EBM, and XGB. Left: Models trained on EUH data only. Right: Models trained on same EUH samples supplemented with full MIMIC-IV dataset (additional 55832 samples). Shaded areas represent standard deviationCC BY · PubMed Central
Fig. 15 Break-even points for LR, EBM, and XGB models. Solid lines represent models trained on EUH data only; dashed lines represent models trained on EUH data supplemented with MIMIC-IV data. The x-axis shows the number of EUH samples used, while the y-axis displays AUROC values for length of stay predictions. Vertical lines indicate the break-even points
Fig. 15 Break-even points for LR, EBM, and XGB models. Solid lines represent models trained on EUH data only; dashed lines represent models trained on EUH data supplemented with MIMIC-IV data. The x-axis shows the number of EUH samples used, while the y-axis displays AUROC values for length of stay predictions. Vertical lines indicate the break-even pointsCC BY · PubMed Central
Fig. 16 Break-even points for LR, EBM, and XGB models. Solid lines represent models trained on EUH data only; dashed lines represent models trained on EUH data supplemented with MIMIC-IV data. The x-axis shows the number of EUH samples used, while the y-axis displays AUROC values for readmission predictions. Vertical lines indicate the break-even points
Fig. 16 Break-even points for LR, EBM, and XGB models. Solid lines represent models trained on EUH data only; dashed lines represent models trained on EUH data supplemented with MIMIC-IV data. The x-axis shows the number of EUH samples used, while the y-axis displays AUROC values for readmission predictions. Vertical lines indicate the break-even pointsCC BY · PubMed Central
Fig. 17 Length of stay model insights
Fig. 17 Length of stay model insightsCC BY · PubMed Central
Fig. 18 Readmission model insights
Fig. 18 Readmission model insightsCC BY · PubMed Central
Fig. 19 Comparison of feature-mortality relationships learned by LR, EBM, and XGB trained on the combined MIMIC-IV and EUH dataset. Plots demonstrate consistent trends across model types for each feature, indicating robust underlying relationships in the data
Fig. 19 Comparison of feature-mortality relationships learned by LR, EBM, and XGB trained on the combined MIMIC-IV and EUH dataset. Plots demonstrate consistent trends across model types for each feature, indicating robust underlying relationships in the dataCC BY · PubMed Central
Fig. 20 Boxplots for regret r for the three instance sizes
Fig. 20 Boxplots for regret r for the three instance sizesCC BY · PubMed Central
Fig. 21 Kernel density estimate plot illustrating the distribution of A_{max} and A_{min} for \mathrm{XGB\_ext} and \mathrm{XGB\_loc} as acuity estimate input models
Fig. 21 Kernel density estimate plot illustrating the distribution of A_{max} and A_{min} for \mathrm{XGB\_ext} and \mathrm{XGB\_loc} as acuity estimate input modelsCC BY · PubMed Central

Verification

Claims the paper makes

15 Valid · 5 Not valid · 8 Not checked · 123 Own results and interpretations

92 more claims · 1 with their cited sources checked

Create a free account to see all 151 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 19
  • It may state two separate claims as one. 40

Create a free account to see these claims.

Citations

Cites

Show all 91 references

Research works printed but not linked: 5

  1. Zschech P, Weinzierl S, Hambauer N, Zilker S, Kraus M (2022) GAM(e) changer or not? An evaluation of interpretable machine learning models based on additive model constraints. In: Proceedings of the 30th European conference on information systems (ECIS)
  2. Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. In: Proceedings of the 31st international conference on neural information processing systems, pp 4768–4777
  3. McDermott MBA, Zhang H, Hansen LH, Angelotti G, Gallifant J (2024) A closer look at AUROC and AUPRC under class imbalance. In: Proceedings of the 38th international conference on neural information processing systems
  4. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30
  5. Loscalzo J, Fauci AS, Kasper DL, Hauser SL, Longo DL, Jameson JL (eds) (2022) Harrison’s principles of internal medicine, 21st edn. McGraw Hill

Metadata

Authors
Bohlen, L., Rosenberger, J., Hambauer, N., Zähringer, D., Franz, V., Zschech, P., & Kraus, M.
Affiliations
  • Technische Universität Dresden (DE)
  • University of Regensburg (DE)
  • University Hospital Carl Gustav Carus (DE)
Corresponding author
  • Bohlen, Lasse
  • Rosenberger, Julian
  • Hambauer, Nico
  • Zähringer, Daniel
  • Franz, Volkmar
  • Zschech, Patrick
  • Kraus, Mathias
Venue
Health Care Management Science
Published
2026-06
Volume
29
Issue
2
ISSN
  • 1386-9620 (ISSN-L)
  • 1572-9389
OpenAlex
W7164133478
Cited by
0
References
91
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Health Care Management Science (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • Econstor (Econstor) (submittedVersion, cc-by)
Funders
Technische Universität Dresden
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
Keywords
model generalization, interpretability, intensive care unit, mortality prediction, length of hospital stay, hospital readmission, external validation, data availability, model complexity, generalized additive models, MIMIC-IV, hospital resource allocation, health management
MeSH terms
Machine Learning, Predictive Learning Models, Prediction Algorithms, Germany, Humans, Intensive Care Units, Length of Stay, Patient Readmission, Hospital Mortality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions. https://app.cochat.ai/papers/10.1007/s10729-026-09760-y
The paper
Bohlen, L., Rosenberger, J., Hambauer, N., Zähringer, D., Franz, V., Zschech, P., & Kraus, M. (2026). Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions. Health Care Management Science. https://doi.org/10.1007/s10729-026-09760-y

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.