The DOI matches its Crossref record; 86 of the 91 references it cites were found in a registry or an index; 23 of the 35 sources its claims cite were checked for support.
Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions
Open accessAbstract
The application of machine learning (ML) models in healthcare management offers high potential. In particular, resource allocation and operational decision-making in intensive care units (ICUs) can benefit from ML predictions, leading to improvements in patient outcomes and operational efficiency. However, the generalizability of these models across diverse hospital settings with potentially different patient populations remains a critical challenge. This study examines the generalizability of ML-based ICU outcome prediction models built using external data. We utilize data from two sources: a European University Hospital (EUH) dataset from Universitätsklinikum Carl Gustav Carus Dresden, Germany and the Medical Information Mart for Intensive Care (MIMIC)-IV database, representing different healthcare systems and patient populations. Our approach evaluates multiple models of varying architectures and complexity across three common prediction tasks in ICU settings (mortality, length of stay, and readmission), analyzes the impact of data availability on model performance, and applies interpretability techniques to identify features and scenarios where models succeed or fail in new environments. We found that locally trained models generally outperform those using external data when sufficient local data is available. Low and medium complexity models, such as generalized additive models, demonstrate significantly superior generalizability compared to high complexity models and require substantially less local data for high-quality predictions, offering evidence-based guidance for healthcare managers dealing with limited data resources. Our results demonstrate how interpretability techniques can identify dataset differences that hinder generalizability, providing valuable insights for healthcare practitioners in implementing ML solutions across diverse hospitals. This research contributes to the development of more generalizable and interpretable ML models in healthcare.
Overview
CoChat Summary
## TL;DR
This paper compares six ML models across three ICU prediction tasks using two hospital datasets and finds that simpler, interpretable models generalize better across hospitals than complex ones. It also shows that external data only helps when local data are very limited, and that interpretability can pinpoint features that undermine cross-hospital transfer.
## Key findings
- Across mortality, length of stay, and readmission, local models outperformed external models, and the paper reports that model complexity was inversely related to cross-hospital generalizability.
- For generalizability loss, logistic regression (LR) was the most stable model across all tasks, with ranks of 1.5, 1.2, and 1.4; multilayer perceptrons (MLP) ranked worst with 5.3, 5.5, and 4.6.
- In comparative generalizability, only LR and EBM achieved best external performance (Δcomp = 0.0) across all tasks and directions.
- Data-availability thresholds for when local-only models surpassed models supplemented with external MIMIC-IV data were reported as about 500 EUH samples for LR, 1250 for EBM, and 4250 for XGB.
- Interpretability analysis found some features were consistent across hospitals (e.g., heart rate and partial pressure of CO2), while others differed substantially (e.g., fraction of inspired O2 and albumin), helping explain transfer failures.
## Methods
- The study used a bidirectional cross-hospital design comparing a European University Hospital (EUH) dataset (25,062 patients) with MIMIC-IV (94,458 de-identified records).
- It evaluated three binary ICU prediction tasks: mortality (first 24 hours), length of stay (≥7 days vs shorter stays, first 24 hours), and readmission within 72 hours after discharge (using the last 24 hours before discharge; deaths in the initial stay excluded).
- Six models were tested across three complexity levels: low (LR, decision trees), medium (EBM, IGANN), and high (XGB, MLP), with AUROC as the main metric; statistical comparisons used Friedman and Nemenyi tests, and interpretability used direct model inspection or SHAP for complex models.
## Limitations
- The paper states that the analysis is limited to two datasets representing European and North American healthcare systems, which may limit applicability to other regions and care settings.
- It states that only structured tabular data were modeled; the text notes that real-world ICU decision-making is multimodal and temporal, so the findings may not extend to such settings.
- Evidence made evident by the text: the interpretability analysis is based mainly on mortality examples, with the paper saying length of stay and readmission are comparable but less detailed in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





















Verification
Claims the paper makes
15 Valid · 5 Not valid · 8 Not checked · 123 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This quality becomes crucial when predictive models are deployed across different hospitals or healthcare systems, where variations in patient demographics, clinical practices, and data collection procedures can lead to inaccurate predictions [17, 18].”
§ Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions > Introduction
What the sources say
- Not checked — no full text of the sourcecites [17] Justice AC, Covinsky KE, Berlin JA (1999) Assessing the generalizability of prognostic information. Ann Intern Med 130(6):515–524. https://doi.org/10.7326/0003-4819-130-6-199903160-00016 source
- Partly supportedcites [18] Finlayson SG, Subbaswamy A, Singh K, Bowers J, Kupke A, Zittrain J, Kohane IS, Saria S (2021) The clinician and dataset shift in artificial intelligence. N Engl J Med 385(3):283–286. https://doi.org/10.1056/NEJMc2104626 sourceExperimental — accuracy not yet measured
In “To the Editor” (page unavailable), the paper says dataset shift can make a clinical AI system underperform when deployment data differ from development data. It identifies changing patient demographics and clinical behavior or practice as potential causes, and describes a hospital sepsis model producing spurious alerts. Our copy does not specifically identify differences in data-collection procedures as a cause. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As a remedy, many healthcare institutions, especially those smaller or with constrained data and computational resources, increasingly seek to leverage models developed at facilities with greater data resources [15].”
§ Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions > Introduction
What the sources say
- Partly supportedcites [15] Alberto IRI, Alberto NRI, Ghosh AK, Jain B, Jayakumar S, Martinez-Martin N, McCague N, Moukheiber D, Moukheiber L, Moukheiber M, Moukheiber S, Yaghy A, Zhang A, Celi LA (2023) The impact of commercial health datasets on medical research and health-care algorithms. Lancet Digit Health 5(5):288–294. https://doi.org/10.1016/S2589-7500(23)00025-0 sourceExperimental — accuracy not yet measured
The section on reproducibility and generalisability reports that the Epic Sepsis Model, developed using data from three hospital systems, was implemented at hundreds of US hospitals. It does not say that smaller or resource-constrained institutions increasingly seek such models, or that resource constraints motivate their adoption; it attributes this model’s widespread use to ease of EHR integration. (LLM, conf=0.80)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Achieving high predictive performance in healthcare often requires ML models with high flexibility to capture complex patterns in high-dimensional data [41].”
§ Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions > Research background > Model complexity and interpretability
What the sources say
- Partly supportedcites [41] Bohr A, Memarzadeh K (2020) The rise of artificial intelligence in healthcare applications. In: Artificial intelligence in healthcare. Academic Press, imprint of Elsevier, pp 25–60. https://doi.org/10.1016/B978-0-12-818438-7.00002-2 sourceExperimental — accuracy not yet measured
The Precision medicine > Drug discovery and development section (page unavailable) describes ML prediction from large biomedical datasets and says deep-learning models often have many parameters. The Deep learning and medical image recognition section (page unavailable) says CNN layers learn image features and adjust weights to improve performance. Neither passage establishes that high predictive performance in healthcare often requires highly flexible models to capture complex patterns in high-dimensional data. (LLM, conf=0.83)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, post-hoc explanations are approximations to the underlying model rather than exact representations [65].”
§ Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions > Research background > Model complexity and interpretability
What the sources say
- Supportedcites [65] Rudin C (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mac Intell 1(5):206–215. https://doi.org/10.1038/s42256-019-0048-x source
In “Key Issues with Explainable ML,” the paper says explanations cannot have perfect fidelity with the original model and explicitly calls them approximations to black-box model predictions. This supports the citing sentence’s distinction between an approximation and an exact representation. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“a pattern that generally suggests strong potential for cross-hospital generalizability [72]”
§ Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions > Methodology > Data sources and preprocessing
What the sources say
- Partly supportedcites [72] Sharafutdinov K, Bhat JS, Fritsch SJ, Nikulina K, Samadi ME, Polzin R, Mayer H, Marx G, Bickenbach J, Schuppert A (2022) Application of convex hull analysis for the evaluation of data heterogeneity between patient populations of different origin and implications of hospital bias in downstream machine-learning-based data processing: a comparison of 4 critical-care patient datasets. Front Big Data 5:603429. https://doi.org/10.3389/fdata.2022.603429 sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says convex-hull overlap can provide a first estimate of whether populations are similar enough for model generalization. But it also says that even extensive overlap may conceal distribution differences that impair generalization; overlap alone does not establish the claimed strong potential. (LLM, conf=0.88)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
92 more claims · 1 with their cited sources checked
Create a free account to see all 151 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 19
- It may state two separate claims as one. 40
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
ASSESSMENT OF COMA AND IMPAIRED CONSCIOUSNESS
Teasdale, G., & Jennett, B. (1974) · The Lancet · doi:10.1016/s0140-6736(74)91639-0 · Cited by 13,215 · cited 1×
“For LR and EBM, the plots directly reveal the models’ decision logic, whereas for XGB, we use SHAP values computed for individual patients in the EUH test set.”
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 3×
“Interpretable ML refers to ML models and techniques that provide explanations for their predic - tions in terms understandable to human users, enabling them to comprehend, trust, and effectively manage the prediction process [65 ].”
The potential for artificial intelligence in healthcare
Davenport, T., & Kalakota, R. (2019) · Future Healthcare Journal · doi:10.7861/futurehosp.6-2-94 · Cited by 3,800
MIMIC-IV, a freely accessible electronic health record dataset
Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. (2023) · Scientific Data · doi:10.1038/s41597-022-01899-x · Cited by 3,401
Zimmerman, J. E., Kramer, A. A., McNair, D. S., & Malila, F. M. (2006) · Critical Care Medicine · doi:10.1097/01.ccm.0000215112.84523.f0 · Cited by 1,764 · cited 2×
“This consistency aligns with established mortality scoring systems used in intensive care, such as APACHE-IV and SAPS III [ 50], and suggests that heart rate contributes positively to model generalizability across hospitals.”
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
The rise of artificial intelligence in healthcare applications
Bohr, A., & Memarzadeh, K. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00002-2 · Cited by 1,636 · cited 2×
“ML has shown immense promise in healthcare, offering powerful tools to support clinical decision-making by creating rapid and accurate predictions based on complex, high-dimensional data [41–44 ].”
Moreno, R. P., Metnitz, P. G. H., Almeida, E., Jordan, B., Bauer, P., Campos, R. A., Iapichino, G., Edbrooke, D., Capuzzo, M., Le Gall, J.-R., et al. (2005) · Intensive Care Medicine · doi:10.1007/s00134-005-2763-5 · Cited by 1,413
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160
Research works printed but not linked: 5
- Zschech P, Weinzierl S, Hambauer N, Zilker S, Kraus M (2022) GAM(e) changer or not? An evaluation of interpretable machine learning models based on additive model constraints. In: Proceedings of the 30th European conference on information systems (ECIS)
- Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. In: Proceedings of the 31st international conference on neural information processing systems, pp 4768–4777
- McDermott MBA, Zhang H, Hansen LH, Angelotti G, Gallifant J (2024) A closer look at AUROC and AUPRC under class imbalance. In: Proceedings of the 38th international conference on neural information processing systems
- Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30
- Loscalzo J, Fauci AS, Kasper DL, Hauser SL, Longo DL, Jameson JL (eds) (2022) Harrison’s principles of internal medicine, 21st edn. McGraw Hill
Metadata
- Authors
- Bohlen, L., Rosenberger, J., Hambauer, N., Zähringer, D., Franz, V., Zschech, P., & Kraus, M.
- Affiliations
- Technische Universität Dresden (DE)
- University of Regensburg (DE)
- University Hospital Carl Gustav Carus (DE)
- Corresponding author
- Bohlen, Lasse
- Rosenberger, Julian
- Hambauer, Nico
- Zähringer, Daniel
- Franz, Volkmar
- Zschech, Patrick
- Kraus, Mathias
- Venue
- Health Care Management Science
- Published
- 2026-06
- Volume
- 29
- Issue
- 2
- ISSN
- 1386-9620 (ISSN-L)
- 1572-9389
- PMC
- PMC13253713
- OpenAlex
- W7164133478
- Cited by
- 0
- References
- 91
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 9 d)
- Locations
- Health Care Management Science (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Econstor (Econstor) (submittedVersion, cc-by)
- Funders
- Technische Universität Dresden
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Keywords
- model generalization, interpretability, intensive care unit, mortality prediction, length of hospital stay, hospital readmission, external validation, data availability, model complexity, generalized additive models, MIMIC-IV, hospital resource allocation, health management
- MeSH terms
- Machine Learning, Predictive Learning Models, Prediction Algorithms, Germany, Humans, Intensive Care Units, Length of Stay, Patient Readmission, Hospital Mortality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions. https://app.cochat.ai/papers/10.1007/s10729-026-09760-y
- The paper
- Bohlen, L., Rosenberger, J., Hambauer, N., Zähringer, D., Franz, V., Zschech, P., & Kraus, M. (2026). Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions. Health Care Management Science. https://doi.org/10.1007/s10729-026-09760-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.