CoChat

Transferability and interpretability of the sepsis prediction models in the intensive care unit

Open access
Chen, Qiyu, Li, Ranran, Lin, ChihChe, Lai, Chiming, Chen, Dechang, Qu, Hongping, Huang, Yaling, Lu, Wenlian, Tang, Yaoqing, Li, Lei · 2022 · BMC Medical Informatics and Decision Making

Published version of a preprint

The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.

Abstract

Background: We aimed to develop an early warning system for real-time sepsis prediction in the ICU by machine learning methods, with tools for interpretative analysis of the predictions. In particular, we focus on the deployment of the system in a target medical center with small historical samples. Methods: Light Gradient Boosting Machine (LightGBM) and multilayer perceptron (MLP) were trained on Medical Information Mart for Intensive Care (MIMIC-III) dataset and then finetuned on the private Historical Database of local Ruijin Hospital (HDRJH) using transfer learning technique. The Shapley Additive Explanations (SHAP) analysis was employed to characterize the feature importance in the prediction inference. Ultimately, the performance of the sepsis prediction system was further evaluated in the real-world study in the ICU of the target Ruijin Hospital. Results: The datasets comprised 6891 patients from MIMIC-III, 453 from HDRJH, and 67 from Ruijin real-world data. The area under the receiver operating characteristic curves (AUCs) for LightGBM and MLP models derived from MIMIC-III were 0.98 − 0.98 and 0.95 − 0.96 respectively on MIMIC-III dataset, and, in comparison, 0.82 − 0.86 and 0.84 − 0.87 respectively on HDRJH, from 1 to 5 h preceding. After transfer learning and ensemble learning, the AUCs of the final ensemble model were enhanced to 0.94 − 0.94 on HDRJH and to 0.86 − 0.9 in the real-world study in the ICU of the target Ruijin Hospital. In addition, the SHAP analysis illustrated the importance of age, antibiotics, net balance, and ventilation for sepsis prediction, making the model interpretable. Conclusions: Our machine learning model allows accurate real-time prediction of sepsis within 5-h preceding. Transfer learning can effectively improve the feasibility to deploy the prediction model in the target cohort, and ameliorate the model performance for external validation. SHAP analysis indicates that the role of antibiotic usage and fluid management needs further investigation. We argue that our system and methodology have the potential to improve ICU management by helping medical practitioners identify at-sepsis-risk patients and prepare for timely diagnosis and intervention. Trial registration : NCT05088850 (retrospectively registered).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12911-022-02090-3

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 The results of models trained on sampled HDRJH dataset. The training set of HDRJH was sampled at different ratios to simulate a medical center with fewer records. These trained models were tested on the common MIMIC-III test set (top) and the HDRJH test set (bottom), respectively. The shaded part in the figure represents the 95% confidence interval. These models predict sepsis in 4-h preceding and the complete results can be found in Additional file 1: Figs. S5–S8
Fig. 1 The results of models trained on sampled HDRJH dataset. The training set of HDRJH was sampled at different ratios to simulate a medical center with fewer records. These trained models were tested on the common MIMIC-III test set (top) and the HDRJH test set (bottom), respectively. The shaded part in the figure represents the 95% confidence interval. These models predict sepsis in 4-h preceding and the complete results can be found in Additional file 1: Figs. S5–S8CC BY · PubMed Central
Fig. 2 Feature importance of LightGBM model in the sepsis early warning module. The model predicted sepsis in 4-h preceding and the complete results can be found in Additional file 1: Fig. S10
Fig. 2 Feature importance of LightGBM model in the sepsis early warning module. The model predicted sepsis in 4-h preceding and the complete results can be found in Additional file 1: Fig. S10CC BY · PubMed Central
Fig. 3 Some illustrative examples of prediction. Each subplot described the confidence index for multiple models (Y-axis) at the indicated time (X-axis). (i) The condition of the patient aggravated in the early morning, with multiple organ dysfunction, and the patient was diagnosed with sepsis at noon. Our model prediction exceeded the warning threshold of 0.7 for the prediction at 9:00 AM. (ii) Despite the high SOFA score (7.0), there was no evidence of ΔSOFA ≥ 2 within 72 h. Consistently, the predictions were all lower than the threshold. (iii) Although the patient’s SOFA score was stable at 6.0, our model made incorrect predictions of sepsis. (iv) The SOFA score showed an increase from 6.0 to 9.0 at 06:00 PM. In combination with evidence of infection, the patient was diagnosed with sepsis. However, the prediction was below the warning threshold
Fig. 3 Some illustrative examples of prediction. Each subplot described the confidence index for multiple models (Y-axis) at the indicated time (X-axis). (i) The condition of the patient aggravated in the early morning, with multiple organ dysfunction, and the patient was diagnosed with sepsis at noon. Our model prediction exceeded the warning threshold of 0.7 for the prediction at 9:00 AM. (ii) Despite the high SOFA score (7.0), there was no evidence of ΔSOFA ≥ 2 within 72 h. Consistently, the predictions were all lower than the threshold. (iii) Although the patient’s SOFA score was stable at 6.0, our model made incorrect predictions of sepsis. (iv) The SOFA score showed an increase from 6.0 to 9.0 at 06:00 PM. In combination with evidence of infection, the patient was diagnosed with sepsis. However, the prediction was below the warning thresholdCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 1

  1. Moor M, Horn M, Rieck B, Roqueiro D, Borgwardt K. Early recognition of sepsis with Gaussian process temporal convolutional networks and dynamic time warping. Mach Learn Healthc Conf PMLR. 2019;106:2–26.

Cited by

See all 28

Metadata

Authors
Chen, Q., Li, R., Lin, C., Lai, C., Chen, D., Qu, H., Huang, Y., Lu, W., Tang, Y., & Li, L.
Affiliations
  • Fudan University (CN)
  • Shanghai Jiao Tong University (CN)
  • Shanghai Electric (China) (CN)
Corresponding author
Li, Lei
Venue
BMC Medical Informatics and Decision Making
Published
2022-12-29
Volume
22
Issue
1
Pages
343-343
ISSN
1472-6947 (ISSN-L)
OpenAlex
W4313242224
Cited by
28
References
47
Citations by year
2026: 9 · 2025: 7 · 2024: 9 · 2023: 1 · 2022: 2
FWCI
3.43
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
  • medRxiv (acceptedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • the Science and Technology Commission of Shanghai Municipality (19JC1420101)
  • Shanghai Municipal Science and Technology Major Project, the ZHANGJIANG LAB (2018SHZDZX01)
Topics
  • Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
sepsis prediction, intensive care unit, early warning systems, transfer learning, model interpretability, SHAP, LightGBM, multilayer perceptron, ensemble learning, MIMIC-III, antibiotic use, fluid management, real-time prediction, external validation, net balance, mechanical ventilation, age as risk factor
MeSH terms
Critical Care, Humans, Intensive Care Units, Databases, Factual, Sepsis, Area Under Curve

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.