CoChat

Prediction model of in-hospital mortality in intensive care unit patients with heart failure: machine learning-based, retrospective analysis of the MIMIC-III database

Open access
Li, Fuhai, Xin, Hui, Zhang, Jidong, Fu, Mingqiang, Zhou, Jingmin, Lian, Zhexun · 2021 · BMJ Open

The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.

Abstract

Objective: The predictors of in-hospital mortality for intensive care units (ICUs)-admitted heart failure (HF) patients remain poorly characterised. We aimed to develop and validate a prediction model for all-cause in-hospital mortality among ICU-admitted HF patients. Design: A retrospective cohort study. Setting and participants: Data were extracted from the Medical Information Mart for Intensive Care (MIMIC-III) database. Data on 1177 heart failure patients were analysed. Methods: Patients meeting the inclusion criteria were identified from the MIMIC-III database and randomly divided into derivation (n=825, 70%) and a validation (n=352, 30%) group. Independent risk factors for in-hospital mortality were screened using the extreme gradient boosting (XGBoost) and the least absolute shrinkage and selection operator (LASSO) regression models in the derivation sample. Multivariate logistic regression analysis was used to build prediction models in derivation group, and then validated in validation cohort. Discrimination, calibration and clinical usefulness of the predicting model were assessed using the C-index, calibration plot and decision curve analysis. After pairwise comparison, the best performing model was chosen to build a nomogram according to the regression coefficients. Results: Among the 1177 admissions, in-hospital mortality was 13.52%. In both groups, the XGBoost, LASSO regression and Get With the Guidelines-Heart Failure (GWTG-HF) risk score models showed acceptable discrimination. The XGBoost and LASSO regression models also showed good calibration. In pairwise comparison, the prediction effectiveness was higher with the XGBoost and LASSO regression models than with the GWTG-HF risk score model (p<0.05). The XGBoost model was chosen as our final model for its more concise and wider net benefit threshold probability range and was presented as the nomogram. Conclusions: Our nomogram enabled good prediction of in-hospital mortality in ICU-admitted HF patients, which may help clinical decision-making for such patients.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjopen-2020-044779

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 (A) Flowchart of patient selection (n=1177). (B) Model development flowchart. ICD-9, 9th revision of the International Classification of Diseases; ICU, intensive care unit; LASSO, least absolute shrinkage and selection operator; MIMIC-III, Medical Information Mart for Intensive Care III; NT-proBNP, N-terminal pro-brain natriuretic peptide; XGBoost, extreme gradient boosting.
Figure 1 (A) Flowchart of patient selection (n=1177). (B) Model development flowchart. ICD-9, 9th revision of the International Classification of Diseases; ICU, intensive care unit; LASSO, least absolute shrinkage and selection operator; MIMIC-III, Medical Information Mart for Intensive Care III; NT-proBNP, N-terminal pro-brain natriuretic peptide; XGBoost, extreme gradient boosting.CC BY-NC · PubMed Central
Figure 2 Demographic and clinical feature selection using the least absolute shrinkage and selection operator (LASSO) binary logistic regression model. (A) Tuning parameter (λ) selection in the LASSO model used 10-fold cross-validation via minimum criteria. The partial likelihood deviance (binomial deviance) curve was plotted versus log(λ). Dotted vertical lines were drawn at the optimal values by using the minimum criteria and the one SE of the minimum criteria (the 1-SE criteria). λ value of 0.0192, with log (λ), −3.9545 was chosen (1-SE criteria) according to 10-fold cross-validation. (B) LASSO coefficient profiles of the 52 features. A coefficient profile plot was produced against the log (λ) sequence. The vertical line was drawn at the value selected using 10-fold cross-validation, where optimal resulted in 20 features with non-zero coefficients.
Figure 2 Demographic and clinical feature selection using the least absolute shrinkage and selection operator (LASSO) binary logistic regression model. (A) Tuning parameter (λ) selection in the LASSO model used 10-fold cross-validation via minimum criteria. The partial likelihood deviance (binomial deviance) curve was plotted versus log(λ). Dotted vertical lines were drawn at the optimal values by using the minimum criteria and the one SE of the minimum criteria (the 1-SE criteria). λ value of 0.0192, with log (λ), −3.9545 was chosen (1-SE criteria) according to 10-fold cross-validation. (B) LASSO coefficient profiles of the 52 features. A coefficient profile plot was produced against the log (λ) sequence. The vertical line was drawn at the value selected using 10-fold cross-validation, where optimal resulted in 20 features with non-zero coefficients.CC BY-NC · PubMed Central
Figure 3 Predictor variables selection. (A) Importance of the predictor variables selected by XGBoost algorithm scaled to a maximum of 1.0. (B) Predictor variables selected by LASSO. LASSO, least absolute shrinkage and selection operator; MCH, mean corpuscular haemoglobin; MCV, mean corpuscular volume; NT-proBNP, N-terminal pro-brain natriuretic peptide; PaCO2, partial pressure of carbon dioxide in the artery; RDW, red blood cell distribution width; SPO2, saturation pulse oxygen; XGBoost, extreme gradient boosting.
Figure 3 Predictor variables selection. (A) Importance of the predictor variables selected by XGBoost algorithm scaled to a maximum of 1.0. (B) Predictor variables selected by LASSO. LASSO, least absolute shrinkage and selection operator; MCH, mean corpuscular haemoglobin; MCV, mean corpuscular volume; NT-proBNP, N-terminal pro-brain natriuretic peptide; PaCO2, partial pressure of carbon dioxide in the artery; RDW, red blood cell distribution width; SPO2, saturation pulse oxygen; XGBoost, extreme gradient boosting.CC BY-NC · PubMed Central
Figure 4 Unadjusted relationship between anion gap (A), calcium (B), blood urea nitrogen (C), lactate (D), diastolic blood pressure (E), MCV (F) and the risk of in-hospital mortality by using Lowess smoothing technique. The y-axis represents the risk of in-hospital mortality. The x-axis spans the range of the variable of interest. MCV, mean corpuscular volume.
Figure 4 Unadjusted relationship between anion gap (A), calcium (B), blood urea nitrogen (C), lactate (D), diastolic blood pressure (E), MCV (F) and the risk of in-hospital mortality by using Lowess smoothing technique. The y-axis represents the risk of in-hospital mortality. The x-axis spans the range of the variable of interest. MCV, mean corpuscular volume.CC BY-NC · PubMed Central
Figure 5 The discrimination and calibration performance of XGBoost model. Plots (A) and (C) show the ROC curves of the XGBoost model in the derivation and validation groups, respectively (AUC=0.8515 versus 0.8029). The light yellow area represents the 95% CIs. 500 bootstrap resamples was used to calculate a relatively corrected AUC and 95% CI. The light yellow area represents the 95% CIs. Calibration curves of the XGBoost model in the derivation (B) and validation (D) groups. Calibration curves depicted the calibration of the XGBoost model in terms of the agreement between the predicted risk of in-hospital mortality and observed in-hospital mortality. The 45° red line represents a perfect prediction, and the green lines represent the predictive performance of the XGBoost model. The closer the green line fit is to the ideal line, the better the predictive accuracy of the XGBoost model is. The light yellow area represents the 95% CIs. AUC, area under the curve; ROC, receiver operating characteristic; XGBoost, extreme gradient boosting.
Figure 5 The discrimination and calibration performance of XGBoost model. Plots (A) and (C) show the ROC curves of the XGBoost model in the derivation and validation groups, respectively (AUC=0.8515 versus 0.8029). The light yellow area represents the 95% CIs. 500 bootstrap resamples was used to calculate a relatively corrected AUC and 95% CI. The light yellow area represents the 95% CIs. Calibration curves of the XGBoost model in the derivation (B) and validation (D) groups. Calibration curves depicted the calibration of the XGBoost model in terms of the agreement between the predicted risk of in-hospital mortality and observed in-hospital mortality. The 45° red line represents a perfect prediction, and the green lines represent the predictive performance of the XGBoost model. The closer the green line fit is to the ideal line, the better the predictive accuracy of the XGBoost model is. The light yellow area represents the 95% CIs. AUC, area under the curve; ROC, receiver operating characteristic; XGBoost, extreme gradient boosting.CC BY-NC · PubMed Central
Figure 6 The discrimination and calibration performance of the LASSO model. Plots (A) and (C) show the ROC curves of the LASSO model in the derivation and validation groups, respectively (AUC=0.8646 versus 0.8194). 500 bootstrap resamples used to calculate a relatively corrected AUC and 95% CI. The light yellow area represents the 95% CIs. Calibration curves of the LASSO model in the derivation (B) and validation (D) groups. The light yellow area represents the 95% CIs. AUC, area under the curve; LASSO, least absolute shrinkage and selection operator; ROC, receiver operating characteristic.
Figure 6 The discrimination and calibration performance of the LASSO model. Plots (A) and (C) show the ROC curves of the LASSO model in the derivation and validation groups, respectively (AUC=0.8646 versus 0.8194). 500 bootstrap resamples used to calculate a relatively corrected AUC and 95% CI. The light yellow area represents the 95% CIs. Calibration curves of the LASSO model in the derivation (B) and validation (D) groups. The light yellow area represents the 95% CIs. AUC, area under the curve; LASSO, least absolute shrinkage and selection operator; ROC, receiver operating characteristic.CC BY-NC · PubMed Central
Figure 7 The discrimination and calibration performance of GWTG-HF risk score model. Plots (A) and (C) show the ROC curves of the GWTG-HF risk score model in the derivation and validation groups, respectively (AUC=0.7856 versus 0.7510). 500 bootstrap resamples used to calculate a relatively corrected AUC and 95% CI. Calibration curves of the GWTG-HF risk score model in the derivation (B) and validation (D) groups. The area between two green dotted lines represents the 95% CIs. AUC, area under the curve; GWTG-HF, Get With The Guidelines-Heart Failure; ROC, receiver operating characteristic.
Figure 7 The discrimination and calibration performance of GWTG-HF risk score model. Plots (A) and (C) show the ROC curves of the GWTG-HF risk score model in the derivation and validation groups, respectively (AUC=0.7856 versus 0.7510). 500 bootstrap resamples used to calculate a relatively corrected AUC and 95% CI. Calibration curves of the GWTG-HF risk score model in the derivation (B) and validation (D) groups. The area between two green dotted lines represents the 95% CIs. AUC, area under the curve; GWTG-HF, Get With The Guidelines-Heart Failure; ROC, receiver operating characteristic.CC BY-NC · PubMed Central
Figure 8 (A) Pairwise comparison of ROC curves for XGBoost model, LASSO model and GWTG-HF risk score model. (B) Decision curve analysis for XGBoost model, LASSO model and GWTG-HF risk score model. The y-axis measures the net benefit. The red line represents the XGBoost model. The light green line represents the LASSO model. The purple line represents the GWTG-HF risk score model. The grey line represents the assumption that all patients die in the hospital. The black line represents the assumption that no patients die in the hospital. Plot (B) shows the net benefit threshold probability range of the three models respectively, between which using the models in the current study to predict in-hospital mortality adds more benefit than the die-all-patients scheme or the die-none scheme. GWTG-HF, Get With The Guidelines-Heart Failure; LASSO, least absolute shrinkage and selection operator; ROC, receiver operating characteristic; XGBoost, extreme gradient boosting.
Figure 8 (A) Pairwise comparison of ROC curves for XGBoost model, LASSO model and GWTG-HF risk score model. (B) Decision curve analysis for XGBoost model, LASSO model and GWTG-HF risk score model. The y-axis measures the net benefit. The red line represents the XGBoost model. The light green line represents the LASSO model. The purple line represents the GWTG-HF risk score model. The grey line represents the assumption that all patients die in the hospital. The black line represents the assumption that no patients die in the hospital. Plot (B) shows the net benefit threshold probability range of the three models respectively, between which using the models in the current study to predict in-hospital mortality adds more benefit than the die-all-patients scheme or the die-none scheme. GWTG-HF, Get With The Guidelines-Heart Failure; LASSO, least absolute shrinkage and selection operator; ROC, receiver operating characteristic; XGBoost, extreme gradient boosting.CC BY-NC · PubMed Central
Figure 9 Developed risk of in-hospital mortality nomogram. Each predictor corresponds to a specific point by drawing a line straight upward to the Points axis. After sum of the scores is located on the Total Points axis, the sum represents the probability of in-hospital mortality by drawing straight down to the risk of in-hospital mortality axis. BUN, blood urea nitrogen.
Figure 9 Developed risk of in-hospital mortality nomogram. Each predictor corresponds to a specific point by drawing a line straight upward to the Points axis. After sum of the scores is located on the Total Points axis, the sum represents the probability of in-hospital mortality by drawing straight down to the risk of in-hospital mortality axis. BUN, blood urea nitrogen.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 41 references

Cited by

See all 129

Metadata

Authors
Li, F., Xin, H., Zhang, J., Fu, M., Zhou, J., & Lian, Z.
Affiliations
  • Fudan University (CN)
  • Affiliated Hospital of Qingdao University (CN)
Corresponding author
  • Zhou, Jingmin
  • Lian, Zhexun
Venue
BMJ Open
Publisher
BMJ
Published
2021-07
Volume
11
Issue
7
Pages
e044779
ISSN
2044-6055 (ISSN-L)
OpenAlex
W3184022450
Cited by
128
References
41
Citations by year
2026: 9 · 2025: 36 · 2024: 46 · 2023: 25 · 2022: 11 · 2021: 1
FWCI
13.91
Type
journal article
Language
en
Open access
Locations
  • BMJ Open (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Basic Research Program of China (973 350 Program, 2012CB518605)
  • National Natural Science Foundation of China (81370199)
  • the Program for the Outstanding Academic Leaders supported by Shanghai Science and Technology Commission (16XD1400700)
Topics
  • Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Heart Failure Treatment and Management
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
Keywords
heart failure, in-hospital mortality, mortality prediction model, intensive care unit patients, nomogram, XGBoost, LASSO, risk factors, MIMIC-III
MeSH terms
Machine Learning, Critical Care, Heart Failure, Humans, Intensive Care Units, Retrospective Studies, Hospital Mortality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Prediction model of in-hospital mortality in intensive care unit patients with heart failure: machine learning-based, retrospective analysis of the MIMIC-III database. https://app.cochat.ai/papers/10.1136/bmjopen-2020-044779
The paper
Li, F., Xin, H., Zhang, J., Fu, M., Zhou, J., & Lian, Z. (2021). Prediction model of in-hospital mortality in intensive care unit patients with heart failure: machine learning-based, retrospective analysis of the MIMIC-III database. BMJ Open. https://doi.org/10.1136/bmjopen-2020-044779

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.