Using a machine learning approach to predict mortality in critically ill influenza patients: A cross-sectional retrospective multicentre study in Taiwan
Open accessAbstract
OBJECTIVES: Current mortality prediction models used in the intensive care unit (ICU) have a limited role for specific diseases such as influenza, and we aimed to establish an explainable machine learning (ML) model for predicting mortality in critically ill influenza patients using a real-world severe influenza data set. STUDY DESIGN: A cross-sectional retrospective multicentre study in Taiwan SETTING: Eight medical centres in Taiwan. PARTICIPANTS: A total of 336 patients requiring ICU-admission for virology-proven influenza at eight hospitals during an influenza epidemic between October 2015 and March 2016. PRIMARY AND SECONDARY OUTCOME MEASURES: We employed extreme gradient boosting (XGBoost) to establish the prediction model, compared the performance with logistic regression (LR) and random forest (RF), demonstrated the feature importance categorised by clinical domains, and used SHapley Additive exPlanations (SHAP) for visualised interpretation. RESULTS: The data set contained 76 features of the 336 patients with severe influenza. The severity was apparently high, as shown by the high Acute Physiology and Chronic Health Evaluation II score (22, 17 to 29) and pneumonia severity index score (118, 88 to 151). XGBoost model (area under the curve (AUC): 0.842; 95% CI 0.749 to 0.928) outperformed RF (AUC: 0.809; 95% CI 0.629 to 0.891) and LR (AUC: 0.701; 95% CI 0.573 to 0.825) for predicting 30-day mortality. To give clinicians an intuitive understanding of feature exploitation, we stratified features by the clinical domain. The cumulative feature importance in the fluid balance domain, ventilation domain, laboratory data domain, demographic and symptom domain, management domain and severity score domain was 0.253, 0.113, 0.177, 0.140, 0.152 and 0.165, respectively. We further used SHAP plots to illustrate associations between features and 30-day mortality in critically ill influenza patients. CONCLUSIONS: We used a real-world data set and applied an ML approach, mainly XGBoost, to establish a practical and explainable mortality prediction model in critically ill influenza patients.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Perturbation-based models that use model-agnostic explanations are common in the literature and have been used in two of the proposed CDSS.
“In terms of post-hoc explainability, perturbation-based models that use model-agnostic explanations are common in the literature [98,99,118,120,127] and have been used in two of the proposed CDSS [118,120].”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [120]
Most studies used visualization as a key aspect to enhance explainability, either with SHAP plots, regional abnormality maps, rainbow boxes, or the attention mechanism that highlighted important words leading to a prediction.
“Most studies used visualization as a key aspect to enhance explainability, either with SHAP plots [118,120], regional abnormality maps [119], rainbow boxes [123,126], or the attention mechanism that highlighted the important words that lead to a prediction [122].”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [120]
Hu et al. [120] used SHAP for summary plot and partial dependence plot; the method was model-agnostic, post-hoc, and global.
“Hu et al. [120] SHAP [79] for summary plot and partial dependence plot agnostic post-hoc global”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [120]
Among the reviewed CDSS, Hu et al. [120] was the only one that focused on global explanations, using a model-agnostic post-hoc technique.
“The work by Hu et al. [120] was the only one that focused on global explanations, using a model-agnostic post-hoc technique.”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [120]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · see citing works
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
APACHE II-A Severity of Disease Classification System
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1986) · Critical Care Medicine · doi:10.1097/00003246-198608000-00028 · Cited by 13,379 · see citing works
APACHE II: a severity of disease classification system.
Knaus, W. A., Draper, E. S., Wagner, D. P., & Zimmerman, J. E. (1985) · PubMed · Cited by 6,034 · see citing works
A Prediction Rule to Identify Low-Risk Patients with Community-Acquired Pneumonia
Fine, M. J., Auble, T. E., Yealy, D. M., Hanusa, B. H., Weissfeld, L. A., Singer, D. E., Coley, C. M., Marrie, T. J., & Kapoor, W. N. (1997) · New England Journal of Medicine · doi:10.1056/nejm199701233360402 · Cited by 4,560 · see citing works
Fleischmann, C., Scherag, A., Adhikari, N. K. J., Hartog, C. S., Tsaganos, T., Schlattmann, P., Angus, D. C., & Reinhart, K. (2016) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201504-0781oc · Cited by 3,548 · see citing works
Logistic regression and Artificial Neural Network Classification models: A methodology review
Dreiseitl, S., & Ohno-Machado, L. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00034-0 · Cited by 2,177 · see citing works
Practice guidelines in acute pancreatitis
Banks, P. A., Freeman, M. L., & the Practice Parameters Committee of the American College of Gastroenterology (2006) · The American Journal of Gastroenterology · doi:10.1111/j.1572-0241.2006.00856.x · Cited by 2,050 · see citing works
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · see citing works
Hospital deaths in patients with sepsis from 2 independent cohorts.
Liu, V. X., Escobar, G., Greene, J., Soule, J., Whippy, A., Angus, D., & Iwashyna, T. (2014) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.2014.5804 · Cited by 946 · see citing works
Clinical review: Scoring systems in the critically ill
Vincent, J.-L., & Moreno, R. (2010) · Critical Care · doi:10.1186/cc8204 · Cited by 634 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
Taylor, R. A., Pare, J. R., Venkatesh, A. K., Mowafi, H. O., Melnick, E. R., Fleischman, W. M., & Hall, M. K. (2015) · Academic Emergency Medicine · doi:10.1111/acem.12876 · Cited by 496 · see citing works
Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets
Bryll, R., Gutiérrez‐Osuna, R., & Quek, F. K. H. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00121-8 · Cited by 481 · see citing works
Rosenberg, A. L., Dechert, R. E., Park, P. K., et al. (2008) · Journal of Intensive Care Medicine · doi:10.1177/0885066608329850 · Cited by 332 · see citing works
Horng, S., Sontag, D. A., Halpern, Y., Jernite, Y., Shapiro, N. I., & Nathanson, L. A. (2017) · PLOS ONE · doi:10.1371/journal.pone.0174708 · Cited by 310 · see citing works
Zhang, Z., Beck, M. W., Winkler, D. A., Huang, B., Sibanda, W., & Goyal, H. (2018) · Annals of Translational Medicine · doi:10.21037/atm.2018.05.32 · Cited by 293 · see citing works
Needham, D. M., Yang, T., Dinglas, V. D., Méndez-Téllez, P. A., Shanholtz, C. B., Sevransky, J., Brower, R. G., Pronovost, P. J., & Colantuoni, E. (2014) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201409-1598oc · Cited by 259 · see citing works
Loke, Y. K., Kwok, C. S., Niruban, A., & Myint, P. K. (2010) · Thorax · doi:10.1136/thx.2009.134072 · Cited by 200 · see citing works
Allyn, J., Allou, N., Augustin, P., Philip, I., Martinet, O., Belghiti, M., Provenchère, S., Montravers, P., & Ferdynus, C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0169772 · Cited by 188 · see citing works
Variation in ICU Risk-Adjusted Mortality
Kuzniewicz, M. W., Vasilevskis, E. E., Lane, R., Dean, M. L., Trivedi, N. G., Rennie, D. J., Clay, T., Kotler, P. L., & Dudley, R. A. (2008) · CHEST Journal · doi:10.1378/chest.07-3061 · Cited by 133 · see citing works
Importance Sampled Learning Ensembles
Friedman, J. H., & Popescu, B. E. (2003) · Cited by 110 · see citing works
Oliva, J. A., Delgado‐Sanz, C., Larrauri, A., et al. (2017) · Influenza and Other Respiratory Viruses · doi:10.1111/irv.12499 · Cited by 54 · see citing works
Severity Scores in Respiratory Intensive Care: APACHE II Predicted Mortality Better than SAPS II
Bufalo, C. D., Morelli, A., Bassein, L., et al. (1995) · Respiratory Care · doi:10.1177/194336549504001003 · Cited by 53 · see citing works
Liu, V. X., Turk, B. J., Ragins, A. I., Kipnis, P., & Escobar, G. J. (2012) · Critical Care Medicine · doi:10.1097/ccm.0b013e318267636e · Cited by 46 · see citing works
Estimates of the Demand for Mechanical Ventilation in the United States During an Influenza Pandemic
Meltzer, M. I., Patel, A. K., Ajao, A. O., Nystrom, S. V., & Koonin, L. M. (2015) · Clinical Infectious Diseases · doi:10.1093/cid/civ089 · Cited by 46 · see citing works
Chao, W., Tseng, C., Chien, Y., Sheu, C., Tsai, M., Fang, W., Chen, Y., Kao, K., Hu, H., Perng, W., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0190952 · Cited by 43 · see citing works
Self-citationTree Space Prototypes: Another Look at Making Tree Ensembles InterpretablePreprint
Tan, S., Soloviev, M., Hooker, G., & Wells, M. T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07115 · Cited by 42 · see citing works
Severity assessment tools in ICU patients with 2009 Influenza A (H1N1) pneumonia
Pereira, J. M., Moreno, R. P., Matos, R., Rhodes, A., Martín‐Loeches, I., Cecconi, M., Lisboa, T. C., & Rello, J. (2011) · Clinical Microbiology and Infection · doi:10.1111/j.1469-0691.2011.03736.x · Cited by 41 · see citing works
Comparison of six outcome prediction models in an adult burn population in a developing country.
Salehi, S. H., Aśadi, K., Abbaszadeh-Kasbi, A., Isfeedvajani, M. S., & Khodaei, N. (2017) · Europe PMC (PubMed Central) · Cited by 38 · see citing works
Development of the Post Cardiac Surgery (POCAS) prognostic score
Tamayo, E., Fierro, I., Bustamante‐Munguira, J., et al. (2013) · Critical Care · doi:10.1186/cc13017 · Cited by 31 · see citing works
Peltola, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02678 · Cited by 31 · see citing works
Bjarnason, A., Thorleifsdottir, G., Löve, A., Gudnason, J. F., Asgeirsson, H., Hallgrimsson, K. L., Kristjansdottir, B. S., Haraldsson, G., Baldursson, O., Kristinsson, K. G., et al. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046816 · Cited by 30 · see citing works
Chan, M.-C., Chao, W., Liang, S., Tseng, C., Wang, H., Chien, Y., Yang, K., Chen, W., Perng, W., Kao, K., et al. (2018) · Journal of the Formosan Medical Association · doi:10.1016/j.jfma.2018.06.010 · Cited by 27 · see citing works
Self-citationScoring-Systeme in der Intensivmedizin
Fleig, V., Brenck, F., Wolff, M., & Weigand, M. A. (2011) · Der Anaesthesist · doi:10.1007/s00101-011-1942-8 · Cited by 21 · see citing works
Machine learning and deep analytics for biocomputing: call for better explainability
Petković, D., Kobzik, L., & Ré, C. (2017) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789813235533_0058 · Cited by 13 · see citing works
Knowledge-Analytics Synergy in Clinical Decision Support
Slonim, N., Carmeli, B., Goldsteen, A., et al. (2012) · Studies in health technology and informatics · doi:10.3233/978-1-61499-101-4-703 · Cited by 8 · see citing works
Rasmussen, S. A., & Redd, S. C. (2015) · Clinical Infectious Diseases · doi:10.1093/cid/civ090 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Loh, H. W., Ooi, C. P., Seoni, S., Barua, P. D., Molinari, F., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107161 · Cited by 734 · see citing works
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629 · see citing works
Explainable Artificial Intelligence for Tabular Data: A Survey
Sahakyan, M., Aung, Z., & Rahwan, T. (2021) · IEEE Access · doi:10.1109/access.2021.3116481 · Cited by 135 · see citing works
“A different application of SHAP was proposed by Hu et al. [196], who used it to identify the individual-level features that had the greatest impact on the 30-day mortality rate of influenza patients with critical illness in Taiwan.”
MethodSheu, R.-K., & Pardeshi, M. S. (2022) · Sensors · doi:10.3390/s22208068 · Cited by 129 · see citing works
Du, Y., Antoniadi, A. M., McNestry, C., McAuliffe, F. M., & Mooney, C. (2022) · Applied Sciences · doi:10.3390/app122010323 · Cited by 47 · see citing works
Lin, M.-Y., Li, C.-C., Lin, P.-H., Wang, J.-L., Chan, M.-C., Wu, C.-L., & Chao, W.-C. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.663739 · Cited by 45 · see citing works
Self-citationLi, S., Dou, R., Song, X., Lui, K., Xu, J., Guo, Z., Hu, X., Guan, X., & Cai, C. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12030915 · Cited by 25 · see citing works
XAI-based Clinical Decision Support System: A Systematic Review
Kim, S., Kim, D., Kim, M., Ko, H., & Jeong, O. (2024) · doi:10.20944/preprints202406.0721.v1 · Cited by 72 · see citing works
- Authors
- Hu, C.-A., Chen, C.-M., Fang, Y.-C., Liang, S.-J., Wang, H.-C., Fang, W.-F., Sheu, C.-C., Perng, W.-C., Yang, K.-Y., Kao, K.-C., Wu, C.-L., Tsai, C.-S., Lin, M.-Y., & Chao, W.-C.
- Affiliations
- Feng Chia University (TW)
- National Chung Hsing University (TW)
- China Medical University Hospital (TW)
- National Taiwan University Hospital (TW)
- Chang Gung University of Science and Technology (TW)
- Kaohsiung Medical University (TW)
- Tri-Service General Hospital (TW)
- Taipei Veterans General Hospital (TW)
- Chang Gung University (TW)
- Taichung Veterans General Hospital (TW)
- Corresponding author
- Lin, Ming-Yen
- Chao, Wen-Cheng
- ORCID
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2020-02
- Volume
- 10
- Issue
- 2
- Pages
- e033898
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC7045134
- OpenAlex
- W3007001459
- Cited by
- 106
- References
- 39
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 12 · 2023: 15 · 2022: 31 · 2021: 25 · 2020: 7
- FWCI
- 14.34
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMJ Open (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- HighWire Press Open Archive (submittedVersion)
- Funders
- Veterans General Hospitals and the University System of Taiwan Joint Research Program (VGHUST108-G2-4-2)
- Topics
- Health Sciences › Medicine › Epidemiology › Influenza Virus Research Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Keywords
- influenza, mortality prediction, critically ill patients, intensive care unit, extreme gradient boosting, SHAP, explainable machine learning, 30-day mortality, fluid balance, mechanical ventilation, APACHE II score, Pneumonia Severity Index, feature importance, Taiwan, real-world data, clinical decision support
- MeSH terms
- Machine Learning, Adult, Cross-Sectional Studies, Female, Humans, Influenza, Human, Male, Middle Aged, Models, Theoretical, Retrospective Studies, Taiwan, Critical Illness, Hospital Mortality, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
