The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Machine learning for real-time prediction of complications in critical care: a retrospective study
Abstract
Background The large amount of clinical signals in intensive care units can easily overwhelm health-care personnel and can lead to treatment delays, suboptimal care, or clinical errors. The aim of this study was to apply deep machine learning methods to predict severe complications during critical care in real time after cardiothoracic surgery.Methods We used deep learning methods (recurrent neural networks) to predict several severe complications (mortality, renal failure with a need for renal replacement therapy, and postoperative bleeding leading to operative revision) in post cardiosurgical care in real time. Adult patients who underwent major open heart surgery from Jan 1, 2000, to Dec 31, 2016, in a German tertiary care centre for cardiovascular diseases formed the main derivation dataset. We measured the accuracy and timeliness of the deep learning model's forecasts and compared predictive quality to that of established standard-of-care clinical reference tools (clinical rule for postoperative bleeding, Simplified Acute Physiology Score II for mortality, and the Kidney Disease: Improving Global Outcomes staging criteria for acute renal failure) using positive predictive value (PPV), negative predictive value, sensitivity, specificity, area under the curve (AUC), and the F 1 measure (which computes a harmonic mean of sensitivity and PPV). Results were externally retrospectively validated with 5898 cases from the published MIMIC-III dataset.Findings Of 47 559 intensive care admissions (corresponding to 42 007 patients), we included 11 492 (corresponding to 9269 patients). The deep learning models yielded accurate predictions with the following PPV and sensitivity scores: PPV 0·90 and sensitivity 0·85 for mortality, 0·87 and 0·94 for renal failure, and 0·84 and 0·74 for bleeding. The predictions significantly outperformed the standard clinical reference tools, improving the absolute complication prediction AUC by 0·29 (95% CI 0·23-0·35) for bleeding, by 0·24 (0·19-0·29) for mortality, and by 0·24 (0·13-0·35) for renal failure (p<0·0001 for all three analyses). The deep learning methods showed accurate predictions immediately after patient admission to the intensive care unit. We also observed an increase in performance in our validation cohort when the machine learning approach was tested against clinical reference tools, with absolute improvements in AUC of 0·09 (95% CI 0·03-0·15; p=0·0026) for bleeding, of 0·18 (0·07-0·29; p=0·0013) for mortality, and of 0·25 (0·18-0·32; p<0·0001) for renal failure.Interpretation The observed improvements in prediction for all three investigated clinical outcomes have the potential to improve critical care. These findings are noteworthy in that they use routinely collected clinical data exclusively, without the need for any manual processing. The deep machine learning method showed AUC scores that significantly surpass those of clinical reference tools, especially soon after admission. Taken together, these properties are encouraging for prospective deployment in critical care settings to direct the staff's attention towards patients who are most at risk.Funding No specific funding.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Although various research projects are focusing on CDSS in the ICU, translation into the clinical routine is lagging far behind [15,46-49].”
From Clinical requirements of future patient monitoring in the intensive care unit: qualitative study (2019) · cites this paper as [15]
In the citing paper
“Models for real-time prediction have been trained in clinical contexts such as circulatory failure in critical care24 and post-operative complications25,26.”
From Machine learning for real-time aggregated prediction of hospital admission for emergency patients (2022) · cites this paper as [26]
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1985) · Critical Care Medicine · doi:10.1097/00003246-198510000-00009 · Cited by 15,518
APACHE II-A Severity of Disease Classification System
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1986) · Critical Care Medicine · doi:10.1097/00003246-198608000-00028 · Cited by 13,379
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
KDIGO Clinical Practice Guidelines for Acute Kidney Injury
Khwaja, A. (2012) · Nephron Clinical Practice · doi:10.1159/000339789 · Cited by 8,297
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Le Gall, J.-R. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.270.24.2957 · Cited by 7,773
Kumar, A., Roberts, D., Wood, K. E., Light, B., Parrillo, J. E., Sharma, S., Suppes, R., Feinstein, D., Zanotti, S., Taiberg, L., et al. (2006) · Critical Care Medicine · doi:10.1097/01.ccm.0000217961.75225.e9 · Cited by 6,109
Predicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652
Recurrent Neural Networks for Multivariate Time Series with Missing Values
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2018) · Scientific Reports · doi:10.1038/s41598-018-24271-9 · Cited by 2,370
A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation
Goutte, C., & Gaussier, E. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-31865-1_25 · Cited by 2,285
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131
Machine Learning and Prediction in Medicine — Beyond the Peak of Inflated Expectations
Chen, J. H., & Asch, S. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1702071 · Cited by 1,171
Artificial Intelligence in Precision Cardiovascular Medicine
Krittanawong, C., Zhang, H., Wang, Z., Aydar, M., & Kitai, T. (2017) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2017.03.571 · Cited by 1,126
Gaieski, D. F., Mikkelsen, M. E., Band, R. A., Pines, J. M., Massone, R., Furia, F. F., Shofer, F. S., & Goyal, M. (2010) · Critical Care Medicine · doi:10.1097/ccm.0b013e3181cc4824 · Cited by 970
Halpern, N. A., & Pastores, S. M. (2009) · Critical Care Medicine · doi:10.1097/ccm.0b013e3181b090d0 · Cited by 818
Bell, G., Hey, T., & Szalay, A. (2009) · Science · doi:10.1126/science.1170411 · Cited by 537
What This Computer Needs Is a Physician: Humanism and Artificial Intelligence
Verghese, A., Shah, N. H., & Harrington, R. A. (2018) · JAMA · doi:10.1001/jama.2017.19198 · Cited by 468
Machine Learning and Decision Support in Critical Care
Johnson, A. E. W., Ghassemi, M. M., Nemati, S., Niehaus, K. E., Clifton, D. A., & Clifford, G. D. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2501978 · Cited by 459
How Many Variables Can Humans Process?
Halford, G. S., Baker, R., McCredden, J., & Bain, J. (2005) · Psychological Science · doi:10.1111/j.0956-7976.2005.00782.x · Cited by 452
Learning from Big Health Care Data
Schneeweiß, S. (2014) · New England Journal of Medicine · doi:10.1056/nejmp1401111 · Cited by 321
The hostile environment of the intensive care unit
Donchin, Y., & Seagull, F. J. (2002) · Current Opinion in Critical Care · doi:10.1097/00075198-200208000-00008 · Cited by 297
Lost in thought—the limits of the human mind and the future of medicine
Obermeyer, Z., & Lee, T. H. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1705348 · Cited by 252
Interpretable Deep Models for ICU Outcome Prediction.
Che, Z., Purushotham, S., Khemani, R. G., & Liu, Y. (2016) · PubMed · Cited by 243
Recurrent Neural Networks for Multivariate Time Series with Missing ValuesPreprint
Che, Z., Purushotham, S., Cho, K., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01865 · Cited by 230
Modeling Missing Data in Clinical Time Series with RNNsPreprint
Lipton, Z. C., Kale, D. C., & Wetzel, R. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04130 · Cited by 216
“Big Data” in the Intensive Care Unit. Closing the Data Loop
Celi, L. A., Mark, R. G., Stone, D. J., & Montgomery, R. A. (2013) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201212-2311ed · Cited by 214
Multicenter, randomized, controlled trials evaluating mortality in intensive care: Doomed to fail?
Ospina-Tascón, G. A., Büchele, G. L., & Vincent, J. (2008) · Critical Care Medicine · doi:10.1097/ccm.0b013e318168ea3e · Cited by 174
Sakr, Y. L., Lobo, S. M., Moreno, R. P., Gerlach, H., Ranieri, V. M., Michalopoulos, A. S., Vincent, J., & the SOAP Investigators (2012) · Critical Care · doi:10.1186/cc11868 · Cited by 149
Clinical Intervention Prediction and Understanding using Deep NetworksPreprint
Suresh, H., Hunt, N. J., Johnson, A. E. W., Celi, L. A., Szolovits, P., & Ghassemi, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08498 · Cited by 130
Artificial intelligence in health care: within touching distance
Lancet, T. (2017) · The Lancet · doi:10.1016/s0140-6736(17)31540-4 · Cited by 117
A path to precision in the ICU
Maslove, D. M., Lamontagne, F., Marshall, J. C., & Heyland, D. K. (2017) · Critical Care · doi:10.1186/s13054-017-1653-x · Cited by 112
Manual of Perioperative Care in Adult Cardiac Surgery
Bojar, R. M. (2020) · Wiley eBooks · doi:10.1002/9781119582540 · Cited by 89
Advances in Information Retrieval
Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., & Johnson, D. (2005) · Lecture notes in computer science · doi:10.1007/b107096 · Cited by 82
Acute abdomen in the medical intensive care unit
Gajic, O., Urrutia, L., Sewani, H. H., Schroeder, D. R., Cullinane, D. C., & Peters, S. G. (2002) · Critical Care Medicine · doi:10.1097/00003246-200206000-00001 · Cited by 73
Manual of Perioperative Care in Adult Cardiac Surgery
Rosenbloom, M. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e31822d4495 · Cited by 54
Costs of Critical Care Medicine
Pastores, S. M., Dakwar, J., & Halpern, N. A. (2011) · Critical Care Clinics · doi:10.1016/j.ccc.2011.10.003 · Cited by 53
Digital medicine: empowering both patients and clinicians
Topol, E. J. (2016) · The Lancet · doi:10.1016/s0140-6736(16)31355-1 · Cited by 49
Neural Document Embeddings for Intensive Care Patient Mortality PredictionPreprint
Grnarova, P., Schmidt, F., Hyland, S. L., & Eickhoff, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00467 · Cited by 46
Self-citationMcKenna, J. (2017) · European Heart Journal · doi:10.1093/eurheartj/ehx021 · Cited by 2
Cited by
Sorted by citations
Early prediction of circulatory failure in the intensive care unit using machine learning
Hyland, S. L., Faltys, M., Hüser, M., Lyu, X., Gumbsch, T., Esteban, C., Bock, C., Horn, M., Moor, M., Rieck, B., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0789-4 · Cited by 477
Xue, B., Li, D., Lu, C., King, C. R., Wildes, T. S., Avidan, M. S., Kannampallil, T., & Abraham, J. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.2240 · Cited by 373
Thorsen-Meyer, H.-C., Nielsen, A. B., Nielsen, A. P., Kaas-Hansen, B. S., Toft, P., Schierbeck, J., Strøm, T., Chmura, P. J., Heimann, M., Dybdahl, L., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30018-2 · Cited by 336
Sendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287
Mobile Health in Remote Patient Monitoring for Chronic Diseases: Principles, Trends, and Challenges
El-Rashidy, N. M., El–Sappagh, S., Islam, S. M. R., El-Bakry, H. M., & Abdelrazek, S. M. (2021) · Diagnostics · doi:10.3390/diagnostics11040607 · Cited by 274
Metadata
- Authors
- Meyer, A., Zverinski, D., Pfahringer, B., Kempfert, J., Kuehne, T., Sündermann, S. H., Stamm, C., Hofmann, T., Falk, V., & Eickhoff, C.
- Affiliations
- Deutsches Herzzentrum der Charité (DE)
- ETH Zurich (CH)
- Brown University (US)
- Corresponding author
- Meyer, Alexander
- ORCID
- Venue
- The Lancet Respiratory Medicine
- Publisher
- Elsevier BV
- Published
- 2018-12
- Volume
- 6
- Issue
- 12
- Pages
- 905-914
- ISSN
- 2213-2600 (ISSN-L)
- OpenAlex
- W2892592994
- Cited by
- 350
- References
- 41
- Citations by year
- 2026: 29 · 2025: 40 · 2024: 41 · 2023: 51 · 2022: 58 · 2021: 54 · 2020: 57 · 2019: 15 · 2018: 4
- FWCI
- 18.32
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The Lancet Respiratory Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Charité – Universitätsmedizin Berlin
- Berlin Institute of Health
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (174025)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Cardiac, Anesthesia and Surgical Outcomes
- Keywords
- complication prediction, intensive care, cardiothoracic surgery, postoperative mortality, acute kidney injury, renal replacement therapy, postoperative bleeding, real-time prediction, recurrent neural networks, intensive care unit, MIMIC-III, early warning systems, model validation, external validation, clinical decision support, treatment delay, medical errors
- MeSH terms
- Deep Learning, Aged, Female, Cardiac Surgical Procedures, Humans, Intensive Care Units, Male, Middle Aged, Predictive Value of Tests, Retrospective Studies, ROC Curve, Outcome Assessment, Health Care, Postoperative Hemorrhage, Renal Insufficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
