A Machine Learning Sepsis Prediction Algorithm for Intended Intensive Care Unit Use (NAVOY Sepsis): Proof-of-Concept Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
Background: Despite decades of research, sepsis remains a leading cause of mortality and morbidity in intensive care units worldwide. The key to effective management and patient outcome is early detection, for which no prospectively validated machine learning prediction algorithm is currently available for clinical use in Europe. Objective: We aimed to develop a high-performance machine learning sepsis prediction algorithm based on routinely collected intensive care unit data, designed to be implemented in European intensive care units. Methods: The machine learning algorithm was developed using convolutional neural networks, based on Massachusetts Institute of Technology Lab for Computational Physiology MIMIC-III clinical data from intensive care unit patients aged 18 years or older. The model uses 20 variables to produce hourly predictions of onset of sepsis, defined by international Sepsis-3 criteria. Predictive performance was externally validated using hold-out test data. Results: The algorithm—NAVOY Sepsis—uses 4 hours of input and can identify patients with high risk of developing sepsis, with high performance (area under the receiver operating characteristics curve 0.90; area under the precision-recall curve 0.62) for predictions up to 3 hours before sepsis onset. Conclusions: The prediction performance of NAVOY Sepsis was superior to that of existing sepsis early warning scoring systems and comparable with those of other prediction algorithms designed to predict sepsis onset. The algorithm has excellent predictive properties and uses variables that are routinely collected in intensive care units.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“The algorithm was developed using convolutional neural networks [23].”
MethodThe third international consensus definitions for sepsis and septic shock (Sepsis-3).
Singer, M., Deutschman, C. S., Seymour, C. W., Shankar-Hari, M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0287 · Cited by 28,250
Zhou, F., Yu, T., Du, R., Fan, G., Liu, Y., Liu, Z., Xiang, J., Wang, Y., Song, B., Gu, X., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30566-3 · Cited by 27,263 · cited 1×
“During the coronavirus disease 2019 (COVID-19) pandemic, sepsis was the most frequently observed complication among adult inpatients at Jinyintan Hospital and Wuhan Pulmonary Hospital (Wuhan, China) who had been discharged or had died (as of January 31, 2020) [5].”
BackgroundBone, R. C., Balk, R. A., Cerra, F. B., Dellinger, R. P., Fein, A. M., Knaus, W. A., Schein, R. M. H., & Sibbald, W. J. (1992) · Chest · doi:10.1378/chest.101.6.1644 · Cited by 13,198 · cited 1×
“The following scores were included in this study: Systemic Inflammatory Response Syndrome criteria, at least 2 of 4 criteria present [29]; Quick Sepsis-Related Organ Failure Assessment score, at least 2 of 3 criteria present [1]; Sepsis-Related Organ Failure Assessment score, total score ≥2 [1]; Modified Early Warning…”
MethodMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 1×
“The algorithm for prediction of sepsis was developed based on Massachusetts Institute of Technology Lab for Computational Physiology MIMIC-III Clinical Database [20].”
MethodRudd, K. E., Johnson, S. C., Agesa, K. M., Shackelford, K. A., Tsoi, D., Kievlan, D. R., Colombara, D. V., Ikuta, K. S., Kissoon, N., Finfer, S., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(19)32989-7 · Cited by 7,971 · cited 1×
“Globally, it affects approximately 49 million people every year [4].”
BackgroundKumar, A., Roberts, D., Wood, K. E., Light, B., Parrillo, J. E., Sharma, S., Suppes, R., Feinstein, D., Zanotti, S., Taiberg, L., et al. (2006) · Critical Care Medicine · doi:10.1097/01.ccm.0000217961.75225.e9 · Cited by 6,109 · cited 2×
“Early diagnosis of sepsis has been shown to reduce delays in treatment, increase appropriate care, and reduce mortality [6-9].”
BackgroundSeymour, C. W., Liu, V. X., Iwashyna, T. J., Brunkhorst, F. M., Rea, T. D., Scherag, A., Rubenfeld, G., Kahn, J. M., Shankar-Hari, M., Singer, M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0288 · Cited by 3,939 · cited 1×
“Suspected infection [21] was defined as instances when antibiotics had been prescribed and when body fluid cultures were present in the electronic health record within a specific time window; if a culture is ordered within 24 hours after antibiotics, or antibiotics had been prescribed less than 72 hours after a cultur…”
BackgroundSurviving Sepsis Campaign: International guidelines for management of sepsis and septic shock: 2016.
Rhodes, A., Evans, L. E., Alhazzani, W., Levy, M. M., Antonelli, M., Ferrer, R., Kumar, A., Sevransky, J. E., Sprung, C. L., Nunnally, M. E., et al. (2017) · Critical Care Medicine · doi:10.1097/ccm.0000000000002255 · Cited by 3,652
Time to treatment and mortality during mandated emergency care for sepsis.
Seymour, C. W., Gesten, F., Prescott, H. C., Friedrich, M. E., Iwashyna, T. J., Phillips, G. S., Lemeshow, S., Osborn, T., Terry, K. M., & Levy, M. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmoa1703058 · Cited by 2,225 · cited 2×
“Early diagnosis of sepsis has been shown to reduce delays in treatment, increase appropriate care, and reduce mortality [6-9].”
BackgroundPollard, T. J., Johnson, A. E. W., Raffa, J. D., Celi, L. A., Mark, R. G., & Badawi, O. (2018) · Scientific Data · doi:10.1038/sdata.2018.178 · Cited by 1,993 · cited 1×
“Third, it would have been valuable to test the performance of the algorithm with an additional external validation cohort, for example, data from the PhysioNet Challenge [38] or the eICU Collaborative Research Database [40].”
MethodIncidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.
Rhee, C., Dantes, R., Epstein, L., Murphy, D. J., Seymour, C. W., Iwashyna, T. J., Kadri, S. S., Angus, D. C., Danner, R. L., Fiore, A. E., et al. (2017) · JAMA · doi:10.1001/jama.2017.13836 · Cited by 1,884 · cited 1×
“It has been estimated that sepsis is present in 6% of adult hospital admissions [2] and in approximately one-third of intensive care unit (ICU) patients [3].”
BackgroundValidation of a modified Early Warning Score in medical admissions
Subbe, C. P. (2001) · QJM · doi:10.1093/qjmed/94.10.521 · Cited by 1,762 · cited 1×
“The following scores were included in this study: Systemic Inflammatory Response Syndrome criteria, at least 2 of 4 criteria present [29]; Quick Sepsis-Related Organ Failure Assessment score, at least 2 of 3 criteria present [1]; Sepsis-Related Organ Failure Assessment score, total score ≥2 [1]; Modified Early Warning…”
MethodFerrer, R., Martin-Loeches, I., Phillips, G., Osborn, T. M., Townsend, S., Dellinger, R. P., Artigas, A., Schorr, C., & Levy, M. M. (2014) · Critical Care Medicine · doi:10.1097/ccm.0000000000000330 · Cited by 1,480 · cited 1×
“A retrospective analysis of 17,000 patients has shown that there is a linear increase in the risk of mortality for each hour of delay in antibiotic administration [10].”
BackgroundPierrakos, C., & Vincent, J.-L. (2010) · Critical Care · doi:10.1186/cc8872 · Cited by 1,377 · cited 1×
“Hundreds of biomarkers have been tested as prognostic markers in sepsis [12-14]; however, none has demonstrated sufficient specificity or sensitivity to be routinely used in clinical practice [12].”
BackgroundAn interpretable machine learning model for accurate prediction of sepsis in the ICU
Nemati, S., Holder, A., Razmi, F., Stanley, M. D., Clifford, G. D., & Buchman, T. G. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000002936 · Cited by 832 · cited 3×
“Only Futoma et al [35] used a comparable sepsis prevalence (21%), with other prevalences ranging between 6% and 9% (or not specified) [17,19,36-38].”
BackgroundMethodFleuren, L. M., Klausch, T. L. T., Zwager, C. L., Schoonmade, L. J., Guo, T., Roggeveen, L. F., Swart, E. L., Girbes, A. R. J., Thoral, P., Ercole, A., et al. (2020) · Intensive Care Medicine · doi:10.1007/s00134-019-05872-y · Cited by 755
Biomarkers of sepsis: time for a reappraisal
Pierrakos, C., Velissaris, D., Bisdorff, M., Marshall, J. C., & Vincent, J.-L. (2020) · Critical Care · doi:10.1186/s13054-020-02993-5 · Cited by 713
Sepsis in Intensive Care Unit Patients: Worldwide Data From the Intensive Care over Nations Audit
Sakr, Y., Jaschinski, U., Wittebole, X., Szakmany, T., Lipman, J., Ñamendys-Silva, S. A., Martin-Loeches, I., Leone, M., Lupu, M.-N., Vincent, J.-L., et al. (2018) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofy313 · Cited by 534 · cited 3×
“Data were sampled to maintain a prevalence of sepsis of 20% in both training and test data, to resemble the prevalence of sepsis in ICU patients in North America and Western Europe [3].”
BackgroundMethodDesautels, T., Calvert, J., Hoffman, J., Jay, M., Kerem, Y., Shieh, L., Shimabukuro, D., Chettipally, U., Feldman, M. D., Barton, C., et al. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.5909 · Cited by 509 · cited 6×
“ICU stays logged using the CareVue (Philips) electronic health record system were excluded, that is, only ICU stays logged using the Metavision (iMDSoft) electronic health record system were included, since negative blood cultures are underreported with CareVue, which means that suspicion of infection is underrepresen…”
BackgroundMethodResultInfluentialEarly Prediction of Sepsis From Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019
Reyna, M. A., Josef, C. S., Jeter, R., Shashikumar, S. P., Westover, M. B., Nemati, S., Clifford, G. D., & Sharma, A. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004145 · Cited by 383
Shimabukuro, D. W., Barton, C. W., Feldman, M. D., Mataraso, S. J., & Das, R. (2017) · BMJ Open Respiratory Research · doi:10.1136/bmjresp-2017-000234 · Cited by 382 · cited 1×
“To date, and to the best of our knowledge, only 1 ICU algorithm is available for clinical use [17,18], and another is planned to be prospectively validated [19].”
MethodWidgren, B. R., & Jourak, M. (2011) · The Journal of Emergency Medicine · doi:10.1016/j.jemermed.2008.04.003 · Cited by 288 · cited 1×
“The following scores were included in this study: Systemic Inflammatory Response Syndrome criteria, at least 2 of 4 criteria present [29]; Quick Sepsis-Related Organ Failure Assessment score, at least 2 of 3 criteria present [1]; Sepsis-Related Organ Failure Assessment score, total score ≥2 [1]; Modified Early Warning…”
MethodEarly Prediction of Sepsis in the ICU Using Machine Learning: A Systematic Review
Moor, M., Rieck, B., Horn, M., Jutzeler, C. R., & Borgwardt, K. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.607952 · Cited by 231 · cited 5×
“Random onset matching [16]—randomly chosen 4-hour sequences, with the last time point up to 3 hours before onset, for patients with sepsis, or at any point during the whole ICU stay, for patients without sepsis—was used.”
BackgroundMethodInfluentialCho, S., & Choi, J. (2014) · Infection and Chemotherapy · doi:10.3947/ic.2014.46.1.1 · Cited by 171
Husabø, G., Nilsen, R. M., Flaatten, H. K., Solligård, E., Frich, J. C., Bondevik, G. T., Braut, G. S., Walshe, K., Harthug, S., & Hovlid, E. (2020) · PLoS ONE · doi:10.1371/journal.pone.0227652 · Cited by 161 · cited 2×
“Early diagnosis of sepsis has been shown to reduce delays in treatment, increase appropriate care, and reduce mortality [6-9].”
BackgroundAn Explainable Artificial Intelligence Predictor for Early Detection of Sepsis
Yang, M., Liu, C., Wang, X., Li, Y., Gao, H., Liu, X., & Li, J. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004550 · Cited by 118 · cited 3×
“Only Futoma et al [35] used a comparable sepsis prevalence (21%), with other prevalences ranging between 6% and 9% (or not specified) [17,19,36-38].”
BackgroundMethodBayer, O., Schwarzkopf, D., Stumme, C., Stacke, A., Hartog, C. S., Hohenstein, C., Kabisch, B., Reichel, J., Reinhart, K., & Winning, J. (2015) · Academic Emergency Medicine · doi:10.1111/acem.12707 · Cited by 105 · cited 1×
“The following scores were included in this study: Systemic Inflammatory Response Syndrome criteria, at least 2 of 4 criteria present [29]; Quick Sepsis-Related Organ Failure Assessment score, at least 2 of 3 criteria present [1]; Sepsis-Related Organ Failure Assessment score, total score ≥2 [1]; Modified Early Warning…”
MethodData-driven discovery of a novel sepsis pre-shock state predicts impending septic shock in the ICU
Liu, R., Greenstein, J. L., Granite, S. J., Fackler, J. C., Bembea, M. M., Sarma, S. V., & Winslow, R. L. (2019) · Scientific Reports · doi:10.1038/s41598-019-42637-5 · Cited by 79 · cited 1×
“Sepsis-relevant antibiotics and body fluids were chosen as the indicators based on methods used by Liu et al [22]—they used blood cultures and a defined list of antibiotics (presented in the code repository referred to in their paper).”
MethodTime to Administration of Antibiotics among Inpatients with Severe Sepsis or Septic Shock
Mok, K., Christian, M. D., Nelson, S., & Burry, L. D. (2014) · The Canadian Journal of Hospital Pharmacy · doi:10.4212/cjhp.v67i3.1358 · Cited by 45 · cited 2×
“Early diagnosis of sepsis has been shown to reduce delays in treatment, increase appropriate care, and reduce mortality [6-9].”
BackgroundSepsis Alert – a triage model that reduces time to antibiotics and length of hospital stay
Rosenqvist, M., Fagerstrand, E., Lanbeck, P., Melander, O., & Åkesson, P. (2017) · Infectious Diseases · doi:10.1080/23744235.2017.1293840 · Cited by 33 · cited 2×
“The following scores were included in this study: Systemic Inflammatory Response Syndrome criteria, at least 2 of 4 criteria present [29]; Quick Sepsis-Related Organ Failure Assessment score, at least 2 of 3 criteria present [1]; Sepsis-Related Organ Failure Assessment score, total score ≥2 [1]; Modified Early Warning…”
MethodResultBi-Directional Gated Recurrent Unit Based Ensemble Model for the Early Detection of Sepsis
Wickramaratne, S., & Mahmud, M. S. (2020) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc44109.2020.9175223 · Cited by 26 · cited 6×
“” In other words, their model [39] seems to be a self-learning model, which would be the first of its kind if used in practice.”
BackgroundMethodInfluentialGraph Convolutional Networks-Based Noisy Data Imputation in Electronic Health Record
Lee, B. T., Kwon, O., Park, H., Cho, K.-J., Kwon, J., & Lee, Y. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004583 · Cited by 16 · cited 4×
“Only Futoma et al [35] used a comparable sepsis prevalence (21%), with other prevalences ranging between 6% and 9% (or not specified) [17,19,36-38].”
BackgroundMethodResultInfluentialSepsis early warning scoring systems: The ideal tool remains elusive!
Postelnicu, R., Pastores, S. M., Chong, D. H., & Evans, L. (2018) · Journal of Critical Care · doi:10.1016/j.jcrc.2018.07.009 · Cited by 5 · cited 1×
“Performance of the algorithm was compared with a number of illness severity risk scores currently used in clinical practice to predict sepsis in the same time frame (for a summary of sepsis early warning scoring systems, see Postelnicu et al [27] and Rosenqvist [28]).”
Result
Research works printed but not linked: 4
- AbadiMBarhamPChenJTensorFlow: a system for large-scale machine learningProceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16)201612th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16)November 2-4Savannah, GA, USA265283
- 2017, Updated Report of a Working Party
- Moor, M, 2019, Proc Mach Learn Res, 106, 2
- Futoma, J, 2017, Proc Mach Learn Res, 68, 243
Cited by
Predicting sepsis onset in ICU using machine learning models: a systematic review and meta-analysis
Yang, Z., Cui, X., & Song, Z. (2023) · BMC Infectious Diseases · doi:10.1186/s12879-023-08614-0 · Cited by 68
Artificial Intelligence in Sepsis Management: An Overview for Clinicians
Bignami, E. G., Berdini, M., Panizzi, M., Domenichetti, T., Bezzi, F., Allai, S., Damiano, T., & Bellini, V. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14010286 · Cited by 53
“Persson et al. [14] constructed a high-performance MLM for sepsis prediction called NAVOY Sepsis by using convolutional neural networks.”
Persson, I., Macura, A., Becedas, D., & Sjövall, F. (2024) · Journal of Critical Care · doi:10.1016/j.jcrc.2023.154400 · Cited by 51
Self-citationYadgarov, M. Y., Landoni, G., Бондоевич, Б. Л., Polyakov, P. A., Kadantseva, K., Smirnova, A. V., Kuznetsov, I. V., Shemetova, M. M., Yakovlev, A. А., & Likhvantsev, V. V. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1491358 · Cited by 64
Zubair, M., Din, I., Sarwar, N., Boltayevich, E. B., Mаkhmudov, S., & Trabelsi, Z. (2025) · BMC Infectious Diseases · doi:10.1186/s12879-025-11423-2 · Cited by 49
Metadata
- Authors
- Persson, I., Östling, A., Arlbrandt, M., Söderberg, J., & Becedas, D.
- Affiliations
- Uppsala University (SE)
- Stockholm South General Hospital (SE)
- Corresponding author
- Persson, Inger
- Venue
- JMIR Formative Research
- Publisher
- JMIR Publications Inc.
- Published
- 2021-09-30
- Volume
- 5
- Issue
- 9
- Pages
- e28000
- ISSN
- 2561-326X (ISSN-L)
- PMC
- PMC8517825
- OpenAlex
- W3193078598
- Cited by
- 58
- References
- 34
- Citations by year
- 2026: 13 · 2025: 14 · 2024: 13 · 2023: 11 · 2022: 7
- FWCI
- 5.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Formative Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Uppsala Universitet
- Skånes universitetssjukhus
- Topics
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- sepsis prediction, early sepsis detection, intensive care unit, convolutional neural networks, MIMIC-III, Sepsis-3 criteria, early warning scores, clinical decision support, routinely collected clinical data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.