CoChat
Sign Up
Cited by 251Machine Learning in HealthcareArtificial Intelligence in HealthcareSepsis Diagnosis and Treatment

Use of machine learning to analyse routinely collected intensive care unit data: a systematic review

Open access
Shillan, Duncan, Sterne, Jonathan A. C., Champneys, Alan, Gibbison, Ben·2019·Critical Care
DOI 10.1186/s13054-019-2564-9PMC PMC6704673
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Intensive care units (ICUs) face financial, bed management, and staffing constraints. Detailed data covering all aspects of patients' journeys into and through intensive care are now collected and stored in electronic health records: machine learning has been used to analyse such data in order to provide decision support to clinicians. METHODS: Systematic review of the applications of machine learning to routinely collected ICU data. Web of Science and MEDLINE databases were searched to identify candidate articles: those on image processing were excluded. The study aim, the type of machine learning used, the size of dataset analysed, whether and how the model was validated, and measures of predictive accuracy were extracted. RESULTS: Of 2450 papers identified, 258 fulfilled eligibility criteria. The most common study aims were predicting complications (77 papers [29.8% of studies]), predicting mortality (70 [27.1%]), improving prognostic models (43 [16.7%]), and classifying sub-populations (29 [11.2%]). Median sample size was 488 (IQR 108-4099): 41 studies analysed data on > 10,000 patients. Analyses focused on 169 (65.5%) papers that used machine learning to predict complications, mortality, length of stay, or improvement of health. Predictions were validated in 161 (95.2%) of these studies: the area under the ROC curve (AUC) was reported by 97 (60.2%) but only 10 (6.2%) validated predictions using independent data. The median AUC was 0.83 in studies of 1000-10,000 patients, rising to 0.94 in studies of > 100,000 patients. The most common machine learning methods were neural networks (72 studies [42.6%]), support vector machines (40 [23.7%]), and classification/decision trees (34 [20.1%]). Since 2015 (125 studies [48.4%]), the most common methods were support vector machines (37 studies [29.6%]) and random forests (29 [23.2%]). CONCLUSIONS: The rate of publication of studies using machine learning to analyse routinely collected ICU data is increasing rapidly. The sample sizes used in many published studies are too small to exploit the potential of these methods. Methodological and reporting guidelines are needed, particularly with regard to the choice of method and validation of predictions, to increase confidence in reported findings and aid in translating findings towards routine use in clinical practice.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13054-019-2564-9

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most AI tools remain poorly validated and untested in real settings.

    “Despite the need to support overwhelmed ICUs facing staff shortages, increasing case complexity, and rising costs, most AI tools remain poorly validated and untested in real settings. [3, 45]”

    From Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22. (2025) · cites this paper as [3]

    The Discussion (Implications) says only a small minority of reviewed ICU machine-learning studies validated algorithms on independent data, and that studies rarely discussed implementation or whether the algorithms improved care. This supports concerns about validation, but the review concerns ICU machine-learning studies rather than most AI tools generally; rarely discussing implementation does not establish that the tools were untested in real settings. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • AUROC is widely adopted for binary classification in ICU prediction tasks and offers threshold-independent evaluation, enabling fair comparison across hospitals without requiring model calibration.

    “AUROC is widely adopted for binary classification in ICU prediction tasks [12, 75] and offers threshold-independent evaluation, enabling fair comparison across hospitals without requiring model calibration.”

    From Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions (2026) · cites this paper as [12]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 44 · Cited by 251

Research works printed but not linked: 29

  1. [1]Xu H Wu W Nemati S Zha H Patient flow prediction via discriminative learning of mutually-correcting processes IEEE Trans Knowl Data Eng 2017 29 1 157 171
  2. [3]Ruyssinck J van der Herten J Houthooft R Ongenae F Couckuyt I Gadeyne B Random survival forests for predicting the bed occupancy in the intensive care unit Comput. 2016 2016 7087053 PMC5081505 27818706
  3. [4]Ngufor C Murphree D Upadhyaya S Madde N Pathak J Carter R Predicting prolonged stay in the ICU attributable to bleeding in patients offered plasma transfusion AMIA Annu Symp Proc 2016 2016 954 963 28269892 PMC5333266
  4. [6]Johnson AEW Ghassemi MM Nemati S Niehaus KE Clifton DA Clifford GD Machine learning and decision support in critical care Proc IEEE 2016 104 2 444 466 PMC5066876 27765959
  5. [7]Rajkomar A Dean J Kohane I Machine learning in medicine N Engl J Med 2019 380 14 1347 1358 30943338
  6. [9]Hanson CW 3rd Marshall BE Artificial intelligence applications in the intensive care unit Crit Care Med 2001 29 2 427 435 11269246
  7. [10]Johnson AE Pollard TJ Shen L Li-wei HL Feng M Ghassemi M MIMIC-III, a freely accessible critical care database Scientific Data 2016 3 160035 27219127 PMC4878278
  8. [11]Moher D Liberati A Tetzlaff J Altman DG The PRISMA Group Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement PLoS Med 2009 6 7 e1000097 19621072 PMC2707599
  9. [13]Koyner JL Carey KA Edelson DP Churpek MM The development of a machine learning inpatient acute kidney injury prediction model Crit Care Med 2018 46 7 1070 1077 29596073
  10. [15]Liu Y Traskin M Lorch SA George EI Small D Ensemble of trees approaches to risk adjustment for evaluating a hospital’s performance Health Care Manag Sci 2015 18 1 58 66 24777832
  11. [16]Mao Q Jay M Hoffman JL Calvert J Barton C Shimabukuro D Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU BMJ Open 2018 8 1 e017833 29374661 PMC5829820
  12. [17]Rowan M Ryan T Hegarty F O'Hare N The use of artificial neural networks to stratify the length of stay of cardiac patients based on preoperative and initial postoperative factors Artif Intell Med 2007 40 3 211 221 17580112
  13. [18]Meyfroidt G Guiza F Cottem D De Becker W Van Loon K Aerts JM Computerized prediction of intensive care unit discharge after cardiac surgery: development and validation of a Gaussian processes model BMC Med Inf Decis Mak. 2011 11 64 PMC3228706 22027016
  14. [19]Tu JV Guerriere MR Use of a neural network as a predictive instrument for length of stay in the intensive care unit following cardiac surgery Comput Biomed Res 1993 26 3 220 229 8325002
  15. [20]Houthooft R Ruyssinck J van der Herten J Stijven S Couckuyt I Gadeyne B Predictive modelling of survival and length of stay in critically ill patients using sequential organ failure scores Artif Intell Med 2015 63 3 191 207 25579436
  16. [21]Barbini E Cevenini G Scolletta S Biagioli B Giomarelli P Barbini P A comparative analysis of predictive models of morbidity in intensive care unit after cardiac surgery - part I: model planning BMC Med Inf Decis Mak 2007 7 35 PMC2212627 18034872
  17. [22]Awad A Bader-El-Den M McNicholas J Patient length of stay and mortality prediction: a survey Health Serv Manag Res 2017 30 2 105 120 28539083
  18. [24]Lisboa PJ A review of evidence of health benefit from artificial neural networks in medical intervention Neural Netw 2002 15 1 11 39 11958484
  19. [25]Beam AL Kohane IS Big data and machine learning in health care Jama. 2018 319 13 1317 1318 29532063
  20. [26]Kamio T Van T Masamune K Use of machine-learning approaches to predict clinical deterioration in critically ill patients: a systematic review Int J Med Res Health Sci 2017 6 6 1 7
  21. [27]Iacobucci G. Patient data were shared with Google on an “inappropriate legal basis,” says NHS data guardian. BMJ. 2017;357:j2439. 28522583
  22. [28]de Araujo JM de Menezes JM Moura de Albuquerque AA da Mota Almeida O Ugulino de Araujo FM Assessment and certification of neonatal incubator sensors through an inferential neural network Sensors (Basel) 2013 13 11 15613 15632 24248278 PMC3871086
  23. [30]Van Loon K Guiza F Meyfroidt G Aerts JM Ramon J Blockeel H Dynamic data analysis and data mining for prediction of clinical stability Stud Health Technol Inform 2009 150 590 594 19745380
  24. [31]Bailly Sébastien Meyfroidt Geert Timsit Jean-François What’s new in ICU in 2050: big data and machine learning Intensive Care Medicine 2017 44 9 1524 1527 29279970
  25. [36]Moher D Jones A Lepage L Group ftC Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation Jama. 2001 285 15 1992 1995 11308436
  26. [37]Kane RL Wang J Garrard J Reporting in randomized clinical trials improved after adoption of the CONSORT statement J Clin Epidemiol 2007 60 3 241 249 17292017
  27. [38]Collins GS Reitsma JB Altman DG Moons KGM Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement Ann Intern Med 2015 162 1 55 63 25560714
  28. [39]Wolff RF Moons KGM Riley RD Whiting PF Westwood M Collins GS PROBAST: a tool to assess the risk of bias and applicability of prediction model studies PROBAST (Prediction model Risk Of Bias ASsessment Tool) Ann Intern Med 2019 170 1 51 58 30596875
  29. [40]Johnson AE Pollard TJ Mark RG Reproducibility in critical care: a mortality prediction case study. Machine Learning for Healthcare Conference 2017

Authors
Shillan, D., Sterne, J. A. C., Champneys, A., & Gibbison, B.
Affiliations
  • University of Bristol (GB)
  • Bristol Royal Infirmary (GB)
Corresponding author
Gibbison, Ben
ORCID
Venue
Critical Care
Publisher
Springer Science and Business Media LLC
Published
2019-08-22
Volume
23
Issue
1
Pages
284-284
ISSN
1364-8535 (ISSN-L)
DOI
10.1186/s13054-019-2564-9
PMC
PMC6704673
OpenAlex
W2969097171
Cited by
251
References
44
Citations by year
2026: 34 · 2025: 47 · 2024: 26 · 2023: 46 · 2022: 44 · 2021: 32 · 2020: 22
FWCI
1.63
Type
journal article
Language
en
Open access
Licence
Locations
  • Critical Care (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Explore Bristol Research (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Institute for Health Research (BRC at University of Bristol and UH Bristol NHSFT, BRC at the University of Bristol and UH Bristol NHS FT, BRC at The University of Bristol and UH Bristol NHS FT)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
Keywords
machine learning, intensive care unit, electronic health records, clinical decision support, mortality prediction, complication prediction, prognostic model, model validation, sample size, random forest, support vector machine, artificial neural networks, length of hospital stay, reporting guidelines, methodological quality, health technology assessment
MeSH terms
Machine Learning, Data Analysis, Adult, Female, Humans, Intensive Care Units, Male, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review