CoChat

Prediction of Postoperative ICU Requirements: Closing the Translational Gap with a Real-World Clinical Benchmark for Artificial Intelligence Approaches

Open access

The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index; 31 of the 35 sources its claims cite were checked for support.

Abstract

Background: Accurate prediction of postoperative care requirements is critical for patient safety and resource allocation. Although numerous approaches involving artificial intelligence (AI) and machine learning (ML) have been proposed to support such predictions, their implementation in practice has so far been insufficiently successful. One reason for this is that the performance of the algorithms is difficult to assess in practical use, as the accuracy of clinical decisions has not yet been systematically quantified. As a result, models are often assessed purely from a technical perspective, neglecting the socio-technical context. Methods: We conducted a retrospective, single-center observational study at the University Hospital Augsburg, including 35,488 elective surgical cases documented between August 2023 and January 2025. For each case, preoperative care-level predictions by surgical and anesthesiology teams were compared with the actual postoperative care provided. Predictive performance was evaluated using accuracy and sensitivity. Since this is a highly imbalanced dataset, in addition to sensitivity and specificity, the balanced accuracy and the Fβ-score were also calculated. The results were contrasted with published Machine-Learning (ML)-based approaches. Results: Overall prediction accuracy was high (surgery: 91.2%; anesthesiology: 87.1%). However, sensitivity for identifying patients requiring postoperative intensive care was markedly lower than reported for ML models in the literature, with the largest discrepancies observed in patients ultimately admitted to the ICU (surgery: 38.05%; anesthesiology: 56.84%; ML: 70%). Nevertheless, clinical judgment demonstrated a superior F1-score, indicating a more balanced performance between sensitivity and precision (surgery: 0.527; anesthesiology: 0.551; ML: 0.28). Conclusions: This study provides the first real-world benchmark of clinical expertise in postoperative care prediction and shows a way in which modern ML approaches must be evaluated in a specific sociotechnical context. By quantifying the predictive performance of surgeons and anesthesiologists, it enables an evaluation of existing ML approaches. Thus the strength of our work is the provision of a real-world benchmark against which all ML methods for preoperative prediction of ICU demand can be systematically evaluated. This enables, for the first time, a comparison of different approaches on a common, practice-oriented basis and thus significantly facilitates translation into clinical practice, thereby closing the translational gap. Furthermore it offers a data-driven framework to support the integration of ML into preoperative decision-making.

Overview

CoChat Summary

TL;DR

This retrospective single-center study benchmarks preoperative clinician predictions of postoperative care level against actual care in 35,488 elective surgical cases, finding high overall accuracy but low sensitivity for ICU-level needs. It proposes this clinical performance as a reference benchmark for future AI/ML models.

Key findings

  • Overall prediction accuracy was 91.17% for surgery and 87.12% for anesthesiology; approximately 9% and 13% of cases, respectively, did not match the actual postoperative level of care.
  • For patients ultimately requiring Level 3 ICU care, sensitivity was 38.05% for surgery and 56.84% for anesthesiology, compared with 70% reported for ML models in the literature; F1-scores were 0.527 for surgery, 0.551 for anesthesiology, and 0.28 for the cited ML model.
  • Level 0 dominated the dataset (90.3% prevalence): sensitivity for Level 0 was 98.21% for surgery and 91.14% for anesthesiology, while specificity differed (41.78% vs. 69.46%). Level 1 sensitivity was very low (9.88% surgery, 26.59% anesthesiology), and interrater agreement was low (Kappa = 0.0263 surgery, 0.0408 anesthesiology).
  • Among 28,976 cases where both specialties predicted Level 0 and this was confirmed, some patients still required higher care (Level 1: n = 485; Level 2: n = 114; Level 3: n = 161).

Methods

  • Retrospective, single-center observational study at University Hospital Augsburg. From 46,830 recorded cases, 11,342 emergency cases were excluded, leaving 35,488 elective procedures from 1 August 2023 to 31 January 2025. For each case, surgical and anesthesiologic preoperative predictions were compared with the actual postoperative level of care.
  • Four care levels were defined: Level 0 = standard recovery room/PACU followed by timely transfer to general ward; Level 1 = prolonged PACU stay of at least 4 h; Level 2 = postoperative IMC monitoring; Level 3 = postoperative ICU care.
  • Performance metrics included accuracy, sensitivity, specificity, balanced accuracy, precision, Fβ-score (β = 0.5, 1, 2), Cohen’s Kappa, and Chi-square tests; analyses used R and Python. No ML model was trained; published ML results were used only for contextual comparison.

Limitations

  • Stated in the paper: overall accuracy is influenced by the high proportion of Level 0 cases and the system default to Level 0; physician training level was not systematically recorded; physicians’ reasons for predictions were not recorded; the single-center design limits generalizability; center-specific protocols may influence postoperative care levels; reasons for the actual care level realized were unavailable; no ML model was developed or trained; and the comparison with published ML results is illustrative rather than a direct evaluation.
  • Evident from the text: the retrospective design and severe class imbalance limit inference, Level 1 was almost indistinguishable in practice, and no judgment could be made about the medical appropriateness of the observed care allocation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/info16100888

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

17 Valid · 5 Not valid · 4 Not checked · 86 Own results and interpretations

49 more claims · 5 with their cited sources checked

Create a free account to see all 112 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 37
  • It may state two separate claims as one. 28

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 41 references

Research works printed but not linked: 1

  1. Devereaux, 2011, An international prospective cohort study evaluating major vascular complications among patients undergoing noncardiac surgery: The VISION Pilot Study, Open Med., 5, e193

Metadata

Authors
Althammer, A., Berger, F., Spring, O., Simon, P., Girrbach, F., Dieing, M., Brunner, J. O., Shmygalev, S., Bartenschlager, C. C., & Heller, A. R.
Affiliations
University of Augsburg (DE)
Corresponding author
Althammer, Alexander
Venue
Information
Publisher
MDPI AG
Published
2025-10-13
Volume
16
Issue
10
Pages
888
ISSN
2078-2489 (ISSN-L)
OpenAlex
W4415280985
Cited by
0
References
37
FWCI
0.00
Type
journal article
Language
en
Open access
Locations
  • Information (publishedVersion, cc-by)
  • Preprints.org (acceptedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
German Federal Ministry of Education and Research (16SV9030)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
ICU admission prediction, elective surgery, clinical decision-making, machine learning model evaluation, translational gap, predictive performance, sensitivity and specificity, resource allocation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Prediction of Postoperative ICU Requirements: Closing the Translational Gap with a Real-World Clinical Benchmark for Artificial Intelligence Approaches. https://app.cochat.ai/papers/10.3390/info16100888
The paper
Althammer, A., Berger, F., Spring, O., Simon, P., Girrbach, F., Dieing, M., Brunner, J. O., Shmygalev, S., Bartenschlager, C. C., & Heller, A. R. (2025). Prediction of Postoperative ICU Requirements: Closing the Translational Gap with a Real-World Clinical Benchmark for Artificial Intelligence Approaches. Information. https://doi.org/10.3390/info16100888

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.