CoChat
Sign Up
Cited by 14Sepsis Diagnosis and TreatmentInflammation biomarkers and pathwaysHyperglycemia and glycemic control in critically ill and hospitalized patients

Ground truth labels challenge the validity of sepsis consensus definitions in critical illness.

Open access
Lindner, Holger A., Schamoni, Shigehiko, Kirschning, Thomas, Worm, Corinna, Hahn, Bianka, Centner, Franz-Simon, Schoettler, Jochen J., Hagmann, Michael +6 more·2022·Journal of Translational Medicine
DOI 10.1186/s12967-022-03228-7PMC PMC8760797
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Sepsis is the leading cause of death in the intensive care unit (ICU). Expediting its diagnosis, largely determined by clinical assessment, improves survival. Predictive and explanatory modelling of sepsis in the critically ill commonly bases both outcome definition and predictions on clinical criteria for consensus definitions of sepsis, leading to circularity. As a remedy, we collected ground truth labels for sepsis. METHODS: In the Ground Truth for Sepsis Questionnaire (GTSQ), senior attending physicians in the ICU documented daily their opinion on each patient's condition regarding sepsis as a five-category working diagnosis and nine related items. Working diagnosis groups were described and compared and their SOFA-scores analyzed with a generalized linear mixed model. Agreement and discriminatory performance measures for clinical criteria of sepsis and GTSQ labels as reference class were derived. RESULTS: We analyzed 7291 questionnaires and 761 complete encounters from the first survey year. Editing rates for all items were > 90%, and responses were consistent with current understanding of critical illness pathophysiology, including sepsis pathogenesis. Interrater agreement for presence and absence of sepsis was almost perfect but only slight for suspected infection. ICU mortality was 19.5% in encounters with SIRS as the "worst" working diagnosis compared to 5.9% with sepsis and 5.9% with severe sepsis without differences in admission and maximum SOFA. Compared to sepsis, proportions of GTSQs with SIRS plus acute organ dysfunction were equal and macrocirculatory abnormalities higher (p < 0.0001). SIRS proportionally ranked above sepsis in daily assessment of illness severity (p < 0.0001). Separate analyses of neurosurgical referrals revealed similar differences. Discriminatory performance of Sepsis-1/2 and Sepsis-3 compared to GTSQ labels was similar with sensitivities around 70% and specificities 92%. Essentially no difference between the prevalence of SIRS and SOFA ≥ 2 yielded sensitivities and specificities for detecting sepsis onset close to 55% and 83%, respectively. CONCLUSIONS: GTSQ labels are a valid measure of sepsis in the ICU. They reveal suspicion of infection as an unclear clinical concept and refute an illness severity hierarchy in the SIRS-sepsis-severe sepsis spectrum. Ground truth challenges the accuracy of Sepsis-1/2 and Sepsis-3 in detecting sepsis onset. It is an indispensable intermediate step towards advancing diagnosis and therapy in the ICU and, potentially, other health care settings.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12967-022-03228-7

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • This study used Sepsis-3 clinical criteria to label sepsis cases, an approach that has imperfect concordance with the criterion standard approach of manual physician review.

    “First, this study used Sepsis-3 clinical criteria to label sepsis cases, an approach that has imperfect concordance with the criterion standard approach of manual physician review.25,26,27,28”

    From Multicenter Prospective Validation of an Updated Proprietary Sepsis Prediction Model (2026) · cites this paper as [25]

    The Results section ('Validation of Sepsis-1/2 and Sepsis-3 clinical criteria against ground truth labels'; page unavailable) reports only fair agreement between Sepsis-3 criteria and expert sepsis-onset labels, supporting the imperfect-concordance point. The Methods describe those reference labels as daily judgments by senior physicians using a questionnaire, not sepsis labels established through manual physician chart review. The cited paper also cannot establish how the citing study labeled its own cases. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 58

  1. [1]Rudd KE Johnson SC Agesa KM Shackelford KA Tsoi D Kievlan DR Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the Global Burden of Disease Study Lancet 2020 395 10219 200 211 10.1016/S0140-6736(19)32989-7 31954465 PMC6970225 · DOI · Paper page
  2. [2]van Vught LA Klein Klouwenberg PM Spitoni C Scicluna BP Wiewel MA Horn J Incidence, risk factors, and attributable mortality of secondary infections in the intensive care unit after admission for sepsis JAMA 2016 315 14 1469 1479 10.1001/jama.2016.2691 26975785 · DOI
  3. [3]Sakr Y Jaschinski U Wittebole X Szakmany T Lipman J Namendys-Silva SA Sepsis in intensive care unit patients: worldwide data from the Intensive Care over Nations audit Open Forum Infect Dis 2018 5 12 ofy313 10.1093/ofid/ofy313 30555852 PMC6289022 · DOI
  4. [4]Kumar A Roberts D Wood KE Light B Parrillo JE Sharma S Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock Crit Care Med 2006 34 6 1589 1596 10.1097/01.CCM.0000217961.75225.E9 16625125 · DOI · Paper page
  5. [6]Ferrer R Martin-Loeches I Phillips G Osborn TM Townsend S Dellinger RP Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour: results from a guideline-based performance improvement program Crit Care Med 2014 42 8 1749 1755 10.1097/CCM.0000000000000330 24717459 · DOI
  6. [7]Whiles BB Deis AS Simpson SQ Increased time to initial antimicrobial administration is associated with progression to septic shock in severe sepsis patients Crit Care Med 2017 45 4 623 629 10.1097/CCM.0000000000002262 28169944 PMC5374449 · DOI
  7. [8]Peltan ID Mitchell KH Rudd KE Mann BA Carlbom DJ Hough CL Physician variation in time to antimicrobial treatment for septic patients presenting to the emergency department Crit Care Med 2017 45 6 1011 1018 10.1097/CCM.0000000000002436 28426466 PMC5439956 · DOI
  8. [10]Giacobbe DR Signori A Del Puente F Mora S Carmisciano L Briano F Early detection of sepsis with machine learning techniques: a brief clinical perspective Front Med (Lausanne). 2021 8 617486 10.3389/fmed.2021.617486 33644097 PMC7906970 · DOI
  9. [11]Bates B Gathering data. A guide to clinical thinking 1995 Philadelphia J.B. Lippincott 1 8
  10. [12]Bone RC Balk RA Cerra FB Dellinger RP Fein AM Knaus WA Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine Chest 1992 101 6 1644 1655 10.1378/chest.101.6.1644 1303622 · DOI
  11. [13]Rangel-Frausto MS Pittet D Costigan M Hwang T Davis CS Wenzel RP The natural history of the systemic inflammatory response syndrome (SIRS). A prospective study JAMA 1995 273 2 117 123 10.1001/jama.1995.03520260039030 7799491 · DOI
  12. [15]Simpson SQ SIRS in the time of sepsis-3 Chest 2018 153 1 34 38 10.1016/j.chest.2017.10.006 29037526 · DOI
  13. [16]Levy MM Fink MP Marshall JC Abraham E Angus D Cook D 2001 SCCM/ESICM/ACCP/ATS/SIS international sepsis definitions conference Crit Care Med 2003 31 4 1250 1256 10.1097/01.CCM.0000050454.01978.3B 12682500 · DOI
  14. [17]Dellinger RP Levy MM Rhodes A Annane D Gerlach H Opal SM Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012 Intensive Care Med 2013 39 2 165 228 10.1007/s00134-012-2769-8 23361625 PMC7095153 · DOI
  15. [18]Singer M Deutschman CS Seymour CW Shankar-Hari M Annane D Bauer M The third international consensus definitions for sepsis and septic shock (Sepsis-3) JAMA 2016 315 8 801 810 10.1001/jama.2016.0287 26903338 PMC4968574 · DOI · Paper page
  16. [19]Rhee C Kadri SS Danner RL Suffredini AF Massaro AF Kitch BT Diagnosing sepsis is subjective and highly variable: a survey of intensivists using case vignettes Crit Care 2016 20 89 10.1186/s13054-016-1266-9 27048508 PMC4822273 · DOI · Paper page
  17. [20]Levy MM Ward NS Ward NS Levy MM Introduction Sepsis 2017 Providence Humana Press 3 5
  18. [22]Cosgriff CV Celi LA Stone DJ Critical care, critical data Biomed Eng Comput Biol 2019 10 1179597219856564 10.1177/1179597219856564 31217702 PMC6563388 · DOI
  19. [24]Yoon JH Pinsky MR Predicting adverse hemodynamic events in critically ill patients Curr Opin Crit Care 2018 24 3 196 203 10.1097/MCC.0000000000000496 29601321 PMC6007856 · DOI
  20. [25]Saria S Henry KE Too many definitions of sepsis: can machine learning leverage the electronic health record to increase accuracy and bring consensus? Crit Care Med 2020 48 2 137 141 10.1097/CCM.0000000000004144 31939780 · DOI
  21. [26]Gaieski DF Edwards JM Kallan MJ Carr BG Benchmarking the incidence and mortality of severe sepsis in the United States Crit Care Med 2013 41 5 1167 1174 10.1097/CCM.0b013e31827c09f8 23442987 · DOI
  22. [27]Jolley RJ Sawka KJ Yergens DW Quan H Jette N Doig CJ Validity of administrative data in recording sepsis: a systematic review Crit Care 2015 19 139 10.1186/s13054-015-0847-3 25887596 PMC4403835 · DOI
  23. [28]Mariansdatter SE Eiset AH Sogaard KK Christiansen CF Differences in reported sepsis incidence according to study design: a literature review BMC Med Res Methodol 2016 16 1 137 10.1186/s12874-016-0237-9 27733132 PMC5062833 · DOI
  24. [29]Shankar-Hari M Harrison DA Rowan KM Differences in impact of definitional elements on mortality precludes international comparisons of sepsis epidemiology-a cohort study illustrating the need for standardized reporting Crit Care Med 2016 44 12 2223 2230 10.1097/CCM.0000000000001876 27352126 · DOI
  25. [30]Wulff A Montag S Marschollek M Jack T Clinical decision-support systems for detection of systemic inflammatory response syndrome, sepsis, and septic shock in critically ill patients: a systematic review Methods Inf Med 2019 58 S 02 e43 e57 10.1055/s-0039-1695717 31499571 · DOI
  26. [31]Rhee C Dantes RB Epstein L Klompas M Using objective clinical data to track progress on preventing and treating sepsis: CDC's new 'Adult Sepsis Event' surveillance strategy BMJ Qual Saf 2019 28 4 305 309 10.1136/bmjqs-2018-008331 30254095 PMC6557151 · DOI
  27. [32]Yu SC Betthauser KD Gupta A Lyons PG Lai AM Kollef MH Comparison of sepsis definitions as automated criteria Crit Care Med 2021 49 4 e433 e443 10.1097/CCM.0000000000004875 33591014 · DOI · Paper page
  28. [33]Seymour CW Liu VX Iwashyna TJ Brunkhorst FM Rea TD Scherag A Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (sepsis-3) JAMA 2016 315 8 762 774 10.1001/jama.2016.0288 26903335 PMC5433435 · DOI · Paper page
  29. [34]Nemati S Holder A Razmi F Stanley MD Clifford GD Buchman TG An interpretable machine learning model for accurate prediction of sepsis in the ICU Crit Care Med 2018 46 4 547 553 10.1097/CCM.0000000000002936 29286945 PMC5851825 · DOI · Paper page
  30. [35]Liu R Greenstein JL Granite SJ Fackler JC Bembea MM Sarma SV Data-driven discovery of a novel sepsis pre-shock state predicts impending septic shock in the ICU Sci Rep 2019 9 1 6145 10.1038/s41598-019-42637-5 30992534 PMC6467982 · DOI
  31. [36]Ghalati PF Samal SS Bhat JS Deisz R Marx G Schuppert A Critical transitions in intensive care units: a sepsis case study Sci Rep 2019 9 1 12888 10.1038/s41598-019-49006-2 31501451 PMC6733794 · DOI
  32. [37]Desautels T Calvert J Hoffman J Jay M Kerem Y Shieh L Prediction of sepsis in the intensive care unit with minimal electronic health record data: a machine learning approach JMIR Med Inform 2016 4 3 e28 10.2196/medinform.5909 27694098 PMC5065680 · DOI · Paper page
  33. [39]Barton C Chettipally U Zhou Y Jiang Z Lynn-Palevsky A Le S Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs Comput Biol Med 2019 109 79 84 10.1016/j.compbiomed.2019.04.027 31035074 PMC6556419 · DOI · Paper page
  34. [40]Kim J Chang H Kim D Jang DH Park I Kim K Machine learning for prediction of septic shock at initial triage in emergency department J Crit Care 2020 55 163 170 10.1016/j.jcrc.2019.09.024 31734491 · DOI
  35. [41]Reyna MA Josef CS Jeter R Shashikumar SP Westover MB Nemati S Early prediction of sepsis from clinical data: the PhysioNet/computing in cardiology challenge 2019 Crit Care Med 2020 48 2 210 217 10.1097/CCM.0000000000004145 31939789 PMC6964870 · DOI · Paper page
  36. [42]Norman G Young M Brooks L Non-analytical models of clinical reasoning: the role of experience Med Educ 2007 41 12 1140 1145 10.1111/j.1365-2923.2007.02914.x 18004990 · DOI
  37. [43]Islam R Weir CR Jones M Del Fiol G Samore MH Understanding complex clinical reasoning in infectious diseases for improving clinical decision support design BMC Med Inform Decis Mak 2015 15 101 10.1186/s12911-015-0221-z 26620881 PMC4665869 · DOI
  38. [44]Schamoni S Lindner HA Schneider-Lindner V Thiel M Riezler S Leveraging implicit expert knowledge for non-circular machine learning in sepsis prediction Artif Intell Med 2019 100 101725 10.1016/j.artmed.2019.101725 31607345 · DOI
  39. [45]Hagmann M, Riezler S. False perfection in machine prediction: detecting and assessing circularity problems in machine learning 2021 June 01. 2021:[arXiv:2106.12417 p.]. https://ui.adsabs.harvard.edu/abs/2021arXiv210612417H. · arXiv
  40. [46]Fagerström J Bang M Wilhelms D Chew MS LiSep LSTM: a machine learning algorithm for early detection of septic shock Sci Rep 2019 9 1 15132 10.1038/s41598-019-51219-4 31641162 PMC6805937 · DOI
  41. [47]Meisel C Schwab JM Prass K Meisel A Dirnagl U Central nervous system injury-induced immune deficiency syndrome Nat Rev Neurosci 2005 6 10 775 786 10.1038/nrn1765 16163382 · DOI
  42. [48]Chen S Li Q Wu H Krafft PR Wang Z Zhang JH The harmful effects of subarachnoid hemorrhage on extracerebral organs Biomed Res Int 2014 2014 858496 10.1155/2014/858496 25110700 PMC4109109 · DOI
  43. [49]Vincent JL Moreno R Takala J Willatts S De Mendonca A Bruining H The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine Intensive Care Med 1996 22 7 707 710 10.1007/BF01709751 8844239 · DOI · Paper page
  44. [50]Centner FS Schoettler JJ Fairley AM Lindner HA Schneider-Lindner V Weiss C Impact of different consensus definition criteria on sepsis diagnosis in a cohort of critically ill patients-Insights from a new mathematical probabilistic approach to mortality-based validation of sepsis criteria PLoS ONE 2020 15 9 e0238548 10.1371/journal.pone.0238548 32898161 PMC7478755 · DOI
  45. [51]Byrt T Bishop J Carlin JB Bias, prevalence and kappa J Clin Epidemiol 1993 46 5 423 429 10.1016/0895-4356(93)90018-v 8501467 · DOI · Paper page
  46. [52]Landis JR Koch GG The measurement of observer agreement for categorical data Biometrics 1977 33 1 159 174 10.2307/2529310 843571 · DOI · Paper page
  47. [53]Sharshar T Hopkinson NS Orlikowski D Annane D Science review: the brain in sepsis–culprit and victim Crit Care 2005 9 1 37 44 10.1186/cc2951 15693982 PMC1065096 · DOI
  48. [54]Vincent JL Jones G David S Olariu E Cadwell KK Frequency and mortality of septic shock in Europe and North America: a systematic review and meta-analysis Crit Care 2019 23 1 196 10.1186/s13054-019-2478-6 31151462 PMC6545004 · DOI
  49. [55]Engel C Brunkhorst FM Bone HG Brunkhorst R Gerlach H Grond S Epidemiology of sepsis in Germany: results from a national prospective multicenter study Intensive Care Med 2007 33 4 606 618 10.1007/s00134-006-0517-7 17323051 · DOI
  50. [56]Vincent JL Rello J Marshall J Silva E Anzueto A Martin CD International study of the prevalence and outcomes of infection in intensive care units JAMA 2009 302 21 2323 2329 10.1001/jama.2009.1754 19952319 · DOI
  51. [57]Shankar-Hari M Harrison DA Rubenfeld GD Rowan K Epidemiology of sepsis and septic shock in critical care units: comparison between sepsis-2 and sepsis-3 populations using a national critical care database Br J Anaesth 2017 119 4 626 636 10.1093/bja/aex234 29121281 · DOI
  52. [58]Dhar R Diringer MN The burden of the systemic inflammatory response predicts vasospasm and outcome after subarachnoid hemorrhage Neurocrit Care 2008 8 3 404 412 10.1007/s12028-008-9054-2 18196475 PMC2538678 · DOI
  53. [59]Boehme AK Hays AN Kicielinski KP Arora K Kapoor N Lyerly MJ Systemic inflammatory response syndrome and outcomes in intracerebral hemorrhage Neurocrit Care 2016 25 1 133 140 10.1007/s12028-016-0255-9 26920909 PMC4955667 · DOI
  54. [60]Rass V Gaasch M Kofler M Schiefecker AJ Ianosi BA Rhomberg P Systemic inflammatory response syndrome as predictor of poor outcome in nontraumatic subarachnoid hemorrhage patients Crit Care Med 2018 46 12 e1152 e1159 10.1097/CCM.0000000000003429 30252711 · DOI
  55. [61]Angus DC Seymour CW Coopersmith CM Deutschman CS Klompas M Levy MM A framework for the development and interpretation of different sepsis definitions and clinical criteria Crit Care Med 2016 44 3 e113 e121 10.1097/CCM.0000000000001730 26901559 PMC4765912 · DOI
  56. [63]Zhao X Liu JS Deng K Assumptions behind intercoder reliability indices Ann Int Commun Assoc 2013 36 1 419 480 10.1080/23808985.2013.11679142 · DOI · Paper page
  57. [64]Lindner HA Balaban U Sturm T Weiss C Thiel M Schneider-Lindner V An algorithm for systemic inflammatory response syndrome criteria-based prediction of sepsis in a polytrauma cohort Crit Care Med 2016 44 12 2199 2207 10.1097/CCM.0000000000001955 27441906 · DOI
  58. [65]Angus DC Opening the debate on the new sepsis definition defining sepsis: a case of bounded rationality and fuzzy thinking? Am J Respir Crit Care Med 2016 194 1 14 15 10.1164/rccm.201604-0879ED 27166833 PMC4960635 · DOI

Cited by

Sorted by citations

Authors
Lindner, H. A., Schamoni, S., Kirschning, T., Worm, C., Hahn, B., Centner, F.-S., Schoettler, J. J., Hagmann, M., Krebs, J., Mangold, D., Nitsch, S., Riezler, S., Thiel, M., & Schneider-Lindner, V.
Affiliations
Heidelberg University (DE)
Corresponding author
Lindner, Holger A.
ORCID
Venue
Journal of Translational Medicine
Publisher
Springer Science and Business Media LLC
Published
2022-01-15
Volume
20
Issue
1
Pages
27-27
ISSN
1479-5876 (ISSN-L)
DOI
10.1186/s12967-022-03228-7
PMC
PMC8760797
OpenAlex
W4206809756
Cited by
14
References
7
Citations by year
2026: 5 · 2024: 3 · 2023: 5 · 2022: 1
FWCI
1.62
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Translational Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Mspace (University of Manitoba) (submittedVersion)
Funders
  • Klaus Tschira Stiftung (00.0277.2015)
  • Ruprecht-Karls-Universität Heidelberg
Topics
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
  • Life Sciences › Immunology and Microbiology › Immunology › Inflammation biomarkers and pathways
  • Health Sciences › Medicine › Endocrinology, Diabetes and Metabolism › Hyperglycemia and glycemic control in critically ill and hospitalized patients
Keywords
sepsis, sepsis definitions, Sepsis-3, systemic inflammatory response syndrome, SOFA score, ground truth labels, intensive care unit, suspected infection, organ dysfunction, interobserver agreement, neurosurgical patients, ICU mortality, disease severity
MeSH terms
Consensus Statements as Topic, Delivery of Health Care, Humans, Prognosis, Retrospective Studies, Critical Illness, Hospital Mortality, Systemic Inflammatory Response Syndrome, Sepsis, Consensus, Organ Dysfunction Scores

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review