CoChat
Sign Up
Cited by 205Machine Learning in HealthcareCardiac, Anesthesia and Surgical OutcomesArtificial Intelligence in Healthcare

The DOI matches its Crossref record; 37 of the 39 references it cites were found in a registry or an index.

Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study

Open access
Corey, Kristin M., Kashyap, Sehj, Lorenzi, Elizabeth, Lagoo-Deenadayalan, Sandhya A., Heller, Katherine, Whalen, Krista, Balu, Suresh, Heflin, Mitchell T., McDonald, Shelley R., Swaminathan, Madhav, Sendak, Mark·2018·PLOS Medicine
DOI 10.1371/journal.pmed.1002701PMC PMC6258507
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Pythia is an automated, clinically curated surgical data pipeline and repository housing all surgical patient electronic health record (EHR) data from a large, quaternary, multisite health institute for data science initiatives. In an effort to better identify high-risk surgical patients from complex data, a machine learning project trained on Pythia was built to predict postoperative complication risk. METHODS AND FINDINGS: A curated data repository of surgical outcomes was created using automated SQL and R code that extracted and processed patient clinical and surgical data across 37 million clinical encounters from the EHRs. A total of 194 clinical features including patient demographics (e.g., age, sex, race), smoking status, medications, comorbidities, procedure information, and proxies for surgical complexity were constructed and aggregated. A cohort of 66,370 patients that had undergone 99,755 invasive procedural encounters between January 1, 2014, and January 31, 2017, was studied further for the purpose of predicting postoperative complications. The average complication and 30-day postoperative mortality rates of this cohort were 16.0% and 0.51%, respectively. Least absolute shrinkage and selection operator (lasso) penalized logistic regression, random forest models, and extreme gradient boosted decision trees were trained on this surgical cohort with cross-validation on 14 specific postoperative outcome groupings. Resulting models had area under the receiver operator characteristic curve (AUC) values ranging between 0.747 and 0.924, calculated on an out-of-sample test set from the last 5 months of data. Lasso penalized regression was identified as a high-performing model, providing clinically interpretable actionable insights. Highest and lowest performing lasso models predicted postoperative shock and genitourinary outcomes with AUCs of 0.924 (95% CI: 0.901, 0.946) and 0.780 (95% CI: 0.752, 0.810), respectively. A calculator requiring input of 9 data fields was created to produce a risk assessment for the 14 groupings of postoperative outcomes. A high-risk threshold (15% risk of any complication) was determined to identify high-risk surgical patients. The model sensitivity was 76%, with a specificity of 76%. Compared to heuristics that identify high-risk patients developed by clinical experts and the ACS NSQIP calculator, this tool performed superiorly, providing an improved approach for clinicians to estimate postoperative risk for patients. Limitations of this study include the missingness of data that were removed for analysis. CONCLUSIONS: Extracting and curating a large, local institution's EHR data for machine learning purposes resulted in models with strong predictive performance. These models can be used in clinical settings as decision support tools for identification of high-risk patients as well as patient evaluation and care management. Further work is necessary to evaluate the impact of the Pythia risk calculator within the clinical workflow on postoperative outcomes and to optimize this data flow for future machine learning efforts.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pmed.1002701

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 SQL and R code were written to extract, clean, and curate patient data from the electronic health record (EHR) data warehouse. Extracted raw data elements included inpatient and outpatient encounter information, Current Procedural Terminology (CPT) codes, patient demographics, International Classification of Diseases (ICD) codes, medications, and vitals. These elements were curated into clinical features to feed into a cleaning data pipeline to populate Pythia’s data repository. Vars, variables.
Fig 1 SQL and R code were written to extract, clean, and curate patient data from the electronic health record (EHR) data warehouse. Extracted raw data elements included inpatient and outpatient encounter information, Current Procedural Terminology (CPT) codes, patient demographics, International Classification of Diseases (ICD) codes, medications, and vitals. These elements were curated into clinical features to feed into a cleaning data pipeline to populate Pythia’s data repository. Vars, variables.CC BY · PubMed Central
Fig 2 Graphs displaying the resulting receiver operator characteristic curves for each modeling method across all 14 complications. comp, complication; endo, endocrine; gastro, gastrointestinal; genit, genitourinary; hemat, hematological; lasso, least absolute shrinkage and selection operator; neuro, neurological; pulm, pulmonary.
Fig 2 Graphs displaying the resulting receiver operator characteristic curves for each modeling method across all 14 complications. comp, complication; endo, endocrine; gastro, gastrointestinal; genit, genitourinary; hemat, hematological; lasso, least absolute shrinkage and selection operator; neuro, neurological; pulm, pulmonary.CC BY · PubMed Central
Fig 3 Boxplots displaying the model predicted probabilities for patients who experienced a postoperative complication versus those who did not. Any postoperative complication (category = 1) versus no complications (category = 0). Thresholds (dashed lines) were chosen based on these population delineations. lasso, least absolute shrinkage and selection operator.
Fig 3 Boxplots displaying the model predicted probabilities for patients who experienced a postoperative complication versus those who did not. Any postoperative complication (category = 1) versus no complications (category = 0). Thresholds (dashed lines) were chosen based on these population delineations. lasso, least absolute shrinkage and selection operator.CC BY · PubMed Central
Fig 4 Predicted 30-day complication rate based on lasso model versus observed complication rates. January 2014–September 2016 shows predicted complication rates of the training data, and October 2016–January 2017 displays projected complication rates based on a non-random, out-of-sample test set from a different time period. lasso, least absolute shrinkage and selection operator.
Fig 4 Predicted 30-day complication rate based on lasso model versus observed complication rates. January 2014–September 2016 shows predicted complication rates of the training data, and October 2016–January 2017 displays projected complication rates based on a non-random, out-of-sample test set from a different time period. lasso, least absolute shrinkage and selection operator.CC BY · PubMed Central
Fig 5 Plotted AUCs for 30-day postoperative mortality in Pythia versus ACS NSQIP. ACS, American College of Surgeons; AUC, area under the receiver operator characteristic curve; NSQIP, National Surgical Quality Improvement Program.
Fig 5 Plotted AUCs for 30-day postoperative mortality in Pythia versus ACS NSQIP. ACS, American College of Surgeons; AUC, area under the receiver operator characteristic curve; NSQIP, National Surgical Quality Improvement Program.CC BY · PubMed Central
Fig 6 R Shiny application hosting the preoperative risk prediction calculator to use for preoperative patient evaluation in clinic. The calculator includes 9 input fields and calculates a percent risk for each of the 14 postoperative outcomes per patient. The calculator also identifies patients who meet the determined high-risk threshold.
Fig 6 R Shiny application hosting the preoperative risk prediction calculator to use for preoperative patient evaluation in clinic. The calculator includes 9 input fields and calculates a percent risk for each of the 14 postoperative outcomes per patient. The calculator also identifies patients who meet the determined high-risk threshold.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

Sorted by citations

See all 205

Related

See all

Metadata

Authors
Corey, K. M., Kashyap, S., Lorenzi, E., Lagoo-Deenadayalan, S. A., Heller, K., Whalen, K., Balu, S., Heflin, M. T., McDonald, S. R., Swaminathan, M., & Sendak, M.
Affiliations
Duke University (US)
Corresponding author
Corey, Kristin M.
ORCID
Venue
PLOS Medicine
Publisher
Public Library of Science (PLoS)
Published
2018-11-27
Volume
15
Issue
11
Pages
e1002701
ISSN
  • 1549-1277 (ISSN-L)
  • 1549-1676
DOI
10.1371/journal.pmed.1002701
PMC
PMC6258507
OpenAlex
W2901997892
Cited by
204
References
39
Citations by year
2026: 14 · 2025: 17 · 2024: 25 · 2023: 34 · 2022: 30 · 2021: 40 · 2020: 28 · 2019: 14 · 2018: 2
FWCI
11.82
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS Medicine (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Funders
  • Duke Institue for Health Innovation (Clinical Research and Innovation Scholarship)
  • Duke Forge (Graduate Fellowship Award)
  • Health Resources and Services Administration (U1QHP28708)
  • Duke Institute for Health Innovation (Internal Pilot Grant)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Cardiac, Anesthesia and Surgical Outcomes
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
postoperative complication prediction, automated data processing pipeline, random forest, extreme gradient boosting, 30-day postoperative mortality, postoperative shock, genitourinary complications, surgical risk calculator, clinical decision support, Pythia, model validation, surgical complexity
MeSH terms
Machine Learning, Adolescent, Adult, Aged, Automation, Female, Health Status, Humans, Male, Middle Aged, Postoperative Complications, Retrospective Studies, Risk Factors, Surgical Procedures, Operative, Reproducibility of Results, Comorbidity, Risk Assessment, Young Adult, Data Mining, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review