CoChat
Sign Up
Cited by 228Gastrointestinal Bleeding Diagnosis and TreatmentHelicobacter pylori-related gastroenterology studiesArtificial Intelligence in Healthcare

Validation of a Machine Learning Model That Outperforms Clinical Risk Scoring Systems for Upper Gastrointestinal Bleeding

Open access
Shung, Dennis L., Au, Benjamin, Taylor, Richard Andrew, Tay, J. Kenneth, Laursen, Stig B., Stanley, Adrian J., Dalton, Harry R., Ngu, Jeffrey +2 more·2020·Gastroenterology
DOI 10.1053/j.gastro.2019.09.009PMC PMC7004228
Open this paper in CoChat — free accountOpen PDF

Abstract

Background & Aims: Scoring systems are suboptimal for determining risk in patients with gastrointestinal bleeding (UGIB); these might be improved by a machine learning model. We used machine learning to develop a model to calculate risk of hospital-based intervention or death in patients with UGIB and compared its performance with other scoring systems. Methods: We analyzed data collected from consecutive unselected patients with UGIB from medical centers in 4 countries (United States, Scotland, England, Denmark; n=1958) from March 2014 through March 2015. We used the data to derive and internally validate a gradient-boosting machine learning model to identify patients who met a composite endpoint of hospital-based intervention (transfusion or hemostatic intervention) or death within 30 days. We compared the performance of the machine learning prediction model with validated pre-endoscopic clinical risk scoring systems (the Glasgow-Blatchford score [GBS], admission-Rockall score, and AIMS65). We externally validated the machine learning model using data from 2 Asia-Pacific sites (Singapore and New Zealand; n=399). Performance was measured by area under receiver operating characteristic curve (AUC) analysis. Results: The machine learning model identified patients who met the composite endpoint with an AUC of 0.91 in the in the internal validation set; the clinical scoring systems identified patients who met the composite endpoint with AUC values of 0.88 for GBS (P=.001), 0.73 for Rockall score (P<.001), and 0.78 for AIMS65 score (P<.001). In the external validation cohort, the machine learning model identified patients who met the composite endpoint with an AUC of 0.90, the GBS with an AUC of 0.87 (P=.004), the Rockall score with an AUC of 0.66 (P<.001), and the AIMS65 with an AUC of 0.64 (P<.001). At cutoff scores at which the machine learning model and GBS identified patients who met the composite endpoint with 100% sensitivity, the specificity values were 26% with the machine learning model vs 12% with GBS (P<.001). Conclusions: We developed a machine learning model that identifies patients with UGIB who met a composite endpoint of hospital-based intervention or death within 30 days with a greater AUC and higher levels of specificity, at 100% sensitivity, than validated clinical risk scoring systems. This model could increase identification of low-risk patients who can be safely discharged from the emergency department for outpatient management.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1053/j.gastro.2019.09.009

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI-assisted tools have been successfully applied in various clinical settings to assist diagnosis, improve therapy, and predict risk of mortality.

    “AI-assisted tools have been successfully applied in various clinical settings to assist diagnosis [5], improve therapy [6], and predict risk of mortality [7].”

    From Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review (2022) · cites this paper as [7]

    The Methods (page unavailable) say the model predicts a composite outcome that includes 30-day mortality, and the Results report successful validation of that composite prediction. The paper does not validate prediction of mortality alone; its Future Directions section describes clinical implementation as a next step rather than an established application. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • There have been well-designed studies where more complex ML approaches have outperformed regression-derived clinical risk scores on external validation.

    “there have been well-designed studies where more “complex” ML approaches have outperformed regression-derived clinical risk scores on external validation [7,59,60].”

    From Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review (2022) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

4 of the 54 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. [25]Chen T, Guestrin C. Xgboost: A scalable tree boosting system, In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, ACM, 2016.
  2. [42]Rotondano G, Cipolletta L, Grossi E, Artificial neural networks accurately predict mortality in patients with nonvariceal upper GI bleeding. Gastrointestinal Endoscopy 2011; 73: 218– 26, 226.e1. 21295635
  3. [46]Shung D, Simonov M, Au B, Machine Learning to Predict Outcomes in Patients with Acute Gastrointestinal Bleeding: Systematic Review and Meta-Analysis. Gastroenterology 2018; 154. 31055722

Cited by

Sorted by citations

Related

Authors
Shung, D. L., Au, B., Taylor, R. A., Tay, J. K., Laursen, S. B., Stanley, A. J., Dalton, H. R., Ngu, J., Schultz, M., & Laine, L.
Affiliations
  • Yale University (US)
  • Stanford University (US)
  • Odense University Hospital (DK)
  • Glasgow Royal Infirmary (GB)
  • Royal Cornwall Hospital (GB)
  • Christchurch Hospital (NZ)
  • Dunedin Public Hospital (NZ)
Corresponding author
  • Shung, Dennis L.
  • Laine, Loren
ORCID
Venue
Gastroenterology
Publisher
Elsevier BV
Published
2020-01
Volume
158
Issue
1
Pages
160-167
ISSN
0016-5085 (ISSN-L)
DOI
10.1053/j.gastro.2019.09.009
PMC
PMC7004228
OpenAlex
W2976369463
Semantic Scholar
49e263c97948d8177c777a1106e0ab0795effa58
Cited by
227
References
54
Citations by year
2026: 22 · 2025: 30 · 2024: 49 · 2023: 29 · 2022: 31 · 2021: 37 · 2020: 25 · 2019: 3
FWCI
21.87
Type
journal article
Language
en
Open access
Licence
Locations
  • Gastroenterology (publishedVersion)
  • PubMed (publishedVersion)
  • University of Southern Denmark Research Portal (University of Southern Denmark) (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
Funders
National Institutes of Health (T32 DK007017)
Topics
  • Health Sciences › Medicine › Gastroenterology › Gastrointestinal Bleeding Diagnosis and Treatment
  • Health Sciences › Medicine › Surgery › Helicobacter pylori-related gastroenterology studies
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
upper gastrointestinal bleeding, machine learning prediction models, Glasgow-Blatchford score, Rockall score, AIMS65, gradient boosting, hospital-based intervention, 30-day mortality, external validation, risk stratification, endoscopic hemostasis, blood transfusion, internal validation, sensitivity and specificity
MeSH terms
Machine Learning, Adult, Aged, Aged, 80 and over, Blood Transfusion, Emergency Service, Hospital, Female, Gastrointestinal Hemorrhage, Hemostatic Techniques, Humans, Male, Middle Aged, Models, Biological, Prognosis, ROC Curve, Risk Assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review