CoChat
Sign Up
Cited by 75Machine Learning in HealthcareArtificial Intelligence in HealthcareHospital Admissions and Outcomes

The effects of data sources, cohort selection, and outcome definition on a predictive model of risk of thirty-day hospital readmissions

Open access
Walsh, Colin, Hripcsak, George·2014·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2014.08.006PMC PMC4261028
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Hospital readmission risk prediction remains a motivated area of investigation and operations in light of the hospital readmissions reduction program through CMS. Multiple models of risk have been reported with variable discriminatory performances, and it remains unclear how design factors affect performance. OBJECTIVES: To study the effects of varying three factors of model development in the prediction of risk based on health record data: (1) reason for readmission (primary readmission diagnosis); (2) available data and data types (e.g. visit history, laboratory results, etc); (3) cohort selection. METHODS: Regularized regression (LASSO) to generate predictions of readmissions risk using prevalence sampling. Support Vector Machine (SVM) used for comparison in cohort selection testing. Calibration by model refitting to outcome prevalence. RESULTS: Predicting readmission risk across multiple reasons for readmission resulted in ROC areas ranging from 0.92 for readmission for congestive heart failure to 0.71 for syncope and 0.68 for all-cause readmission. Visit history and laboratory tests contributed the most predictive value; contributions varied by readmission diagnosis. Cohort definition affected performance for both parametric and nonparametric algorithms. Compared to all patients, limiting the cohort to patients whose index admission and readmission diagnoses matched resulted in a decrease in average ROC from 0.78 to 0.55 (difference in ROC 0.23, p value 0.01). Calibration plots demonstrate good calibration with low mean squared error. CONCLUSION: Targeting reason for readmission in risk prediction impacted discriminatory performance. In general, laboratory data and visit history data contributed the most to prediction; data source contributions varied by reason for readmission. Cohort selection had a large impact on model performance, and these results demonstrate the difficulty of comparing results across different studies of predictive risk modeling.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2014.08.006

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Differences in AUROC across the two hospitals cannot be ascribed to the presence or absence of notes given the difference in cohorts.

    “In the current study, we caution that the differences in AUROC across the two hospitals (one with and one without notes) cannot be ascribed to the presence or absence of notes given the difference in cohorts.45”

    From Scalable and accurate deep learning with electronic health records (2018) · cites this paper as [45]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. [2]McAlister Fa Decreasing readmissions: it can be done but one size does not fit all BMJ quality & safety 2013 1 2 24006457
  2. [3]Amarasingham R Patel PC Toto K Nelson LL Swanson TS Moore BJ Allocating scarce resources in real-time to reduce heart failure readmissions: a prospective, controlled study BMJ quality & safety 2013 1 8 PMC3888600 23904506
  3. [4]Kansagara D Englander H Salanitro A Kagen D Theobald C Freeman M Risk prediction models for hospital readmission: a systematic review JAMA: the journal of the American Medical Association 2011 306 1688 98 PMC3603349 22009101
  4. [5]Almagro P Barreiro B Ochoa de Echaguen A Quintana S Rodriguez Carballeira M Heredia JL Risk factors for hospital readmission in patients with chronic obstructive pulmonary disease Respiration; international review of thoracic diseases 2006 73 311 7 16155352
  5. [6]Rasmussen JN Chong A Alter DA Relationship between adherence to evidence-based pharmacotherapy and long-term mortality after acute myocardial infarction JAMA: the journal of the American Medical Association 2007 297 177 86 17213401
  6. [7]Lee DS Austin PC Rouleau JL Liu PP Naimark D Tu JV Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model JAMA: the journal of the American Medical Association 2003 290 2581 7 14625335
  7. [8]Hannan EL Racz MJ Walford G Ryan TJ Isom OW Bennett E Predictors of readmission for complications of coronary artery bypass graft surgery JAMA: the journal of the American Medical Association 2003 290 773 80 12915430
  8. [9]Rich MW Beckham V Wittenberg C Leven CL Freedland KE Carney RM A multidisciplinary intervention to prevent the readmission of elderly patients with congestive heart failure The New England journal of medicine 1995 333 1190 5 7565975
  9. [10]Burns R Nichols LO Factors predicting readmission of older general medicine patients Journal of general internal medicine 1991 6 389 93 1744751
  10. [11]Graham H Livesley B Can readmissions to a geriatric medical unit be prevented? Lancet 1983 1 404 6 6130389
  11. [12]Singh M Guth JC Liotta E Kosteva AR Bauer RM Prabhakaran S Predictors of 30-Day Readmission After Subarachnoid Hemorrhage Neurocritical care 2013 24037248
  12. [13]Rambachan A Smith TR Saha S Eskandari MK Bendok B Kim JY Reasons for readmission after carotid endarterectomy World neurosurgery 2013 23973514
  13. [14]Lissauer ME Diaz JJ Narayan M Shah PK Hanna NN Surgical intensive care unit admission variables predict subsequent readmission The American surgeon 2013 79 583 8 23711267
  14. [15]Hazratjee N Agito M Lopez R Lashner B Rizk MK Hospital readmissions in patients with inflammatory bowel disease The American journal of gastroenterology 2013 108 1024 32 23820989
  15. [16]Coller RJ Klitzner TS Lerner CF Chung PJ Predictors of 30-Day Readmission and Association with Primary Care Follow-Up Plans The Journal of pediatrics 2013 1 7 23706518
  16. [17]Hannan EL Samadashvili Z Walford G Holmes DR Jacobs A Sharma S Predictors and outcomes of ad hoc versus non-ad hoc percutaneous coronary interventions JACC Cardiovascular interventions 2009 2 350 6 19463449
  17. [18]He D Mathews SC Kalloo aN Hutfless S Mining high-dimensional administrative claims data to predict early hospital readmissions Journal of the American Medical Informatics Association 2013 PMC3932470 24076748
  18. [19]Hechenbleikner EM Makary Ma Samarov DV Bennett JL Gearhart SL Efron JE Hospital readmission by method of data collection Journal of the American College of Surgeons 2013 216 1150 8 23583617
  19. [20]Liao Y McGee DL Cooper RS Sutkowski MB How generalizable are coronary risk prediction models? Comparison of Framingham and two national cohorts American heart journal 1999 137 837 45 10220632
  20. [21]Steyerberg EW Clinical prediction models: a practical approach to development, validation, and updating New York, NY Springer 2009
  21. [22]Hansen CDG Fox CJP Gross CP Bruun LCJS Hospital readmissions and emergency department visits following laparoscopic and open colon resection for cancer Diseases of the colon and rectum 2013 56 1053 61 23929014
  22. [23]Berkowitz SA Anderson GF Medicare beneficiaries most likely to be readmitted Journal of hospital medicine: an official publication of the Society of Hospital Medicine 2013 24038901
  23. [24]Bradley EH Curry L Horwitz LI Sipsma H Wang Y Walsh MN Hospital strategies associated with 30-day readmission rates for patients with heart failure Circulation Cardiovascular quality and outcomes 2013 6 444 50 23861483 PMC3802532
  24. [25]Bradley EH Yakusheva O Horwitz LI Sipsma H Fletcher J Identifying patients at increased risk for unplanned readmission Medical care 2013 51 761 6 23942218 PMC3771868
  25. [26]Donzé J Aujesky D Williams D Schnipper JL Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model JAMA internal medicine 2013 173 632 8 23529115
  26. [28]Hastie T Tibshirani R Friedman JH The elements of statistical learning: data mining, inference, and prediction: with 200 full-color illustrations New York Springer 2001
  27. [29]Tibshirani R Regression Shrinkage and Selection via the Lasso Journal of the Royal Statistical Society (Series B) 1996 58 267 88
  28. [30]Staples JA Thiruchelvam D Redelmeier DA Site of hospital readmission and mortality: a population-based retrospective cohort study Canadian Medical Association Open Access Journal 2014 2 E77 E85 PMC4084742 25077133
  29. [32]Meyer D Hornik K Feinerer I Text mining infrastructure in R Journal of Statistical Software 2008 25 1 54
  30. [33]Moons KG Kengne AP Woodward M Royston P Vergouwe Y Altman DG Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker Heart 2012 98 683 90 22397945
  31. [34]Altman DG Vergouwe Y Royston P Moons KG Prognosis and prognostic research: validating a prognostic model Bmj 2009 338 b605 19477892
  32. [36]Weiss GM Mining with rarity: a unifying framework ACM SIGKDD Explorations Newsletter 2004 6 7 19
  33. [37]Longadge R Dongre S Class Imbalance Problem in Data Mining Review arXiv preprint arXiv:13051707 2013
  34. [38]Guo X Yin Y Dong C Yang G Zhou G On the class imbalance problem Natural Computation, 2008 ICNC’08 Fourth International Conference on: IEEE 2008 192 201
  35. [39]He H Garcia EA Learning from Imbalanced Data IEEE Trans on Knowl and Data Eng 2009 21 1263 84
  36. [40]Cohen G Hilario M Sax H Hugonnet S Geissbuhler A Learning from imbalanced data in surveillance of nosocomial infection Artificial intelligence in medicine 2006 37 7 18 16233974
  37. [42]Zou H Hastie T Regularization and variable selection via the elastic net Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2005 67 301 20
  38. [44]R. FJHTT Regularization Paths for Generalized Linear Models via Coordinate Descent Journal of statistical software 2010 1 22 PMC2929880 20808728
  39. [47]Ip E Visualizing Multiple Regression Journal of Statistics Education 2001 9
  40. [48]Kennedy P More on Venn Diagrams for Regression Journal of Statistics Education 2002 10
  41. [49]Harrell FE Jr Lee KL Mark DB Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors Statistics in medicine 1996 15 361 87 8668867
  42. [50]Steyerberg EW Borsboom GJ van Houwelingen HC Eijkemans MJ Habbema JD Validation and updating of predictive logistic regression models: a study on sample size and shrinkage Statistics in medicine 2004 23 2567 86 15287085
  43. [51]Meier L Van De Geer S Bühlmann P The group lasso for logistic regression Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2008 70 53 71

Cited by

Sorted by citations

Authors
Walsh, C., & Hripcsak, G.
Affiliations
Columbia University (US)
Corresponding author
Walsh, Colin
ORCID
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2014-12
Volume
52
Pages
418-426
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2014.08.006
PMC
PMC4261028
OpenAlex
W2072647517
Cited by
75
References
1
Citations by year
2026: 1 · 2025: 1 · 2024: 2 · 2022: 8 · 2021: 6 · 2020: 13 · 2019: 11 · 2018: 13 · 2017: 8 · 2016: 7 · 2015: 4 · 2014: 1
FWCI
6.65
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion, other-oa)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
U.S. National Library of Medicine (R01LM006910, T15LM007079)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Emergency Medicine › Hospital Admissions and Outcomes
Keywords
cohort selection, data sources, outcome definitions, all-cause readmission, laboratory test data, model calibration, LASSO, support vector machine
MeSH terms
Adolescent, Adult, Aged, Female, Humans, Male, Middle Aged, Patient Readmission, Research Design, Retrospective Studies, Reproducibility of Results, Models, Statistical, Risk Assessment, Young Adult, Data Mining, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review