CoChat
Sign Up
Cited by 19Hospital Admissions and OutcomesEmergency and Acute Care StudiesMachine Learning in Healthcare

External validation of EPIC’s Risk of Unplanned Readmission model, the LACE+ index and SQLape as predictors of unplanned hospital readmissions: A monocentric, retrospective, diagnostic cohort study in Switzerland

Open access
Hwang, Aljoscha Benjamin, Schuepfer, Guido, Pietrini, Mario, Boes, Stefan·2021·PLOS ONE
DOI 10.1371/journal.pone.0258338PMC PMC8589185
Open this paper in CoChat — free account

Abstract

Introduction: Readmissions after an acute care hospitalization are relatively common, costly to the health care system, and are associated with significant burden for patients. As one way to reduce costs and simultaneously improve quality of care, hospital readmissions receive increasing interest from policy makers. It is only relatively recently that strategies were developed with the specific aim of reducing unplanned readmissions using prediction models to identify patients at risk. EPIC’s Risk of Unplanned Readmission model promises superior performance. However, it has only been validated for the US setting. Therefore, the main objective of this study is to externally validate the EPIC’s Risk of Unplanned Readmission model and to compare it to the internationally, widely used LACE+ index, and the SQLAPE® tool, a Swiss national quality of care indicator. Methods: A monocentric, retrospective, diagnostic cohort study was conducted. The study included inpatients, who were discharged between the 1 st of January 2018 and the 31 st of December 2019 from the Lucerne Cantonal Hospital, a tertiary-care provider in Central Switzerland. The study endpoint was an unplanned 30-day readmission. Models were replicated using the original intercept and beta coefficients as reported. Otherwise, score generator provided by the developers were used. For external validation, discrimination of the scores under investigation were assessed by calculating the area under the receiver operating characteristics curves (AUC). Calibration was assessed with the Hosmer-Lemeshow X2 goodness-of-fit test This report adheres to the TRIPOD statement for reporting of prediction models. Results: At least 23,116 records were included. For discrimination, the EPIC´s prediction model, the LACE+ index and the SQLape® had AUCs of 0.692 (95% CI 0.676–0.708), 0.703 (95% CI 0.687–0.719) and 0.705 (95% CI 0.690–0.720). The Hosmer-Lemeshow X2 tests had values of p<0.001. Conclusion: In summary, the EPIC´s model showed less favorable performance than its comparators. It may be assumed with caution that the EPIC´s model complexity has hampered its wide generalizability—model updating is warranted.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0258338

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Hwang 2021, conducted at Swiss Hospitals with differences in demographic populations and disease prevalence, may not reflect performance in the United States.

    “Hwang 2021 27 is a validation study conducted at Swiss Hospitals with innate differences amongst demographic populations and disease prevalence, and as such, results of this validation study may not reflect performance in the United States.”

    From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as [27]

    The Discussion (Limitations; page not available) describes a single-center Swiss validation study and warns that its findings may not generalize to organizations outside Switzerland, which supports the caution about U.S. performance. It does not establish differences in demographics or disease prevalence between its patients and U.S. patients; the claim also calls the setting 'Swiss Hospitals' although the validation was conducted at one hospital. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 51

  1. [1]Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. New England Journal of Medicine. 2009; 360(14): 1418– 28. doi: 10.1056/NEJMsa0803563 19339721 · DOI · Paper page
  2. [2]Donzé J, Lipsitz S, Bates DW, Schnipper JL. Causes and patterns of readmissions in patients with common comorbidities: retrospective cohort study. Bmj. 2013; 347: f7171. doi: 10.1136/bmj.f7171 24342737 PMC3898702 · DOI
  3. [3]Bosco JA, Karkenny AJ, Hutzler LH, Slover JD, Iorio R. Cost Burden of 30-Day Readmissions Following Medicare Total Hip and Knee Arthroplasty. The Journal of Arthroplasty. 2014; 29(5): 903– 5. doi: 10.1016/j.arth.2013.11.006 24332969 · DOI
  4. [4]Smedira NG, Hoercher KJ, Lima B, Mountis MM, Starling RC, Thuita L, et al. Unplanned hospital readmissions after HeartMate II implantation: frequency, risk factors, and impact on resource use and survival. JACC: Heart Failure. 2013; 1(1): 31– 9. doi: 10.1016/j.jchf.2012.11.001 24621797 · DOI
  5. [5]Wong EL, Cheung AW, Leung MC, Yam CH, Chan FW, Wong FY, et al. Unplanned readmission rates, length of hospital stay, mortality, and medical costs of ten common medical conditions: a retrospective analysis of Hong Kong hospital data. BMC Health Services Research. 2011; 11(1): 149. doi: 10.1186/1472-6963-11-149 21679471 PMC3146405 · DOI
  6. [6]Halfon P, Eggli Y, Pêtre-Rohrbach I, Meylan D, Marazzi A, Burnand B. Validation of the Potentially Avoidable Hospital Readmission Rate as a Routine Indicator of the Quality of Hospital Care. Medical Care. 2006; 44(11): 972– 81. doi: 10.1097/01.mlr.0000228002.43688.c2 17063128 · DOI
  7. [7]Wish JB. The role of 30-day readmission as a measure of quality. Am Soc Nephrol; 2014. doi: 10.2215/CJN.00240114 24509293 PMC3944749 · DOI
  8. [10]Kristensen SR, Bech M, Quentin W. A roadmap for comparing readmission policies with application to Denmark, England, Germany and the United States. Health Policy. 2015; 119(3): 264– 73. doi: 10.1016/j.healthpol.2014.12.009 25547401 · DOI
  9. [11]Halfon P, Eggli Y, van Melle G, Chevalier J, Wasserfallen J-B, Burnand B. Measuring potentially avoidable hospital readmissions. Journal of Clinical Epidemiology. 2002; 55(6): 573– 87. doi: 10.1016/s0895-4356(01)00521-2 12063099 · DOI
  10. [12]Leppin AL, Gionfriddo MR, Kessler M, Brito JP, Mair FS, Gallacher K, et al. Preventing 30-Day Hospital Readmissions: A Systematic Review and Meta-analysis of Randomized TrialsPreventing 30-Day Hospital ReadmissionsPreventing 30-Day Hospital Readmissions. JAMA Internal Medicine. 2014; 174(7): 1095– 107. doi: 10.1001/jamainternmed.2014.1608 24820131 PMC4249925 · DOI
  11. [13]Van Walraven C, Bennett C, Jennings A, Austin PC, Forster AJ. Proportion of hospital readmissions deemed avoidable: a systematic review. Canadian Medical Association Journal. 2011; 183(7): E391– E402. doi: 10.1503/cmaj.101860 21444623 PMC3080556 · DOI
  12. [14]Zhou H, Della PR, Roberts P, Goh L, Dhaliwal SS. Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review. BMJ open. 2016; 6(6): e011060. Epub 2016/06/30. doi: 10.1136/bmjopen-2016-011060; PubMed Central PMCID: PMC4932323. 27354072 PMC4932323 · DOI
  13. [15]Kansagara D, Englander H, Salanitro A, Kagen D, Theobald C, Freeman M, et al. Risk Prediction Models for Hospital Readmission: A Systematic Review. Jama. 2011; 306(15): 1688– 98. doi: 10.1001/jama.2011.1515 22009101 PMC3603349 · DOI
  14. [16]Moons KG, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012; 98(9): 691– 8. doi: 10.1136/heartjnl-2011-301247 22397946 · DOI · Paper page
  15. [18]Struja T, Baechli C, Koch D, Haubitz S, Eckart A, Kutz A, et al. What Are They Worth? Six 30-Day Readmission Risk Scores for Medical Inpatients Externally Validated in a Swiss Cohort. Journal of General Internal Medicine. 2020. doi: 10.1007/s11606-020-05638-z 31965531 PMC7351934 · DOI
  16. [19]Donzé JD, Williams MV, Robinson EJ, Zimlichman E, Aujesky D, Vasilevskis EE, et al. International Validity of the HOSPITAL Score to Predict 30-Day Potentially Avoidable Hospital ReadmissionsThe HOSPITAL Score and Hospital ReadmissionsThe HOSPITAL Score and Hospital Readmissions. JAMA Internal Medicine. 2016; 176(4): 496– 502. doi: 10.1001/jamainternmed.2015.8462 26954698 PMC5070968 · DOI
  17. [20]Blanc A-L, Fumeaux T, Stirnemann J, Dupuis Lozeron E, Ourhamoune A, Desmeules J, et al. Development of a predictive score for potentially avoidable hospital readmissions for general internal medicine patients. PLOS ONE. 2019; 14(7): e0219348. doi: 10.1371/journal.pone.0219348 31306461 PMC6629067 · DOI
  18. [21]Aubert CE, Folly A, Mancinetti M, Hayoz D, Donzé J. Prospective validation and adaptation of the HOSPITAL score to predict high risk of unplanned readmission of medical patients. Swiss medical weekly. 2016; 146: w14335. doi: 10.4414/smw.2016.14335 27497141 · DOI
  19. [22]Uhlmann M, Lécureux E, Griesser A-C, Duong HD, Lamy O. Prediction of potentially avoidable readmission risk in a division of general internal medicine. Swiss medical weekly. 2017; 147. 28750417
  20. [25]Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological). 1996; 58(1): 267– 88.
  21. [27]van Walraven C, Wong J, Forster AJ. LACE+ index: extension of a validated index to predict early death or urgent readmission after hospital discharge using administrative data. Open medicine: a peer-reviewed, independent, open-access journal. 2012; 6(3): e80– 90. Epub 2012/01/01.; PubMed Central PMCID: PMC3659212. 23696773 PMC3659212
  22. [28]van Walraven C, Dhalla IA, Bell C, Etchells E, Stiell IG, Zarnke K, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ: Canadian Medical Association journal = journal de l’Association medicale canadienne. 2010; 182(6): 551– 7. Epub 2010/03/03. doi: 10.1503/cmaj.091117; PubMed Central PMCID: PMC2845681. 20194559 PMC2845681 · DOI
  23. [29]Hosmer DW Jr, Lemeshow S, Sturdivant RX. Applied logistic regression: John Wiley & Sons; 2013.
  24. [30]Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi J-C, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Medical care. 2005: 1130– 9. doi: 10.1097/01.mlr.0000182534.19832.83 16224307 · DOI · Paper page
  25. [34]Winter A, Takabayashi K, Jahn F, Kimura E, Engelbrecht R, Haux R, et al. Quality Requirements for Electronic Health Record Systems*. A Japanese-German Information Management Perspective. Methods Inf Med. 2017; 56(7): e92– e104. doi: 10.3414/ME17-05-0002. 28925415 PMC6291988 · DOI
  26. [35]Little RJ, D’Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, et al. The Prevention and Treatment of Missing Data in Clinical Trials. New England Journal of Medicine. 2012; 367(14): 1355– 60. doi: 10.1056/NEJMsr1203730. 23034025 PMC3771340 · DOI · Paper page
  27. [36]Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of the Odds Ratio in Gauging the Performance of a Diagnostic, Prognostic, or Screening Marker. American Journal of Epidemiology. 2004; 159(9): 882– 90. doi: 10.1093/aje/kwh101 15105181 · DOI · Paper page
  28. [37]Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010; 21(1): 128– 38. doi: 10.1097/EDE.0b013e3181c30fb2. 20010215 PMC3575184 · DOI · Paper page
  29. [39]van Walraven C, Jennings A, Forster AJ. A meta-analysis of hospital 30-day avoidable readmission rates. Journal of Evaluation in Clinical Practice. 2012; 18(6): 1211– 8. doi: 10.1111/j.1365-2753.2011.01773.x 22070191 · DOI
  30. [40]Vasilevskis EE, Ouslander JG, Mixon AS, Bell SP, Jacobsen JML, Saraf AA, et al. Potentially Avoidable Readmissions of Patients Discharged to Post-Acute Care: Perspectives of Hospital and Skilled Nursing Facility Staff. Journal of the American Geriatrics Society. 2017; 65(2): 269– 76. doi: 10.1111/jgs.14557 27981557 PMC5311021 · DOI
  31. [41]Donzé J, Lipsitz S, Bates DW, Schnipper JL. Causes and patterns of readmissions in patients with common comorbidities: retrospective cohort study. BMJ: British Medical Journal. 2013; 347: f7171. doi: 10.1136/bmj.f7171 24342737 PMC3898702 · DOI
  32. [43]Gilliard N, Eggli Y, Halfon P. A methodology to estimate the potential to move inpatient to one day surgery. BMC health services research. 2006; 6(1): 78. doi: 10.1186/1472-6963-6-78 16784523 PMC1552063 · DOI
  33. [45]Vilpert S. Konsultationen in Schweizer Notfallstationen. Obsan Bulletin [Internet]. 2013 [cited 2020 May 23]; 3. Available from: https://www.obsan.admin.ch/sites/default/files/publications/2015/obsan_bulletin_2013-03_d.pdf.
  34. [46]Hospital discharge rates [Internet]. Organisation for Economic Cooperation and Development (OECD); 2018 [cited 2020 May 24]. Available from: https://www.oecd-ilibrary.org/social-issues-migration-health/hospital-discharge-rates/indicator/english_5880c955-en.
  35. [48]Glauser G, Winter E, Caplan IF, Goodrich S, McClintock SD, Guzzo TJ, et al. The LACE+ Index as a Predictor of 30-Day Patient Outcomes in a Urologic Surgery Population: A Coarsened Exact Match Study. Urology. 2019; 134: 109– 15. doi: 10.1016/j.urology.2019.08.030 31487509 · DOI
  36. [49]Garrison GM, Robelia PM, Pecina JL, Dawson NL. Comparing performance of 30-day readmission risk classifiers among hospitalized primary care patients. Journal of Evaluation in Clinical Practice. 2017; 23(3): 524– 9. doi: 10.1111/jep.12656 27696638 · DOI
  37. [50]Ibrahim AM, Koester C, Al-Akchar M, Tandan N, Regmi M, Bhattarai M, et al. HOSPITAL Score, LACE Index and LACE+ Index as predictors of 30-day readmission in patients with heart failure. BMJ Evidence-Based Medicine. 2019: bmjebm-2019-111271. doi: 10.1136/bmjebm-2019-111271 31771947 · DOI
  38. [51]Caplan IF, Sullivan PZ, Kung D, O’Rourke DM, Choudhri O, Glauser G, et al. LACE+ Index as Predictor of 30-Day Readmission in Brain Tumor Population. World Neurosurgery. 2019; 127: e443– e8. doi: 10.1016/j.wneu.2019.03.169 30926557 · DOI
  39. [52]Caplan IF, Zadnik Sullivan P, Glauser G, Choudhri O, Kung D, O’Rourke DM, et al. The LACE+ index fails to predict 30–90 day readmission for supratentorial craniotomy patients: A retrospective series of 238 surgical procedures. Clinical Neurology and Neurosurgery. 2019; 182: 79– 83. doi: 10.1016/j.clineuro.2019.04.026 31102908 · DOI
  40. [53]Kerr KF, Wang Z, Janes H, McClelland RL, Psaty BM, Pepe MS. Net reclassification indices for evaluating risk-prediction instruments: a critical review. Epidemiology. 2014; 25(1): 114. doi: 10.1097/EDE.0000000000000018 24240655 PMC3918180 · DOI
  41. [55]Cook NR, Paynter NP. Performance of reclassification statistics in comparing risk prediction models. Biometrical Journal. 2011; 53(2): 237– 58. doi: 10.1002/bimj.201000078 21294152 PMC3395053 · DOI
  42. [56]Pavon JM, Zhao Y, McConnell E, Hastings SN. Identifying risk of readmission in hospitalized elderly adults through inpatient medication exposure. Journal of the American Geriatrics Society. 2014; 62(6): 1116– 21. doi: 10.1111/jgs.12829 24802165 PMC4157051 · DOI
  43. [57]Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW, Bossuyt P, et al. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019; 17(1): 230. doi: 10.1186/s12916-019-1466-7 31842878 PMC6912996 · DOI · Paper page
  44. [58]Tao H, Ellenbecker CH, Chen J, Zhan L, Dalton J. The influence of social environmental factors on rehospitalization among patients receiving home health care services. Advances in nursing science. 2012; 35(4): 346– 58. doi: 10.1097/ANS.0b013e318271d2ad 23107991 · DOI
  45. [59]Shih SL, Zafonte R, Bates DW, Gerrard P, Goldstein R, Mix J, et al. Functional Status Outperforms Comorbidities as a Predictor of 30-Day Acute Care Readmissions in the Inpatient Rehabilitation Population. Journal of the American Medical Directors Association. 2016; 17(10): 921– 6. doi: 10.1016/j.jamda.2016.06.003 27424092 · DOI
  46. [60]Tonkikh O, Shadmi E, Flaks-Manov N, Hoshen M, Balicer RD, Zisberg A. Functional status before and during acute hospitalization and readmission risk identification. Journal of Hospital Medicine. 2016; 11(9): 636– 41. doi: 10.1002/jhm.2595 27130176 · DOI
  47. [61]Arbaje AI, Wolff JL, Yu Q, Powe NR, Anderson GF, Boult C. Postdischarge Environmental and Socioeconomic Factors and the Likelihood of Early Hospital Readmission Among Community-Dwelling Medicare Beneficiaries. The Gerontologist. 2008; 48(4): 495– 504. doi: 10.1093/geront/48.4.495 18728299 · DOI
  48. [62]Morgan DJ, Bame B, Zimand P, Dooley P, Thom KA, Harris AD, et al. Assessment of Machine Learning vs Standard Prediction Rules for Predicting Hospital Readmissions. JAMA Network Open. 2019; 2(3): e190348– e. doi: 10.1001/jamanetworkopen.2019.0348 30848808 PMC6484642 · DOI
  49. [63]Mišić VV, Gabel E, Hofer I, Rajaram K, Mahajan A. Machine Learning Prediction of Postoperative Emergency Department Hospital Readmission. Anesthesiology: The Journal of the American Society of Anesthesiologists. 2020; 132(5): 968– 80. doi: 10.1097/ALN.0000000000003140 32011336 · DOI
  50. [64]Jamei M, Nisnevich A, Wetchler E, Sudat S, Liu E. Predicting all-cause risk of 30-day hospital readmission using artificial neural networks. PLOS ONE. 2017; 12(7): e0181173. doi: 10.1371/journal.pone.0181173 28708848 PMC5510858 · DOI
  51. [65]Hao S, Wang Y, Jin B, Shin AY, Zhu C, Huang M, et al. Development, Validation and Deployment of a Real Time 30 Day Hospital Readmission Risk Assessment Tool in the Maine Healthcare Information Exchange. PLOS ONE. 2015; 10(10): e0140271. doi: 10.1371/journal.pone.0140271 26448562 PMC4598005 · DOI

Cited by

Sorted by citations

Authors
Hwang, A. B., Schuepfer, G., Pietrini, M., & Boes, S.
Affiliations
  • University of Lucerne (CH)
  • Luzerner Kantonsspital (CH)
Corresponding author
Hwang, Aljoscha Benjamin
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2021-11-12
Volume
16
Issue
11
Pages
e0258338
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0258338
PMC
PMC8589185
OpenAlex
W3214032254
Cited by
19
References
2
Citations by year
2026: 5 · 2025: 3 · 2024: 3 · 2023: 6 · 2022: 2
FWCI
2.88
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Emergency Medicine › Hospital Admissions and Outcomes
  • Health Sciences › Medicine › Emergency Medicine › Emergency and Acute Care Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
LACE index, risk prediction model, external validation, 30-day readmission, model discrimination, model calibration, quality of care indicators, Swiss hospitals
MeSH terms
Adolescent, Adult, Aged, Aged, 80 and over, Child, Child, Preschool, Female, Humans, Infant, Male, Middle Aged, Patient Discharge, Patient Readmission, Prognosis, Retrospective Studies, Risk Factors, ROC Curve, Switzerland, Models, Statistical, Risk Assessment, Area Under Curve, Young Adult, Tertiary Care Centers

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review