Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review
Open accessThe DOI matches its Crossref record; 141 of the 144 references it cites were found in a registry or an index.
Abstract
Objective: Electronic health records (EHRs) are an increasingly common data source for clinical risk prediction, presenting both unique analytic opportunities and challenges. We sought to evaluate the current state of EHR based risk prediction modeling through a systematic review of clinical prediction studies using EHR data. Methods: We searched PubMed for articles that reported on the use of an EHR to develop a risk prediction model from 2009 to 2014. Articles were extracted by two reviewers, and we abstracted information on study design, use of EHR data, model building, and performance from each publication and supplementary documentation. Results: We identified 107 articles from 15 different countries. Studies were generally very large (median sample size = 26 100) and utilized a diverse array of predictors. Most used validation techniques (n = 94 of 107) and reported model coefficients for reproducibility (n = 83). However, studies did not fully leverage the breadth of EHR data, as they uncommonly used longitudinal information (n = 37) and employed relatively few predictor variables (median = 27 variables). Less than half of the studies were multicenter (n = 50) and only 26 performed validation across sites. Many studies did not fully address biases of EHR data such as missing data or loss to follow-up. Average c-statistics for different outcomes were: mortality (0.84), clinical prediction (0.83), hospitalization (0.71), and service utilization (0.71). Conclusions: EHR data present both opportunities and challenges for clinical risk prediction. There is room for improvement in designing such studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In spite of the richness and potential of available data, scaling the development of predictive models is difficult because, for traditional predictive modeling techniques, each outcome to be predicted requires the creation of a custom dataset with specific variables.7”
From Scalable and accurate deep learning with electronic health records (2018) · cites this paper as [7]
In the citing paper
“Despite the rich data now digitized in EHRs,20 a recent systematic review of the medical literature7 found that predictive models built with EHR data use a median of only 27 variables, rely on traditional generalized linear models, and are built using data at a single center.”
From Scalable and accurate deep learning with electronic health records (2018) · cites this paper as [7]
In the citing paper
“Although external validation is recognized as an area of improvement for clinical risk prediction models [85]”
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Self-citationPrediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581 · cited 2×
“Traditionally, risk pre- diction algorithms have been developed from large cohort studies such as the Framingham Heart Study.3 These studies were designed to follow people for years or even decades.”
MethodMoons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Joint modelling of longitudinal measurements and event time data
Henderson, R. (2000) · Biostatistics · doi:10.1093/biostatistics/1.4.465 · Cited by 913
The Electronic Medical Records and Genomics (eMERGE) Network: past, present, and future
Gottesman, O., Kuivaniemi, H., Tromp, G., Faucett, W. A., Li, R., Manolio, T. A., Sanderson, S. C., Kannry, J., Zinberg, R., Basford, M. A., et al. (2013) · Genetics in Medicine · doi:10.1038/gim.2013.72 · Cited by 698
Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve
Cook, N. R. (2008) · Clinical Chemistry · doi:10.1373/clinchem.2007.096529 · Cited by 691
Launching PCORnet, a national patient-centered clinical research network
Fleurence, R. L., Curtis, L. H., Califf, R. M., Platt, R., Selby, J. V., & Brown, J. S. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002747 · Cited by 637
Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., et al. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1dbd · Cited by 567 · cited 1×
“As others have noted, EHR data come with many challenges.(4) EHRs include all patients that touch a medical system, primarily capture data only when patients are ill, and collect metrics that clinicians deem to be necessary at each clinic visit.”
BackgroundSaltzman, J. R., Tabak, Y. P., Hyett, B. H., Sun, X., Travis, A. C., & Johannes, R. S. (2011) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2011.06.024 · Cited by 541
Collins, G. S., Mallett, S., Omar, O., & Yu, L.-M. (2011) · BMC Medicine · doi:10.1186/1741-7015-9-103 · Cited by 484 · cited 1×
“9 In this review, we focus specifically on the unique components of EHR data and how they have been utilized.”
BackgroundHippisley-Cox, J., & Coupland, C. (2009) · BMJ · doi:10.1136/bmj.b4229 · Cited by 479
Siontis, G. C. M., Tzoulaki, I., Castaldi, P. J., & Ioannidis, J. P. A. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.007 · Cited by 459
Self-citationAmarasingham, R., Moore, B. J., Tabak, Y. P., Drazner, M. H., Clark, C. A., Zhang, S., Reed, W. G., Swanson, T. S., Ma, Y., & Halm, E. A. (2010) · Medical Care · doi:10.1097/mlr.0b013e3181ef60d9 · Cited by 437
Estimating the Probability of Neonatal Early-Onset Infection on the Basis of Maternal Risk Factors
Puopolo, K. M., Draper, D., Wi, S., Newman, T. B., Zupancic, J., Lieberman, E., Smith, M., & Escobar, G. J. (2011) · Pediatrics · doi:10.1542/peds.2010-3464 · Cited by 393
Wu, J., Roy, J., & Stewart, W. F. (2010) · Medical Care · doi:10.1097/mlr.0b013e3181de9e17 · Cited by 373
Hippisley-Cox, J., Coupland, C., Robson, J., Sheikh, A., & Brindle, P. (2009) · BMJ · doi:10.1136/bmj.b880 · Cited by 372
Hippisley-Cox, J., Coupland, C., Robson, J., & Brindle, P. (2010) · BMJ · doi:10.1136/bmj.c6624 · Cited by 364
Hippisley-Cox, J., & Coupland, C. (2012) · BMJ · doi:10.1136/bmj.e3427 · Cited by 351
Geersing, G.-J., Bouwmeester, W., Zuithoff, P., Spijker, R., Leeflang, M., & Moons, K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0032844 · Cited by 350 · cited 2×
“suggested search terms for clinical prediction papers that were later updated by Geersing et al.(12) reaching a sensitivity of 0.”
BackgroundInfluentialStill, C. D., Wood, G. C., Benotti, P., Petrick, A. T., Gabrielsen, J., Strodel, W. E., Ibele, A., Seiler, J., Irving, B. A., Celaya, M. P., et al. (2014) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(13)70070-6 · Cited by 350
Cardiovascular Disease Risk Assessment: Insights from Framingham
D’Agostino Sr., R. B., Pencina, M. J., Massaro, J. M., & Coady, S. (2013) · Global Heart · doi:10.1016/j.gheart.2013.01.001 · Cited by 329
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.010 · Cited by 324
Wang, L., Porter, B., Maynard, C., Evans, G., Bryson, C., Sun, H., Gupta, I., Lowy, E., McDonell, M., Frisbee, K., et al. (2013) · Medical Care · doi:10.1097/mlr.0b013e31827da95a · Cited by 319
Multicenter Development and Validation of a Risk Stratification Tool for Ward Patients
Churpek, M. M., Yuen, T. C., Winslow, C., Robicsek, A. A., Meltzer, D. O., Gibbons, R. D., & Edelson, D. P. (2014) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201406-1022oc · Cited by 259
Systematic Review of Prognostic Models in Patients with Acute Stroke
Counsell, C., & Dennis, M. (2001) · Cerebrovascular Diseases · doi:10.1159/000047699 · Cited by 250
Rothman, M. J., Rothman, S. I., & Beals, J. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.06.011 · Cited by 233
Hippisley-Cox, J., & Coupland, C. (2012) · British Journal of General Practice · doi:10.3399/bjgp12x616355 · Cited by 208
Derivation of a cardiac arrest prediction model using ward vital signs*
Churpek, M. M., Yuen, T. C., Park, S. Y., Meltzer, D. O., Hall, J. B., & Edelson, D. P. (2012) · Critical Care Medicine · doi:10.1097/ccm.0b013e318250aa5a · Cited by 199
Escobar, G. J., Gardner, M. N., Greene, J. D., Draper, D., & Kipnis, P. (2013) · Medical Care · doi:10.1097/mlr.0b013e3182881c8e · Cited by 199
Medical decision support using machine learning for early detection of late-onset neonatal sepsis
Mani, S., Ozdas, A., Aliferis, C., Varol, H. A., Chen, Q., Carnevale, R., Chen, Y., Romano-Keeler, J., Nian, H., & Weitkamp, J.-H. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001854 · Cited by 196
Reporting performance of prognostic models in cancer: a review.
Mallett, S., Royston, P., Waters, R., Dutton, S., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-21 · Cited by 195
Gultepe, E., Green, J. P., Nguyen, H., Adams, J., Albertson, T., & Tagkopoulos, I. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001815 · Cited by 187
Escobar, G. J., LaGuardia, J. C., Turk, B. J., Ragins, A., Kipnis, P., & Draper, D. (2012) · Journal of Hospital Medicine · doi:10.1002/jhm.1929 · Cited by 185
Searching for Clinical Prediction Rules in MEDLINE
Ingui, B. J., & Rogers, M. A. M. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080391 · Cited by 184
Derivation and Diagnostic Accuracy of the Surgical Lung Injury Prediction Model
Kor, D. J., Warner, D. O., Alsara, A., Fernández-Pérez, E. R., Malinchoc, M., Kashyap, R., Li, G., & Gajic, O. (2011) · Anesthesiology · doi:10.1097/aln.0b013e31821b5839 · Cited by 176
Hippisley-Cox, J., & Coupland, C. (2010) · BMC Family Practice · doi:10.1186/1471-2296-11-49 · Cited by 176
Goldstein, B. A., Bhavsar, N. A., Phelan, M., & Pencina, M. J. (2016) · American Journal of Epidemiology · doi:10.1093/aje/kww112 · Cited by 176
Self-citationVan Driest, S. L., Wells, Q. S., Stallings, S., Bush, W. S., Gordon, A., Nickerson, D. A., Kim, J. H., Crosslin, D. R., Jarvik, G. P., Carrell, D. S., et al. (2016) · JAMA · doi:10.1001/jama.2015.17701 · Cited by 170
Hippisley-Cox, J., Coupland, C., & Brindle, P. (2013) · BMJ · doi:10.1136/bmj.f2573 · Cited by 153
Tran, T., Luo, W., Phung, D., Harvey, R., Berk, M., Kennedy, R. L., & Venkatesh, S. (2014) · BMC Psychiatry · doi:10.1186/1471-244x-14-76 · Cited by 150
Type 2 diabetes risk forecasting from EMR data using machine learning.
Mani, S., Chen, Y., Elasy, T. A., Clayton, W., & Denny, J. C. (2012) · PubMed · Cited by 147
Carter, E. M., & Potts, H. W. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-26 · Cited by 145
Rapsomaniki, E., Shah, A. D., Perel, P., Denaxas, S., George, J., Nicholas, O., Udumyan, R., Feder, G. S., Hingorani, A. D., Timmis, A., et al. (2013) · European Heart Journal · doi:10.1093/eurheartj/eht533 · Cited by 144
Himes, B. E., Dai, Y., Kohane, I. S., Weiss, S. T., & Ramoni, M. F. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2846 · Cited by 143
Hanauer, D. A., & Huang, Y. (2014) · Applied Clinical Informatics · doi:10.4338/aci-2014-04-ra-0026 · Cited by 142
Prediction of hospitalization due to heart diseases by supervised learning methods
Dai, W., Brisimi, T. S., Adams, W. G., Mela, T., Saligrama, V., & Paschalidis, I. C. (2015) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2014.10.002 · Cited by 142
Hippisley‐Cox, J., & Coupland, C. A. C. (2012) · British Journal of General Practice · doi:10.3399/bjgp13x660733 · Cited by 142
Crane, S. J., Tung, E. E., Hanson, G. J., Cha, S., Chaudhry, R., & Takahashi, P. Y. (2010) · BMC Health Services Research · doi:10.1186/1472-6963-10-338 · Cited by 141
Churpek, M. M., Yuen, T. C., Park, S. Y., Gibbons, R., & Edelson, D. P. (2014) · Critical Care Medicine · doi:10.1097/ccm.0000000000000038 · Cited by 138
Rusanov, A., Weiskopf, N. G., Wang, S., & Weng, C. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-51 · Cited by 137
Zhao, D., & Weng, C. (2011) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.05.004 · Cited by 135
Hippisley‐Cox, J., & Coupland, C. A. C. (2011) · BMJ · doi:10.1136/bmj.d4656 · Cited by 132
Woller, S. C., Stevens, S. M., Jones, J. P., Lloyd, J. F., Evans, R. S., Aston, V. T., & Elliott, C. G. (2011) · The American Journal of Medicine · doi:10.1016/j.amjmed.2011.06.004 · Cited by 130
A comparison of phenotype definitions for diabetes mellitus
Richesson, R., Rusincovitch, S. A., Wixted, D., Batch, B. C., Feinglos, M. N., Miranda, M. L., Hammond, W. E., Califf, R. M., & Spratt, S. E. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001952 · Cited by 130
Future of Electronic Health Records: Implications for Decision Support
Rothman, B., Leonard, J. C., & Vigoda, M. M. (2012) · Mount Sinai Journal of Medicine: A Journal of Translational and Personalized Medicine · doi:10.1002/msj.21351 · Cited by 124
Eapen, Z. J., Liang, L., Fonarow, G. C., Heidenreich, P. A., Curtis, L. H., Peterson, E. D., & Hernandez, A. F. (2013) · JACC: Heart Failure · doi:10.1016/j.jchf.2013.01.008 · Cited by 123
Wiens, J., Guttag, J., & Horvitz, E. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002162 · Cited by 123
Hippisley-Cox, J., & Coupland, C. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003482 · Cited by 122
Lipsky, B. A., Weigelt, J. A., Sun, X., Johannes, R. S., Derby, K. G., & Tabak, Y. P. (2011) · Diabetes Care · doi:10.2337/dc11-0331 · Cited by 119
Gupta, S., Tran, T., Luo, W., Phung, D., Kennedy, R. L., Broad, A., Campbell, D., Kipp, D., Singh, M., Khasraw, M., et al. (2014) · BMJ Open · doi:10.1136/bmjopen-2013-004007 · Cited by 118
Hippisley‐Cox, J., & Coupland, C. A. C. (2011) · British Journal of General Practice · doi:10.3399/bjgp12x616346 · Cited by 110
Toward personalizing treatment for depression: predicting diagnosis and severity
Huang, S. H., LePendu, P., Iyer, S. V., Tai-Seale, M., Carrell, D., & Shah, N. H. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002733 · Cited by 109
Uyar, A., Bener, A., & Ciray, H. N. (2014) · Medical Decision Making · doi:10.1177/0272989x14535984 · Cited by 109
Practical challenges in integrating genomic data into the electronic health record
Kho, A. N., Rasmussen, L. V., Connolly, J. J., Peissig, P. L., Starren, J., Hakonarson, H., & Hayes, M. G. (2013) · Genetics in Medicine · doi:10.1038/gim.2013.131 · Cited by 108
Hippisley‐Cox, J., & Coupland, C. A. C. (2012) · British Journal of General Practice · doi:10.3399/bjgp13x660724 · Cited by 108
Tabak, Y. P., Sun, X., Nunez, C. M., & Johannes, R. S. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001790 · Cited by 107
Matheny, M. E., Miller, R. A., Ikizler, T. A., Waitman, L. R., Denny, J. C., Schildcrout, J. S., Dittus, R. S., & Peterson, J. F. (2010) · Medical Decision Making · doi:10.1177/0272989x10364246 · Cited by 105
Hippisley‐Cox, J., & Coupland, C. A. C. (2014) · BMJ · doi:10.1136/bmj.g4606 · Cited by 104
Predicting clinical deterioration in the hospital: The impact of outcome selection
Churpek, M. M., Yuen, T. C., & Edelson, D. P. (2012) · Resuscitation · doi:10.1016/j.resuscitation.2012.09.024 · Cited by 99
Risk Prediction of Emergency Department Revisit 30 Days Post Discharge: A Prospective Study
Hao, S., Jin, B., Shin, A. Y., Zhao, Y., Zhu, C., Li, Z., Hu, Z., Fu, C., Ji, J., Wang, Y., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112944 · Cited by 96
Singal, A. G., Rahimi, R. S., Clark, C., Ma, Y., Cuthbert, J. A., Rockey, D. C., & Amarasingham, R. (2013) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2013.03.022 · Cited by 94
Hippisley‐Cox, J., & Coupland, C. A. C. (2011) · British Journal of General Practice · doi:10.3399/bjgp11x606627 · Cited by 92
QRESEARCH: a new general practice database for research
Hippisley‐Cox, J., Stables, D., & Pringle, M. (2004) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v12i1.108 · Cited by 90
Predicting Virologic Failure in an HIV Clinic
Robbins, G. K., Johnson, K. L., Chang, Y., Jackson, K. E., Sax, P. E., Meigs, J. B., & Freedberg, K. A. (2010) · Clinical Infectious Diseases · doi:10.1086/650537 · Cited by 89
Liu, V., Kipnis, P., Gould, M. K., & Escobar, G. J. (2010) · Medical Care · doi:10.1097/mlr.0b013e3181e359f3 · Cited by 87
Atchison, C. M., Arlikar, S., Amankwah, E. K., Ayala, I. A., Barrett, L., Branchford, B. R., Streiff, M. B., Takemoto, C. M., & Goldenberg, N. A. (2014) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2014.05.053 · Cited by 85
All At-Risk Patients Are Not Created Equal
Tescher, A. N., Branda, M. E., Byrne, T., & Naessens, J. M. (2012) · Journal of Wound Ostomy and Continence Nursing · doi:10.1097/won.0b013e3182435715 · Cited by 83
Zhai, H., Brady, P., Li, Q., Lingren, T., Ni, Y., Wheeler, D. S., & Solti, I. (2014) · Resuscitation · doi:10.1016/j.resuscitation.2014.04.009 · Cited by 82
Kennedy, E. H., Wiitala, W. L., Hayward, R. A., & Sussman, J. B. (2012) · Medical Care · doi:10.1097/mlr.0b013e31827da594 · Cited by 75
Choudhry, S. A., Li, J., Davis, D. A., Erdmann, C., Sikka, R., & Sutariya, B. B. (2013) · Online Journal of Public Health Informatics · doi:10.5210/ojphi.v5i2.4726 · Cited by 74
Hippisley‐Cox, J., & Coupland, C. A. C. (2011) · BMJ · doi:10.1136/bmj.d8009 · Cited by 73
Sun, J., McNaughton, C. D., Zhang, P., Perer, A., Gkoulalas-Divanis, A., Denny, J. C., Kirby, J., Lasko, T., Saip, A., & Malin, B. A. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002033 · Cited by 73
Mathias, J. S., Agrawal, A., Feinglass, J., Cooper, A. J., Baker, D. W., & Choudhary, A. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001360 · Cited by 67
Álvarez, C., Clark, C. A., Zhang, S., Halm, E. A., Shannon, J. J., Girod, C. E., Cooper, L., & Amarasingham, R. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-28 · Cited by 66
Baillie, C. A., VanZandbergen, C., Tait, G., Hanish, A., Leas, B., French, B., William Hanson, C., Behta, M., & Umscheid, C. A. (2013) · Journal of Hospital Medicine · doi:10.1002/jhm.2106 · Cited by 65
Nijhawan, A. E., Clark, C., Kaplan, R., Moore, B., Halm, E. A., & Amarasingham, R. (2012) · JAIDS Journal of Acquired Immune Deficiency Syndromes · doi:10.1097/qai.0b013e31826ebc83 · Cited by 62
Prospective EHR-Based Clinical Trials: The Challenge of Missing Data
Kharrazi, H. H. K., Wang, C., & Scharfstein, D. O. (2014) · Journal of General Internal Medicine · doi:10.1007/s11606-014-2883-0 · Cited by 60
Hebert, C., Shivade, C., Foraker, R., Wasserman, J., Roth, C., Mekhjian, H., Lemeshow, S., & Embi, P. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-65 · Cited by 59
Billings, J., Georghiou, T., Blunt, I., & Bardsley, M. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003352 · Cited by 58
Goldstein, B. A., Knowles, J. W., Salfati, E., Ioannidis, J. P. A., & Assimes, T. L. (2014) · Frontiers in Genetics · doi:10.3389/fgene.2014.00254 · Cited by 57
Self-citationPredicting Hospital-Acquired Infections by Scoring System with Simple Parameters
Chang, Y.-J., Yeh, M.-L., Li, Y.-C., Hsu, C.-Y., Lin, C.-C., Hsu, M.-S., & Chiu, W.-T. (2011) · PLoS ONE · doi:10.1371/journal.pone.0023137 · Cited by 55
Learning to predict post-hospitalization VTE risk from EHR data.
Kawaler, E. A., Cobian, A. G., Peissig, P. L., Cross, D., Yale, S., & Craven, M. W. (2012) · PubMed Central · Cited by 52
Barrett, T. W., Martin, A. R., Storrow, A. B., Jenkins, C. A., Harrell, F. E., Russ, S., Roden, D. M., & Darbar, D. (2011) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2010.05.031 · Cited by 52
Electronic Health Record–Based Detection of Risk Factors for Clostridium difficile Infection Relapse
Hebert, C., Du, H., Peterson, L. R., & Robicsek, A. (2013) · Infection Control & Hospital Epidemiology · doi:10.1086/669864 · Cited by 51
Hippisley‐Cox, J., & Coupland, C. A. C. (2011) · British Journal of General Practice · doi:10.3399/bjgp11x606609 · Cited by 50
Rana, S., Tran, T., Luo, W., Phung, D., Kennedy, R. L., & Venkatesh, S. (2014) · Australian Health Review · doi:10.1071/ah14059 · Cited by 49
Predicting the risk of hyperkalemia in patients with chronic kidney disease starting lisinopril
Johnson, E. S., Weinstein, J. R., Thorp, M. L., Platt, R. W., Petrik, A. F., Yang, X., Anderson, S., & Smith, D. H. (2010) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1923 · Cited by 46
Skevofilakas, M., Zarkogianni, K., Karamanos, B. G., & Nikita, K. S. (2010) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.2010.5626245 · Cited by 45
Predictability of persistent frequent attendance: a historic 3-year cohort study
Smits, F. T., Brouwer, H. J., van Weert, H. C., Schene, A. H., & ter Riet, G. (2009) · British Journal of General Practice · doi:10.3399/bjgp09x395120 · Cited by 44
Hemoglobin A1c as a Predictor of Incident Diabetes
Cheng, P., Neugaard, B. I., Foulis, P., & Conlin, P. R. (2011) · Diabetes Care · doi:10.2337/dc10-0625 · Cited by 43
Electronic Prediction Rules for Methicillin-ResistantStaphylococcus aureusColonization
Robicsek, A. A., Beaumont, J. L., Wright, M., Thomson, R. B., Kaul, K. L., & Peterson, L. R. (2010) · Infection Control and Hospital Epidemiology · doi:10.1086/657631 · Cited by 43
Wang, L., Porter, B., Maynard, C. C., Bryson, C. L., Sun, H., Lowy, E., McDonell, M. B., Frisbee, K. L., Nielson, C., & Fihn, S. D. (2012) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2012.06.038 · Cited by 41
Roubinian, N. H., Murphy, E. L., Swain, B. E., Gardner, M. N., Liu, V. X., & Escobar, G. J. (2014) · BMC Health Services Research · doi:10.1186/1472-6963-14-213 · Cited by 40
Cholleti, S. R., Post, A. R., Gao, J., Lin, X., Bornstein, W. A., Cantrell, D., & Saltz, J. H. (2012) · PubMed · Cited by 38
Meyfroidt, G., Güiza, F., Cottem, D., De Becker, W., Van Loon, K., Aerts, J.-M., Berckmans, D., Ramon, J., Bruynooghe, M., & Van den Berghe, G. (2011) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-11-64 · Cited by 38
Predicting mortality of elderly patients acutely admitted to the Department of Internal Medicine
Smolin, B., Levy, Y., Sabbach-Cohen, E., Levi, L., & Mashiach, T. (2014) · International Journal of Clinical Practice · doi:10.1111/ijcp.12564 · Cited by 38
Tabak, Y. P., Sun, X., Derby, K. G., Kurtz, S. G., & Johannes, R. S. (2010) · Health Services Research · doi:10.1111/j.1475-6773.2010.01126.x · Cited by 37
Predicting patient acuity from electronic patient records
Kontio, E., Airola, A., Pahikkala, T., Lundgrén‐Laine, H., Junttila, K., Korvenranta, H., Salakoski, T., & Salanterä, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.04.001 · Cited by 37
Sho, S., Neal, M. D., Sperry, J. L., & Hackam, D. J. (2013) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2013.10.023 · Cited by 36
Kotz, D., Simpson, C. R., Viechtbauer, W., van Schayck, O. C. P., & Sheikh, A. (2014) · npj Primary Care Respiratory Medicine · doi:10.1038/npjpcrm.2014.11 · Cited by 36
Westra, B. L., Savik, K., Oancea, S. C., Choromanski, L., Holmes, J. H., & Bliss, D. Z. (2011) · Journal of Wound Ostomy and Continence Nursing · doi:10.1097/won.0b013e318202e4a6 · Cited by 34
Goldstein, B. A., Chang, T. I., Mitani, A., Assimes, T. L., & Winkelmayer, W. C. (2013) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.03050313 · Cited by 34
Self-citationRamchandran, K. J., Shega, J. W., Roenn, J. V., Schumacher, M., Szmuilowicz, E., RADEMAKER, A. W., Weitner, B. B., Loftus, P. D., Chu, I. M., & Weitzman, S. A. (2013) · Cancer · doi:10.1002/cncr.27974 · Cited by 32
Tabak, Y. P., Sun, X., Johannes, R. S., Hyde, L. A., Shorr, A. F., & Lindenauer, P. K. (2013) · Medical Care · doi:10.1097/mlr.0b013e3182901982 · Cited by 32
O'Leary, E. A., Desale, S. Y., Yi, W. S., Fujita, K. A., Hynes, C. F., Chandra, S. K., & Sava, J. A. (2014) · The American Surgeon · doi:10.1177/000313481408000621 · Cited by 32
Lipsky, B. A., Kollef, M. H., Miller, L. G., Sun, X., Johannes, R. S., & Tabak, Y. P. (2010) · Infection Control and Hospital Epidemiology · doi:10.1086/654007 · Cited by 31
Predicting Poor Outcomes in Heart Failure
Smith, D. H., Johnson, E. S., Thorp, M. L., Yang, X., Petrik, A. F., Platt, R. W., & Crispell, K. A. (2011) · The Permanente Journal · doi:10.7812/tpp/11-100 · Cited by 29
Hippisley‐Cox, J., & Coupland, C. A. C. (2012) · British Journal of General Practice · doi:10.3399/bjgp12x636074 · Cited by 27
Relationship of Predictive Modeling to Receiver Operating Characteristics
Mandic, S., Go, C. K., Aggarwal, I., MYERS, J. N., & Froelicher, V. F. (2008) · Journal of Cardiopulmonary Rehabilitation and Prevention · doi:10.1097/hcr.0b013e31818c3c78 · Cited by 24
Puttkammer, N. H., Zeliadt, S. B., Balan, J. G., Baseman, J. G., Destiné, R., Domerçant, J. W., France, G., Hyppolite, N., Pelletier, V., Raphael, N. A., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112261 · Cited by 23
Perkins, R. M., Rahman, A., Bucaloiu, I. D., Norfolk, E. R., Difilippo, W., Hartle, J. E., & Kirchner, H. L. (2013) · Clinical Nephrology · doi:10.5414/cn107961 · Cited by 23
Casarett, D., Farrington, S., Craig, T. R., Slattery, J., Harrold, J. K., Oldanie, B., Roy, J., Biehl, R. E., & Teno, J. M. (2012) · Journal of Palliative Medicine · doi:10.1089/jpm.2011.0531 · Cited by 22
A new statistical approach to predict bacteremia using electronic medical records
Jin, S. J., Kim, M., Yoon, J. H., & Song, Y. G. (2013) · Scandinavian Journal of Infectious Diseases · doi:10.3109/00365548.2013.799287 · Cited by 21
Exploring the Value of Clinical Data Standards to Predict Hospitalization of Home Care Patients
Swanberg, H., Oancea, S. C., Westra, B. L., & Monsen, K. A. (2012) · Applied Clinical Informatics · doi:10.4338/aci-2012-05-ra-0016 · Cited by 20
Prediction of morbidity and mortality in patients with type 2 diabetes
Wells, B. J., Roth, R., Nowacki, A. S., Arrigain, S., Yu, C., Rosenkrans, W. A., & Kattan, M. W. (2013) · PeerJ · doi:10.7717/peerj.87 · Cited by 18
Storing and interpreting genomic information in widely deployed electronic health record systems
Ury, A. G. (2013) · Genetics in Medicine · doi:10.1038/gim.2013.111 · Cited by 18
Predicting atrial fibrillation and flutter using Electronic Health Records
Karnik, S., Tan, S. L., Berg, B., Glurich, I., Zhang, J., Vidaillet, H. J., Jr., Page, C. D., & Chowdhary, R. (2012) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2012.6347254 · Cited by 16
Pulse pressure and stroke risk: development and validation of a new stroke risk model
Ayyagari, R., Vekeman, F., Lefèbvre, P., Ong, S. H., Faust, E., Trahey, A., Machnicki, G., & Duh, M. S. (2014) · Current Medical Research and Opinion · doi:10.1185/03007995.2014.971357 · Cited by 16
Navarro, A. E., Enguidanos, S. M., & Wilber, K. H. (2012) · Home Health Care Services Quarterly · doi:10.1080/01621424.2012.681561 · Cited by 14
Hivert, M., Dusseault-Bélanger, F., Cohen, A. A., Courteau, J., & Vanasse, A. (2012) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2012.02.009 · Cited by 14
Khan, A., Malone, M. L., Pagel, P., Vollbrecht, M., & Baumgardner, D. J. (2012) · PubMed · Cited by 12
Tangri, N., Inker, L. A., & Levey, A. S. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.11.012 · Cited by 12
Development and validation of the excess mortality ratio–based Emergency Severity Index
Hong, K. J., Shin, S. D., Ro, Y. S., Song, K. J., & Singer, A. J. (2012) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2011.12.011 · Cited by 8
Busch, A. B., Neelon, B., Zelevinsky, K., He, Y., & Normand, S. T. (2011) · Medical Care · doi:10.1097/mlr.0b013e3182422aec · Cited by 7
Tabak, Y. P., Sun, X., Hyde, L. A., Yaitanes, A., Derby, K. G., & Johannes, R. S. (2013) · Medical Care · doi:10.1097/mlr.0b013e318287d57d · Cited by 4
Sairamesh, J., Rajagopal, R., Nemana, R., & Argenbright, K. E. (2009) · PubMed · Cited by 1
Research works printed but not linked: 1
- Liu, 2010, Length of stay predictions: improvements through the use of automated laboratory and comorbidity variables, Med Care., 48, 739, 10.1097/mlr.0b013e3181e359f3 · DOI · Paper page
Cited by
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648
“In spite of the richness and potential of available data, scaling the development of predictive models is difficult because, for traditional predictive modeling techniques, each outcome to be predicted requires the creation of a custom dataset with specific variables [17].”
BackgroundInfluentialPotential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395
“A 2017 review 13 found that only 54% of studies that generated prediction algorithms based on the electronic health record accountedformissingdata.”
BackgroundClinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Khan, S. S., Coresh, J., Pencina, M. J., Ndumele, C. E., Rangaswami, J., Chow, S. L., Palaniappan, L. P., Sperling, L. S., Virani, S. S., Ho, J. E., et al. (2023) · Circulation · doi:10.1161/cir.0000000000001191 · Cited by 797
Self-citation
Metadata
- Authors
- Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A.
- Affiliations
- Duke University (US)
- Center for Predictive Medicine, Duke Clinical Research Institute, Duke University, Durham, NC 27710, USA
- Stanford University (US)
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-05-17
- Volume
- 24
- Issue
- 1
- Pages
- 198-208
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC5201180
- OpenAlex
- W2395172628
- Cited by
- 936
- References
- 144
- Citations by year
- 2026: 132 · 2025: 115 · 2024: 97 · 2023: 98 · 2022: 107 · 2021: 122 · 2020: 100 · 2019: 82 · 2018: 59 · 2017: 15 · 2016: 6
- FWCI
- 9.07
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Funders
- National Institute of Diabetes and Digestive and Kidney Diseases (K25DK097279)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Keywords
- electronic health records, clinical risk prediction, risk prediction model, model validation, multicenter validation, longitudinal data, missing data, loss to follow-up, model reproducibility, concordance index, mortality prediction, hospitalization prediction, service utilization
- MeSH terms
- Humans, Models, Statistical, Risk Assessment, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.