Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review
Open accessThe DOI matches its Crossref record; 141 of the 144 references it cites were found in a registry or an index.
Abstract
Objective: Electronic health records (EHRs) are an increasingly common data source for clinical risk prediction, presenting both unique analytic opportunities and challenges. We sought to evaluate the current state of EHR based risk prediction modeling through a systematic review of clinical prediction studies using EHR data. Methods: We searched PubMed for articles that reported on the use of an EHR to develop a risk prediction model from 2009 to 2014. Articles were extracted by two reviewers, and we abstracted information on study design, use of EHR data, model building, and performance from each publication and supplementary documentation. Results: We identified 107 articles from 15 different countries. Studies were generally very large (median sample size = 26 100) and utilized a diverse array of predictors. Most used validation techniques (n = 94 of 107) and reported model coefficients for reproducibility (n = 83). However, studies did not fully leverage the breadth of EHR data, as they uncommonly used longitudinal information (n = 37) and employed relatively few predictor variables (median = 27 variables). Less than half of the studies were multicenter (n = 50) and only 26 performed validation across sites. Many studies did not fully address biases of EHR data such as missing data or loss to follow-up. Average c-statistics for different outcomes were: mortality (0.84), clinical prediction (0.83), hospitalization (0.71), and service utilization (0.71). Conclusions: EHR data present both opportunities and challenges for clinical risk prediction. There is room for improvement in designing such studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In spite of the richness and potential of available data, scaling the development of predictive models is difficult because, for traditional predictive modeling techniques, each outcome to be predicted requires the creation of a custom dataset with specific variables.7”
From Scalable and accurate deep learning with electronic health records (2018) · cites this paper as [7]
In the citing paper
“Despite the rich data now digitized in EHRs,20 a recent systematic review of the medical literature7 found that predictive models built with EHR data use a median of only 27 variables, rely on traditional generalized linear models, and are built using data at a single center.”
From Scalable and accurate deep learning with electronic health records (2018) · cites this paper as [7]
In the citing paper
“Although external validation is recognized as an area of improvement for clinical risk prediction models [85]”
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Self-citationPrediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581 · cited 2×
“Traditionally, risk pre- diction algorithms have been developed from large cohort studies such as the Framingham Heart Study.3 These studies were designed to follow people for years or even decades.”
MethodMoons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Joint modelling of longitudinal measurements and event time data
Henderson, R. (2000) · Biostatistics · doi:10.1093/biostatistics/1.4.465 · Cited by 913
The Electronic Medical Records and Genomics (eMERGE) Network: past, present, and future
Gottesman, O., Kuivaniemi, H., Tromp, G., Faucett, W. A., Li, R., Manolio, T. A., Sanderson, S. C., Kannry, J., Zinberg, R., Basford, M. A., et al. (2013) · Genetics in Medicine · doi:10.1038/gim.2013.72 · Cited by 698
Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve
Cook, N. R. (2008) · Clinical Chemistry · doi:10.1373/clinchem.2007.096529 · Cited by 691
Research works printed but not linked: 1
- Liu, 2010, Length of stay predictions: improvements through the use of automated laboratory and comorbidity variables, Med Care., 48, 739, 10.1097/mlr.0b013e3181e359f3 · DOI · Paper page
Cited by
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648
“In spite of the richness and potential of available data, scaling the development of predictive models is difficult because, for traditional predictive modeling techniques, each outcome to be predicted requires the creation of a custom dataset with specific variables [17].”
BackgroundInfluentialPotential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395
“A 2017 review 13 found that only 54% of studies that generated prediction algorithms based on the electronic health record accountedformissingdata.”
BackgroundClinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Khan, S. S., Coresh, J., Pencina, M. J., Ndumele, C. E., Rangaswami, J., Chow, S. L., Palaniappan, L. P., Sperling, L. S., Virani, S. S., Ho, J. E., et al. (2023) · Circulation · doi:10.1161/cir.0000000000001191 · Cited by 797
Self-citationClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618
Natural Language Processing of Clinical Notes on Chronic Diseases: Systematic Review
Sheikhalishahi, S., Miotto, R., Dudley, J. T., Lavelli, A., Rinaldi, F., & Osmani, V. (2019) · JMIR Medical Informatics · doi:10.2196/12239 · Cited by 558
“Methods based in analysis of EHRs [2] are resulting in improved understanding of patient clinical trajectories [3] while enabling better patient stratification and risk prediction [4-6].”
MethodMachine learning in cardiovascular medicine: are we there yet?
Shameer, K., Johnson, K. W., Glicksberg, B. S., Dudley, J. T., & Sengupta, P. P. (2018) · Heart · doi:10.1136/heartjnl-2017-311198 · Cited by 555
“40 However, machine learning algorithms are not a panacea and also have limitations.”
BackgroundGoldstein, B. A., Navar, A. M., & Carter, R. E. (2016) · European Heart Journal · doi:10.1093/eurheartj/ehw302 · Cited by 543
Self-citationRole of artificial intelligence in patient safety outcomes: systematic literature review
Choudhury, A., & Asan, O. (2020) · JMIR Medical Informatics · doi:10.2196/18599 · Cited by 440
Gerry, S., Bonnici, T., Birks, J., Kirtley, S., Virdee, P. S., Watkinson, P. J., & Collins, G. S. (2020) · BMJ · doi:10.1136/bmj.m1501 · Cited by 383
Measuring objective and subjective well-being: dimensions and data sources
Voukelatou, V., Gabrielli, L., Miliou, I., Cresci, S., Sharma, R., Tesconi, M., & Pappalardo, L. (2021) · International Journal of Data Science and Analytics · doi:10.1007/s41060-020-00224-2 · Cited by 365
A predictive analytics approach for stroke prediction using machine learning and neural networks
Dev, S., Wang, H., Nwosu, C. S., Jain, N., Veeravalli, B., & John, D. (2022) · Healthcare Analytics · doi:10.1016/j.health.2022.100032 · Cited by 260
Ye, C., Fu, T., Hao, S., Zhang, Y., Wang, O., Jin, B., Xia, M., Liu, M., Zhou, X., Wu, Q., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9268 · Cited by 250
“clinical history and can be used to build risk models that can potentially help predict disease progression, revealing the evolution of disease, and thus, offering great promise for accelerating clinical research and predictive analysis on a population [13,14].”
Velupillai, S., Suominen, H., Liakata, M., Roberts, A., Shah, A. D., Morley, K., Osborn, D., Hayes, J., Stewart, R., Downs, J., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.10.005 · Cited by 247
Steele, A. J., Denaxas, S. C., Shah, A. D., Hemingway, H., & Luscombe, N. M. (2018) · PLOS ONE · doi:10.1371/journal.pone.0202344 · Cited by 241
Time Series Prediction Using Deep Learning Methods in Healthcare
Morid, M. A., Sheng, O. R. L., & Dunbar, J. (2023) · ACM Transactions on Management Information Systems · doi:10.1145/3531326 · Cited by 231
A systematic review of the prediction of hospital length of stay: Towards a unified framework
Stone, K., Zwiggelaar, R., Jones, P., & Mac Parthaláin, N. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000017 · Cited by 231
Machine learning for comprehensive forecasting of Alzheimer’s Disease progression
Fisher, C. K., Smith, A. M., Walsh, J. R., Diseases, C. A. M., Simon, A. J., Edgar, C., Jack, C. R., Holtzman, D., Russell, D., Hill, D., et al. (2019) · Scientific Reports · doi:10.1038/s41598-019-49656-2 · Cited by 223
Bernert, R. A., Hilberg, A. M., Melia, R., Kim, J. P., Shah, N. H., & Abnousi, F. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17165929 · Cited by 222
Reps, J. M., Schuemie, M. J., Suchard, M. A., Ryan, P. B., & Rijnbeek, P. R. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy032 · Cited by 205
Exploration, Inference, and Prediction in Neuroscience and Biomedicine
Bzdok, D., & Ioannidis, J. P. A. (2019) · Trends in Neurosciences · doi:10.1016/j.tins.2019.02.001 · Cited by 204
Self-citationA Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?
Bharati, S., Mondal, M. R. H., & Podder, P. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3266418 · Cited by 203
Machine learning and artificial intelligence in research and healthcare
Rubinger, L., Gazendam, A., Ekhtiari, S., & Bhandari, M. (2023) · Injury · doi:10.1016/j.injury.2022.01.046 · Cited by 203
Readmission prediction using deep learning on electronic health records
Ashfaq, A., Sant’Anna, A., Lingman, M., & Nowaczyk, S. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103256 · Cited by 194
Kuziemsky, C., Maeder, A. J., John, O., Gogia, S. B., Basu, A., Meher, S., & Ito, M. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677897 · Cited by 183
A Path for Translation of Machine Learning Products into Healthcare Delivery
(2020) · EMJ Innovations · doi:10.33590/emjinnov/19-00172 · Cited by 172
“10,21 Configurations are categorised as either centralised or decentralised.”
BackgroundZhang, Y., Wang, S., Hermann, A., Joly, R., & Pathak, J. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.09.113 · Cited by 164
Minimal Impact of Implemented Early Warning Score and Best Practice Alert for Patient Deterioration*
Bedoya, A. D., Clement, M. E., Phelan, M., Steorts, R. C., O’Brien, C., & Goldstein, B. A. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003439 · Cited by 162
Self-citationGRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
Brouwer, E. D., Simm, J., Arany, Á., & Moreau, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.12374 · Cited by 162
Machine Learning in Healthcare
Callahan, A., & Shah, N. H. (2017) · Key Advances in Clinical Informatics · doi:10.1016/b978-0-12-809523-2.00019-4 · Cited by 160
Predicting the onset of type 2 diabetes using wide and deep learning with electronic health records
Nguyen, B. P., Pham, H. N., Tran, H., Nghiem, N., Nguyen, Q. H., Do, T. T. T., Tran, C. T., & Simpson, C. R. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.105055 · Cited by 158
Frailty Screening Using the Electronic Health Record Within a Medicare Accountable Care Organization
Pajewski, N. M., Lenoir, K. M., Wells, B. J., Williamson, J. D., & Callahan, K. E. (2019) · The Journals of Gerontology Series A · doi:10.1093/gerona/glz017 · Cited by 158
Li, Y., Mamouei, M., Salimi-Khorshidi, G., Rao, S., Hassaine, A., Canoy, D., Lukasiewicz, T., & Rahimi, K. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3224727 · Cited by 156
Xie, F., Yuan, H., Ning, Y., Ong, M. E. H., Feng, M., Hsu, W., Chakraborty, B., & Liu, N. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103980 · Cited by 153
Murala, D. K., Panda, S. K., & Dash, S. P. (2023) · IEEE Access · doi:10.1109/access.2023.3340791 · Cited by 152
Putting the data before the algorithm in big data addressing personalized healthcare
Cahan, E. M., Hernandez-Boussard, T., Thadaney-Israni, S., & Rubin, D. L. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0157-2 · Cited by 149
Clinician checklist for assessing suitability of machine learning applications in healthcare
Scott, I., Carter, S., & Coiera, E. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100251 · Cited by 143
A novel machine learning approach for diagnosing diabetes with a self-explainable interface
Dharmarathne, G., Jayasinghe, T. N., Bogahawaththa, M., Meddage, D. P. P., & Rathnayake, U. (2024) · Healthcare Analytics · doi:10.1016/j.health.2024.100301 · Cited by 98
Xie, F., Chakraborty, B., Ong, M. E. H., Goldstein, B. A., & Liu, N. (2020) · JMIR Medical Informatics · doi:10.2196/21798 · Cited by 140
Self-citationPredictive Modeling of the Progression of Alzheimer’s Disease with Recurrent Neural Networks
Wang, T., Qiu, R. G., & Yu, M. (2018) · Scientific Reports · doi:10.1038/s41598-018-27337-w · Cited by 138
Lin, H., Tang, X., Shen, P., Zhang, D., Wu, J., Zhang, J., Lu, P., Si, Y., & Gao, P. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019698 · Cited by 136
Lee, T. C., Shah, N. U., Haack, A., & Baxter, S. L. (2020) · Informatics · doi:10.3390/informatics7030025 · Cited by 133
“[17] examined risk prediction models utilizing EHR data, but focused on model development and validation rather than implementation in clinical settings.”
MethodLanguage models are an effective representation learning technique for electronic health record data
Steinberg, E., Jung, K., Fries, J. A., Corbin, C. K., Pfohl, S. R., & Shah, N. H. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103637 · Cited by 132
Sharma, V., Ali, I., van der Veer, S. N., Martin, G. P., Ainsworth, J. D., & Augustine, T. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100253 · Cited by 131
Bishara, A. M., Chiu, C., Whitlock, E. L., Douglas, V. C., Lee, S., Butte, A. J., Leung, J. M., & Donovan, A. L. (2022) · BMC Anesthesiology · doi:10.1186/s12871-021-01543-y · Cited by 128
Li, L., Zhou, J., Gao, Z., Hua, W., Fan, L., Yu, H., Hagen, L., Zhang, Y., Assimes, T. L., Hemphill, L., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.03066 · Cited by 127
Benedictis, A. D., Lettieri, E., Gastaldi, L., Masella, C., Urgu, A., & Tartaglini, D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0234108 · Cited by 120
Wardi, G., Carlile, M., Holder, A., Shashikumar, S., Hayden, S. R., & Nemati, S. (2021) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2020.11.007 · Cited by 96
Applications of machine learning in routine laboratory medicine: Current state and future directions
Rabbani, N., Kim, G. Y. E., Suarez, C. J., & Chen, J. H. (2022) · Clinical Biochemistry · doi:10.1016/j.clinbiochem.2022.02.011 · Cited by 118
Ethics and Epistemology in Big Data Research
Lipworth, W. L., Mason, P. H., Kerridge, I., & Ioannidis, J. P. A. (2017) · Journal of Bioethical Inquiry · doi:10.1007/s11673-017-9771-3 · Cited by 117
Self-citationDiao, J. A., Shi, I., Murthy, V. L., Buckley, T. A., Patel, C. J., Pierson, E., Yeh, R. W., Kazi, D. S., Wadhera, R. K., & Manrai, A. K. (2024) · JAMA · doi:10.1001/jama.2024.12537 · Cited by 116
Lin, C.-S., Liu, W.-T., Tsai, D.-J., Lou, Y.-S., Chang, C.-H., Lee, C.-C., Fang, W.-H., Wang, C.-C., Chen, Y.-Y., Lin, W.-S., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02961-4 · Cited by 115
Prognostic models will be victims of their own success, unless…
Lenert, M. C., Matheny, M. E., & Walsh, C. G. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz145 · Cited by 92
Li, Q., Yang, X., Xu, J., Guo, Y., He, X., Hu, H., Lyu, T., Marra, D., Miller, A., Smith, G., et al. (2023) · Alzheimer's & Dementia · doi:10.1002/alz.12967 · Cited by 113
“22,23 Many risk factors of ADRD identified by previous studies (e.g., obesity, hypertension, high cholesterol) are routinely captured in patients’ 25 andthesamplesizeswerealsosmall(i.e., n = 614fordefinitiveAD,and n = 2026 for probable AD).”
BackgroundInfluentialSwinckels, L., Bennis, F. C., Ziesemer, K. A., Scheerman, J. F. M., Bijwaard, H., de Keijzer, A., & Bruers, J. J. M. (2024) · Journal of Medical Internet Research · doi:10.2196/48320 · Cited by 109
Beesley, L. J., Salvatore, M., Fritsche, L. G., Pandit, A., Rao, A., Brummett, C., Willer, C. J., Lisabeth, L. D., & Mukherjee, B. (2020) · Statistics in Medicine · doi:10.1002/sim.8445 · Cited by 104
Missing data should be handled differently for prediction than for description or causal explanation
Sperrin, M., Martin, G. P., Sisk, R., & Peek, N. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.028 · Cited by 103
Automating Electronic Health Record Data Quality Assessment
Ozonze, O., Scott, P. J., & Hopgood, A. A. (2023) · Journal of Medical Systems · doi:10.1007/s10916-022-01892-2 · Cited by 69
Chen, J. H., Alagappan, M., Goldstein, M. K., Asch, S. M., & Altman, R. B. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.03.006 · Cited by 101
Ejiyi, C. J., Qin, Z., Amos, J., Ejiyi, M. B., Nnani, A. O., Ejiyi, T. U., Agbesi, V. K., Diokpo, C., & Okpara, C. (2023) · Healthcare Analytics · doi:10.1016/j.health.2023.100166 · Cited by 100
Du, Z., Yang, Y., Zheng, J., Li, Q., Lin, D., Li, Y., Fan, J., Cheng, W., Chen, X.-H., & Cai, Y. (2020) · JMIR Medical Informatics · doi:10.2196/17257 · Cited by 99
Early risk assessment for COVID-19 patients from emergency department data using machine learning
Heldt, F. S., Vizcaychipi, M. P., Peacock, S., Cinelli, M., McLachlan, L., Andreotti, F., Jovanović, S., Dürichen, R., Lipunova, N., Fletcher, R. A., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-83784-y · Cited by 97
A framework for making predictive models useful in practice
Jung, K., Kashyap, S., Avati, A., Harman, S., Shaw, H., Li, R., Smith, M., Shum, K., Javitz, J., Vetteth, Y., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa318 · Cited by 78
Machine Learning for Predicting Readmission Risk Among the Frail - Explainable AI for Healthcare
Mohanty, S. D., Lekan, D. A., McCoy, T. P., Jenkins, M., & Manda, P. (2021) · Patterns · doi:10.1016/j.patter.2021.100395 · Cited by 97
Phelan, M., Bhavsar, N., & Goldstein, B. A. (2017) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.243 · Cited by 96
Self-citationLindberg, D. S., Prosperi, M., Bjarnadottir, R. I., Thomas, J., Crane, M., Chen, Z., Shear, K., Solberg, L. M., Snigurska, U. A., Wu, Y., et al. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104272 · Cited by 95
Bhavsar, N. A., Gao, A., Phelan, M., Pagidipati, N. J., & Goldstein, B. A. (2018) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.2716 · Cited by 93
Self-citationJauk, S., Kramer, D., Großauer, B., Rienmüller, S., Avian, A., Berghold, A., Leodolter, W., & Schulz, S. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa113 · Cited by 83
Liu, S., Wang, X., Hou, Y., Li, G., Wang, H., Xu, H., Xiang, Y., & Tang, B. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3217810 · Cited by 93
Primary care visits increase utilization of evidence-based preventative health measures
Hostetter, J. E., Schwarz, N., Klug, M. G., Wynne, J., & Basson, M. D. (2020) · BMC Family Practice · doi:10.1186/s12875-020-01216-8 · Cited by 92
Debray, T. P. A., Collins, G. S., Riley, R. D., Snell, K. I. E., Van Calster, B., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071058 · Cited by 57
Prabhu, L. A. J., Sengan, S., Kamalam, G. K., Vellingiri, J., Gopal, J., Velayutham, P., & Subramaniyaswamy, V. (2020) · Microprocessors and Microsystems · doi:10.1016/j.micpro.2020.103344 · Cited by 91
Liao, W., Coupland, C. A. C., Burchardt, J., Baldwin, D., Gleeson, F. V., Baldwin, D., Batchkala, G., Buchanan, J. P., Burchardt, J., Chakraborty, R., et al. (2023) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(23)00050-4 · Cited by 91
Yang, C. B., Kors, J. A., Ioannou, S., John, L. H., Markus, A. F., Rekkas, A., de Ridder, M. A. J., Seinen, T. M., Williams, R. D., & Rijnbeek, P. R. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac002 · Cited by 90
Inclusion of Unstructured Clinical Text Improves Early Prediction of Death or Prolonged ICU Stay*
Weissman, G. E., Hubbard, R. A., Ungar, L. H., Harhay, M. O., Greene, C. S., Himes, B. E., & Halpern, S. D. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000003148 · Cited by 89
Predictors of 30-Day Mortality Among Dutch Patients Undergoing Colorectal Cancer Surgery, 2011-2016
van den Bosch, T., Warps, A. K., Babberich, M. P. M. D. N. T., Stamm, C., Geerts, B. F., Vermeulen, L., Wouters, M. W. J. M., Dekker, J. W. T., Tollenaar, R. A. �. �., Tanis, P. J., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.7737 · Cited by 87
Transforming healthcare with data analytics: Predictive models for patient outcomes
Nwaimo, C. S., Adegbola, A. E., & Adegbola, M. D. (2024) · GSC Biological and Pharmaceutical Sciences · doi:10.30574/gscbps.2024.27.3.0190 · Cited by 87
van Daalen, K. R., Zhang, D., Kaptoge, S. K., Paige, E., Angelantonio, E. D., & Pennells, L. A. M. (2024) · The Lancet Global Health · doi:10.1016/s2214-109x(24)00210-9 · Cited by 87
Tsai, M., Chen, K., & Chen, P. (2025) · Journal of the American Heart Association · doi:10.1161/jaha.124.036946 · Cited by 86
Cavedoni, S., Chirico, A., Pedroli, E., Cipresso, P., & Riva, G. (2020) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.00245 · Cited by 85
Ross, E., Jung, K., Dudley, J. T., Li, L., Leeper, N. J., & Shah, N. H. (2019) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.118.004741 · Cited by 85
Shung, D. L., Simonov, M., Gentry, M., Au, B., & Laine, L. (2019) · Digestive Diseases and Sciences · doi:10.1007/s10620-019-05645-z · Cited by 85
Systemic nature of spinal muscular atrophy revealed by studying insurance claims
Lipnick, S., Agniel, D. M., Aggarwal, R., Makhortova, N. R., Finlayson, S. G., Brocato, A., Palmer, N., Darras, B. T., Kohane, I. S., & Rubin, L. L. (2019) · PLoS ONE · doi:10.1371/journal.pone.0213680 · Cited by 85
Wang, H.-H., Wang, Y.-H., Liang, C.-W., & Li, Y. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2019.2335 · Cited by 85
Hong, C., Rush, E., Liu, M., Zhou, D., Sun, J., Sonabend, A., Castro, V. M., Schubert, P., Panickan, V. A., Cai, T., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00519-z · Cited by 64
Clinical code set engineering for reusing EHR data for research: A review
Williams, R., Kontopantelis, E., Buchan, I. E., & Peek, N. B. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.04.010 · Cited by 83
Tian, M., Chen, B., Guo, A., Jiang, S., & Zhang, A. R. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae229 · Cited by 81
Bull, L. M., Lunt, M., Martin, G. P., Hyrich, K. L., & Sergeant, J. C. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00078-z · Cited by 80
Giddings, R., Joseph, A., Callender, T., Janes, S. M., van der Schaar, M., Sheringham, J., & Navani, N. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00241-8 · Cited by 80
Silverio, A., Cavallo, P., Rosa, R. D., & Galasso, G. (2019) · Frontiers in Medicine · doi:10.3389/fmed.2019.00036 · Cited by 80
Use of unstructured text in prognostic clinical prediction models: a systematic review
Seinen, T. M., Fridgeirsson, E. A., Ioannou, S., Jeannetot, D., John, L. H., Kors, J. A., Markus, A. F., Pera, V., Rekkas, A., Williams, R. D., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac058 · Cited by 79
Canales, L., Menke, S., Marchesseau, S., D’Agostino, A., Rio‐Bermudez, C. D., Taberna, M., & Tello, J. (2021) · JMIR Medical Informatics · doi:10.2196/20492 · Cited by 78
Ethical dilemmas posed by mobile health and machine learning in psychiatry research
Jacobson, N. C., Bentley, K. H., Walton, A., Wang, S. B., Fortgang, R. G., Millner, A. J., Coombs, G., Rodman, A. M., & Coppersmith, D. D. L. (2020) · Bulletin of the World Health Organization · doi:10.2471/blt.19.237107 · Cited by 76
Ebert, D. D., Harrer, M., Apolinário-Hagen, J., & Baumeister, H. (2019) · Advances in Experimental Medicine and Biology · doi:10.1007/978-981-32-9721-0_29 · Cited by 76
Sisk, R., Lin, L., Sperrin, M., Barrett, J., Tom, B. D. M., Diaz‐Ordaz, K., Peek, N. B., & Martin, G. P. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa242 · Cited by 76
Care Models for Acute Chest Pain That Improve Outcomes and Efficiency
Dawson, L. P., Smith, K. L., Cullen, L., Nehme, Z., Lefkovits, J., Taylor, A. J., & Stub, D. A. (2022) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2022.03.380 · Cited by 75
Cho, S. M., Austin, P. C., Ross, H. J., Abdel-Qadir, H. M., Chicco, D., TOMLINSON, G. A., Taheri, C., Foroutan, F., Lawler, P. R., Billia, F., et al. (2021) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.02.020 · Cited by 74
Missing data was handled inconsistently in UK prediction models: a review of method used
Tsvetanova, A., Sperrin, M., Peek, N. B., Buchan, I. E., Hyland, S. L., & Martin, G. P. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.09.008 · Cited by 72
Simonov, M., Ugwuowo, U. C., Moreira, E., Yamamoto, Y., Biswas, A., Martin, M. C., Testani, J. M., & Wilson, F. P. (2019) · PLoS Medicine · doi:10.1371/journal.pmed.1002861 · Cited by 72
Metadata
- Authors
- Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A.
- Affiliations
- Duke University (US)
- Center for Predictive Medicine, Duke Clinical Research Institute, Duke University, Durham, NC 27710, USA
- Stanford University (US)
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-05-17
- Volume
- 24
- Issue
- 1
- Pages
- 198-208
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC5201180
- OpenAlex
- W2395172628
- Cited by
- 936
- References
- 144
- Citations by year
- 2026: 132 · 2025: 115 · 2024: 97 · 2023: 98 · 2022: 107 · 2021: 122 · 2020: 100 · 2019: 82 · 2018: 59 · 2017: 15 · 2016: 6
- FWCI
- 9.07
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Funders
- National Institute of Diabetes and Digestive and Kidney Diseases (K25DK097279)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Keywords
- electronic health records, clinical risk prediction, risk prediction model, model validation, multicenter validation, longitudinal data, missing data, loss to follow-up, model reproducibility, concordance index, mortality prediction, hospitalization prediction, service utilization
- MeSH terms
- Humans, Models, Statistical, Risk Assessment, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.