Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review
Open accessThe DOI matches its Crossref record; 141 of the 144 references it cites were found in a registry or an index.
Abstract
Objective: Electronic health records (EHRs) are an increasingly common data source for clinical risk prediction, presenting both unique analytic opportunities and challenges. We sought to evaluate the current state of EHR based risk prediction modeling through a systematic review of clinical prediction studies using EHR data. Methods: We searched PubMed for articles that reported on the use of an EHR to develop a risk prediction model from 2009 to 2014. Articles were extracted by two reviewers, and we abstracted information on study design, use of EHR data, model building, and performance from each publication and supplementary documentation. Results: We identified 107 articles from 15 different countries. Studies were generally very large (median sample size = 26 100) and utilized a diverse array of predictors. Most used validation techniques (n = 94 of 107) and reported model coefficients for reproducibility (n = 83). However, studies did not fully leverage the breadth of EHR data, as they uncommonly used longitudinal information (n = 37) and employed relatively few predictor variables (median = 27 variables). Less than half of the studies were multicenter (n = 50) and only 26 performed validation across sites. Many studies did not fully address biases of EHR data such as missing data or loss to follow-up. Average c-statistics for different outcomes were: mortality (0.84), clinical prediction (0.83), hospitalization (0.71), and service utilization (0.71). Conclusions: EHR data present both opportunities and challenges for clinical risk prediction. There is room for improvement in designing such studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In spite of the richness and potential of available data, scaling the development of predictive models is difficult because, for traditional predictive modeling techniques, each outcome to be predicted requires the creation of a custom dataset with specific variables.7”
From Scalable and accurate deep learning with electronic health records (2018) · cites this paper as [7]
In the citing paper
“Despite the rich data now digitized in EHRs,20 a recent systematic review of the medical literature7 found that predictive models built with EHR data use a median of only 27 variables, rely on traditional generalized linear models, and are built using data at a single center.”
From Scalable and accurate deep learning with electronic health records (2018) · cites this paper as [7]
In the citing paper
“Although external validation is recognized as an area of improvement for clinical risk prediction models [85]”
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Self-citationPrediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581 · cited 2×
“Traditionally, risk pre- diction algorithms have been developed from large cohort studies such as the Framingham Heart Study.3 These studies were designed to follow people for years or even decades.”
MethodMoons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Joint modelling of longitudinal measurements and event time data
Henderson, R. (2000) · Biostatistics · doi:10.1093/biostatistics/1.4.465 · Cited by 913
The Electronic Medical Records and Genomics (eMERGE) Network: past, present, and future
Gottesman, O., Kuivaniemi, H., Tromp, G., Faucett, W. A., Li, R., Manolio, T. A., Sanderson, S. C., Kannry, J., Zinberg, R., Basford, M. A., et al. (2013) · Genetics in Medicine · doi:10.1038/gim.2013.72 · Cited by 698
Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve
Cook, N. R. (2008) · Clinical Chemistry · doi:10.1373/clinchem.2007.096529 · Cited by 691
Research works printed but not linked: 1
- Liu, 2010, Length of stay predictions: improvements through the use of automated laboratory and comorbidity variables, Med Care., 48, 739, 10.1097/mlr.0b013e3181e359f3 · DOI · Paper page
Cited by
Items 201–300 of 1,000
Decentralized Clinical Trials in the Era of Real‐World Evidence: A Statistical Perspective
Chen, J., Di, J., Daizadeh, N. S., Lü, Y., Wang, H., Shen, Y. L., Kirk, J., Rockhold, F. W., Pang, H., Zhao, J., et al. (2025) · Clinical and Translational Science · doi:10.1111/cts.70117 · Cited by 38
Goshen, R., Mizrahi, B., Akiva, P., Kinar, Y., Choman, E. N., Shalev, V., Sopik, V., Kariv, R., & Narod, S. A. (2017) · British Journal of Cancer · doi:10.1038/bjc.2017.53 · Cited by 38
Bavishi, A. A., Bruce, M., Ning, H., Freaney, P. M., Glynn, P. A., Ahmad, F. S., Yancy, C. W., Shah, S. J., Allen, N. B., Vupputuri, S. X., et al. (2020) · Circulation Heart Failure · doi:10.1161/circheartfailure.120.007462 · Cited by 38
MACHINE LEARNING TECHNIQUES FOR EARLY HEART FAILURE PREDICTION
Huang, N. S. M., Ibrahim, Z., & Diah, N. M. (2021) · MALAYSIAN JOURNAL OF COMPUTING · doi:10.24191/mjoc.v6i2.13708 · Cited by 38
Lentzen, M., Lindén, T., Veeranki, S. P. K., Madan, S., Kramer, D., Leodolter, W., & Fröhlich, H. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3288768 · Cited by 38
Performing an Informatics Consult: Methods and Challenges
Schuler, A., Callahan, A., Jung, K., & Shah, N. H. (2018) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.12.023 · Cited by 37
Castiñeira, D., Schlosser, K. R., Geva, A., Rahmani, A. R., Fiore, G. A., Walsh, B. K., Smallwood, C. D., Arnold, J. H., & Santillana, M. (2020) · Respiratory Care · doi:10.4187/respcare.07561 · Cited by 37
A Novel Deep Similarity Learning Approach to Electronic Health Records Data
Gupta, V., Sachdeva, S., & Bhalla, S. (2020) · IEEE Access · doi:10.1109/access.2020.3037710 · Cited by 37
Prediction models used in the progression of chronic kidney disease: A scoping review
Lim, D., Boyd, J., Thomas, E., Chakera, A., Tippaya, S., Irish, A. B., Manuel, J., Betts, K. S., & Robinson, S. (2022) · PLoS ONE · doi:10.1371/journal.pone.0271619 · Cited by 37
COMMUTE: Communication-efficient transfer learning for multi-site risk prediction
Gu, T., Lee, P. H., & Duan, R. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104243 · Cited by 37
Fraccaro, P., van der Veer, S. N., Brown, B. C., Prosperi, M., O'Donoghue, D. J., Collins, G. S., Buchan, I. E., & Peek, N. B. (2016) · BMC Medicine · doi:10.1186/s12916-016-0650-2 · Cited by 37
Kom, I. Y., Dongelmans, D. A., de Keizer, N. F., Jager, K. J., Schut, M. C., Abu–Hanna, A., & Klopotowska, J. E. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad014 · Cited by 36
Clinical risk prediction using language models: benefits and considerations
Acharya, A., Shrestha, S., Chen, A., Conte, J., Avramovic, S., Sikdar, S., Anastasopoulos, A., & Das, S. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae030 · Cited by 36
Presentation of lung cancer in primary care
Weller, D., Peake, M. D., & Field, J. K. (2019) · npj Primary Care Respiratory Medicine · doi:10.1038/s41533-019-0133-y · Cited by 36
Nesaragi, N., Patidar, S., & Veerakumar, T. (2021) · Journal of Applied Biomedicine · doi:10.1016/j.bbe.2021.06.009 · Cited by 36
Xie, F., Ning, Y., Yuan, H., Goldstein, B. A., Ong, M. E. H., Liu, N., & Chakraborty, B. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103959 · Cited by 36
Self-citationZhang, H., Zhang, J., Ni, W., Jiang, Y., Liu, K., Sun, D., & Li, J. (2022) · JMIR Medical Informatics · doi:10.2196/35239 · Cited by 36
Towards an automated multimodal clinical decision support system at the post anesthesia care unit
Olsen, R. M., Aasvang, E. K., Meyhoff, C. S., & Sørensen, H. B. D. (2018) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2018.07.018 · Cited by 35
Identifying patients at highest-risk: the best timing to apply a readmission predictive model
Flaks‐Manov, N., Topaz, M., Hoshen, M. B., Balicer, R. D., & Shadmi, E. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0836-6 · Cited by 35
Prediction Accuracy With Electronic Medical Records Versus Administrative Claims
Zeltzer, D., Balicer, R. D., Shir, T., Flaks-Manov, N., Einav, L., & Shadmi, E. (2019) · Medical Care · doi:10.1097/mlr.0000000000001135 · Cited by 34
Shung, D. L., Chan, C., You, K., Nakamura-Sakai, S., Saarinen, T., Zheng, N. S., Simonov, M., Li, D. K., Tsay, C. J., Kawamura, Y. I., et al. (2024) · Gastroenterology · doi:10.1053/j.gastro.2024.06.030 · Cited by 34
Jahan, N., & Akther, J. (2024) · American Journal of Interdisciplinary Studies · doi:10.63125/5gh5df13 · Cited by 34
Baralou, V., Kalpourtzi, N., & Touloumi, G. (2022) · Biometrical Journal · doi:10.1002/bimj.202100380 · Cited by 34
Chen, X., Chen, H., Nan, S., Kong, X., Duan, H., & Zhu, H. (2023) · JMIR Medical Informatics · doi:10.2196/38590 · Cited by 34
Park, J., Kim, J., Ryu, B., Heo, E., Jung, S. Y., & Yoo, S. (2018) · Journal of Medical Internet Research · doi:10.2196/11757 · Cited by 34
Edgcomb, J. B., Thiruvalluru, R. K., Pathak, J., & Brooks, J. O. (2021) · Medical Care · doi:10.1097/mlr.0000000000001467 · Cited by 34
On predictions in critical care: The individual prognostication fallacy in elderly patients
Beil, M., Sviri, S., Flaatten, H. K., de Lange, D. W., Jung, C., Szczeklik, W., Leaver, S. K., Rhodes, A., Guidet, B., & van Heerden, P. V. (2020) · Journal of Critical Care · doi:10.1016/j.jcrc.2020.10.006 · Cited by 35
Goldstein, B. A., Navar, A. M., & Pencina, M. J. (2016) · JAMA Cardiology · doi:10.1001/jamacardio.2016.3826 · Cited by 33
Self-citationDormosh, N., van de Loo, B. P. A., Heymans, M. W., Schut, M. C., Medlock, S. K., van Schoor, N. M., van der Velde, N., & Abu–Hanna, A. (2024) · Age and Ageing · doi:10.1093/ageing/afae131 · Cited by 33
Shung, D. L., Huang, J., Castro, E., Tay, J. K., Simonov, M., Laine, L., Batra, R., & Krishnaswamy, S. (2021) · Scientific Reports · doi:10.1038/s41598-021-88226-3 · Cited by 33
A scalable approach for developing clinical risk prediction applications in different hospitals
Sun, H., Depraetere, K., Meesseman, L., Roo, J. D., Vanbiervliet, M., Baerdemaeker, J. D., Muys, H., von Dossow, V., Hulde, N., & Szymanowsky, R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103783 · Cited by 33
Mathis, M. R., Steffner, K. R., Subramanian, H., Gill, G., Girardi, N. I., Bansal, S., Bartels, K., Khanna, A. K., & Huang, J. (2024) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2024.02.004 · Cited by 33
Fong, A., Adams, K., Gaunt, M. J., Howe, J. L., Kellogg, K. M., & Ratwani, R. M. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.09.007 · Cited by 33
Forecasting Mortality Risk for Patients Admitted to Intensive Care Units Using Machine Learning
Darabi, H. R., Tsinis, D., Zecchini, K., Whitcomb, W. F., & Liss, A. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.10.313 · Cited by 33
Plate, J. D. J., van de Leur, R. R., Leenen, L. P. H., Hietbrink, F., Peelen, L. M., & Eijkemans, M. J. C. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0847-0 · Cited by 33
Three controversies in health data science
Peek, N. B., & Rodrigues, P. P. (2018) · International Journal of Data Science and Analytics · doi:10.1007/s41060-018-0109-y · Cited by 32
A deep learning approach for length of stay prediction in clinical settings from medical records
Zebin, T., Rezvy, S., & Chaussalet, T. (2019) · IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) · doi:10.1109/cibcb.2019.8791477 · Cited by 32
Al-Shwaheen, T. I., Hau, Y. W., Ass'Ad, N., & AbuAlSamen, M. M. (2020) · IEEE Access · doi:10.1109/access.2020.3047186 · Cited by 32
Modelling of longitudinal data to predict cardiovascular disease risk: a methodological review
Stevens, D., Lane, D. A., Harrison, S. L., Lip, G. Y. H., & Kolamunnage‐Dona, R. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01472-x · Cited by 32
Predicting mortality over different time horizons: which data elements are needed?
Goldstein, B. A., Pencina, M. J., Montez‐Rath, M. E., & Winkelmayer, W. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw057 · Cited by 32
Self-citationSchofield, I., Brodbelt, D. C., Niessen, S., Church, D. B., Geddes, R. F., Kennedy, N., & O'Neill, D. G. (2020) · Journal of Veterinary Internal Medicine · doi:10.1111/jvim.15851 · Cited by 31
Liang, C., Qiao, S., Olatosi, B., Lyu, T., & Li, X. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104558 · Cited by 31
Chen, Y., Hao, L., Zou, V. Z., Hollander, Z., Ng, R. T., & Isaac, K. V. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01583-z · Cited by 31
Nguyen, H., Vasconcellos, H. D. D., Keck, K., Reis, J. P., Lewis, C. E., Sidney, S., Lloyd‐Jones, D. M., Schreiner, P. J., Güallar, E., Wu, C. O., et al. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-01845-4 · Cited by 31
Wang, X., Li, L., Wang, L., Feng, W., & Zhang, P. (2019) · Statistics in Medicine · doi:10.1002/sim.8457 · Cited by 31
Machine learning and medicine: book review and commentary
Koprowski, R., & Foster, K. R. (2018) · BioMedical Engineering OnLine · doi:10.1186/s12938-018-0449-9 · Cited by 31
Cho, I., Boo, E.-H., Chung, E., Bates, D. W., & Dykes, P. C. (2018) · Journal of Medical Internet Research · doi:10.2196/11505 · Cited by 30
Considerations for Drug Dosing in Premature Infants
van den Anker, J. N., & Allegaert, K. (2021) · The Journal of Clinical Pharmacology · doi:10.1002/jcph.1884 · Cited by 30
Curating a longitudinal research resource using linked primary care EHR data—a UK Biobank case study
Darke, P., Cassidy, S., Catt, M., Taylor, R., Missier, P., & Bacardit, J. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab260 · Cited by 30
Ho, N.-H., Yang, H.-J., Kim, J., Dao, D.-P., Park, H.-R., & Pant, S. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.03.016 · Cited by 30
Jung, K., Sudat, S. E. K., Kwon, N., Stewart, W. F., & Shah, N. H. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103115 · Cited by 30
Predicting outcomes of psychotherapy for depression with electronic health record data
Coley, R. Y., Boggs, J. M., Beck, A., & Simon, G. E. (2021) · Journal of Affective Disorders Reports · doi:10.1016/j.jadr.2021.100198 · Cited by 29
“While there is reasonable apprehension about the adequacy of EHR data for some research questions, concerns about data quality are unlikely to explain our findings considering the success of suicide prediction models using similar data (Goldstein et al., 2017).”
ResultA Deep Learning Model to Predict Knee Osteoarthritis Based on Nonimage Longitudinal Medical Record
Ningrum, D. N. A., Kung, W., Tzeng, I., Yuan, S., Wu, C.-C., Huang, C.-Y., Muhtar, M. S., Nguyen, P., Li, Y., & Wang, Y. (2021) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s325179 · Cited by 29
Liang, Y., Xie, Y., Du, M., Shi, J. C., Yang, J.-F., & Wang, H. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.731445 · Cited by 29
Filipow, N., Main, E., Sebire, N. J., Booth, J., Taylor, A. M., Davies, G., & Stanojevic, S. (2022) · BMJ Open Respiratory Research · doi:10.1136/bmjresp-2021-001165 · Cited by 29
Mora, D., Mateo, J., Nieto, J. A., Bikdeli, B., Yamashita, Y., Barco, S., Jiménez, D., Rodríguez, P. D., Rosa, V., Yoo, H. H. B., et al. (2023) · British Journal of Haematology · doi:10.1111/bjh.18737 · Cited by 29
Viippola, E., Kuitunen, S., Rodosthenous, R. S., Vabalas, A., Hartonen, T., Vartiainen, P., Demmler, J. C., Vuorinen, A., Liu, A., Havulinna, A. S., et al. (2023) · International Journal of Epidemiology · doi:10.1093/ije/dyad091 · Cited by 29
Goodwin, T. R., & Demner‐Fushman, D. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa004 · Cited by 29
Sylvain, M. H., Nyabyenda, E. C., Uwase, M., Komezusenge, I., Ndikumana, F., & Ngaruye, I. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02921-z · Cited by 28
Gong, W., Pu, Y., Ning, T., Zhu, Y., Mu, G., & Li, J. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1591832 · Cited by 28
Quantifying the impact of addressing data challenges in prediction of length of stay
Naemi, A., Schmidt, T., Mansourvar, M., Ebrahimi, A., & Wiil, U. K. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01660-1 · Cited by 28
Reps, J., Ryan, P. B., & Rijnbeek, P. R. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-050146 · Cited by 28
Coombs, L., Orlando, A., Wang, X., Shaw, P., Rich, A. S., Lakhtakia, S., Titchener, K., Adamson, B., Miksad, R. A., & Mooney, K. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00660-3 · Cited by 27
Ruan, X., Li, Y., Jin, X., Deng, P., Xu, J., Li, N., Li, X., Liu, Y., Hu, Y., Xie, J., et al. (2021) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2021.100110 · Cited by 27
The power of big data mining to improve the health care system in the United Arab Emirates
Alhajaj, K. E., & Moonesar, I. A. (2023) · Journal Of Big Data · doi:10.1186/s40537-022-00681-5 · Cited by 27
Tsiampalis, T., & Panagiotakos, D. B. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-02004-5 · Cited by 27
Analysing an imbalanced stroke prediction dataset using machine learning techniques
Elangovan, V. S., Rajeswari, D., Khalaf, O. I., Sharif, M. S., & Elmedany, W. M. (2024) · Karbala International Journal of Modern Science · doi:10.33640/2405-609x.3355 · Cited by 27
Artificial Intelligence and Data Mining Methods for Cardiovascular Risk Prediction
Georga, E. I., Tachos, N. S., Sakellarios, A. I., Kigka, V. I., Exarchos, T. P., Pelosi, G., Parodi, O., Michalis, L. K., & Fotiadis, D. I. (2019) · Series in bioengineering · doi:10.1007/978-981-10-5092-3_14 · Cited by 27
Gupta, A., Liu, T., & Crick, C. (2020) · Statistical Methods in Medical Research · doi:10.1177/0962280220929045 · Cited by 27
Fitzgerald, O., Perez‐Concha, O., Gallego, B., Saxena, M. K., Rudd, L., Metke‐Jimenez, A., & Jorm, L. R. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab060 · Cited by 27
Walsh, C. G., McKillop, M. M., Lee, P., Harris, J. W., Simpson, C., & Novak, L. L. (2021) · JAMIA Open · doi:10.1093/jamiaopen/ooab092 · Cited by 26
Health system utilization before age 1 among children later diagnosed with autism or ADHD
Engelhard, M., Berchuck, S. I., Garg, J., Henao, R., Olson, A., Rusincovitch, S. A., Dawson, G., & Kollins, S. H. (2020) · Scientific Reports · doi:10.1038/s41598-020-74458-2 · Cited by 26
Christopherson, K. M., Das, P., Berlind, C. G., Lindsay, W. D., Ahern, C. A., Smith, B. D., Subbiah, I. M., Koay, E. J., Koong, A. C., Holliday, E. B., et al. (2021) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2021.04.019 · Cited by 26
Withrow, D. R., Shine, B. S. F., Oke, J. L., Tamm, A., James, T. J., Morris, E. J. A., Davies, J., Harris, S., East, J. E., & Nicholson, B. D. (2022) · BMC Medicine · doi:10.1186/s12916-022-02272-w · Cited by 26
John, L. H., Kors, J. A., Reps, J., Ryan, P. B., & Rijnbeek, P. R. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104762 · Cited by 26
Navigating the machine learning pipeline: a scoping review of inpatient delirium prediction models
Strating, T., Hanjani, L. S., Tornvall, I., Hubbard, R. E., & Scott, I. A. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100767 · Cited by 26
Interpretable Disease Progression Prediction Based on Reinforcement Reasoning Over a Knowledge Graph
Sun, Z., Dong, W., Shi, J., & Huang, Z. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3331847 · Cited by 26
Vogelsang, R. P., Bojesen, R. D., Hölmich, E. R., Orhan, A., Buzquurz, F., Cai, L., Grube, C., Zahid, J. A., Allakhverdiiev, E., Raskov, H., et al. (2021) · BJS Open · doi:10.1093/bjsopen/zrab023 · Cited by 26
Ding, R., Deng, M., Wei, H., Zhang, Y., Wei, L., Jiang, G., Zhu, H., Huang, X., Fu, H., Zhao, S., et al. (2024) · CNS Neuroscience & Therapeutics · doi:10.1111/cns.14848 · Cited by 25
Miao, Z., Sealey, M. D., Sathyanarayanan, S., Delen, D., Zhu, L., & Shepherd, S. (2022) · Information Systems · doi:10.1016/j.is.2022.102130 · Cited by 25
Lee, S. J., Lee, S., Choi, H., Lee, J., Jeong, Y. W., Kang, D. R., & Sung, K. (2022) · Journal of Clinical Medicine · doi:10.3390/jcm11226677 · Cited by 25
Jauk, S., Kramer, D., Veeranki, S. P. K., Siml-Fraissler, A., Lenz-Waldbauer, A., Tax, E., Leodolter, W., & Gugatschka, M. (2023) · Dysphagia · doi:10.1007/s00455-022-10548-9 · Cited by 25
Palmowski, L., Nowak, H., Witowski, A., Koos, B., Wolf, A., Weber, M., Kleefisch, D., Unterberg, M., Haberl, H., von Busch, A., et al. (2024) · PLoS ONE · doi:10.1371/journal.pone.0300739 · Cited by 25
Zhang, L., Ding, X., Ma, Y., Muthu, N., Ajmal, I., Moore, J. H., Herman, D. S., & Chen, J. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz170 · Cited by 25
Transferring Clinical Prediction Models Across Hospitals and Electronic Health Record Systems
Curth, A., Thoral, P., van den Wildenberg, W., Bijlstra, P. J., de Bruin, D., Elbers, P., & Fornasa, M. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43823-4_48 · Cited by 25
Heins, M. J., Korevaar, J. C., Schellevis, F., & Rijken, M. (2020) · European Journal of General Practice · doi:10.1080/13814788.2020.1854719 · Cited by 25
Machine Learning Prediction of Death in Critically Ill Patients With Coronavirus Disease 2019
Churpek, M. M., Gupta, S., Spicer, A. B., Hayek, S. S., Srivastava, A., Chan, L., Melamed, M. L., Brenner, S. K., Radbel, J. M., MADHANI-LOVELY, F. K., et al. (2021) · Critical Care Explorations · doi:10.1097/cce.0000000000000515 · Cited by 25
Zhao, Y., Jia, L., Jia, R., Han, H., Feng, C., Li, X., Wei, Z., Wang, H., Zhang, H., Pan, S., et al. (2021) · Shock · doi:10.1097/shk.0000000000001842 · Cited by 25
Predicting the risk of cancer in adults using supervised machine learning: a scoping review
Alfayez, A. A., Kunz, H., & Lai, A. G. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-047755 · Cited by 25
Implementing Cardiovascular Risk Prediction in Clinical Practice: The Future Is Now
Karmali, K. N., & Lloyd‐Jones, D. M. (2017) · Journal of the American Heart Association · doi:10.1161/jaha.117.006019 · Cited by 25
Estimating real-world performance of a predictive model: a case-study in predicting mortality
Major, V. J., Jethani, N., & Aphinyanaphongs, Y. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa008 · Cited by 24
Machine Learning-Based Wearable Devices for Smart Healthcare Application With Risk Factor Monitoring
Alex, S. A., Ponkamali, S., Andrew, T. R., Jhanjhi, N. Z., & Tayyab, M. A. (2022) · Advances in logistics, operations, and management science book series · doi:10.4018/978-1-7998-9201-4.ch009 · Cited by 24
Is it all about you or your driving? Designing IoT‐enabled risk assessments
Ho, Y., Liu, S., Pu, J., & Zhang, D. (2022) · Production and Operations Management · doi:10.1111/poms.13816 · Cited by 24
The landscape of AI implementation in US hospitals
Hwang, Y.-M., Ng, M. Y., Pillai, M., Sahai, M. P., & Hernandez-Boussard, T. (2026) · Nature Health · doi:10.1038/s44360-025-00016-7 · Cited by 23
Use of machine learning in pediatric surgical clinical prediction tools: A systematic review
Bianco, A., Al‐Azzawi, Z. A. M., Guadagno, E., Osmanlliu, E., Gravel, J., & Poenaru, D. (2023) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2023.01.020 · Cited by 23
Cerebral Palsy classification based on multi-feature analysis using machine learning
Al-Sowi, A. M., Almasri, N. A., HAMMO, B. H., & Al-Qwaqzeh, F. A.-Z. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101197 · Cited by 23
Participant flow diagrams for health equity in AI
Ellen, J., de Matos, J. C. R. G., Viola, M., Gallifant, J., Quion, J., Celi, L. A., & Hussein, N. S. A. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104631 · Cited by 23
Bergquist, T., Yan, Y., Schaffter, T., Yu, T. V., Pejaver, V. R., Hammarlund, N. E., Prosser, J., Guinney, J., & Mooney, S. D. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa083 · Cited by 23
Dormosh, N., Schut, M. C., Heymans, M. W., Maarsingh, O. R., Bouman, J., van der Velde, N., & Abu–Hanna, A. (2023) · Age and Ageing · doi:10.1093/ageing/afad046 · Cited by 22
González‐Juanatey, C., Anguita-Sánchez, M. P., Barrios, V., Nuñez-Gil, I. J., Gómez‐Doblas, J. J., García–Moll, X., Gormaz, C. L., Rollán-Gómez, M. J., Peral-Disdier, V., Martínez‐Dolz, L., et al. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12165218 · Cited by 22
Metadata
- Authors
- Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A.
- Affiliations
- Duke University (US)
- Center for Predictive Medicine, Duke Clinical Research Institute, Duke University, Durham, NC 27710, USA
- Stanford University (US)
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-05-17
- Volume
- 24
- Issue
- 1
- Pages
- 198-208
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC5201180
- OpenAlex
- W2395172628
- Cited by
- 936
- References
- 144
- Citations by year
- 2026: 132 · 2025: 115 · 2024: 97 · 2023: 98 · 2022: 107 · 2021: 122 · 2020: 100 · 2019: 82 · 2018: 59 · 2017: 15 · 2016: 6
- FWCI
- 9.07
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Funders
- National Institute of Diabetes and Digestive and Kidney Diseases (K25DK097279)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Keywords
- electronic health records, clinical risk prediction, risk prediction model, model validation, multicenter validation, longitudinal data, missing data, loss to follow-up, model reproducibility, concordance index, mortality prediction, hospitalization prediction, service utilization
- MeSH terms
- Humans, Models, Statistical, Risk Assessment, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.