Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review
Open accessThe DOI matches its Crossref record; 141 of the 144 references it cites were found in a registry or an index.
Abstract
Objective: Electronic health records (EHRs) are an increasingly common data source for clinical risk prediction, presenting both unique analytic opportunities and challenges. We sought to evaluate the current state of EHR based risk prediction modeling through a systematic review of clinical prediction studies using EHR data. Methods: We searched PubMed for articles that reported on the use of an EHR to develop a risk prediction model from 2009 to 2014. Articles were extracted by two reviewers, and we abstracted information on study design, use of EHR data, model building, and performance from each publication and supplementary documentation. Results: We identified 107 articles from 15 different countries. Studies were generally very large (median sample size = 26 100) and utilized a diverse array of predictors. Most used validation techniques (n = 94 of 107) and reported model coefficients for reproducibility (n = 83). However, studies did not fully leverage the breadth of EHR data, as they uncommonly used longitudinal information (n = 37) and employed relatively few predictor variables (median = 27 variables). Less than half of the studies were multicenter (n = 50) and only 26 performed validation across sites. Many studies did not fully address biases of EHR data such as missing data or loss to follow-up. Average c-statistics for different outcomes were: mortality (0.84), clinical prediction (0.83), hospitalization (0.71), and service utilization (0.71). Conclusions: EHR data present both opportunities and challenges for clinical risk prediction. There is room for improvement in designing such studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In spite of the richness and potential of available data, scaling the development of predictive models is difficult because, for traditional predictive modeling techniques, each outcome to be predicted requires the creation of a custom dataset with specific variables.7”
From Scalable and accurate deep learning with electronic health records (2018) · cites this paper as [7]
In the citing paper
“Despite the rich data now digitized in EHRs,20 a recent systematic review of the medical literature7 found that predictive models built with EHR data use a median of only 27 variables, rely on traditional generalized linear models, and are built using data at a single center.”
From Scalable and accurate deep learning with electronic health records (2018) · cites this paper as [7]
In the citing paper
“Although external validation is recognized as an area of improvement for clinical risk prediction models [85]”
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Self-citationPrediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581 · cited 2×
“Traditionally, risk pre- diction algorithms have been developed from large cohort studies such as the Framingham Heart Study.3 These studies were designed to follow people for years or even decades.”
MethodMoons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Joint modelling of longitudinal measurements and event time data
Henderson, R. (2000) · Biostatistics · doi:10.1093/biostatistics/1.4.465 · Cited by 913
The Electronic Medical Records and Genomics (eMERGE) Network: past, present, and future
Gottesman, O., Kuivaniemi, H., Tromp, G., Faucett, W. A., Li, R., Manolio, T. A., Sanderson, S. C., Kannry, J., Zinberg, R., Basford, M. A., et al. (2013) · Genetics in Medicine · doi:10.1038/gim.2013.72 · Cited by 698
Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve
Cook, N. R. (2008) · Clinical Chemistry · doi:10.1373/clinchem.2007.096529 · Cited by 691
Research works printed but not linked: 1
- Liu, 2010, Length of stay predictions: improvements through the use of automated laboratory and comorbidity variables, Med Care., 48, 739, 10.1097/mlr.0b013e3181e359f3 · DOI · Paper page
Cited by
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648
“In spite of the richness and potential of available data, scaling the development of predictive models is difficult because, for traditional predictive modeling techniques, each outcome to be predicted requires the creation of a custom dataset with specific variables [17].”
BackgroundInfluentialPotential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395
“A 2017 review 13 found that only 54% of studies that generated prediction algorithms based on the electronic health record accountedformissingdata.”
BackgroundClinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Khan, S. S., Coresh, J., Pencina, M. J., Ndumele, C. E., Rangaswami, J., Chow, S. L., Palaniappan, L. P., Sperling, L. S., Virani, S. S., Ho, J. E., et al. (2023) · Circulation · doi:10.1161/cir.0000000000001191 · Cited by 797
Self-citation
Metadata
- Authors
- Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A.
- Affiliations
- Duke University (US)
- Center for Predictive Medicine, Duke Clinical Research Institute, Duke University, Durham, NC 27710, USA
- Stanford University (US)
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-05-17
- Volume
- 24
- Issue
- 1
- Pages
- 198-208
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC5201180
- OpenAlex
- W2395172628
- Cited by
- 936
- References
- 144
- Citations by year
- 2026: 132 · 2025: 115 · 2024: 97 · 2023: 98 · 2022: 107 · 2021: 122 · 2020: 100 · 2019: 82 · 2018: 59 · 2017: 15 · 2016: 6
- FWCI
- 9.07
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Funders
- National Institute of Diabetes and Digestive and Kidney Diseases (K25DK097279)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Keywords
- electronic health records, clinical risk prediction, risk prediction model, model validation, multicenter validation, longitudinal data, missing data, loss to follow-up, model reproducibility, concordance index, mortality prediction, hospitalization prediction, service utilization
- MeSH terms
- Humans, Models, Statistical, Risk Assessment, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. https://app.cochat.ai/papers/10.1093/jamia/ocw042
- The paper
- Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017). Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocw042
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.