Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Objective: To develop a conceptual prediction model framework containing standardized steps and describe the corresponding open-source software developed to consistently implement the framework across computational environments and observational healthcare databases to enable model sharing and reproducibility. Methods: Based on existing best practices we propose a 5 step standardized framework for: (1) transparently defining the problem; (2) selecting suitable datasets; (3) constructing variables from the observational data; (4) learning the predictive model; and (5) validating the model performance. We implemented this framework as open-source software utilizing the Observational Medical Outcomes Partnership Common Data Model to enable convenient sharing of models and reproduction of model evaluation across multiple observational datasets. The software implementation contains default covariates and classifiers but the framework enables customization and extension. Results: As a proof-of-concept, demonstrating the transparency and ease of model dissemination using the software, we developed prediction models for 21 different outcomes within a target population of people suffering from depression across 4 observational databases. All 84 models are available in an accessible online repository to be implemented by anyone with access to an observational database in the Common Data Model format. Conclusions: The proof-of-concept study illustrates the framework’s ability to develop reproducible models that can be readily shared and offers the potential to perform extensive external validation of models, and improve their likelihood of clinical uptake. In future work the framework will be applied to perform an “all-by-all” prediction analysis to assess the observational data prediction domain across numerous target populations, outcomes and time, and risk settings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“This also ensures reproducibility of the results.”
In the citing paper
“The Observational Health Data Sciences and Informatics (OHDSI) network has developed standards and tools that allow patientlevel prediction models to be developed and externally validated at a large scale in a transparent way following accepted best practices [87, 88].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. (2015) · BMC Medicine · doi:10.1186/s12916-014-0241-z · Cited by 1,927
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,542
Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · cited 1×
“For example, the prognostic research strategy consists of 4 papers stating a variety of best practices aimed at general considerations, 6 prognostic factor discovery, 7 model development, 8 and making clinical impacts by implementing the model to enable stratified medicine.”
BackgroundCan machine-learning improve cardiovascular risk prediction using routine clinical data?
Weng, S. F., Reps, J., Kai, J., Garibaldi, J. M., & Qureshi, N. (2017) · PLOS ONE · doi:10.1371/journal.pone.0174944 · Cited by 1,346
Self-citationSchnabel, R. B., Sullivan, L. M., Levy, D., Pencina, M. J., Massaro, J. M., D'Agostino, R. B., Newton-Cheh, C., Yamamoto, J. F., Magnani, J. W., Tadros, T. M., et al. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60443-8 · Cited by 1,041
Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936
Prognosis Research Strategy (PROGRESS) 2: Prognostic Factor Research
Riley, R. D., Hayden, J. A., Steyerberg, E. W., Moons, K. G. M., Abrams, K., Kyzas, P. A., Malats, N., Briggs, A., Schroter, S., Altman, D. G., et al. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001380 · Cited by 823
Collins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729 · cited 1×
“Many published efforts in patient-level-prediction do not follow the model development guidelines, 1,2 fail to perform extensive external validation, 3,4 or provide insufficient model details 5 that limits the ability of independent researchers to reproduce the models and perform external…”
BackgroundPrognosis research strategy (PROGRESS) 1: A framework for researching clinical outcomes
Hemingway, H., Croft, P., Perel, P., Hayden, J. A., Abrams, K., Timmis, A., Briggs, A., Udumyan, R., Moons, K. G. M., Steyerberg, E. W., et al. (2013) · BMJ · doi:10.1136/bmj.e5595 · Cited by 663
Onukwugha, E., Yong, C., Hussain, A., Seal, B., & Mullins, C. D. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-1 · Cited by 589
Research works printed but not linked: 1
- Schuemie, 2017, SqlRender: Rendering Parameterized SQL and Translation to Dialects. R Package Version 1.2.0
Cited by
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Self-citationThe National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment
Haendel, M. A., Chute, C. G., Bennett, T. D., Eichmann, D. A., Guinney, J., Kibbe, W. A., Payne, P. R. O., Pfaff, E. R., Robinson, P. N., Saltz, J. H., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa196 · Cited by 553
Thorsen-Meyer, H.-C., Nielsen, A. B., Nielsen, A. P., Kaas-Hansen, B. S., Toft, P., Schierbeck, J., Strøm, T., Chmura, P. J., Heimann, M., Dybdahl, L., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30018-2 · Cited by 336
Predictive analytics in health care: how can we know it works?
Van Calster, B., Wynants, L., Timmerman, D., Steyerberg, E. W., & Collins, G. S. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz130 · Cited by 223
Big data hurdles in precision medicine and precision public health
Prosperi, M., Min, J. S., Bian, J., & Modave, F. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0719-2 · Cited by 200
“…that leverages OHDSI and OMOP not only for “ transparently defining the problem and selecting suitable datasets, ” (i.e. semantics) but also for “ constructing variables from the observational data, learning the predictive model, and validating the model performance ” (i.e. modelling) [106].”
Background
Metadata
- Authors
- Reps, J. M., Schuemie, M. J., Suchard, M. A., Ryan, P. B., & Rijnbeek, P. R.
- Affiliations
- Janssen Research & Development (United States) (US)
- University of California, Los Angeles (US)
- Erasmus MC (NL)
- Corresponding author
- Reps, Jenna M
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-04-27
- Volume
- 25
- Issue
- 8
- Pages
- 969-975
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC6077830
- OpenAlex
- W2799695199
- Cited by
- 205
- References
- 19
- Citations by year
- 2026: 22 · 2025: 18 · 2024: 32 · 2023: 19 · 2022: 41 · 2021: 26 · 2020: 22 · 2019: 21 · 2018: 4
- FWCI
- 10.28
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- eScholarship (California Digital Library) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by-nc)
- Funders
- National Science Foundation (1251151)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- model reproducibility, model sharing, depression, open source software, model validation, external validation
- MeSH terms
- Machine Learning, Adult, Female, Health Services Needs and Demand, Humans, Male, Models, Theoretical, Prognosis, Software, Treatment Outcome, Risk Assessment, Observation, Observational Studies as Topic, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.