Feasibility and evaluation of a large-scale external validation approach for patient-level prediction in an international data network: validation of models predicting stroke in female patients newly diagnosed with atrial fibrillation
Open accessPublished version of a preprint
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Background: To demonstrate how the Observational Healthcare Data Science and Informatics (OHDSI) collaborative network and standardization can be utilized to scale-up external validation of patient-level prediction models by enabling validation across a large number of heterogeneous observational healthcare datasets. Methods: Five previously published prognostic models (ATRIA, CHADS 2 , CHADS 2 VASC, Q-Stroke and Framingham) that predict future risk of stroke in patients with atrial fibrillation were replicated using the OHDSI frameworks. A network study was run that enabled the five models to be externally validated across nine observational healthcare datasets spanning three countries and five independent sites. Results: The five existing models were able to be integrated into the OHDSI framework for patient-level prediction and they obtained mean c-statistics ranging between 0.57–0.63 across the 6 databases with sufficient data to predict stroke within 1 year of initial atrial fibrillation diagnosis for females with atrial fibrillation. This was comparable with existing validation studies. The validation network study was run across nine datasets within 60 days once the models were replicated. An R package for the study was published at https://github.com/OHDSI/StudyProtocolSandbox/tree/master/ExistingStrokeRiskExternalValidation . Conclusion: This study demonstrates the ability to scale up external validation of patient-level prediction models using a collaboration of researchers and a data standardization that enable models to be readily shared across data sites. External validation is necessary to understand the transportability or reproducibility of a prediction model, but without collaborative approaches it can take three or more years for a model to be validated by one independent researcher. In this paper we show it is possible to both scale-up and speed-up external validation by showing how validation can be done across multiple databases in less than 2 months. We recommend that researchers developing new prediction models use the OHDSI network to externally validate their models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This also ensures reproducibility of the results.”
From The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies (2021) · cites this paper as [88]
Experimental — accuracy not yet measuredIn the citing paper
“The Observational Health Data Sciences and Informatics (OHDSI) network has developed standards and tools that allow patientlevel prediction models to be developed and externally validated at a large scale in a transparent way following accepted best practices [87, 88].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lip, G. Y. H., Nieuwlaat, R., Pisters, R., Lane, D. A., & Crijns, H. J. G. M. (2010) · Chest · doi:10.1378/chest.09-1584 · Cited by 6,879 · cited 2×
“To demonstrate this we selected the prediction problem of 1-year risk of stroke in newly diagnosed atrial fibrillation patients as there are multiple existing models that are used clinically, namely Anticoagulation and Risk Factors in Atrial Fibrillation (ATRIA) (no prior stroke model) [8], Framingham (no prior stroke…”
MethodGage, B. F., Waterman, A. D., Shannon, W., Boechler, M., Rich, M. W., & Radford, M. J. (2001) · JAMA · doi:10.1001/jama.285.22.2864 · Cited by 4,832 · cited 11×
“The CHADS2 score obtained a c-statistic of 0.81 on this population.”
MethodInfluentialPredicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652 · cited 2×
“There has been increased interest in learning patient-level prediction models using these big realworld datasets with the aim of improving healthcare [2].”
BackgroundGeorge, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,542 · cited 1×
“community of researchers that are working towards the common goal of improving the analysis of observational data [6].”
BackgroundSelf-citationPrognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Wang, T. J., Massaro, J. M., Levy, D., Vasan, R. S., Wolf, P. A., D'Agostino, R. B., Larson, M. G., Kannel, W. B., & Benjamin, E. J. (2003) · JAMA · doi:10.1001/jama.290.8.1049 · Cited by 780 · cited 2×
“To demonstrate this we selected the prediction problem of 1-year risk of stroke in newly diagnosed atrial fibrillation patients as there are multiple existing models that are used clinically, namely Anticoagulation and Risk Factors in Atrial Fibrillation (ATRIA) (no prior stroke model) [8], Framingham (no prior stroke…”
MethodPrimary prevention of cardiovascular disease: A review of contemporary guidance and literature
Stewart, J., Manmathan, G., & Wilkinson, P. (2017) · JRSM Cardiovascular Disease · doi:10.1177/2048004016687211 · Cited by 464 · cited 1×
“For example, the cardiovascular model QRISK2, that was developed using a UK primary care database, is used to identify patients who may benefit from lipid -lowering medication [3].”
MethodSiontis, G. C. M., Tzoulaki, I., Castaldi, P. J., & Ioannidis, J. P. A. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.007 · Cited by 459 · cited 3×
“Few published patient-level prediction models are externally validated, and research has shown that it often takes three or more years for external validation to occur once a model is published [1].”
BackgroundMethodInfluentialSinger, D. E., Chang, Y., Borowsky, L. H., Fang, M. C., Pomernacki, N. K., Udaltsova, N., Reynolds, K., & Go, A. S. (2013) · Journal of the American Heart Association · doi:10.1161/jaha.113.000250 · Cited by 405 · cited 11×
“The existing models we replicated were ATRIA, CHADS2, CHA2DS2 −VASc, Framingham and Q-Stroke.”
MethodInfluentialVAN STAA, T. P., SETAKIS, E., DI TANNA, G. L., LANE, D. A., & LIP, G. Y. H. (2011) · Journal of Thrombosis and Haemostasis · doi:10.1111/j.1538-7836.2010.04085.x · Cited by 272
Cited by
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Self-citationYang, C. B., Fridgeirsson, E. A., Kors, J. A., Reps, J., & Rijnbeek, P. R. (2024) · Journal Of Big Data · doi:10.1186/s40537-023-00857-7 · Cited by 145
“…the classifier (lasso logistic regression, random forest, XGBoost), the sampling strategy (random over-sampling, random undersampling), and the target IR [1, 2, 9, 20] and applied each combination across 58 prediction tasks (each a combination of a prediction problem and one of the four databases).”
MethodSelf-citationPrivacy-preserving data sharing infrastructures for medical research: systematization and comparison
Wirth, F. N., Meurers, T., Johns, M., & Praßer, F. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01602-x · Cited by 137
Machine Learning and Real-World Data to Predict Lung Cancer Risk in Routine Care
Chandran, U., Reps, J., Yang, R., Vachani, A., Maldonado, F., & Kalsekar, I. (2023) · Cancer Epidemiology, Biomarkers & Prevention · doi:10.1158/1055-9965.epi-22-0873 · Cited by 51
Self-citationvan der Endt, V. H. W., Milders, J., de Vries, B. B. L. P., Trines, S. A. I. P., Groenwold, R. H. H., Dekkers, O. M., Trevisan, M., Carrero, J. J., van Diepen, M., Dekker, F. W., et al. (2022) · EP Europace · doi:10.1093/europace/euac096 · Cited by 61
Metadata
- Authors
- Reps, J. M., Williams, R. D., You, S. C., Falconer, T., Minty, E., Callahan, A., Ryan, P. B., Park, R. W., Lim, H.-S., & Rijnbeek, P.
- Affiliations
- Erasmus MC (NL)
- Ajou University (KR)
- Columbia University Irving Medical Center (US)
- University of Calgary (CA)
- Stanford Medicine (US)
- Ajou University Hospital (KR)
- Corresponding author
- Reps, Jenna M.
- ORCID
- Venue
- BMC Medical Research Methodology
- Published
- 2020-05-06
- Volume
- 20
- Issue
- 1
- Pages
- 102-102
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC7201646
- OpenAlex
- W3021150726
- Cited by
- 40
- References
- 17
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 10 · 2023: 2 · 2022: 11 · 2021: 6 · 2020: 2
- FWCI
- 3.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
- AJOU Open Repository (Ajou University) (submittedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Libraries and Cultural Resources (University of Calgary) (submittedVersion)
- RePub (Erasmus University Rotterdam) (submittedVersion)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- Funders
- Health Promotion Administration, Ministry of Health and Welfare (HI16C0992)
- Innovative Medicines Initiative Joint Undertaking (806968)
- Topics
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Atrial Fibrillation Management and Outcomes
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- stroke risk prediction, atrial fibrillation, female patients, external validation, OHDSI, data standardization, model replication, model portability, Framingham Stroke Risk Score
- MeSH terms
- Atrial Fibrillation, Feasibility Studies, Female, Humans, Prognosis, Reproducibility of Results, Stroke
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Feasibility and evaluation of a large-scale external validation approach for patient-level prediction in an international data network: validation of models predicting stroke in female patients newly diagnosed with atrial fibrillation. https://app.cochat.ai/papers/10.1186/s12874-020-00991-3
- The paper
- Reps, J. M., Williams, R. D., You, S. C., Falconer, T., Minty, E., Callahan, A., Ryan, P. B., Park, R. W., Lim, H.-S., & Rijnbeek, P. (2020). Feasibility and evaluation of a large-scale external validation approach for patient-level prediction in an international data network: validation of models predicting stroke in female patients newly diagnosed with atrial fibrillation. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-020-00991-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.