CoChat

Feasibility and evaluation of a large-scale external validation approach for patient-level prediction in an international data network: validation of models predicting stroke in female patients newly diagnosed with atrial fibrillation

Open access

Published version of a preprint

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Abstract

Background: To demonstrate how the Observational Healthcare Data Science and Informatics (OHDSI) collaborative network and standardization can be utilized to scale-up external validation of patient-level prediction models by enabling validation across a large number of heterogeneous observational healthcare datasets. Methods: Five previously published prognostic models (ATRIA, CHADS 2 , CHADS 2 VASC, Q-Stroke and Framingham) that predict future risk of stroke in patients with atrial fibrillation were replicated using the OHDSI frameworks. A network study was run that enabled the five models to be externally validated across nine observational healthcare datasets spanning three countries and five independent sites. Results: The five existing models were able to be integrated into the OHDSI framework for patient-level prediction and they obtained mean c-statistics ranging between 0.57–0.63 across the 6 databases with sufficient data to predict stroke within 1 year of initial atrial fibrillation diagnosis for females with atrial fibrillation. This was comparable with existing validation studies. The validation network study was run across nine datasets within 60 days once the models were replicated. An R package for the study was published at https://github.com/OHDSI/StudyProtocolSandbox/tree/master/ExistingStrokeRiskExternalValidation . Conclusion: This study demonstrates the ability to scale up external validation of patient-level prediction models using a collaboration of researchers and a data standardization that enable models to be readily shared across data sites. External validation is necessary to understand the transportability or reproducibility of a prediction model, but without collaborative approaches it can take three or more years for a model to be validated by one independent researcher. In this paper we show it is possible to both scale-up and speed-up external validation by showing how validation can be done across multiple databases in less than 2 months. We recommend that researchers developing new prediction models use the OHDSI network to externally validate their models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12874-020-00991-3

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 17 references

Cited by

See all 48

Metadata

Authors
Reps, J. M., Williams, R. D., You, S. C., Falconer, T., Minty, E., Callahan, A., Ryan, P. B., Park, R. W., Lim, H.-S., & Rijnbeek, P.
Affiliations
  • Erasmus MC (NL)
  • Ajou University (KR)
  • Columbia University Irving Medical Center (US)
  • University of Calgary (CA)
  • Stanford Medicine (US)
  • Ajou University Hospital (KR)
Corresponding author
Reps, Jenna M.
Venue
BMC Medical Research Methodology
Published
2020-05-06
Volume
20
Issue
1
Pages
102-102
ISSN
1471-2288 (ISSN-L)
OpenAlex
W3021150726
Cited by
40
References
17
Citations by year
2026: 3 · 2025: 6 · 2024: 10 · 2023: 2 · 2022: 11 · 2021: 6 · 2020: 2
FWCI
3.40
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
  • AJOU Open Repository (Ajou University) (submittedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Libraries and Cultural Resources (University of Calgary) (submittedVersion)
  • RePub (Erasmus University Rotterdam) (submittedVersion)
  • Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
Funders
  • Health Promotion Administration, Ministry of Health and Welfare (HI16C0992)
  • Innovative Medicines Initiative Joint Undertaking (806968)
Topics
  • Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Atrial Fibrillation Management and Outcomes
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
stroke risk prediction, atrial fibrillation, female patients, external validation, OHDSI, data standardization, model replication, model portability, Framingham Stroke Risk Score
MeSH terms
Atrial Fibrillation, Feasibility Studies, Female, Humans, Prognosis, Reproducibility of Results, Stroke

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Feasibility and evaluation of a large-scale external validation approach for patient-level prediction in an international data network: validation of models predicting stroke in female patients newly diagnosed with atrial fibrillation. https://app.cochat.ai/papers/10.1186/s12874-020-00991-3
The paper
Reps, J. M., Williams, R. D., You, S. C., Falconer, T., Minty, E., Callahan, A., Ryan, P. B., Park, R. W., Lim, H.-S., & Rijnbeek, P. (2020). Feasibility and evaluation of a large-scale external validation approach for patient-level prediction in an international data network: validation of models predicting stroke in female patients newly diagnosed with atrial fibrillation. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-020-00991-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.