Assessing Risk Prediction Models Using Individual Participant Data From Multiple Studies
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Individual participant time-to-event data from multiple prospective epidemiologic studies enable detailed investigation into the predictive ability of risk models. Here we address the challenges in appropriately combining such information across studies. Methods are exemplified by analyses of log C-reactive protein and conventional risk factors for coronary heart disease in the Emerging Risk Factors Collaboration, a collation of individual data from multiple prospective studies with an average follow-up duration of 9.8 years (dates varied). We derive risk prediction models using Cox proportional hazards regression analysis stratified by study and obtain estimates of risk discrimination, Harrell's concordance index, and Royston's discrimination measure within each study; we then combine the estimates across studies using a weighted meta-analysis. Various weighting approaches are compared and lead us to recommend using the number of events in each study. We also discuss the calculation of measures of reclassification for multiple studies. We further show that comparison of differences in predictive ability across subgroups should be based only on within-study information and that combining measures of risk discrimination from case-control studies and prospective studies is problematic. The concordance index and discrimination measure gave qualitatively similar results throughout. While the concordance index was very heterogeneous between studies, principally because of differing age ranges, the increments in the concordance index from adding log C-reactive protein to conventional risk factors were more homogeneous.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299
Self-citationEvaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Interpretation of random effects meta-analyses
Riley, R. D., Higgins, J. P. T., & Deeks, J. J. (2011) · BMJ · doi:10.1136/bmj.d549 · Cited by 2,884
Uno, H., Cai, T., Pencina, M. J., D'Agostino, R. B., & Wei, L. J. (2011) · Statistics in Medicine · doi:10.1002/sim.4154 · Cited by 1,535 · cited 1×
“Only informative pairs (where it is possible to determine which person suffered the first event) are used: This introduces some sensitivity to censoring (25), which we ignore because currently available solutions either assume correct model specification (25) or do not accommodate censoring by the end of follow-up (26…”
MethodSelf-citationSteyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491
The Emerging Risk Factors Collaboration (2011) · The Lancet · doi:10.1016/s0140-6736(11)60105-0 · Cited by 1,216
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160
C-reactive protein, fibrinogen, and cardiovascular disease prediction
The Emerging Risk Factors Collaboration (2012) · New England Journal of Medicine · doi:10.1056/nejmoa1107477 · Cited by 1,150
Cited by
Kaptoge, S., Pennells, L., De Bacquer, D., Cooney, M. T., Kavousi, M., Stevens, G., Riley, L. M., Savin, S., Khan, T., Altay, S., et al. (2019) · The Lancet Global Health · doi:10.1016/s2214-109x(19)30318-3 · Cited by 1,167
Self-citationA guide to systematic review and meta-analysis of prognostic factor studies
Riley, R. D., Moons, K. G. M., Snell, K. I. E., Ensor, J., Hooft, L., Altman, D. G., Hayden, J., Collins, G. S., & Debray, T. P. A. (2019) · BMJ · doi:10.1136/bmj.k4597 · Cited by 742
A guide to systematic review and meta-analysis of prediction model performance
Debray, T. P. A., Damen, J. A. A. G., Snell, K. I. E., Ensor, J., Hooft, L., Reitsma, J. B., Riley, R. D., & Moons, K. G. M. (2017) · BMJ · doi:10.1136/bmj.i6460 · Cited by 657
“…help to retrieve unreported performance measures (eg, calibration slope), estimate the within-study correlation between performance measures, 9 avoid continuity corrections and data transformations, further interpret model generalisability, 8 19 22 31 and tailor the model to populations at hand.”
BackgroundMethodRiley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585
“Other issues might also benefit from big datasets, such as comparing (and even combining 78 ) multiple competing models, 79 and examining the added value of a new predictor, 30 for example, in terms of the net benefit for clinical decision making.”
BackgroundMethodInfluentialEvaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R. D., Archer, L., Snell, K. I. E., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L. J., & Collins, G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074820 · Cited by 401
Metadata
- Authors
- Pennells, L., Kaptoge, S., White, I. R., Thompson, S. G., Wood, A. M., Emerging Risk Factors Collaboration, Tipping, R. W., Folsom, A. R., Couper, D. J., Ballantyne, C. M., Coresh, J., Goya Wannamethee, S., Morris, R. W., Kiechl, S., Willeit, J., Willeit, P., Schett, G., Ebrahim, S., Lawlor, D. A., Yarnell, J. W., Gallacher, J., Cushman, M., Psaty, B. M., Tracy, R., Tybjærg-Hansen, A., Price, J. F., Lee, A. J., McLachlan, S., Khaw, K.-T., Wareham, N. J., Brenner, H., Schöttker, B., Müller, H., Jansson, J.-H., Wennberg, P., Salomaa, V., Harald, K., Jousilahti, P., Vartiainen, E., Woodward, M., D'Agostino, R. B., Bladbjerg, E.-M., Jørgensen, T., Kiyohara, Y., Arima, H., Doi, Y., Ninomiya, T., Dekker, J. M., Nijpels, G., Stehouwer, C. D. A., Kauhanen, J., Salonen, J. T., Meade, T. W., Cooper, J. A., Cushman, M., Folsom, A. R., Psaty, B. M., Shea, S., Döring, A., Kuller, L. H., Grandits, G., Gillum, R. F., Mussolino, M., Rimm, E. B., Hankinson, S. E., Manson, J. E., Pai, J. K., Kirkland, S., Shaffer, J. A., Shimbo, D., Bakker, S. J. L., Gansevoort, R. T., Hillege, H. L., Amouyel, P., Arveiler, D., Evans, A., Ferrières, J., Sattar, N., Westendorp, R. G., Buckley, B. M., Cantin, B., Lamarche, B., Barrett-Connor, E., Wingard, D. L., Bettencourt, R., Gudnason, V., Aspelund, T., Sigurdsson, G., Thorsson, B., Kavousi, M., Witteman, J. C., Hofman, A., Franco, O. H., Howard, B. V., Zhang, Y., Best, L., Umans, J. G., Onat, A., Sundström, J., Michael Gaziano, J., Stampfer, M., Ridker, P. M., Michael Gaziano, J., Ridker, P. M., Marmot, M., Clarke, R., Collins, R., Fletcher, A., Brunner, E., Shipley, M., Kivimäki, M., Ridker, P. M., Buring, J., Cook, N., Ford, I., Shepherd, J., Cobbe, S. M., Robertson, M., Walker, M., Watson, S., Alexander, M., Butterworth, A. S., Angelantonio, E. D., Gao, P., Haycock, P., Kaptoge, S., Pennells, L., Thompson, S. G., Walker, M., Watson, S., White, I. R., Wood, A. M., Wormser, D., & Danesh, J.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2013-12-22
- Volume
- 179
- Issue
- 5
- Pages
- 621-632
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- OpenAlex
- W2133848774
- Cited by
- 63
- References
- 12
- FWCI
- 5.16
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessing Risk Prediction Models Using Individual Participant Data From Multiple Studies. https://app.cochat.ai/papers/10.1093/aje/kwt298
- The paper
- Pennells, L., Kaptoge, S., White, I. R., Thompson, S. G., Wood, A. M., Emerging Risk Factors Collaboration, Tipping, R. W., Folsom, A. R., Couper, D. J., Ballantyne, C. M., Coresh, J., Goya Wannamethee, S., Morris, R. W., Kiechl, S., Willeit, J., Willeit, P., Schett, G., Ebrahim, S., Lawlor, D. A., . . . Danesh, J. (2014). Assessing Risk Prediction Models Using Individual Participant Data From Multiple Studies. American Journal of Epidemiology. https://doi.org/10.1093/aje/kwt298
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.