CoChat

Changing predictor measurement procedures affected the performance of prediction models in clinical examples

Open access
Luijken, Kim, Wynants, Laure, van Smeden, Maarten, Van Calster, Ben, Steyerberg, Ewout W., Groenwold, Rolf H.H., Timmerman, Dirk, Bourne, Tom, Ukaegbu, Chinedu · 2020 · Journal of Clinical Epidemiology
Cited by 50

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Abstract

OBJECTIVES: The aim of this study was to quantify the impact of predictor measurement heterogeneity on prediction model performance. Predictor measurement heterogeneity refers to variation in the measurement of predictor(s) between the derivation of a prediction model and its validation or application. It arises, for instance, when predictors are measured using different measurement instruments or protocols. STUDY DESIGN AND SETTING: We examined the effects of various scenarios of predictor measurement heterogeneity in real-world clinical examples using previously developed prediction models for diagnosis of ovarian cancer, mutation carriers for Lynch syndrome, and intrauterine pregnancy. RESULTS: Changing the measurement procedure of a predictor influenced the performance at validation of the prediction models in nine clinical examples. Notably, it induced model miscalibration. The calibration intercept at validation ranged from -0.70 to 1.43 (0 for good calibration), whereas the calibration slope ranged from 0.50 to 1.67 (1 for good calibration). The difference in C-statistic and scaled Brier score between derivation and validation ranged from -0.08 to +0.08 and from -0.40 to +0.16, respectively. CONCLUSION: This study illustrates that predictor measurement heterogeneity can influence the performance of a prediction model substantially, underlining that predictor measurements used in research settings should resemble clinical practice. Specification of measurement heterogeneity can help researchers explaining discrepancies in predictive performance between derivation and validation setting.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2019.11.001

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 8 references

Research works printed but not linked: 27

  1. Altman, 2000, What do we mean by validating a prognostic model?, Stat Med, 19, 453, 10.1002/(SICI)1097-0258(20000229)19:4<453::AID-SIM350>3.0.CO;2-5 · doi:10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.0.co;2-5
  2. Wynants, 2017, Key steps and common pitfalls in developing and validating risk models, BJOG, 124, 423, 10.1111/1471-0528.14170 · DOI
  3. Steyerberg, 2018, Poor performance of clinical prediction models: the harm of commonly applied methods, J Clin Epidemiol, 98, 133, 10.1016/j.jclinepi.2017.11.013 · DOI · Paper page
  4. Toll, 2008, Validation, updating and impact of clinical prediction rules: a review, J Clin Epidemiol, 61, 1085, 10.1016/j.jclinepi.2008.04.008 · DOI · Paper page
  5. Debray, 2013, A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis, Stat Med, 32, 3158, 10.1002/sim.5732 · DOI · Paper page
  6. Khudyakov, 2015, The impact of covariate measurement error on risk prediction, Stat Med, 34, 2353, 10.1002/sim.6498 · DOI
  7. Rosella, 2012, The influence of measurement error on calibration, discrimination, and overall estimation of a risk prediction model, Popul Health Metr, 10, 20, 10.1186/1478-7954-10-20 · DOI
  8. Luijken, 2019, Impact of predictor measurement heterogeneity across settings on the performance of prediction models: a measurement error perspective, Stat Med, 38, 3444, 10.1002/sim.8183 · DOI · Paper page
  9. Pajouheshnia, 2019, How variation in predictor measurement affects the discriminative ability and transportability of a prediction model, J Clin Epidemiol, 105, 136, 10.1016/j.jclinepi.2018.09.001 · DOI
  10. Pajouheshnia, 2019, When and how to use data from randomised trials to develop or validate prognostic models, BMJ, 365, l2154, 10.1136/bmj.l2154 · DOI
  11. Te Velde, 2013, Comparison of two models predicting IVF success; the effect of time trends on model performance, Hum Reprod, 29, 57, 10.1093/humrep/det393 · DOI · Paper page
  12. Smith, 2019, Comparison of prognostic models to predict the occurrence of colorectal cancer in asymptomatic individuals: a systematic literature review and external validation in the EPIC and UK Biobank prospective cohort studies, Gut, 68, 672, 10.1136/gutjnl-2017-315730 · DOI
  13. Timmerman, 2000, Terms, definitions and measurements to describe the sonographic features of adnexal tumors: a consensus opinion from the International Ovarian Tumor Analysis (IOTA) Group, Ultrasound Obstet Gynecol, 16, 500, 10.1046/j.1469-0705.2000.00287.x · DOI · Paper page
  14. Timmerman, 2005, Logistic regression model to distinguish between the benign and malignant adnexal mass before surgery: a multicenter study by the International Ovarian Tumor Analysis Group, J Clin Oncol, 23, 8794, 10.1200/JCO.2005.01.7632 · DOI · Paper page
  15. Van Holsbeke, 2009, Prospective internal validation of mathematical models to predict malignancy in adnexal masses: results from the international ovarian tumor analysis study, Clin Cancer Res, 15, 684, 10.1158/1078-0432.CCR-08-0113 · DOI · Paper page
  16. Timmerman, 2010, Ovarian cancer prediction in adnexal masses using ultrasound-based logistic regression models: a temporal and external validation study by the IOTA group, Ultrasound Obstet Gynecol, 36, 226, 10.1002/uog.7636 · DOI · Paper page
  17. Nunes, 2013, A prospective validation of the IOTA logistic regression models (LR1 and LR2) in comparison to subjective pattern recognition for the diagnosis of ovarian cancer, Int J Gynecol Cancer, 23, 1583, 10.1097/IGC.0b013e3182a6171a · DOI
  18. Wynants, 2018, Does ignoring clustering in multicenter data influence the performance of prediction models? A simulation study, Stat Methods Med Res, 27, 1723, 10.1177/0962280216668555 · DOI · Paper page
  19. Balmaña, 2006, Prediction of MLH1 and MSH2 mutations in Lynch syndrome, JAMA, 296, 1469, 10.1001/jama.296.12.1469 · DOI · Paper page
  20. Barnetson, 2006, Identification and survival of carriers of mutations in DNA mismatch-repair genes in colon cancer, N Engl J Med, 354, 2751, 10.1056/NEJMoa053493 · DOI · Paper page
  21. Van Calster, 2013, Rationalizing the management of pregnancies of unknown location: temporal and external validation of a risk prediction model on 1962 pregnancies, Hum Reprod, 28, 609, 10.1093/humrep/des440 · DOI
  22. Le Cessie, 1992, Ridge estimators in logistic regression, J R Stat Soc Ser C Appl Stat, 41, 191
  23. Cook, 2008, Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve, Clin Chem, 54, 17, 10.1373/clinchem.2007.096529 · DOI · Paper page
  24. Kundu, 2017, Impact of correlation of predictors on discrimination of risk models in development and external populations, BMC Med Res Methodol, 17, 63, 10.1186/s12874-017-0345-1 · DOI
  25. Wynants, 2013, Screening for data clustering in multicenter studies: the residual intraclass correlation, BMC Med Res Methodol, 13, 128, 10.1186/1471-2288-13-128 · DOI · Paper page
  26. Whittle, 2018, Measurement error and timing of predictor values for multivariable risk prediction models are poorly reported, J Clin Epidemiol, 102, 38, 10.1016/j.jclinepi.2018.05.008 · DOI
  27. McGraw, 1996, Forming inferences about some intraclass correlation coefficients, Psychol Methods, 1, 30, 10.1037/1082-989X.1.1.30 · DOI · Paper page

Cited by

See all 50

Metadata

Authors
Luijken, K., Wynants, L., van Smeden, M., Van Calster, B., Steyerberg, E. W., Groenwold, R. H. H., Timmerman, D., Bourne, T., & Ukaegbu, C.
Venue
Journal of Clinical Epidemiology
Publisher
Elsevier BV
Published
2020-03
Volume
119
Pages
7-18
ISSN
0895-4356 (ISSN-L)
OpenAlex
W2986964657
Cited by
50
References
8
FWCI
4.65
Type
journal article
Language
en
Open access
hybrid
Licence
Funders
  • Netherlands Organisation for Scientific Research
  • Research Foundation–Flanders (G0B4716N)
  • Internal Funds KU Leuven (C24/15/037)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Changing predictor measurement procedures affected the performance of prediction models in clinical examples. https://app.cochat.ai/papers/10.1016/j.jclinepi.2019.11.001
The paper
Luijken, K., Wynants, L., van Smeden, M., Van Calster, B., Steyerberg, E. W., Groenwold, R. H. H., Timmerman, D., Bourne, T., & Ukaegbu, C. (2020). Changing predictor measurement procedures affected the performance of prediction models in clinical examples. Journal of Clinical Epidemiology. https://doi.org/10.1016/j.jclinepi.2019.11.001

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.