Changing predictor measurement procedures affected the performance of prediction models in clinical examples
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: The aim of this study was to quantify the impact of predictor measurement heterogeneity on prediction model performance. Predictor measurement heterogeneity refers to variation in the measurement of predictor(s) between the derivation of a prediction model and its validation or application. It arises, for instance, when predictors are measured using different measurement instruments or protocols. STUDY DESIGN AND SETTING: We examined the effects of various scenarios of predictor measurement heterogeneity in real-world clinical examples using previously developed prediction models for diagnosis of ovarian cancer, mutation carriers for Lynch syndrome, and intrauterine pregnancy. RESULTS: Changing the measurement procedure of a predictor influenced the performance at validation of the prediction models in nine clinical examples. Notably, it induced model miscalibration. The calibration intercept at validation ranged from -0.70 to 1.43 (0 for good calibration), whereas the calibration slope ranged from 0.50 to 1.67 (1 for good calibration). The difference in C-statistic and scaled Brier score between derivation and validation ranged from -0.08 to +0.08 and from -0.40 to +0.16, respectively. CONCLUSION: This study illustrates that predictor measurement heterogeneity can influence the performance of a prediction model substantially, underlining that predictor measurements used in research settings should resemble clinical practice. Specification of measurement heterogeneity can help researchers explaining discrepancies in predictive performance between derivation and validation setting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · cited 1×
“By identifying predictor measurement heterogeneity as a separate explanation of discrepancies in linearpredictor distributions across settings, our findings can facilitate the implementation of the influential TRIPOD statement in clinical prediction research [36].”
MethodSelf-citationClinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Self-citationInternal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. (2015) · BMC Medicine · doi:10.1186/s12916-014-0241-z · Cited by 1,927
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · cited 3×
“Model calibration refers to the agreement between observed outcomes and risk estimates [1,29].”
BackgroundMethodSelf-citationVergouwe, Y., Moons, K. G. M., & Steyerberg, E. W. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq223 · Cited by 243
Self-citation
Research works printed but not linked: 27
- Altman, 2000, What do we mean by validating a prognostic model?, Stat Med, 19, 453, 10.1002/(SICI)1097-0258(20000229)19:4<453::AID-SIM350>3.0.CO;2-5 · doi:10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.0.co;2-5
- Wynants, 2017, Key steps and common pitfalls in developing and validating risk models, BJOG, 124, 423, 10.1111/1471-0528.14170 · DOI
- Steyerberg, 2018, Poor performance of clinical prediction models: the harm of commonly applied methods, J Clin Epidemiol, 98, 133, 10.1016/j.jclinepi.2017.11.013 · DOI · Paper page
- Toll, 2008, Validation, updating and impact of clinical prediction rules: a review, J Clin Epidemiol, 61, 1085, 10.1016/j.jclinepi.2008.04.008 · DOI · Paper page
- Debray, 2013, A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis, Stat Med, 32, 3158, 10.1002/sim.5732 · DOI · Paper page
- Khudyakov, 2015, The impact of covariate measurement error on risk prediction, Stat Med, 34, 2353, 10.1002/sim.6498 · DOI
- Rosella, 2012, The influence of measurement error on calibration, discrimination, and overall estimation of a risk prediction model, Popul Health Metr, 10, 20, 10.1186/1478-7954-10-20 · DOI
- Luijken, 2019, Impact of predictor measurement heterogeneity across settings on the performance of prediction models: a measurement error perspective, Stat Med, 38, 3444, 10.1002/sim.8183 · DOI · Paper page
- Pajouheshnia, 2019, How variation in predictor measurement affects the discriminative ability and transportability of a prediction model, J Clin Epidemiol, 105, 136, 10.1016/j.jclinepi.2018.09.001 · DOI
- Pajouheshnia, 2019, When and how to use data from randomised trials to develop or validate prognostic models, BMJ, 365, l2154, 10.1136/bmj.l2154 · DOI
- Te Velde, 2013, Comparison of two models predicting IVF success; the effect of time trends on model performance, Hum Reprod, 29, 57, 10.1093/humrep/det393 · DOI · Paper page
- Smith, 2019, Comparison of prognostic models to predict the occurrence of colorectal cancer in asymptomatic individuals: a systematic literature review and external validation in the EPIC and UK Biobank prospective cohort studies, Gut, 68, 672, 10.1136/gutjnl-2017-315730 · DOI
- Timmerman, 2000, Terms, definitions and measurements to describe the sonographic features of adnexal tumors: a consensus opinion from the International Ovarian Tumor Analysis (IOTA) Group, Ultrasound Obstet Gynecol, 16, 500, 10.1046/j.1469-0705.2000.00287.x · DOI · Paper page
- Timmerman, 2005, Logistic regression model to distinguish between the benign and malignant adnexal mass before surgery: a multicenter study by the International Ovarian Tumor Analysis Group, J Clin Oncol, 23, 8794, 10.1200/JCO.2005.01.7632 · DOI · Paper page
- Van Holsbeke, 2009, Prospective internal validation of mathematical models to predict malignancy in adnexal masses: results from the international ovarian tumor analysis study, Clin Cancer Res, 15, 684, 10.1158/1078-0432.CCR-08-0113 · DOI · Paper page
- Timmerman, 2010, Ovarian cancer prediction in adnexal masses using ultrasound-based logistic regression models: a temporal and external validation study by the IOTA group, Ultrasound Obstet Gynecol, 36, 226, 10.1002/uog.7636 · DOI · Paper page
- Nunes, 2013, A prospective validation of the IOTA logistic regression models (LR1 and LR2) in comparison to subjective pattern recognition for the diagnosis of ovarian cancer, Int J Gynecol Cancer, 23, 1583, 10.1097/IGC.0b013e3182a6171a · DOI
- Wynants, 2018, Does ignoring clustering in multicenter data influence the performance of prediction models? A simulation study, Stat Methods Med Res, 27, 1723, 10.1177/0962280216668555 · DOI · Paper page
- Balmaña, 2006, Prediction of MLH1 and MSH2 mutations in Lynch syndrome, JAMA, 296, 1469, 10.1001/jama.296.12.1469 · DOI · Paper page
- Barnetson, 2006, Identification and survival of carriers of mutations in DNA mismatch-repair genes in colon cancer, N Engl J Med, 354, 2751, 10.1056/NEJMoa053493 · DOI · Paper page
- Van Calster, 2013, Rationalizing the management of pregnancies of unknown location: temporal and external validation of a risk prediction model on 1962 pregnancies, Hum Reprod, 28, 609, 10.1093/humrep/des440 · DOI
- Le Cessie, 1992, Ridge estimators in logistic regression, J R Stat Soc Ser C Appl Stat, 41, 191
- Cook, 2008, Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve, Clin Chem, 54, 17, 10.1373/clinchem.2007.096529 · DOI · Paper page
- Kundu, 2017, Impact of correlation of predictors on discrimination of risk models in development and external populations, BMC Med Res Methodol, 17, 63, 10.1186/s12874-017-0345-1 · DOI
- Wynants, 2013, Screening for data clustering in multicenter studies: the residual intraclass correlation, BMC Med Res Methodol, 13, 128, 10.1186/1471-2288-13-128 · DOI · Paper page
- Whittle, 2018, Measurement error and timing of predictor values for multivariable risk prediction models are poorly reported, J Clin Epidemiol, 102, 38, 10.1016/j.jclinepi.2018.05.008 · DOI
- McGraw, 1996, Forming inferences about some intraclass correlation coefficients, Psychol Methods, 1, 30, 10.1037/1082-989X.1.1.30 · DOI · Paper page
Cited by
External validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
There is no such thing as a validated prediction model
Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023) · BMC Medicine · doi:10.1186/s12916-023-02779-w · Cited by 286
“Such heterogeneity in measurement procedures will affect model performance [19, 20].”
BackgroundJenkins, D. A., Martin, G. P., Sperrin, M., Riley, R. D., Debray, T. P. A., Collins, G. S., & Peek, N. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00090-3 · Cited by 131
Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings
Davis, S. E., Walsh, C. G., & Matheny, M. E. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.958284 · Cited by 33
Ajuwon, B. I., Richardson, A., Roper, K., & Lidbury, B. A. (2023) · Viruses · doi:10.3390/v15081735 · Cited by 6
Metadata
- Authors
- Luijken, K., Wynants, L., van Smeden, M., Van Calster, B., Steyerberg, E. W., Groenwold, R. H. H., Timmerman, D., Bourne, T., & Ukaegbu, C.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2020-03
- Volume
- 119
- Pages
- 7-18
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2986964657
- Cited by
- 50
- References
- 8
- FWCI
- 4.65
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Funders
- Netherlands Organisation for Scientific Research
- Research Foundation–Flanders (G0B4716N)
- Internal Funds KU Leuven (C24/15/037)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Changing predictor measurement procedures affected the performance of prediction models in clinical examples. https://app.cochat.ai/papers/10.1016/j.jclinepi.2019.11.001
- The paper
- Luijken, K., Wynants, L., van Smeden, M., Van Calster, B., Steyerberg, E. W., Groenwold, R. H. H., Timmerman, D., Bourne, T., & Ukaegbu, C. (2020). Changing predictor measurement procedures affected the performance of prediction models in clinical examples. Journal of Clinical Epidemiology. https://doi.org/10.1016/j.jclinepi.2019.11.001
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.