The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Correcting for Optimistic Prediction in Small Data Sets
Open accessAbstract
The C statistic is a commonly reported measure of screening test performance. Optimistic estimation of the C statistic is a frequent problem because of overfitting of statistical models in small data sets, and methods exist to correct for this issue. However, many studies do not use such methods, and those that do correct for optimism use diverse methods, some of which are known to be biased. We used clinical data sets (United Kingdom Down syndrome screening data from Glasgow (1991-2003), Edinburgh (1999-2003), and Cambridge (1990-2006), as well as Scottish national pregnancy discharge data (2004-2007)) to evaluate different approaches to adjustment for optimism. We found that sample splitting, cross-validation without replication, and leave-1-out cross-validation produced optimism-adjusted estimates of the C statistic that were biased and/or associated with greater absolute error than other available methods. Cross-validation with replication, bootstrapping, and a new method (leave-pair-out cross-validation) all generated unbiased optimism-adjusted estimates of the C statistic and had similar absolute errors in the clinical data set. Larger simulation studies confirmed that all 3 methods performed similarly with 10 or more events per variable, or when the C statistic was 0.9 or greater. However, with lower events per variable or lower C statistics, bootstrapping tended to be optimistic but with lower absolute and mean squared errors than both methods of cross-validation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · Radiology · doi:10.1148/radiol.2261021292 · Cited by 3,215
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Estimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Self-citationPrognosis and prognostic research: Developing a prognostic model
Royston, P., Moons, K. G. M., Altman, D. G., & Vergouwe, Y. (2009) · BMJ · doi:10.1136/bmj.b604 · Cited by 1,172
Self-citationvon Dadelszen, P., Payne, B., Li, J., Ansermino, J. M., Pipkin, F. B., Côté, A.-M., Douglas, M. J., Gruslin, A., Hutcheon, J. A., Joseph, K., et al. (2011) · The Lancet · doi:10.1016/s0140-6736(10)61351-7 · Cited by 586
Self-citationNorth, R. A., McCowan, L. M. E., Dekker, G. A., Poston, L., Chan, E. H. Y., Stewart, A. W., Black, M. A., Taylor, R. S., Walker, J. J., Baker, P. N., et al. (2011) · BMJ · doi:10.1136/bmj.d1875 · Cited by 413
Model‐based screening by risk with application to down's syndrome
Royston, P., & Thompson, S. G. (1992) · Statistics in Medicine · doi:10.1002/sim.4780110211 · Cited by 173
Self-citation
Cited by
Determinants of COVID-19 vaccine acceptance in the US
Malik, A. A., McFadden, S. M., Elharake, J., & Omer, S. B. (2020) · EClinicalMedicine · doi:10.1016/j.eclinm.2020.100495 · Cited by 1,540
Predicting Risk of Suicide Attempts Over Time Through Machine Learning
Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017) · Clinical Psychological Science · doi:10.1177/2167702617691560 · Cited by 567
Austin, P. C., & Steyerberg, E. W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214558972 · Cited by 516
“when the number of EPV was very low).(18) They found that the ‘leave-one-out’ approach resulted in estimates of the c-statistics that were systematically lower than the true c-statistic.”
ResultHumby, F., Lewis, M., Ramamoorthi, N., Hackney, J. A., Barnes, M. R., Bombardieri, M., Setiadi, A. F., Kelly, S., Bene, F., DiCicco, M., et al. (2019) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2018-214539 · Cited by 329
Liu, H., Petukhova, M. V., Sampson, N. A., Aguilar-Gaxiola, S., Alonso, J., Andrade, L. H., Bromet, E. J., de Girolamo, G., Haro, J. M., Hinkov, H., et al. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.3783 · Cited by 270
initiative, F. T. O. T. S., Sauerbrei, W., Perperoglou, A., Schmid, M., Abrahamowicz, M., Becher, H., Binder, H., Dunkler, D., Harrell, F. E., Royston, P., et al. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00074-3 · Cited by 256
Self-citationBaldwin, J. R., Caspi, A., Meehan, A. J., Ambler, A., Arseneault, L., Fisher, H. L., Harrington, H., Matthews, T., Odgers, C. L., Poulton, R., et al. (2021) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2020.5602 · Cited by 229
Valpione, S., Pasquali, S., Campana, L. G., Piccin, L., Mocellin, S., Pigozzo, J., & Chiarion-Sileni, V. (2018) · Journal of Translational Medicine · doi:10.1186/s12967-018-1467-x · Cited by 201
“The predictive effect of the model was then validated using bootstrap methodology (200 replications), as advised for small datasets [18].”
MethodNock, M. K., Millner, A. J., Joiner, T. E., Gutierrez, P. M., Han, G., Hwang, I., King, A., Naifeh, J. A., Sampson, N. A., Zaslavsky, A. M., et al. (2018) · Journal of Abnormal Psychology · doi:10.1037/abn0000317 · Cited by 193
Wilimitis, D., & Walsh, C. G. (2023) · JMIR AI · doi:10.2196/49023 · Cited by 185
Predicting probable Alzheimer's disease using linguistic deficits and biomarkers
Orimaye, S. O., Wong, J. S.-M., Golden, K. J., Wong, C. P., & Soyiri, I. N. (2017) · BMC Bioinformatics · doi:10.1186/s12859-016-1456-0 · Cited by 158
Hirata, T., Arai, Y., Yuasa, S., Abe, Y., Takayama, M., Sasaki, T., Kunitomi, A., Inagaki, H., Endo, M., Morinaga, J., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-17636-0 · Cited by 138
Ruderfer, D. M., Walsh, C. G., Aguirre, M. W., Tanigawa, Y., Ribeiro, J. D., Franklin, J. C., & Rivas, M. A. (2020) · Molecular Psychiatry · doi:10.1038/s41380-018-0326-8 · Cited by 134
“parameters of 500 trees per forest and splits of the square root of the number of predictors at each node in the tree, the clinical phenotyping algorithm was trained via optimism adjustment with the bootstrap using 100 bootstraps [27].”
MethodKessler, R. C. (2018) · Current Opinion in Psychiatry · doi:10.1097/yco.0000000000000377 · Cited by 103
“Internal pseudoreplication methods exist to evaluate prediction shrinkage due to these small samples [71].”
MethodValidation of the AUDIT-C in adults seeking help with their drinking online
Khadjesari, Z., White, I. R., McCambridge, J., Marston, L., Wallace, P., Godfrey, C., & Murray, E. (2017) · Addiction Science & Clinical Practice · doi:10.1186/s13722-016-0066-5 · Cited by 106
“When the same data are used both to select a cut-off score and to evaluate performance (sensitivity, specificity, likelihood ratios or accuracy) at the cut-off, performance tends to be over-estimated—a phenomenon known as “optimism” or “overfitting” [42].”
MethodSelf-citationPredicting imminent suicidal thoughts and nonfatal attempts: The role of complexity
Ribeiro, J. D., Huang, X., Fox, K. R., Walsh, C. G., & Linthicum, K. P. (2019) · Clinical Psychological Science · doi:10.1177/2167702619838464 · Cited by 75
Wood, A. M., Royston, P., & White, I. R. (2015) · Biometrical Journal · doi:10.1002/bimj.201400004 · Cited by 78
Self-citationProspective Validation of an Electronic Health Record–Based, Real-Time Suicide Risk Model
Walsh, C. G., Johnson, K. B., Ripperger, M., Sperry, S., Harris, J., Clark, N., Fielstein, E., Novak, L., Robinson, K., & Stead, W. W. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.1428 · Cited by 78
Hocking, D. C., Mancuso, S. G., & Sundram, S. (2018) · BMC Psychiatry · doi:10.1186/s12888-018-1660-8 · Cited by 74
“This approach outperforms split-sample validation, particularly when the sample size is limited [72, 73].”
MethodMachine learning enhances prediction of illness course: a longitudinal study in eating disorders
Haynos, A. F., Wang, S. B., Lipson, S., Peterson, C. B., Mitchell, J. E., Halmi, K. A., Agras, W. S., & Crow, S. J. (2021) · Psychological Medicine · doi:10.1017/s0033291720000227 · Cited by 73
Orimaye, S. O., Wong, J. S.-M., & Wong, C. P. (2018) · PLOS ONE · doi:10.1371/journal.pone.0205636 · Cited by 52
Predictors for Emergency Cesarean Delivery in Women with Placenta Previa
Eschbach, S., Burgers, M., Rengerink, K., Pampus, M., Goes, B., Mol, B., Graaf, I., Pajkrt, E., & Ruiter, L. (2016) · American Journal of Perinatology · doi:10.1055/s-0036-1584148 · Cited by 46
Stein, D. J., Karam, E. G., Shahly, V., Hill, E. D., King, A., Petukhova, M., Atwoli, L., Bromet, E. J., Florescu, S., Haro, J. M., et al. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-0957-8 · Cited by 40
Metadata
- Authors
- Smith, G. C. S., Seaman, S. R., Wood, A. M., Royston, P., & White, I. R.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-06-24
- Volume
- 180
- Issue
- 3
- Pages
- 318-324
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- OpenAlex
- W2159704023
- Cited by
- 352
- References
- 18
- FWCI
- 22.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.