The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Correcting for Optimistic Prediction in Small Data Sets
Open accessAbstract
The C statistic is a commonly reported measure of screening test performance. Optimistic estimation of the C statistic is a frequent problem because of overfitting of statistical models in small data sets, and methods exist to correct for this issue. However, many studies do not use such methods, and those that do correct for optimism use diverse methods, some of which are known to be biased. We used clinical data sets (United Kingdom Down syndrome screening data from Glasgow (1991-2003), Edinburgh (1999-2003), and Cambridge (1990-2006), as well as Scottish national pregnancy discharge data (2004-2007)) to evaluate different approaches to adjustment for optimism. We found that sample splitting, cross-validation without replication, and leave-1-out cross-validation produced optimism-adjusted estimates of the C statistic that were biased and/or associated with greater absolute error than other available methods. Cross-validation with replication, bootstrapping, and a new method (leave-pair-out cross-validation) all generated unbiased optimism-adjusted estimates of the C statistic and had similar absolute errors in the clinical data set. Larger simulation studies confirmed that all 3 methods performed similarly with 10 or more events per variable, or when the C statistic was 0.9 or greater. However, with lower events per variable or lower C statistics, bootstrapping tended to be optimistic but with lower absolute and mean squared errors than both methods of cross-validation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · Radiology · doi:10.1148/radiol.2261021292 · Cited by 3,215
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Estimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Self-citationPrognosis and prognostic research: Developing a prognostic model
Royston, P., Moons, K. G. M., Altman, D. G., & Vergouwe, Y. (2009) · BMJ · doi:10.1136/bmj.b604 · Cited by 1,172
Self-citationvon Dadelszen, P., Payne, B., Li, J., Ansermino, J. M., Pipkin, F. B., Côté, A.-M., Douglas, M. J., Gruslin, A., Hutcheon, J. A., Joseph, K., et al. (2011) · The Lancet · doi:10.1016/s0140-6736(10)61351-7 · Cited by 586
Self-citationNorth, R. A., McCowan, L. M. E., Dekker, G. A., Poston, L., Chan, E. H. Y., Stewart, A. W., Black, M. A., Taylor, R. S., Walker, J. J., Baker, P. N., et al. (2011) · BMJ · doi:10.1136/bmj.d1875 · Cited by 413
Model‐based screening by risk with application to down's syndrome
Royston, P., & Thompson, S. G. (1992) · Statistics in Medicine · doi:10.1002/sim.4780110211 · Cited by 173
Self-citationAirola, A., Pahikkala, T., Waegeman, W., De Baets, B., & Salakoski, T. (2011) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2010.11.018 · Cited by 166
The Effect of Delaying Childbirth on Primary Cesarean Section Rates
Smith, G. C. S., Cordeaux, Y., White, I. R., Pasupathy, D., Missfelder-Lobos, H., Pell, J. P., Charnock-Jones, D. S., & Fleming, M. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050144 · Cited by 160
Self-citationMolecular Classification of Breast Tumors: Toward Improved Diagnostics and Treatments
Sørlie, T. (2006) · Humana Press eBooks · doi:10.1385/1-59745-165-7:91 · Cited by 91
Stratification bias in low signal microarray studies
Parker, B. J., Günter, S., & Bedő, J. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-326 · Cited by 70
Tumor Markers: The Potential of “Omics” Approach
Sikaroodi, M., Galachiantz, Y., & Baranova, A. V. (2010) · Current Molecular Medicine · doi:10.2174/156652410790963277 · Cited by 37
Prediction of Adverse Maternal Outcomes in Preeclampsia
von Dadelszen, P., Payne, B. A., Li, J., Ansermino, J. M., Pipkin, F. B., Côté, A., Douglas, M. J., Gruslin, A., Hutcheon, J. A., Joseph, K. S., et al. (2012) · Obstetric Anesthesia Digest · doi:10.1097/01.aoa.0000410808.61878.a4 · Cited by 36
Self-citationSystematic review of statistical methods used in molecular marker studies in cancer
Vickers, A. J., Jang, K. S., Sargent, D. J., Lilja, H. G., & Kattan, M. W. (2008) · Cancer · doi:10.1002/cncr.23365 · Cited by 35
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4<361::aid-sim168>3.0.co;2-4
Cited by
Determinants of COVID-19 vaccine acceptance in the US
Malik, A. A., McFadden, S. M., Elharake, J., & Omer, S. B. (2020) · EClinicalMedicine · doi:10.1016/j.eclinm.2020.100495 · Cited by 1,540
Predicting Risk of Suicide Attempts Over Time Through Machine Learning
Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017) · Clinical Psychological Science · doi:10.1177/2167702617691560 · Cited by 567
Austin, P. C., & Steyerberg, E. W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214558972 · Cited by 516
“when the number of EPV was very low).(18) They found that the ‘leave-one-out’ approach resulted in estimates of the c-statistics that were systematically lower than the true c-statistic.”
ResultHumby, F., Lewis, M., Ramamoorthi, N., Hackney, J. A., Barnes, M. R., Bombardieri, M., Setiadi, A. F., Kelly, S., Bene, F., DiCicco, M., et al. (2019) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2018-214539 · Cited by 329
Liu, H., Petukhova, M. V., Sampson, N. A., Aguilar-Gaxiola, S., Alonso, J., Andrade, L. H., Bromet, E. J., de Girolamo, G., Haro, J. M., Hinkov, H., et al. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.3783 · Cited by 270
Metadata
- Authors
- Smith, G. C. S., Seaman, S. R., Wood, A. M., Royston, P., & White, I. R.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-06-24
- Volume
- 180
- Issue
- 3
- Pages
- 318-324
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- OpenAlex
- W2159704023
- Cited by
- 352
- References
- 18
- FWCI
- 22.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.