The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Correcting for Optimistic Prediction in Small Data Sets
Open accessAbstract
The C statistic is a commonly reported measure of screening test performance. Optimistic estimation of the C statistic is a frequent problem because of overfitting of statistical models in small data sets, and methods exist to correct for this issue. However, many studies do not use such methods, and those that do correct for optimism use diverse methods, some of which are known to be biased. We used clinical data sets (United Kingdom Down syndrome screening data from Glasgow (1991-2003), Edinburgh (1999-2003), and Cambridge (1990-2006), as well as Scottish national pregnancy discharge data (2004-2007)) to evaluate different approaches to adjustment for optimism. We found that sample splitting, cross-validation without replication, and leave-1-out cross-validation produced optimism-adjusted estimates of the C statistic that were biased and/or associated with greater absolute error than other available methods. Cross-validation with replication, bootstrapping, and a new method (leave-pair-out cross-validation) all generated unbiased optimism-adjusted estimates of the C statistic and had similar absolute errors in the clinical data set. Larger simulation studies confirmed that all 3 methods performed similarly with 10 or more events per variable, or when the C statistic was 0.9 or greater. However, with lower events per variable or lower C statistics, bootstrapping tended to be optimistic but with lower absolute and mean squared errors than both methods of cross-validation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · Radiology · doi:10.1148/radiol.2261021292 · Cited by 3,215
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Estimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Self-citationPrognosis and prognostic research: Developing a prognostic model
Royston, P., Moons, K. G. M., Altman, D. G., & Vergouwe, Y. (2009) · BMJ · doi:10.1136/bmj.b604 · Cited by 1,172
Self-citationvon Dadelszen, P., Payne, B., Li, J., Ansermino, J. M., Pipkin, F. B., Côté, A.-M., Douglas, M. J., Gruslin, A., Hutcheon, J. A., Joseph, K., et al. (2011) · The Lancet · doi:10.1016/s0140-6736(10)61351-7 · Cited by 586
Self-citationNorth, R. A., McCowan, L. M. E., Dekker, G. A., Poston, L., Chan, E. H. Y., Stewart, A. W., Black, M. A., Taylor, R. S., Walker, J. J., Baker, P. N., et al. (2011) · BMJ · doi:10.1136/bmj.d1875 · Cited by 413
Model‐based screening by risk with application to down's syndrome
Royston, P., & Thompson, S. G. (1992) · Statistics in Medicine · doi:10.1002/sim.4780110211 · Cited by 173
Self-citation
Cited by
Determinants of COVID-19 vaccine acceptance in the US
Malik, A. A., McFadden, S. M., Elharake, J., & Omer, S. B. (2020) · EClinicalMedicine · doi:10.1016/j.eclinm.2020.100495 · Cited by 1,540
Predicting Risk of Suicide Attempts Over Time Through Machine Learning
Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017) · Clinical Psychological Science · doi:10.1177/2167702617691560 · Cited by 567
Austin, P. C., & Steyerberg, E. W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214558972 · Cited by 516
Humby, F., Lewis, M., Ramamoorthi, N., Hackney, J. A., Barnes, M. R., Bombardieri, M., Setiadi, A. F., Kelly, S., Bene, F., DiCicco, M., et al. (2019) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2018-214539 · Cited by 329
Liu, H., Petukhova, M. V., Sampson, N. A., Aguilar-Gaxiola, S., Alonso, J., Andrade, L. H., Bromet, E. J., de Girolamo, G., Haro, J. M., Hinkov, H., et al. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.3783 · Cited by 270
Metadata
- Authors
- Smith, G. C. S., Seaman, S. R., Wood, A. M., Royston, P., & White, I. R.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-06-24
- Volume
- 180
- Issue
- 3
- Pages
- 318-324
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- OpenAlex
- W2159704023
- Cited by
- 352
- References
- 18
- FWCI
- 22.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Correcting for Optimistic Prediction in Small Data Sets. https://app.cochat.ai/papers/10.1093/aje/kwu140
- The paper
- Smith, G. C. S., Seaman, S. R., Wood, A. M., Royston, P., & White, I. R. (2014). Correcting for Optimistic Prediction in Small Data Sets. American Journal of Epidemiology. https://doi.org/10.1093/aje/kwu140
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.