The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Cross-Validation With Confidence
Abstract
Cross-validation is one of the most popular model and tuning parameter selection methods in statistics and machine learning. Despite its wide applicability, traditional cross-validation methods tend to overfit, due to the ignorance of the uncertainty in the testing sample. We develop a novel statistically principled inference tool based on cross-validation that takes into account the uncertainty in the testing sample. This method outputs a set of highly competitive candidate models containing the optimal one with guaranteed probability. As a consequence, our method can achieve consistent variable selection in a classical linear regression setting, for which existing cross-validation methods require unconventional split ratios. When used for tuning parameter selection, the method can provide an alternative trade-off between prediction accuracy and model interpretability than existing variants of cross-validation. We demonstrate the performance of the proposed method in several simulated and real data examples. Supplemental materials for this article can be found online.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004) · The Annals of Statistics · doi:10.1214/009053604000000067 · Cited by 9,357
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014
The Predictive Sample Reuse Method with Applications
Geisser, S. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10479865 · Cited by 2,748 · cited 1×
“Cross-validation [27, 1, 15] is one of the most popular methods for model and tuning parameter selection in statistics and machine learning, due to its conceptual simplicity and wide applicability.”
MethodIntroduction to Nonparametric Estimation
Tsybakov, A. B. (2009) · Springer Series in Statistics · doi:10.1007/b13794 · Cited by 2,486
Simultaneous analysis of Lasso and Dantzig selector
Bickel, P. J., Ritov, Y., & Tsybakov, A. B. (2009) · The Annals of Statistics · doi:10.1214/08-aos620 · Cited by 2,396
Meinshausen, N., & Bühlmann, P. (2010) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2010.00740.x · Cited by 2,215
(2011) · Econometrica · doi:10.3982/ecta5771 · Cited by 2,096
Advances in Neural Information Processing Systems 21
Griffiths, T. L., Lucas, C. G., Williams, J. J., & Kalish, M. L. (2009) · neural information processing systems · Cited by 1,854
Research works printed but not linked: 3
- Chatterjee S., 2015, arXiv no, 1502, 06291
- Chernozhukov V., 2013, arXiv no, 1312, 7614
- Chetverikov D., 2016, arXiv no, 1605, 02214
Cited by
The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century
Maleki Varnosfaderani, S., & Forouzanfar, M. (2024) · Bioengineering · doi:10.3390/bioengineering11040337 · Cited by 931
Machine Learning Aided Design and Optimization of Thermal Metamaterials
Zhu, C., Bamidele, E. A., Shen, X., Zhu, G., & Li, B. (2024) · Chemical Reviews · doi:10.1021/acs.chemrev.3c00708 · Cited by 195
Machine Learning for Perovskite Solar Cells and Component Materials: Key Technologies and Prospects
Liu, Y., Tan, X., Liang, J., Han, H., Xiang, P., & Yan, W. (2023) · Advanced Functional Materials · doi:10.1002/adfm.202214271 · Cited by 174
Chui, K. T., Liu, R. W., Zhao, M., & de Pablos, P. O. (2020) · IEEE Access · doi:10.1109/access.2020.2992869 · Cited by 134
State-of-Charge Balancing of Lithium-Ion Batteries With State-of-Health Awareness Capability
Xia, Z., & Qahouq, J. A. A. (2020) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2020.3029755 · Cited by 123
Metadata
- Authors
- Lei, J.
- Affiliations
- Carnegie Mellon University (US)
- Corresponding author
- Lei, Jing
- ORCID
- Lei, Jing
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2019-10-31
- Volume
- 115
- Issue
- 532
- Pages
- 1978-1997
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2979632005
- Cited by
- 93
- References
- 44
- Citations by year
- 2026: 15 · 2025: 22 · 2024: 17 · 2023: 12 · 2022: 9 · 2021: 8 · 2020: 9 · 2019: 1
- FWCI
- 5.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- NSF (DMS-1407771, DMS-1553884)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- cross-validation, overfitting, model selection, feature selection, hyperparameter optimization, linear regression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Cross-Validation With Confidence. https://app.cochat.ai/papers/10.1080/01621459.2019.1672556
- The paper
- Lei, J. (2020). Cross-Validation With Confidence. Journal of the American Statistical Association. https://doi.org/10.1080/01621459.2019.1672556
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.