Estimation of prediction error by using K-fold cross-validation
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The statistical literature suggests that leave-one-out methods overestimate performance, instead recommending that larger hold-outs be used [73-75].”
From Modeling causes of death: an integrated approach using CODEm (2012) · cites this paper as [73]
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881
Bootstrap Methods and their Application
Davison, A. C., & Hinkley, D. V. (1997) · doi:10.1017/cbo9780511802843 · Cited by 7,350
Bootstrap Methods and Their Application
Loughin, T. M., Davison, A. C., & Hinkley, D. V. (1999) · Journal of the American Statistical Association · doi:10.2307/2669707 · Cited by 5,418
Vaart, A. W. V. D. (1998) · doi:10.1017/cbo9780511802256 · Cited by 4,159
Linear Model Selection by Cross-validation
Shao, J. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476299 · Cited by 1,790
An asymptotic equivalence of choice of model by cross validation and Akaike's criterion
Stone, M. (1977) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1977.tb01603.x · Cited by 1,286
Information Criteria and Statistical Modeling
Konishi, S., & Kitagawa, G. (2008) · Springer Series in Statistics · doi:10.1007/978-0-387-71887-3 · Cited by 959
Cross-Validatory Choice and Assessment of Statistical Predictions (With Discussion)
Stone, M. (1976) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1976.tb01573.x · Cited by 686
BURMAN, P. (1989) · Biometrika · doi:10.1093/biomet/76.3.503 · Cited by 634
Cited by
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.03109 · Cited by 3,159
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
“and obtain relatively more accurate model performance evaluation (Fushiki, 2011); Holdout validation divides the dataset into training and test sets, with a smaller calculation amount but potentially more significant bias; LOOCV is a unique K-fold cross-validation method where only one”
MethodModeling causes of death: an integrated approach using CODEm
Foreman, K. J., Lozano, R., Lopez, A. D., & Murray, C. J. (2012) · Population Health Metrics · doi:10.1186/1478-7954-10-1 · Cited by 564
Chakraborty, K., Bhatia, S., Bhattacharyya, S., Platos, J., Bag, R., & Hassanien, A. E. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106754 · Cited by 375
“…work contains hyper-parametric classifiers from Naïve Bayes Models [48], Ensemble models [49], Support Vector Machine Models [50], Linear Models [51] viz., Multinomial Classifier, Bernoulli Classifier [52], AdaBoost Classifier [53], LinearSVC Classifier [51] and Logistic Regression Classifier [54].”
Delta-radiomics features for the prediction of patient outcomes in non–small cell lung cancer
Fave, X., Zhang, L., Yang, J., Mackin, D., Balter, P., Gomez, D., Followill, D., Jones, A. K., Stingo, F., Liao, Z., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-00665-z · Cited by 353
A survey of Bayesian predictive methods for model assessment, selection and comparison
Vehtari, A., & Ojanen, J. (2012) · Statistics Surveys · doi:10.1214/12-ss102 · Cited by 348
Handling limited datasets with neural networks in medical applications: A small-data approach
Shaikhina, T., & Khovanova, N. A. (2017) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2016.12.003 · Cited by 296
State-of-the-Art and Comparative Review of Adaptive Sampling Methods for Kriging
Fuhg, J. N., Fau, A., & Nackenhorst, U. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-020-09474-6 · Cited by 295
Dense Dilated Network With Probability Regularized Walk for Vessel Detection
Mou, L., Chen, L., Cheng, J., Gu, Z., Zhao, Y., & Liu, J. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2950051 · Cited by 164
“In this paper, we adopt the k-fold (k = 4) crossvalidation [76] for the training and testing phases, similar to that in [77].”
MethodYu, B., & Kumbier, K. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1901326117 · Cited by 164
“CV works more effectively as a tool to select regularization parameters than as an estimate of prediction error, where it can incur high variability due to the often positive dependencies between the estimated prediction errors in the summation of the CV error (31).”
MethodDealing with clustered samples for assessing map accuracy by cross-validation
de Bruin, S., Brus, D. J., Heuvelink, G. B. M., van Ebbenhorst Tengbergen, T., & Wadoux, A. M. J.-C. (2022) · Ecological Informatics · doi:10.1016/j.ecoinf.2022.101665 · Cited by 122
A Survey of Privacy Risks and Mitigation Strategies in the Artificial Intelligence Life Cycle
Shahriar, S., Allana, S., Hazratifard, S. M., & Dara, R. (2023) · IEEE Access · doi:10.1109/access.2023.3287195 · Cited by 114
“For instance, training error and standard k-fold cross-validation contain biases that must be adjusted [150].”
BackgroundOptimization of formulations using robotic experiments driven by machine learning DoE
Cao, L., Russo, D., Felton, K., Salley, D., Sharma, A., Keenan, G., Mauer, W., Gao, H., Cronin, L., & Lapkin, A. A. (2021) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2020.100295 · Cited by 75
Individualized atrophy scores predict dementia onset in familial frontotemporal lobar degeneration
Staffaroni, A. M., Cobigo, Y., Goh, S. M., Kornak, J., Bajorek, L., Chiang, K., Appleby, B., Bove, J., Bordelon, Y., Brannelly, P., et al. (2020) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2019.04.007 · Cited by 40
Bai, J., Sun, Z., Yu, S., Lu, Y., Long, S., Wang, H., Qiu, R., Ou, Z., Zhou, M., Zhi, D., et al. (2022) · Frontiers in Physiology · doi:10.3389/fphys.2022.940150 · Cited by 36
Nilashi, M., Abumalloh, R. A., Ahmadi, H., Samad, S., Alrizq, M., Abosaq, H., & Alghamdi, A. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e21828 · Cited by 30
“Cross-validation is a widely recognized technique for the evaluation of a machine learning model ’ s performance [71,72].”
MethodLiu, S., Zhang, R., Shang, X., & Li, W. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.105302 · Cited by 26
Finney, C. A., Delerue, F., Gold, W. A., Brown, D. A., & Shvetcov, A. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2022.12.018 · Cited by 22
Shvetcov, A., Thomson, S., Spathos, J., Cho, A.-N., Wilkins, H. M., Andrews, S. J., Delerue, F., Couttas, T. A., Issar, J. K., Isik, F., et al. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms241915011 · Cited by 12
Defining Materials Using Laser Signals from Long Distance via Deep Learning
Olgun, N., & Türkoğlu, İ. (2022) · Ain Shams Engineering Journal · doi:10.1016/j.asej.2021.10.001 · Cited by 6
Metadata
- Authors
- Fushiki, T.
- Affiliations
- The Institute of Statistical Mathematics (JP)
- Corresponding author
- Fushiki, Tadayoshi
- ORCID
- Fushiki, Tadayoshi
- Venue
- Statistics and Computing
- Published
- 2009-10-10
- Volume
- 21
- Issue
- 2
- Pages
- 137-146
- ISSN
- 0960-3174 (ISSN-L)
- 1573-1375
- OpenAlex
- W2085281262
- Cited by
- 1,470
- References
- 16
- Citations by year
- 2026: 126 · 2025: 268 · 2024: 267 · 2023: 226 · 2022: 178 · 2021: 159 · 2020: 103 · 2019: 53 · 2018: 28 · 2017: 19 · 2016: 13 · 2015: 11 · 2014: 10 · 2013: 2 · 2012: 6
- FWCI
- 1.66
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Statistics and Computing (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- prediction error estimation, k-fold cross-validation, leave-one-out cross-validation, bias correction, prediction accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.