Cross validatory choice and assessment of statistical predictions

Stone, M.

doi:10.1111/j.2517-6161.1974.tb00994.x

Cited by 10,881Statistical Methods and InferenceStatistical Methods and ApplicationsAdvanced Statistical Methods and Models

Cross validatory choice and assessment of statistical predictions

Stone, M.·1974·Journal of the Royal Statistical Society Series B: Statistical Methodology

DOI 10.1111/j.2517-6161.1974.tb00994.x

Open this paper in CoChat — free account

Abstract

Summary A generalized form of the cross-validation criterion is applied to the choice and assessment of prediction using the data-analytic concept of a prescription. The examples used to illustrate the application are drawn from the problem areas of univariate estimation, linear regression and analysis of variance.

Integrity signals

Registry

Crossref · 10.1111/j.2517-6161.1974.tb00994.x

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

A long-standing practice in the post-selection inference literature is to use different partitions of the data for model estimation and inference, referred to as sample splitting.
“A long-standing practice in that literature is to use different partitions of the data for model estimation and inference, referred to as sample splitting (Larson, 1931; Stone, 1974).”
From Auditing a Dutch Public Sector Risk Profiling Algorithm Using an Unsupervised Bias Detection Tool (2025) · cites this paper as Stone, 1974
The performance of a model in k-fold cross-validation is estimated as the mean of the k validation results.
“The performance of the model on such a setup is estimated to be the mean of the k validation results [19,20,21].”
From Fully Automated Placental Volume Quantification From<scp>3D</scp>Ultrasound for Prediction of Small‐for‐Gestational‐Age Infants (2022) · cites this paper as [20]

The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518 · see citing works
Reviews: Lord, Frederic M., and Novick, Melvin R. (with contributions by Allan Birnbaum). Statistical Theories of Mental Test Scores. Reading, Mass.: Addison-Wesley, 1968. xii + 568 pp. $14.95.
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012 · see citing works
Optimal Statistical Decisions.
Siegmund, D., & DeGroot, M. H. (1972) · Journal of the American Statistical Association · doi:10.2307/2284675 · Cited by 3,461 · see citing works
A Predictive Approach to the Random Effect Model
Geisser, S. (1974) · Biometrika · doi:10.1093/biomet/61.1.101 · Cited by 3,101 · see citing works
Bayes Estimates for the Linear Model
Lindley, D. V., & Smith, A. F. M. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00885.x · Cited by 2,025 · see citing works
Estimation of Error Rates in Discriminant Analysis
Lachenbruch, P. A., & Mickey, M. R. (1968) · Technometrics · doi:10.1080/00401706.1968.10490530 · Cited by 1,508 · see citing works
The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction
Allen, D. M. (1974) · Technometrics · doi:10.1080/00401706.1974.10489157 · Cited by 1,469 · see citing works
Inference and disputed authorship : The Federalist
Mosteller, F. C., & Wallace, D. K. (1964) · Medical Entomology and Zoology · Cited by 916 · see citing works
Mean Square Error of Prediction as a Criterion for Selecting Variables
Allen, D. M. (1971) · Technometrics · doi:10.1080/00401706.1971.10488811 · Cited by 562 · see citing works
Robust Estimates of Location
Robertson, C., Andrews, D. F., Bickel, P. J., Hanpel, F. R., Huber, P. J., Rogers, W. H., & Tukey, J. W. (1972) · Biometrics · doi:10.2307/2528653 · Cited by 462 · see citing works
Optimal Statistical Decisions
Randolph, P. H., & Groot, M. D. (1970) · Technometrics · doi:10.2307/1267221 · Cited by 424 · see citing works
Mean Square Error of Prediction as a Criterion for Selecting Variables
Allen, D. M. (1971) · Technometrics · doi:10.2307/1267161 · Cited by 362 · see citing works
I. Problems and Designs of Cross-Validation 1
Mosier, C. I. (1951) · Educational and Psychological Measurement · doi:10.1177/001316445101100101 · Cited by 325 · see citing works
Statistics and experimental design
(1990) · Reproductive Toxicology · doi:10.1016/0890-6238(90)90011-j · Cited by 305 · see citing works
The prediction of personal adjustment.
Stuit, D. B. (1942) · Psychological Bulletin · doi:10.1037/h0052603 · Cited by 276 · see citing works
A New Formula for Predicting the Shrinkage of the Coefficient of Multiple Correlation
Wherry, R. J. (1931) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177732951 · Cited by 274 · see citing works
Selection of Variables for Fitting Equations to Data
Gorman, J. W., & Toman, R. J. (1966) · Technometrics · doi:10.1080/00401706.1966.10490322 · Cited by 274 · see citing works
Confidence Sets for the Mean of a Multivariate Normal Distribution
Stein, C. (1962) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1962.tb00458.x · Cited by 250 · see citing works
The Choice of Variables in Multiple Regression
Lindley, D. V. (1968) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1968.tb01505.x · Cited by 235 · see citing works
III. The influence of rainfall on the yield of wheat at Rothamsted
Fisher, R. A. (1925) · Philosophical Transactions of the Royal Society of London Series B Containing Papers of a Biological Character · doi:10.1098/rstb.1925.0003 · Cited by 234 · see citing works
The shrinkage of the coefficient of multiple correlation.
Larson, S. C. (1931) · Journal of Educational Psychology · doi:10.1037/h0072400 · Cited by 233 · see citing works
Empirical Bayes on vector observations: An extension of Stein's method
Efron, B., & Morris, C. N. (1972) · Biometrika · doi:10.1093/biomet/59.2.335 · Cited by 224 · see citing works
The Generalized Jackknife Statistic
Strawderman, W. E. (1973) · Journal of Quality Technology · doi:10.1080/00224065.1973.11980580 · Cited by 222 · see citing works
Allocation Rules and Their Error Rates
Hills, M. (1966) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1966.tb00614.x · Cited by 216 · see citing works
The Prediction of Personal Adjustment.
Scates, D. E., & Horst, P. (1942) · Journal of the American Statistical Association · doi:10.2307/2279447 · Cited by 206 · see citing works
Combining Possibly Related Estimation Problems
Efron, B., & Morris, C. N. (1973) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1973.tb00968.x · Cited by 192 · see citing works
Inference in an Authorship Problem
Mosteller, F. C., & Wallace, D. K. (1963) · Journal of the American Statistical Association · doi:10.2307/2283270 · Cited by 187 · see citing works
Topics in the Investigation of Linear Relations Fitted by the Method of Least Squares
Anscombe, F. J. (1967) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1967.tb00672.x · Cited by 141 · see citing works
Some Observations on Robust Estimation
Hogg, R. V. (1967) · Journal of the American Statistical Association · doi:10.1080/01621459.1967.10500924 · Cited by 140 · see citing works
The Generalised Jackknife Statistic.
Springall, A., Gray, H. L., & Schucany, W. R. (1974) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2347008 · Cited by 133 · see citing works
The Parameters of Cross-Validation
Herzberg, P. A. (1967) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 110 · see citing works
Testing Transformations to Normality
Atkinson, A. C. (1973) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1973.tb00975.x · Cited by 109 · see citing works
Validity, Reliability, and Baloney
Cureton, E. E. (1950) · Educational and Psychological Measurement · doi:10.1177/001316445001000107 · Cited by 100 · see citing works
Order Statistics Estimators of the Location of the Cauchy Distribution
BARNETT, V. D. (1966) · Journal of the American Statistical Association · doi:10.1080/01621459.1966.10482205 · Cited by 99 · see citing works
Inadmissibility of Maximum Likelihood Estimators in Some Multiple Regression Problems with Three or More Independent Variables
Baranchik, A. (1973) · The Annals of Statistics · doi:10.1214/aos/1176342368 · Cited by 82 · see citing works
Criteria for Selection of a Subset Regression: Which One Should Be Used?
Hocking, R. R. (1972) · Technometrics · doi:10.1080/00401706.1972.10488992 · Cited by 78 · see citing works
Polynomial Regression from a Bayesian Approach
Halpern, E. F. (1973) · Journal of the American Statistical Association · doi:10.1080/01621459.1973.10481352 · Cited by 42 · see citing works
Evaluation of odd zonal harmonics in the geopotential, of degree less than 33, from the analysis of 22 satellite orbits
King-Hele, D. G., Cook, G., & Scott, D. W. (1969) · Planetary and Space Science · doi:10.1016/0032-0633(69)90185-8 · Cited by 37 · see citing works
Refining the Earth's Pear Shape
King-Hele, D. G., & Cook, G. (1973) · Nature · doi:10.1038/246086a0 · Cited by 33 · see citing works
Statistical Techniques for Diagnosis
Teather, D. (1974) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344551 · Cited by 31 · see citing works
III. Cross-Validation of Item Analyses
Katzell, R. A. (1951) · Educational and Psychological Measurement · doi:10.1177/001316445101100103 · Cited by 26 · see citing works
Analysis of 27 satellite orbits to determine odd zonal harmonics in the geopotential
King-Hele, D. G., & Cook, G. (1974) · Planetary and Space Science · doi:10.1016/0032-0633(74)90138-x · Cited by 24 · see citing works
IV. Comparison of Cross-Validation with Statistical Inference of Betas and Multiple R From a Single Sample
Wherry, R. J. (1951) · Educational and Psychological Measurement · doi:10.1177/001316445101100104 · Cited by 22 · see citing works
II. Approximate Linear Restraints and Best Predictor Weights
Cureton, E. E. (1951) · Educational and Psychological Measurement · doi:10.1177/001316445101100102 · Cited by 21 · see citing works
Commentary on “Estimation of Error Rates in Discriminant Analysis”
Cochran, W. G. (1968) · Technometrics · doi:10.1080/00401706.1968.10490548 · Cited by 20 · see citing works
Asymptotically Robust Estimators of Location
Birnbaum, A., & Miké, V. (1970) · Journal of the American Statistical Association · doi:10.1080/01621459.1970.10481163 · Cited by 13 · see citing works
Commentary on "Estimation of Error Rates in Discriminant Analysis"
Cochran, W. G. (1968) · Technometrics · doi:10.2307/1266237 · Cited by 12 · see citing works

Flagged first, then by citations

Sorted by citations

When to Use and How to Report the Results of PLS-SEM
Hair, J. F., Risher, J. J., Sarstedt, M., & Ringle, C. M. (2019) · European Business Review · doi:10.1108/ebr-11-2018-0203 · Cited by 27,541 · see citing works
Pls-Sem: indeed a silver bullet
Hair, J. F., Ringle, C. M., & Sarstedt, M. (2011) · Journal of Marketing Theory and Practice · doi:10.2753/mtp1069-6679190202 · Cited by 21,631 · see citing works
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Bayesian Measures of Model Complexity and Fit
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van Der Linde, A. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00353 · Cited by 151 · see citing works
Multimodel Inference: Understanding AIC and BIC in Model Selection
Burnham, K. P., & Anderson, D. R. (2004) · Sociological Methods & Research · doi:10.1177/0049124104268644 · Cited by 11,648 · see citing works
Principal component analysis
Abdi, H., & Williams, L. J. (2010) · WIREs Computational Statistics · doi:10.1002/wics.101 · Cited by 11,104 · see citing works
An assessment of the use of partial least squares structural equation modeling in marketing research
Hair, J. F., Sarstedt, M., Ringle, C. M., & Mena, J. A. (2012) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-011-0261-6 · Cited by 8,088 · see citing works
Assessing measurement model quality in PLS-SEM using confirmatory composite analysis
Hair, J. F., Howard, M. C., & Nitzl, C. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2019.11.069 · Cited by 5,446 · see citing works
Predictive model assessment in PLS-SEM: guidelines for using PLSpredict
Shmueli, G., Sarstedt, M., Hair, J. F., Cheah, J.-H., Ting, H., Vaithilingam, S., & Ringle, C. M. (2019) · European Journal of Marketing · doi:10.1108/ejm-02-2019-0189 · Cited by 4,618 · see citing works
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · see citing works
Neural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578 · see citing works
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522 · see citing works
A leisurely look at the bootstrap, the jackknife, and cross-validation
Efron, B., & Gong, G. (1983) · The American Statistician · doi:10.1080/00031305.1983.10483087 · Cited by 3,229 · see citing works
The Predictive Sample Reuse Method with Applications
Geisser, S. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10479865 · Cited by 2,748 · see citing works
The collinearity problem in linear regression. The Partial Least Squares (PLS) approach to generalized inverses
Wold, S., Ruhe, A., Wold, H., & Dunn, III (1984) · SIAM Journal on Scientific and Statistical Computing · doi:10.1137/0905052 · Cited by 2,529 · see citing works
Estimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300 · see citing works
Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation
Rodriguez, J. D., Perez, A., & Lozano, J. A. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.187 · Cited by 1,890 · see citing works
Machine learning algorithm validation with a limited sample size
Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0224365 · Cited by 1,901 · see citing works
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851 · see citing works
Homo Heuristicus: Why Biased Minds Make Better Inferences
Gigerenzer, G., & Brighton, H. (2009) · Topics in Cognitive Science · doi:10.1111/j.1756-8765.2008.01006.x · Cited by 1,801 · see citing works
Linear Model Selection by Cross-validation
Shao, J. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476299 · Cited by 1,790 · see citing works
Introduction to multi-layer feed-forward neural networks
Svozil, D., Kvasnicka, V., & Pospichal, J. (1997) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/s0169-7439(97)00061-0 · Cited by 1,600 · see citing works
Using partial least squares in operations management research: a practical guideline and summary of past research
Peng, D. X., & Lai, F. (2012) · Journal of Operations Management · doi:10.1016/j.jom.2012.06.002 · Cited by 1,532 · see citing works
Estimation of prediction error by using K-fold cross-validation
Fushiki, T. (2011) · Statistics and Computing · doi:10.1007/s11222-009-9153-8 · Cited by 1,470 · see citing works
Assessment of PLSDA Cross Validation
Westerhuis, J. A., Hoefsloot, H. C. J., Smit, S., Vis, D. J., Smilde, A. K., van Velzen, E. J. J., van Duijnhoven, J. P. M., & van Dorsten, F. A. (2008) · Metabolomics · doi:10.1007/s11306-007-0099-6 · Cited by 1,471 · see citing works
Parametric empirical Bayes inference: theory and applications
Morris, C. N. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477920 · Cited by 1,448 · see citing works
An asymptotic equivalence of choice of model by cross validation and Akaike's criterion
Stone, M. (1977) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1977.tb01603.x · Cited by 1,286 · see citing works
Self-citation
Attention to eyes is present but in decline in 2–6-month-old infants later diagnosed with autism
Jones, W., & Klin, A. (2013) · Nature · doi:10.1038/nature12715 · Cited by 1,293 · see citing works
Validation of Regression Models: Methods and Examples
Snee, R. D. (1977) · Technometrics · doi:10.1080/00401706.1977.10489581 · Cited by 1,283 · see citing works
The multivariate calibration problem in chemistry solved by the PLS method
Wold, S., Martens, H., & Wold, H. (1983) · Lecture notes in mathematics · doi:10.1007/bfb0062108 · Cited by 1,185 · see citing works
Some aspects of the spline smoothing approach to non-parametric regression curve fitting
Silverman, B. W. (1985) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1985.tb01327.x · Cited by 1,117 · see citing works
Cross-validation pitfalls when selecting and assessing regression and classification models
Krstajic, D., Buturovic, L. J., Leahy, D. E., & Thomas, S. (2014) · Journal of Cheminformatics · doi:10.1186/1758-2946-6-10 · Cited by 1,090 · see citing works
The determinants of students' perceived learning outcomes and satisfaction in university online education: An empirical investigation
Eom, S. B., Wen, H. J., & Ashill, N. (2006) · Decision Sciences Journal of Innovative Education · doi:10.1111/j.1540-4609.2006.00114.x · Cited by 876 · see citing works
Machine learning in bioinformatics
Larrañaga, P., Calvo, B., Santana, R., Bielza, C., Galdiano, J., Inza, I., Lozano, J. A., Armañanzas, R., Santafé, G., Pérez, A., et al. (2006) · Briefings in Bioinformatics · doi:10.1093/bib/bbk007 · Cited by 867 · see citing works
On the emancipation of PLS-SEM: a commentary on Rigdon (2012)
Sarstedt, M., Ringle, C. M., Henseler, J., & Hair, J. F. (2014) · Long Range Planning · doi:10.1016/j.lrp.2014.02.007 · Cited by 758 · see citing works
Multiple instance learning: A survey of problem characteristics and applications
Carbonneau, M.-A., Cheplygina, V., Granger, E., & Gagnon, G. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.009 · Cited by 740 · see citing works
“The experiments were conducted using a nested cross-fold validation protocol [149].”
Method
Computing Bayes factors using thermodynamic integration.
Lartillot, N., & Philippe, H. (2006) · Systematic Biology · doi:10.1080/10635150500433722 · Cited by 668 · see citing works
An on-chip photonic deep neural network for image classification
Ashtiani, F., Geers, A. J., & Aflatouni, F. (2022) · Nature · doi:10.1038/s41586-022-04714-0 · Cited by 699 · see citing works
Predictive Analytics in Information Systems Research
Shmueli, G., & Koppius, O. (2010) · doi:10.2139/ssrn.1606674 · Cited by 676 · see citing works
Predictive value of statistical models.
Van Houwelingen, J. C., & Le Cessie, S. (1990) · Statistics in Medicine · doi:10.1002/sim.4780091109 · Cited by 621 · see citing works
Internal and external validation of predictive models: A simulation study of bias and precision in small samples
Steyerberg, E. W., Bleeker, S. E., Moll, H. A., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00047-7 · Cited by 611 · see citing works
A Large and Consistent Phylogenomic Dataset Supports Sponges as the Sister Group to All Other Animals
Simion, P., Philippe, H., Baurain, D., Jager, M., Richter, D. J., Di Franco, A., Roure, B., Satoh, N., Quéinnec, É., Ereskovsky, A., et al. (2017) · Current Biology · doi:10.1016/j.cub.2017.02.031 · Cited by 518 · see citing works
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584050 · Cited by 574 · see citing works
Translating stakeholder pressures into environmental performance – the mediating role of green HRM practices
Guerci, M., Longoni, A., & Luzzini, D. (2016) · The International Journal of Human Resource Management · doi:10.1080/09585192.2015.1065431 · Cited by 589 · see citing works
Purchase intention toward organic food among young consumers using theory of planned behavior: role of environmental concerns and environmental awareness
Ahmed, N., Li, C., Khan, A., Qalati, S. A., Naz, S., & Rana, F. (2021) · Journal of Environmental Planning and Management · doi:10.1080/09640568.2020.1785404 · Cited by 393 · see citing works
Understanding the impact of m-banking on individual performance: DeLone & McLean and TTF perspective
Tam, C., & Oliveira, T. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.03.016 · Cited by 383 · see citing works
Genomic data do not support comb jellies as the sister group to all other animals
Pisani, D., Pett, W., Dohrmann, M., Feuda, R., Rota-Stabelli, O., Philippe, H., Lartillot, N., & Wörheide, G. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1518127112 · Cited by 319 · see citing works
The NoSAS score for screening of sleep-disordered breathing: a derivation and validation study
Marti-Soler, H., Hirotsu, C., Marques-Vidal, P., Vollenweider, P., Waeber, G., Preisig, M., Tafti, M., Tufik, S. B., Bittencourt, L., Tufik, S., et al. (2016) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(16)30075-3 · Cited by 317 · see citing works
Improved Modeling of Compositional Heterogeneity Supports Sponges as Sister to All Other Animals
Feuda, R., Dohrmann, M., Pett, W., Philippe, H., Rota-Stabelli, O., Lartillot, N., Wörheide, G., & Pisani, D. (2017) · Current Biology · doi:10.1016/j.cub.2017.11.008 · Cited by 291 · see citing works
Predicting persuasion-induced behavior change from the brain
Falk, E. B., Berkman, E. T., Mann, T., Harrison, B., & Lieberman, M. D. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.0063-10.2010 · Cited by 317 · see citing works
Nested cross-validation when selecting classifiers is overzealous for most practical applications
Wainer, J., & Cawley, G. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115222 · Cited by 309 · see citing works
The predictive validation of ecological and environmental models
Power, M. (1993) · Ecological Modelling · doi:10.1016/0304-3800(93)90106-3 · Cited by 301 · see citing works
An ensemble system for automatic sleep stage classification using single channel EEG signal
Koley, B., & Dey, D. (2012) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2012.09.012 · Cited by 291 · see citing works
Siri, Alexa, and other digital assistants: a study of customer satisfaction with artificial intelligence applications
Brill, T. M., Munoz, L., & Miller, R. J. (2019) · Journal of Marketing Management · doi:10.1080/0267257x.2019.1687571 · Cited by 291 · see citing works
Applying uses and gratifications theory and social influence processes to understand students’ pervasive adoption of social networking sites: Perspectives from the Americas
Ifinedo, P. (2016) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2015.11.007 · Cited by 273 · see citing works
Deep metazoan phylogeny: When different genes tell different stories
Nosenko, T., Schreiber, F., Adamska, M., Adamski, M., Eitel, M., Hammel, J., Maldonado, M., Müller, W. E. G., Nickel, M., Schierwater, B., et al. (2013) · Molecular Phylogenetics and Evolution · doi:10.1016/j.ympev.2013.01.010 · Cited by 254 · see citing works
CRM to Social CRM: The Integration of New Technologies into Customer Relationship Management
Choudhury, M. M., & Harrigan, P. (2014) · Journal of Strategic Marketing · doi:10.1080/0965254x.2013.876069 · Cited by 263 · see citing works
A strategy for risk mitigation of antibodies with fast clearance
Hötzel, I., Theil, F.-P., Bernstein, L. J., Prabhu, S., Deng, R., Quintana, L., Lutman, J., Sibia, R., Chan, P., Bumbaca, D., et al. (2012) · mAbs · doi:10.4161/mabs.22189 · Cited by 245 · see citing works
The Three Pillars of Virtual Reality? Investigating the Roles of Immersion, Presence, and Interactivity
Mütterlein, J. (2018) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2018.174 · Cited by 238 · see citing works
“SEM results Using PLS-SEM, I evaluated relationships of the model [29], significance of effects via bootstrapping [20], predictive relevance [19; 50], convergent and discriminant validity [10; 16], and variance inflation factors [21], amongst others.”
Method
Artificial intelligence: Deep learning in oncological radiomics and challenges of interpretability and data harmonization
Papadimitroulas, P., Brocki, L., Christopher Chung, N., Marchadour, W., Vermet, F., Gaubert, L., Eleftheriadis, V., Plachouris, D., Visvikis, D., Kagadis, G. C., et al. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.03.009 · Cited by 226 · see citing works
Quantitative Structure–Selectivity Relationships in Enantioselective Catalysis: Past, Present, and Future
Zahrt, A. F., Athavale, S. V., & Denmark, S. E. (2020) · Chemical Reviews · doi:10.1021/acs.chemrev.9b00425 · Cited by 212 · see citing works
Consistent cross-validatory model-selection for dependent data: hv-block cross-validation
Racine, J. (2000) · Journal of Econometrics · doi:10.1016/s0304-4076(00)00030-0 · Cited by 205 · see citing works
Deep learning for small and big data in psychiatry
Koppe, G., Meyer-Lindenberg, A., & Durstewitz, D. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-020-0767-z · Cited by 189 · see citing works
Mortality Risk Score Prediction in an Elderly Population Using Machine Learning
Rose, S. (2013) · American Journal of Epidemiology · doi:10.1093/aje/kws241 · Cited by 193 · see citing works
Theory-Guided Machine Learning in Materials Science
Wagner, N., & Rondinelli, J. M. (2016) · Frontiers in Materials · doi:10.3389/fmats.2016.00028 · Cited by 188 · see citing works
Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial
Wilimitis, D., & Walsh, C. G. (2023) · JMIR AI · doi:10.2196/49023 · Cited by 185 · see citing works
BMI as a Modifiable Risk Factor for Type 2 Diabetes: Refining and Understanding Causal Estimates Using Mendelian Randomization
Corbin, L. J., Richmond, R. C., Wade, K. H., Burgess, S., Bowden, J., Smith, G. D., & Timpson, N. J. (2016) · Diabetes · doi:10.2337/db16-0418 · Cited by 184 · see citing works
Cross-Validation Visualized: A Narrative Guide to Advanced Methods
Allgaier, J., & Pryss, R. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6020065 · Cited by 176 · see citing works
Measuring the statistical validity of summary meta-analysis and meta-regression results for use in clinical practice
Willis, B. H., & Riley, R. D. (2017) · Statistics in Medicine · doi:10.1002/sim.7372 · Cited by 171 · see citing works
I tried a bunch of things: The dangers of unexpected overfitting in classification of brain data
Hosseini, M., Powell, M., Collins, J., Callahan-Flintoft, C., Jones, W., Bowman, H., & Wyble, B. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.09.036 · Cited by 165 · see citing works
Phylogenetic-signal dissection of nuclear housekeeping genes supports the paraphyly of sponges and the monophyly of Eumetazoa
Sperling, E. A., Peterson, K. J., & Pisani, D. (2009) · Molecular Biology and Evolution · doi:10.1093/molbev/msp148 · Cited by 172 · see citing works
Generative embedding for model-based classification of fMRI data
Brodersen, K. H., Schofield, T. M., Leff, A. P., Ong, C. S., Lomakina, E. I., Buhmann, J. M., & Stephan, K. E. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002079 · Cited by 165 · see citing works
Adoption of robust business analytics for product innovation and organizational performance: the mediating role of organizational data-driven culture
Chaudhuri, R., Chatterjee, S., Vrontis, D., & Thrassou, A. (2024) · Annals of Operations Research · doi:10.1007/s10479-021-04407-3 · Cited by 164 · see citing works
Multilevel data analysis of a crossover designed human nutritional intervention study
van Velzen, E. J. J., Westerhuis, J. A., van Duynhoven, J. P. M., van Dorsten, F. A., Hoefsloot, H. C. J., Jacobs, D. M., Smit, S., Draijer, R., Kroner, C. I., & Smilde, A. K. (2008) · Journal of Proteome Research · doi:10.1021/pr800145j · Cited by 161 · see citing works
Enterprise resource planning adoption and satisfaction determinants
Costa, C. J., Ferreira, E., Bento, F., & Aparicio, M. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.05.090 · Cited by 161 · see citing works
What makes consumers trust and adopt FinTech? An empirical investigation in China
Roh, T., Yang, Y. S., Xiao, S., & Park, B. I. (2024) · Electronic Commerce Research · doi:10.1007/s10660-021-09527-3 · Cited by 164 · see citing works
Converting freemium customers from free to premium—the role of the perceived premium fit in the case of music as a service
Wagner, T. M., Benlian, A., & Hess, T. (2014) · Electronic Markets · doi:10.1007/s12525-014-0168-4 · Cited by 131 · see citing works
Heterogeneous Models Place the Root of the Placental Mammal Phylogeny
Morgan, C. C., Foster, P. G., Webb, A. E., Pisani, D., McInerney, J. O., & O’Connell, M. J. (2013) · Molecular Biology and Evolution · doi:10.1093/molbev/mst117 · Cited by 123 · see citing works
Towards reinforcing telemedicine adoption amongst clinicians in Nigeria
Adenuga, K. I., Iahad, N. A., & Miskon, S. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.05.008 · Cited by 120 · see citing works
Deep learning-based classification of posttraumatic stress disorder and depression following trauma utilizing visual and auditory markers of arousal and mood
Schultebraucks, K., Yadav, V., Shalev, A. Y., Bonanno, G. A., & Galatzer-Levy, I. R. (2022) · Psychological Medicine · doi:10.1017/s0033291720002718 · Cited by 108 · see citing works
Stability
Yu, B. (2013) · Bernoulli · doi:10.3150/13-bejsp14 · Cited by 113 · see citing works
Effective sense-and-respond strategies: Mediating roles of exploratory and exploitative innovation
Ngo, L. V., Bucic, T., Sinha, A., & Lu, V. N. (2019) · Journal of Business Research · doi:10.1016/j.jbusres.2017.10.050 · Cited by 111 · see citing works
The influence of organizational external factors on construction risk management among Nigerian construction companies
Adeleke, A. Q., Bahaudin, A. Y., Kamaruddeen, A. M., Bamgbade, J. A., Salimon, M. G., Khan, M. W. A., & Sorooshian, S. (2018) · Safety and Health at Work · doi:10.1016/j.shaw.2017.05.004 · Cited by 107 · see citing works
Use of statistical methods to estimate body composition
Guo, S., Chumlea, W., & Cockram, D. (1996) · The American Journal of Clinical Nutrition · doi:10.1093/ajcn/64.3.428s · Cited by 99 · see citing works
Cross-Validation With Confidence
Lei, J. (2020) · Journal of the American Statistical Association · doi:10.1080/01621459.2019.1672556 · Cited by 93 · see citing works
Problematic internet use (PIU): Associations with the impulsive-compulsive spectrum. An application of machine learning in psychiatry
Ioannidis, K., Chamberlain, S. R., Treder, M. S., Kiraly, F., Leppink, E. W., Redden, S. A., Stein, D. J., Lochner, C., & Grant, J. E. (2016) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2016.08.010 · Cited by 90 · see citing works
Selecting Representative Data Sets
Borovicka, T., Jirina, M., Kordik, P., & Jiri, M. (2012) · Advances in Data Mining Knowledge Discovery and Applications · doi:10.5772/50787 · Cited by 94 · see citing works
Corporate social responsibility and sustainability performance measurement systems: implications for organizational performance
Asiaei, K., Bontis, N., Barani, O., & Jusoh, R. (2021) · Journal of Management Control · doi:10.1007/s00187-021-00317-4 · Cited by 91 · see citing works
Why continue sharing: Determinants of behavior in ridesharing services
Arteaga-Sánchez, R., Belda-Ruiz, M., Ros-Galvez, A., & Rosa-Garcia, A. (2020) · International Journal of Market Research · doi:10.1177/1470785318805300 · Cited by 84 · see citing works
Does It Matter Who Exhibits More Green Purchase Behavior of Cosmetic Products in Asian Culture? A Multi-Group Analysis Approach
Quoquab, F., Jaini, A., & Mohammad, J. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17145258 · Cited by 80 · see citing works
Checking and bootstrapping lexical norms by means of word similarity indexes
Bestgen, Y., & Vincze, N. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0195-z · Cited by 75 · see citing works
Do lean manufacturing practices have negative impact on job satisfaction?
Minh, K. S., Zailani, S., Iranmanesh, M., & Heidari, S. (2019) · International Journal of Lean Six Sigma · doi:10.1108/ijlss-11-2016-0072 · Cited by 77 · see citing works
Identifying an optimal cutpoint for the diagnosis of hypertriglyceridemia in the nonfasting state
White, K. T., Moorthy, M. V., Akinkuolie, A. O., Demler, O., Ridker, P. M., Cook, N. R., & Mora, S. (2015) · Clinical Chemistry · doi:10.1373/clinchem.2015.241752 · Cited by 74 · see citing works
A comprehensive study on factors influencing online impulse buying behavior: Evidence from Shopee video platform
Ngo, T. T. A., Nguyen, H. L. T., Nguyen, H. P., Mai, H. T. A., Mai, T. H. T., & Hoang, P. L. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e35743 · Cited by 73 · see citing works
Online social games: The effect of social comparison elements on continuance behaviour
Esteves, J., Valogianni, K., & Greenhill, A. (2021) · Information & Management · doi:10.1016/j.im.2021.103452 · Cited by 73 · see citing works
The influence of psychological, motivational, and behavioral factors on university students’ achievements: the mediating effect of academic adjustment
Raza, S. A., Qazi, W., & Yousufi, S. Q. (2021) · Journal of Applied Research in Higher Education · doi:10.1108/jarhe-03-2020-0065 · Cited by 70 · see citing works
Compulsive smartphone usage and users' ill-being among young Indians: Does personality matter?
Panda, A., & Jain, N. K. (2018) · Telematics and Informatics · doi:10.1016/j.tele.2018.03.006 · Cited by 67 · see citing works
Impact of firm's intellectual capital on firm performance: a study of Indian firms and the moderating effects of age and gender
Chatterjee, S., Chaudhuri, R., Thrassou, A., & Sakka, G. (2022) · Journal of Intellectual Capital · doi:10.1108/jic-12-2020-0378 · Cited by 67 · see citing works
The Impact of Digital Transformation on Supply Chain Procurement for Creating Competitive Advantage: An Empirical Study
Alabdali, M. A., & Salam, M. A. (2022) · Sustainability · doi:10.3390/su141912269 · Cited by 64 · see citing works
Members' site use continuance on Facebook: Examining the role of relational capital
Chen, R., Sharma, S. K., & Raghav Rao, H. (2016) · Decision Support Systems · doi:10.1016/j.dss.2016.07.001 · Cited by 63 · see citing works

Authors: Stone, M.
Affiliations: University College London (GB)
Corresponding author: Stone, M.
Venue: Journal of the Royal Statistical Society Series B: Statistical Methodology
Publisher: Oxford University Press (OUP)
Published: 1974-01-01
Volume: 36
Issue: 2
Pages: 111-133
ISSN: 1369-7412 (ISSN-L)
1467-9868
DOI: 10.1111/j.2517-6161.1974.tb00994.x
OpenAlex: W2112081648
Cited by: 10,881
References: 47
Citations by year: 2026: 812 · 2025: 680 · 2024: 654 · 2023: 758 · 2022: 711 · 2021: 828 · 2020: 638 · 2019: 542 · 2018: 436 · 2017: 376 · 2016: 340 · 2015: 350 · 2014: 314 · 2013: 297 · 2012: 285
FWCI: 17.01
Type: journal article
Language: en
Open access: closed
Licence: academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
Locations: Journal of the Royal Statistical Society Series B (Statistical Methodology) (publishedVersion)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
Keywords: cross-validation, statistical forecasting, drug prescribing, model selection, linear regression, analysis of variance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Cross validatory choice and assessment of statistical predictions

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks