Cited by 409Machine Learning and AlgorithmsMachine Learning and Data ClassificationStochastic Gradient Optimization Techniques

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation

Kearns, Michael, Ron, Dana·1999·Neural Computation

DOI 10.1162/089976699300016304

Open this paper in CoChat — free account

Abstract

In this article we prove sanity-check bounds for the error of the leave-oneout cross-validation estimate of the generalization error: that is, bounds showing that the worst-case error of this estimate is not much worse than that of the training error estimate. The name sanity check refers to the fact that although we often expect the leave-one-out estimate to perform considerably better than the training error estimate, we are here only seeking assurance that its performance will not be considerably worse. Perhaps surprisingly, such assurance has been given only for limited cases in the prior literature on cross-validation. Any nontrivial bound on the error of leave-one-out must rely on some notion of algorithmic stability. Previous bounds relied on the rather strong notion of hypothesis stability, whose application was primarily limited to nearest-neighbor and other local algorithms. Here we introduce the new and weaker notion of error stability and apply it to obtain sanity-check bounds for leave-one-out for other classes of learning algorithms, including training error minimization procedures and Bayesian algorithms. We also provide lower bounds demonstrating the necessity of some form of error stability for proving bounds on the error of the leave-one-out estimate, and the fact that for training error minimization algorithms, in the worst case such bounds must still depend on the Vapnik-Chervonenkis dimension of the hypothesis class.

Integrity signals

Registry

Crossref · 10.1162/089976699300016304

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Stochastic relaxation, gibbs distributions, and the bayesian restoration of images.
Geman, S., & Geman, D. (1984) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1984.4767596 · Cited by 17,949
Untitled work
(1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274
Estimation of Dependences Based on Empirical Data
Vapnik, V. N. (2006) · Information science and statistics · doi:10.1007/0-387-34239-7 · Cited by 2,252
Subset Selection in Regression
Miller, A. J. (2003) · Technometrics · doi:10.1198/tech.2003.s144 · Cited by 1,501
Decision theoretic generalizations of the PAC model for neural net and other learning applications
Haussler, D. H. (1992) · Information and Computation · doi:10.1016/0890-5401(92)90010-d · Cited by 845
Subset Selection in Regression.
RAAB, G. M., & Miller, T. (1991) · Biometrics · doi:10.2307/2532418 · Cited by 547
Statistical mechanics of learning from examples
Seung, H. S., Sompolinsky, H., & Tishby, N. (1992) · Physical Review A · doi:10.1103/physreva.45.6056 · Cited by 532
Toward efficient agnostic learning
Kearns, M. J., Schapire, R. E., & Sellie, L. M. (1992) · Conference on Learning Theory · doi:10.1145/130385.130424 · Cited by 353
Self-citation

Show all 18 references

Sorted by citations

A survey of cross-validation procedures for model selectionPreprint
Arlot, S., & Célisse, A. (2009) · Cited by 3,627
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522
Adversarial machine learning
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194
What can we learn privately?
Kasiviswanathan, S. P., Lee, H. K., Nissim, K., Raskhodnikova, S., & Smith, A. (2011) · SIAM Journal on Computing · doi:10.1137/090756090 · Cited by 898
No Unbiased Estimator of the Variance of K-Fold Cross-ValidationPreprint
Bengio, Y., & Grandvalet, Y. (2003) · RePEc: Research Papers in Economics · Cited by 886
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646
Train faster, generalize better: stability of stochastic gradient descent
Hardt, M., Recht, B., & Singer, Y. (2016) · International Conference on Machine Learning · Cited by 541
Theory of Classification: a Survey of Some Recent Advances
Boucheron, S., Bousquet, O., & Lugosi, G. (2005) · ESAIM Probability and Statistics · doi:10.1051/ps:2005018 · Cited by 481
Stronger generalization bounds for deep nets via a compression approachPreprint
Arora, S., Ge, R., Neyshabur, B., & Zhang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05296 · Cited by 432
Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification
Frome, A., Singer, Y., Sha, F., & Malik, J. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408839 · Cited by 348
Automated detection of brain atrophy patterns based on MRI for the prediction of Alzheimer's disease
Plant, C., Teipel, S. J., Oswald, A., Böhm, C., Meindl, T., Mourão-Miranda, J., Bokde, A. L. W., Hampel, H. J., & Ewers, M. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.046 · Cited by 353
Elevated Triglycerides Correlate With Progression of Diabetic Neuropathy
Wiggin, T. D., Sullivan, K. A., Pop‐Busui, R., Amato, A., Sima, A. A. F., & Feldman, E. L. (2009) · Diabetes · doi:10.2337/db08-1771 · Cited by 346
What Can We Learn Privately?
Kasiviswanathan, S. P., Lee, H. K., Kobbi Nissim, B., Raskhodnikova, S., & Smith, A. (2008) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2008.27 · Cited by 373
Mobile Sensing and Support for People With Depression: A Pilot Trial in the Wild
Wahle, F., Kowatsch, T., Fleisch, E., Rufer, M., & Weidt, S. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.5960 · Cited by 333
Train faster, generalize better: Stability of stochastic gradient descentPreprint
Hardt, M., Recht, B., & Singer, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01240 · Cited by 431
Learnability, Stability and Uniform Convergence
Shalev‐Shwartz, S., Shamir, O., Srebro, N., & Sridharan, K. (2010) · doi:10.5555/1756006.1953019 · Cited by 328
Machine learning-driven new material discovery
Cai, J., Chu, X., Xu, K., Li, H., & Wei, J. (2020) · Nanoscale Advances · doi:10.1039/d0na00388c · Cited by 315
libPLS: An integrated library for partial least squares regression and linear discriminant analysis
Li, H., Xu, Q., & Liang, Y. (2018) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2018.03.003 · Cited by 295
Exhaled volatile organic compounds identify patients with colorectal cancer
Altomare, D. F., Lena, M. D., Porcelli, F., de Trizio, L., Travaglio, E., Tutino, M., Dragonieri, S., Memeo, V., & de Gennaro, G. (2012) · British journal of surgery · doi:10.1002/bjs.8942 · Cited by 289
General conditions for predictivity in learning theory
Poggio, T., Rifkin, R. M., Mukherjee, S., & Niyogi, P. (2004) · Nature Cell Biology · doi:10.1038/nature02341 · Cited by 285
Beating the hold-out: bounds for K-fold and progressive cross-validation
Blum, A., Kalai, A., & Langford, J. (1999) · Proceedings of the twelfth annual conference on Computational learning theory · doi:10.1145/307400.307439 · Cited by 265
Learning in a Large Function Space: Privacy-Preserving Mechanisms for SVM Learning
Rubinstein, B. I. P., Bartlett, P. L., Huang, L., & Taft, N. (2012) · Journal of Privacy and Confidentiality · doi:10.29012/jpc.v4i1.612 · Cited by 298
Measuring the prediction error. A comparison of cross-validation, bootstrap and covariance penalty methods
Borra, S., & Ciaccio, A. D. (2010) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2010.03.004 · Cited by 236
Revealing Real-Time Emotional Responses: a Personalized Assessment based on Heartbeat Dynamics
Valenza, G., Citi, L., Lanatà, A., Scilingo, E. P., & Barbieri, R. (2014) · Scientific Reports · doi:10.1038/srep04998 · Cited by 234
Monitoring of Wheat Growth Status and Mapping of Wheat Yield’s within-Field Spatial Variations Using Color Images Acquired from UAV-camera System
Du, M., & Noguchi, N. (2017) · Remote Sensing · doi:10.3390/rs9030289 · Cited by 223
Recognizing Emotions Induced by Affective Sounds through Heart Rate Variability
Nardelli, M., Valenza, G., Greco, A., Lanatà, A., & Scilingo, E. P. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2432810 · Cited by 227
Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization
Mukherjee, S., Niyogi, P., Poggio, T., & Rifkin, R. M. (2006) · Advances in Computational Mathematics · doi:10.1007/s10444-004-7634-z · Cited by 198
High-density surface EMG maps from upper-arm and forearm muscles
Rojas-Martínez, M., Mañanas, M. Á., & Alonso, J. F. (2012) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-9-85 · Cited by 181
A comparative study of K-Nearest Neighbour, Support Vector Machine and Multi-Layer Perceptron for Thalassemia screening
Amendolia, S. R., Cossu, G. F., Ganadu, M. L., Golosio, B., Masala, G. L., & Mura, G. M. (2003) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/s0169-7439(03)00094-7 · Cited by 173
Wearable Monitoring for Mood Recognition in Bipolar Disorder Based on History-Dependent Long-Term Heart Rate Variability Analysis
Valenza, G., Nardelli, M., Lanatà, A., Gentili, C., Bertschy, G., Paradiso, R., & Scilingo, E. P. (2013) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2013.2290382 · Cited by 166
Conformal Prediction for Reliable Machine Learning: Theory, Adaptations and Applications
Balasubramanian, V. N., Ho, S.-S., & Vovk, V. (2014) · Cited by 155
Algorithmic stability for adaptive data analysis
Bassily, R., Kobbi Nissim, B., Smith, A., Steinke, T., Stemmer, U., & Ullman, J. (2016) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/2897518.2897566 · Cited by 179
Disjunctions of Conjunctions, Cognitive Simplicity, and Consideration Sets
Hauser, J. R., Toubia, O., Evgeniou, T., Befurt, R., & Dzyabura, D. (2010) · Journal of Marketing Research · doi:10.1509/jmkr.47.3.485 · Cited by 154
Synthesizing cognition in neuromorphic electronic systems
Neftci, E., Binas, J., Rutishauser, U., Chicca, E., Indiveri, G., & Douglas, R. J. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212083110 · Cited by 153
Permutation Tests for Classification
Golland, P., Liang, F., Mukherjee, S., & Panchenko, D. (2005) · Lecture notes in computer science · doi:10.1007/11503415_34 · Cited by 139
Generalization Bounds for Ranking Algorithms via Algorithmic Stability
Agarwal, S., & Niyogi, P. (2009) · doi:10.5555/1577069.1577085 · Cited by 131
Leave One Out Error, Stability, and Generalization of Voting Combinations of Classifiers
Evgeniou, T., Pontil, M., & Elisseeff, A. (2004) · Machine Learning · doi:10.1023/b:mach.0000019805.88351.60 · Cited by 132
Algorithm-Dependent Generalization Bounds for Multi-Task Learning
Liu, T., Tao, D., Song, M., & Maybank, S. J. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2544314 · Cited by 131
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646
Low Cost Sensor With IoT LoRaWAN Connectivity and Machine Learning-Based Calibration for Air Pollution Monitoring
Ali, S., Glass, T., Parr, B., Potgieter, J., & Alam, F. (2020) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3034109 · Cited by 129
Machine Learning Based File Entropy Analysis for Ransomware Detection in Backup Systems
Lee, K., Lee, S., & Yim, K. (2019) · IEEE Access · doi:10.1109/access.2019.2931136 · Cited by 128
Stability of Randomized Learning Algorithms
Elisseeff, A., Evgeniou, T., & Pontil, M. (2005) · Cited by 116
Arousal and Valence Recognition of Affective Sounds Based on Electrodermal Activity
Greco, A., Valenza, G., Citi, L., & Scilingo, E. P. (2016) · IEEE Sensors Journal · doi:10.1109/jsen.2016.2623677 · Cited by 121
Analysis of Variance of Cross-Validation Estimators of the Generalization Error
Markatou, M., Tian, H., Biswas, S., & Hripcsak, G. M. (2005) · doi:10.7916/d86d5r2x · Cited by 114
Stability
Yu, B. (2013) · Bernoulli · doi:10.3150/13-bejsp14 · Cited by 113
Foundations of Machine Learning
Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018) · doi:10.7551/mitpress/10290.001.0001 · Cited by 1,122
A Robust Motion Artifact Detection Algorithm for Accurate Detection of Heart Rates From Photoplethysmographic Signals Using Time–Frequency Spectral Features
Dao, D. K., Salehizadeh, S. M. A., Noh, Y., Chong, J. W., Cho, C. H., McManus, D., Darling, C. E., Mendelson, Y., & Chon, K. H. (2016) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2612059 · Cited by 104
Prediction of total viable counts on chilled pork using an electronic nose combined with support vector machine
Wang, D., Wang, X., Liu, T., & Liu, Y. (2011) · Meat Science · doi:10.1016/j.meatsci.2011.07.025 · Cited by 101
Discriminative analysis of schizophrenia using support vector machine and recursive feature elimination on structural MRI images
Lu, X., Yang, Y., Wu, F., Gao, M., Xu, Y., Zhang, Y., Yao, Y., Du, X., Li, C., Wu, L., et al. (2016) · Medicine · doi:10.1097/md.0000000000003973 · Cited by 101
MRI-based clinical-radiomics model predicts tumor response before treatment in locally advanced rectal cancer
Pizzi, A. D., Chiarelli, A. M., Chiacchiaretta, P., d’Annibale, M., Croce, P., Rosa, C., Mastrodicasa, D., Trebeschi, S., Lambregts, D. M. J., Caposiena, D., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-84816-3 · Cited by 99
Leave-One-Out Bounds for Kernel Methods
Zhang, T. (2003) · Neural Computation · doi:10.1162/089976603321780326 · Cited by 95
The Nearest Neighbor Algorithm of Local Probability Centers
Li, B., Chen, Y. W., & Chen, Y. Q. (2008) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2007.908363 · Cited by 95
Concept Drift Detection Through Resampling
Harel, M., Mannor, S., El‐Yaniv, R., & Crammer, K. (2014) · Cited by 92
Supervised Machine Learning Methods and Hyperspectral Imaging Techniques Jointly Applied for Brain Cancer Classification
Urbanos, G., Martín-Pérez, A., Vázquez, G., Villanueva, M., Villa, M., Roldan, L. J., Chavarrías, M., Lagares, A., Juárez, E., & Sanz, C. (2021) · Sensors · doi:10.3390/s21113827 · Cited by 91
On the Existence of Simpler Machine Learning Models
Semenova, L., Rudin, C., & Parr, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533232 · Cited by 90
Molecular Dynamics Simulation, Free Energy Calculation and Structure-Based 3D-QSAR Studies of B-RAF Kinase Inhibitors
Ying, Y., Qin, J., Liu, H., & Yao, X. (2011) · Journal of Chemical Information and Modeling · doi:10.1021/ci100427j · Cited by 90
Application of Machine Learning Approaches for Classifying Sitting Posture Based on Force and Acceleration Sensors
Zemp, R., Tanadini, M., Plüss, S., Schnüriger, K., Singh, N. B., Taylor, W. R., & Lorenzetti, S. (2016) · BioMed Research International · doi:10.1155/2016/5978489 · Cited by 88
Combined network analysis and machine learning allows the prediction of metabolic pathways from tomato metabolomics data
Toubiana, D., Puzis, R., Wen, L., Sikron, N., Kurmanbayeva, A., Soltabayeva, A., Wilhelmi, M. D. M. R., Sade, N., Fait, A., Sagi, M., et al. (2019) · Communications Biology · doi:10.1038/s42003-019-0440-4 · Cited by 89
Insulin resistance and inflammation predict kinetic body weight changes in response to dietary weight loss and maintenance in overweight and obese subjects by using a Bayesian network approach
Kong, L. C., Wuillemin, P.-H., Bastard, J., Sokolovska, N., Gougis, S., Fellahi, S., Darakhshan, F., Bonnefont‐Rousselot, D., Bittar, R., Dore, J. M., et al. (2013) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.113.058099 · Cited by 92
Algorithmic Stability and Generalization Performance
Bousquet, O., Elisseeff, A., & Bioinformatics, B. (2000) · Cited by 87
Machine Learning in Injection Molding: An Industry 4.0 Method of Quality Prediction
Párizs, R. D., Török, D., Ageyeva, T., & Kovács, J. G. (2022) · Sensors · doi:10.3390/s22072704 · Cited by 90
Forming a new small sample deep learning model to predict total organic carbon content by combining unsupervised learning with semisupervised learning
Zhu, L., Zhang, C., Zhang, Z., Zhang, Z., Nie, X., Zhou, X., Liu, W., & Wang, X. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.105596 · Cited by 84
Just-in-Time Adaptive Classifiers—Part II: Designing the Classifier
Alippi, C., & Roveri, M. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2003998 · Cited by 82
The Optimal Phenological Phase of Maize for Yield Prediction with High-Frequency UAV Remote Sensing
Yang, B. T., Zhu, W., Rezaei, E. E., Li, J., Sun, Z., & Zhang, J. (2022) · Remote Sensing · doi:10.3390/rs14071559 · Cited by 84
Identification of isometric contractions based on High Density EMG maps
Rojas-Martínez, M., Mañanas, M. Á., Alonso, J. F., & Merletti, R. (2012) · Journal of Electromyography and Kinesiology · doi:10.1016/j.jelekin.2012.06.009 · Cited by 80
Stability and Generalization of Bipartite Ranking Algorithms
Agarwal, S., & Niyogi, P. (2005) · Lecture notes in computer science · doi:10.1007/11503415_3 · Cited by 77
An adsorbed gas estimation model for shale gas reservoirs via statistical learning
Chen, Y., Jiang, S. Q., Zhang, D., & Liu, C. (2017) · Applied Energy · doi:10.1016/j.apenergy.2017.04.029 · Cited by 75
Learning with Differential Privacy: Stability, Learnability and the Sufficiency and Necessity of ERM PrinciplePreprint
Wang, Y.-X., Lei, J., & Fienberg, S. E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.06309 · Cited by 79
Radiomics and radiogenomics in ovarian cancer: a literature review
Nougaret, S., McCague, C., Tibermacine, H., Vargas, H. A., Rizzo, S., & Sala, E. (2020) · Abdominal Radiology · doi:10.1007/s00261-020-02820-z · Cited by 73
Modeling multivariate distribution of multiple soil parameters using vine copula model
Lü, T.-J., Tang, X.-S., Li, D. Q., & Qi, X. (2019) · Computers and Geotechnics · doi:10.1016/j.compgeo.2019.103340 · Cited by 72
Information-theoretic analysis of stability and bias of learning algorithms
Raginsky, M., Rakhlin, A., Tsao, M., Wu, Y., & Xu, A. (2016) · 2016 IEEE Information Theory Workshop (ITW) · doi:10.1109/itw.2016.7606789 · Cited by 70
Prediction of GFP spectral properties using artificial neural network
Nantasenamat, C., Isarankura‐Na‐Ayudhya, C., Tansila, N., Naenna, T., & Prachayasittikul, V. (2007) · Journal of Computational Chemistry · doi:10.1002/jcc.20656 · Cited by 70
High probability generalization bounds for uniformly stable algorithms with nearly optimal ratePreprint
Feldman, V., & Vondrák, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10710 · Cited by 79
A Review of Matched-pairs Feature Selection Methods for Gene Expression Data Analysis
Liang, S., Ma, A., Yang, S., Wang, Y., & Ma, Q. (2018) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2018.02.005 · Cited by 68
Adversarial Machine Learning
Joseph, A. D., Nelson, B. A., Rubinstein, B. I. P., & Tygar, J. D. (2019) · Cambridge University Press eBooks · doi:10.1017/9781107338548 · Cited by 71
Feasibility study of gross calorific value, carbon content, volatile matter content and ash content of solid biomass fuel using laser-induced breakdown spectroscopy
Lü, Z., Chen, X., Yao, S., Qin, H., Zhang, L., Yao, X., Yu, Z., & Lu, J. (2019) · Fuel · doi:10.1016/j.fuel.2019.116150 · Cited by 65
Active Class Selection
Lomasky, R., Brodley, C. E., Aernecke, M. J., Walt, D. R., & Friedl, M. A. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74958-5_63 · Cited by 64
The need for external validation in machine olfaction: emphasis on health-related applications
Marco, S. (2014) · Analytical and Bioanalytical Chemistry · doi:10.1007/s00216-014-7807-7 · Cited by 62
Detecting middle ear fluid using smartphones
Chan, J., Raju, S. C., Nandakumar, R., Bly, R. A., & Gollakota, S. (2019) · Science Translational Medicine · doi:10.1126/scitranslmed.aav1102 · Cited by 61
Quantifying Uncertainty in Machine Learning-Based Power Outage Prediction Model Training: A Tool for Sustainable Storm Restoration
Yang, F., Wanik, D. W., Cerrai, D., Bhuiyan, M. A. E., & Anagnostou, E. N. (2020) · Sustainability · doi:10.3390/su12041525 · Cited by 60
The Pixels and Sounds of Emotion: General-Purpose Representations of Arousal in Games
Makantasis, K., Liapis, A., & Yannakakis, G. N. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3060877 · Cited by 60
An artificial neural network method for lumen and media-adventitia border detection in IVUS
Su, S., Hu, Z., Lin, Q., Hau, W. K., Gao, Z., & Zhang, H. (2016) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2016.11.003 · Cited by 58
Evaluating the reliability of neurocognitive biomarkers of neurodegenerative diseases across countries: A machine learning approach
Bachli, M. B., Sedeño, L., Ochab, J. K., Piguet, O., Kumfor, F., Reyes, P., Torralva, T., Roca, M., Cardona, J. F., Campo, C. G., et al. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2019.116456 · Cited by 57
Second harmonic generation microscopy provides accurate automated staging of liver fibrosis in patients with non-alcoholic fatty liver disease
Chang, J. P.-E., Goh, G. B., Leow, W. Q., Shen, L., Lim, K. H., & Tan, C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199166 · Cited by 57
Edge deep learning for neural implants: a case study of seizure detection and prediction
Liu, X., & Richardson, A. G. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/abf473 · Cited by 60
Effects of Curative Colorectal Cancer Surgery on Exhaled Volatile Organic Compounds and Potential Implications in Clinical Follow-up
Altomare, D. F., Lena, M. D., Porcelli, F., Travaglio, E., Longobardi, F., Tutino, M., Depalma, N., Tedesco, G., Sardaro, A., Memeo, R., et al. (2015) · Annals of Surgery · doi:10.1097/sla.0000000000001471 · Cited by 55
Design process and preliminary psychometric study of a video game to detect cognitive impairment in senior adults
Valladares‐Rodríguez, S., Perez-Rodriguez, R., Facal-Mayo, D., Iglesias, M. J. F., Anido, L., & Mouriño-García, M. (2017) · PeerJ · doi:10.7717/peerj.3508 · Cited by 57
What Can We Learn Privately?Preprint
Kasiviswanathan, S. P., Lee, H. K., Kobbi Nissim, B., Raskhodnikova, S., & Smith, A. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0803.0924 · Cited by 54
Automated Analysis of Optic Nerve Images for Detection and Staging of Papilledema
Echegaray, S., Zamora, G., Yu, H., Luo, W., Solíz, P., & Kardon, R. H. (2011) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.11-7484 · Cited by 53
Bias in Estimating the Variance of K-Fold Cross-Validation
Bengio, Y., & Grandvalet, Y. (2005) · Springer eBooks · doi:10.1007/0-387-24555-3_5 · Cited by 52
Genomic variation in myeloma: design, content, and initial application of the Bank On A Cure SNP Panel to detect associations with progression-free survival
Ness, B. V., Ramos, C., Haznadar, M., Hoering, A., Haessler, J., Crowley, J. J., Jacobus, S. J., Oken, M. M., Rajkumar, S. V., Greipp, P. R., et al. (2008) · BMC Medicine · doi:10.1186/1741-7015-6-26 · Cited by 52
QSAR analysis of coumarin-based benzamides as histone deacetylase inhibitors using CoMFA, CoMSIA and HQSAR methods
Abdizadeh, R., Hadizadeh, F., & Abdizadeh, T. (2019) · Journal of Molecular Structure · doi:10.1016/j.molstruc.2019.126961 · Cited by 50
Estimating the Confidence Interval for Prediction Errors of Support Vector Machine Classifiers
Jiang, B., Zhang, X. H.-F., & Cai, T. (2008) · Cited by 50
Modeling the effects of litter stoichiometry and soil mineral N availability on soil organic matter formation using CENTURY-CUE (v1.0)
Zhang, H., Goll, D. S., Manzoni, S., Ciais, P., Guenet, B., & Huang, Y. (2018) · Geoscientific model development · doi:10.5194/gmd-11-4779-2018 · Cited by 47
On-Average KL-Privacy and Its Equivalence to Generalization for Max-Entropy Mechanisms
Wang, Y.-X., Lei, J., & Fienberg, S. E. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45381-1_10 · Cited by 48
Brake fault diagnosis using Clonal Selection Classification Algorithm (CSCA) – A statistical learning approach
Jegadeeshwaran, R., & Sugumaran, V. (2014) · Engineering Science and Technology an International Journal · doi:10.1016/j.jestch.2014.08.001 · Cited by 47
A new approach for the estimation of phytoplankton cell counts associated with algal blooms
Nazeer, M., Wong, M. S., & Nichol, J. E. (2017) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2017.02.182 · Cited by 46
Sensor Arrays, Virtual Multisensors, Data Fusion, and Gas Sensor Data Evaluation
Reimann, P., & Schütze, A. (2013) · Springer series on chemical sensors and biosensors · doi:10.1007/5346_2013_52 · Cited by 45
Selection of suitable hand gestures for reliable myoelectric human computer interface
Castro, M. C. F., Arjunan, S. P., & Kumar, D. (2015) · BioMedical Engineering OnLine · doi:10.1186/s12938-015-0025-5 · Cited by 43
Probabilistic Mangrove Species Mapping with Multiple-Source Remote-Sensing Datasets Using Label Distribution Learning in Xuan Thuy National Park, Vietnam
Xia, J., Yokoya, N., & Pham, T. D. (2020) · Remote Sensing · doi:10.3390/rs12223834 · Cited by 42

Authors: Kearns, M., & Ron, D.
Affiliations: AT&T (United States) (US)
Tel Aviv University (IL)
ORCID: Kearns, Michael
Ron, Dana
Venue: Neural Computation
Publisher: MIT Press - Journals
Published: 1999-08-01
Volume: 11
Issue: 6
Pages: 1427-1453
ISSN: 0899-7667 (ISSN-L)
1530-888X
DOI: 10.1162/089976699300016304
OpenAlex: W2138263042
Cited by: 407
References: 18
Citations by year: 2026: 4 · 2025: 7 · 2024: 11 · 2023: 18 · 2022: 22 · 2021: 26 · 2020: 35 · 2019: 46 · 2018: 23 · 2017: 22 · 2016: 27 · 2015: 18 · 2014: 17 · 2013: 24 · 2012: 14
FWCI: 4.55
Type: journal article
Language: en
Open access: closed
Locations: Neural Computation (publishedVersion)
PubMed (publishedVersion)
Funders: National Science Foundation
Office of Naval Research
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
Keywords: leave-one-out cross-validation, generalization error estimation, algorithm stability, VC dimension, lower bounds
MeSH terms: Algorithms, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks