Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation
Abstract
In this article we prove sanity-check bounds for the error of the leave-oneout cross-validation estimate of the generalization error: that is, bounds showing that the worst-case error of this estimate is not much worse than that of the training error estimate. The name sanity check refers to the fact that although we often expect the leave-one-out estimate to perform considerably better than the training error estimate, we are here only seeking assurance that its performance will not be considerably worse. Perhaps surprisingly, such assurance has been given only for limited cases in the prior literature on cross-validation. Any nontrivial bound on the error of leave-one-out must rely on some notion of algorithmic stability. Previous bounds relied on the rather strong notion of hypothesis stability, whose application was primarily limited to nearest-neighbor and other local algorithms. Here we introduce the new and weaker notion of error stability and apply it to obtain sanity-check bounds for leave-one-out for other classes of learning algorithms, including training error minimization procedures and Bayesian algorithms. We also provide lower bounds demonstrating the necessity of some form of error stability for proving bounds on the error of the leave-one-out estimate, and the fact that for training error minimization algorithms, in the worst case such bounds must still depend on the Vapnik-Chervonenkis dimension of the hypothesis class.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Leave-one-out cross validation was conducted for unbiased performance estimation of both classifiers.
“For unbiased performance estimation of both classifiers, leave-one-out cross validation was conducted [51].”
From Mobile Sensing and Support for People With Depression: A Pilot Trial in the Wild (2016) · cites this paper as [51]
Cites
Stochastic relaxation, gibbs distributions, and the bayesian restoration of images.
Geman, S., & Geman, D. (1984) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1984.4767596 · Cited by 17,949 · see citing works
Pattern classification and scene analysis
Duda, R. O., & Hart, P. E. (1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895 · see citing works
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909 · see citing works
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · see citing works
Estimation of Dependences Based on Empirical Data
Vapnik, V. N. (2006) · Information science and statistics · doi:10.1007/0-387-34239-7 · Cited by 2,252 · see citing works
Subset Selection in Regression
Miller, A. J. (2003) · Technometrics · doi:10.1198/tech.2003.s144 · Cited by 1,501 · see citing works
Decision theoretic generalizations of the PAC model for neural net and other learning applications
Haussler, D. H. (1992) · Information and Computation · doi:10.1016/0890-5401(92)90010-d · Cited by 845 · see citing works
Subset Selection in Regression.
RAAB, G. M., & Miller, T. (1991) · Biometrics · doi:10.2307/2532418 · Cited by 547 · see citing works
Statistical mechanics of learning from examples
Seung, H. S., Sompolinsky, H., & Tishby, N. (1992) · Physical Review A · doi:10.1103/physreva.45.6056 · Cited by 512 · see citing works
Toward efficient agnostic learning
Kearns, M. J., Schapire, R. E., & Sellie, L. M. (1992) · Conference on Learning Theory · doi:10.1145/130385.130424 · Cited by 353 · see citing works
Self-citationDistribution-free performance bounds for potential function rules
Devroye, L., & Wagner, T. J. (1979) · IEEE Transactions on Information Theory · doi:10.1109/tit.1979.1056087 · Cited by 233 · see citing works
Toward Efficient Agnostic Learning
Kearns, M. J., Schapire, R. E., & Sellie, L. M. (1994) · Machine Learning · doi:10.1007/bf00993468 · Cited by 172 · see citing works
Self-citationAn experimental and theoretical comparison of model selection methods
Kearns, M. J., Mansour, Y., Ng, A. Y., & Ron, D. (1995) · Conference on Learning Theory · doi:10.1145/225298.225301 · Cited by 167 · see citing works
Self-citationA Finite Sample Distribution-Free Performance Bound for Local Discrimination Rules
Rogers, W. H., & Wagner, T. J. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344196 · Cited by 135 · see citing works
Distribution-free inequalities for the deleted and holdout error estimates
Devroye, L., & Wagner, T. J. (1979) · IEEE Transactions on Information Theory · doi:10.1109/tit.1979.1056032 · Cited by 120 · see citing works
Rigorous learning curve bounds from statistical mechanics
Haussler, D. H., Seung, H. S., Kearns, M. J., & Tishby, N. (1994) · Conference on Learning Theory · doi:10.1145/180139.181018 · Cited by 96 · see citing works
Self-citationPAC-like upper bounds for the sample complexity of leave-one-out cross-validation
Holden, S. B. (1996) · Annual Conference on Computational Learning Theory (COLT) · doi:10.1145/238061.238067 · Cited by 19 · see citing works
Rigorous Learning Curve Bounds from Statistical Mechanics
Haussler, D. H., Kearns, M. J., Seung, H. S., & Tishby, N. (1996) · Machine Learning · doi:10.1023/a:1026499208981 · Cited by 19 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
A survey of cross-validation procedures for model selectionPreprint
Arlot, S., & Célisse, A. (2009) · Cited by 3,627 · see citing works
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522 · see citing works
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194 · see citing works
Kasiviswanathan, S. P., Lee, H. K., Nissim, K., Raskhodnikova, S., & Smith, A. (2011) · SIAM Journal on Computing · doi:10.1137/090756090 · Cited by 898 · see citing works
No Unbiased Estimator of the Variance of K-Fold Cross-ValidationPreprint
Bengio, Y., & Grandvalet, Y. (2003) · RePEc: Research Papers in Economics · Cited by 886 · see citing works
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · see citing works
Train faster, generalize better: stability of stochastic gradient descent
Hardt, M., Recht, B., & Singer, Y. (2016) · International Conference on Machine Learning · Cited by 516 · see citing works
Theory of Classification: a Survey of Some Recent Advances
Boucheron, S., Bousquet, O., & Lugosi, G. (2005) · ESAIM Probability and Statistics · doi:10.1051/ps:2005018 · Cited by 442 · see citing works
Stronger generalization bounds for deep nets via a compression approachPreprint
Arora, S., Ge, R., Neyshabur, B., & Zhang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05296 · Cited by 359 · see citing works
Frome, A., Singer, Y., Sha, F., & Malik, J. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408839 · Cited by 348 · see citing works
Automated detection of brain atrophy patterns based on MRI for the prediction of Alzheimer's disease
Plant, C., Teipel, S. J., Oswald, A., Böhm, C., Meindl, T., Mourão-Miranda, J., Bokde, A. L. W., Hampel, H. J., & Ewers, M. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.046 · Cited by 341 · see citing works
Elevated Triglycerides Correlate With Progression of Diabetic Neuropathy
Wiggin, T. D., Sullivan, K. A., Pop‐Busui, R., Amato, A., Sima, A. A. F., & Feldman, E. L. (2009) · Diabetes · doi:10.2337/db08-1771 · Cited by 342 · see citing works
Kasiviswanathan, S. P., Lee, H. K., Kobbi Nissim, B., Raskhodnikova, S., & Smith, A. (2008) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2008.27 · Cited by 349 · see citing works
Mobile Sensing and Support for People With Depression: A Pilot Trial in the Wild
Wahle, F., Kowatsch, T., Fleisch, E., Rufer, M., & Weidt, S. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.5960 · Cited by 333 · see citing works
Train faster, generalize better: Stability of stochastic gradient descentPreprint
Hardt, M., Recht, B., & Singer, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01240 · Cited by 358 · see citing works
Learnability, Stability and Uniform Convergence
Shalev‐Shwartz, S., Shamir, O., Srebro, N., & Sridharan, K. (2010) · Cited by 304 · see citing works
Machine learning-driven new material discovery
Cai, J., Chu, X., Xu, K., Li, H., & Wei, J. (2020) · Nanoscale Advances · doi:10.1039/d0na00388c · Cited by 303 · see citing works
libPLS: An integrated library for partial least squares regression and linear discriminant analysis
Li, H., Xu, Q., & Liang, Y. (2018) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2018.03.003 · Cited by 295 · see citing works
Exhaled volatile organic compounds identify patients with colorectal cancer
Altomare, D. F., Lena, M. D., Porcelli, F., de Trizio, L., Travaglio, E., Tutino, M., Dragonieri, S., Memeo, V., & de Gennaro, G. (2012) · British journal of surgery · doi:10.1002/bjs.8942 · Cited by 289 · see citing works
General conditions for predictivity in learning theory
Poggio, T., Rifkin, R. M., Mukherjee, S., & Niyogi, P. (2004) · Nature Cell Biology · doi:10.1038/nature02341 · Cited by 276 · see citing works
Beating the hold-out: bounds for K-fold and progressive cross-validation
Blum, A., Kalai, A., & Langford, J. (1999) · Proceedings of the twelfth annual conference on Computational learning theory · doi:10.1145/307400.307439 · Cited by 265 · see citing works
Learning in a Large Function Space: Privacy-Preserving Mechanisms for SVM Learning
Rubinstein, B. I. P., Bartlett, P. L., Huang, L., & Taft, N. (2012) · Journal of Privacy and Confidentiality · doi:10.29012/jpc.v4i1.612 · Cited by 264 · see citing works
Borra, S., & Ciaccio, A. D. (2010) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2010.03.004 · Cited by 236 · see citing works
Revealing Real-Time Emotional Responses: a Personalized Assessment based on Heartbeat Dynamics
Valenza, G., Citi, L., Lanatá, A., Scilingo, E., & Barbieri, R. (2014) · Scientific Reports · doi:10.1038/srep04998 · Cited by 245 · see citing works
Du, M., & Noguchi, N. (2017) · Remote Sensing · doi:10.3390/rs9030289 · Cited by 223 · see citing works
Recognizing Emotions Induced by Affective Sounds through Heart Rate Variability
Nardelli, M., Valenza, G., Greco, A., Lanatà, A., & Scilingo, E. P. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2432810 · Cited by 219 · see citing works
Mukherjee, S., Niyogi, P., Poggio, T., & Rifkin, R. M. (2006) · Advances in Computational Mathematics · doi:10.1007/s10444-004-7634-z · Cited by 198 · see citing works
High-density surface EMG maps from upper-arm and forearm muscles
Rojas-Martínez, M., Mañanas, M. Á., & Alonso, J. F. (2012) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-9-85 · Cited by 181 · see citing works
Amendolia, S. R., Cossu, G. F., Ganadu, M. L., Golosio, B., Masala, G. L., & Mura, G. M. (2003) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/s0169-7439(03)00094-7 · Cited by 173 · see citing works
Valenza, G., Nardelli, M., Lanatá, A., Gentili, C., Bertschy, G., Paradiso, R., & Scilingo, E. (2014) · IEEE journal of biomedical and health informatics · doi:10.1109/jbhi.2013.2290382 · Cited by 158 · see citing works
Conformal Prediction for Reliable Machine Learning: Theory, Adaptations and Applications
Balasubramanian, V. N., Ho, S.-S., & Vovk, V. (2014) · Cited by 155 · see citing works
Algorithmic stability for adaptive data analysis
Bassily, R., Kobbi Nissim, B., Smith, A., Steinke, T., Stemmer, U., & Ullman, J. (2016) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/2897518.2897566 · Cited by 154 · see citing works
Disjunctions of Conjunctions, Cognitive Simplicity, and Consideration Sets
Hauser, J. R., Toubia, O., Evgeniou, T., Befurt, R., & Dzyabura, D. (2010) · Journal of Marketing Research · doi:10.1509/jmkr.47.3.485 · Cited by 152 · see citing works
Synthesizing cognition in neuromorphic electronic systems
Neftci, E. O., Binas, J., Rutishauser, U., Chicca, E., Indiveri, G., & Douglas, R. J. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212083110 · Cited by 144 · see citing works
Permutation Tests for Classification
Golland, P., Liang, F., Mukherjee, S., & Panchenko, D. (2005) · Lecture notes in computer science · doi:10.1007/11503415_34 · Cited by 139 · see citing works
Generalization Bounds for Ranking Algorithms via Algorithmic Stability
Agarwal, S., & Niyogi, P. (2009) · Cited by 134 · see citing works
Leave One Out Error, Stability, and Generalization of Voting Combinations of Classifiers
Evgeniou, T., Pontil, M., & Elisseeff, A. (2004) · Machine Learning · doi:10.1023/b:mach.0000019805.88351.60 · Cited by 132 · see citing works
Algorithm-Dependent Generalization Bounds for Multi-Task Learning
Liu, T., Tao, D., Song, M., & Maybank, S. J. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2544314 · Cited by 131 · see citing works
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646 · see citing works
Ali, S., Glass, T., Parr, B., Potgieter, J., & Alam, F. (2020) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3034109 · Cited by 129 · see citing works
Machine Learning Based File Entropy Analysis for Ransomware Detection in Backup Systems
Lee, K., Lee, S., & Yim, K. (2019) · IEEE Access · doi:10.1109/access.2019.2931136 · Cited by 128 · see citing works
Stability of Randomized Learning Algorithms
Elisseeff, A., Evgeniou, T., & Pontil, M. (2005) · Cited by 116 · see citing works
Arousal and Valence Recognition of Affective Sounds Based on Electrodermal Activity
Greco, A., Valenza, G., Citi, L., & Scilingo, E. (2017) · IEEE Sensors Journal · doi:10.1109/jsen.2016.2623677 · Cited by 105 · see citing works
Analysis of Variance of Cross-Validation Estimators of the Generalization Error
Markatou, M., Tian, H., Biswas, S., & Hripcsak, G. M. (2005) · doi:10.7916/d86d5r2x · Cited by 114 · see citing works
Yu, B. (2013) · Bernoulli · doi:10.3150/13-bejsp14 · Cited by 113 · see citing works
Foundations of Machine Learning
Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018) · doi:10.7551/mitpress/10290.001.0001 · Cited by 1,122 · see citing works
Dao, D. K., Salehizadeh, S. M. A., Noh, Y., Chong, J. W., Cho, C. H., McManus, D., Darling, C. E., Mendelson, Y., & Chon, K. H. (2016) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2612059 · Cited by 104 · see citing works
Wang, D., Wang, X., Liu, T., & Liu, Y. (2011) · Meat Science · doi:10.1016/j.meatsci.2011.07.025 · Cited by 101 · see citing works
Lu, X., Yang, Y., Wu, F., Gao, M., Xu, Y., Zhang, Y., Yao, Y., Du, X., Li, C., Wu, L., et al. (2016) · Medicine · doi:10.1097/md.0000000000003973 · Cited by 101 · see citing works
Pizzi, A. D., Chiarelli, A. M., Chiacchiaretta, P., d’Annibale, M., Croce, P., Rosa, C., Mastrodicasa, D., Trebeschi, S., Lambregts, D. M. J., Caposiena, D., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-84816-3 · Cited by 98 · see citing works
Leave-One-Out Bounds for Kernel Methods
Zhang, T. (2003) · Neural Computation · doi:10.1162/089976603321780326 · Cited by 95 · see citing works
The Nearest Neighbor Algorithm of Local Probability Centers
Li, B., Chen, Y. W., & Chen, Y. Q. (2008) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2007.908363 · Cited by 95 · see citing works
Concept Drift Detection Through Resampling
Harel, M., Mannor, S., El‐Yaniv, R., & Crammer, K. (2014) · Cited by 92 · see citing works
Urbanos, G., Martín-Pérez, A., Vázquez, G., Villanueva, M., Villa, M., Roldan, L. J., Chavarrías, M., Lagares, A., Juárez, E., & Sanz, C. (2021) · Sensors · doi:10.3390/s21113827 · Cited by 91 · see citing works
On the Existence of Simpler Machine Learning Models
Semenova, L., Rudin, C., & Parr, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533232 · Cited by 90 · see citing works
Ying, Y., Qin, J., Liu, H., & Yao, X. (2011) · Journal of Chemical Information and Modeling · doi:10.1021/ci100427j · Cited by 90 · see citing works
Zemp, R., Tanadini, M., Plüss, S., Schnüriger, K., Singh, N. B., Taylor, W. R., & Lorenzetti, S. (2016) · BioMed Research International · doi:10.1155/2016/5978489 · Cited by 88 · see citing works
Toubiana, D., Puzis, R., Wen, L., Sikron, N., Kurmanbayeva, A., Soltabayeva, A., Wilhelmi, M. D. M. R., Sade, N., Fait, A., Sagi, M., et al. (2019) · Communications Biology · doi:10.1038/s42003-019-0440-4 · Cited by 89 · see citing works
Kong, L. C., Wuillemin, P.-H., Bastard, J., Sokolovska, N., Gougis, S., Fellahi, S., Darakhshan, F., Bonnefont‐Rousselot, D., Bittar, R., Dore, J. M., et al. (2013) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.113.058099 · Cited by 87 · see citing works
Algorithmic Stability and Generalization Performance
Bousquet, O., Elisseeff, A., & Bioinformatics, B. (2000) · Cited by 87 · see citing works
Machine Learning in Injection Molding: An Industry 4.0 Method of Quality Prediction
Párizs, R. D., Török, D., Ageyeva, T., & Kovács, J. G. (2022) · Sensors · doi:10.3390/s22072704 · Cited by 85 · see citing works
Zhu, L., Zhang, C., Zhang, Z., Zhang, Z., Nie, X., Zhou, X., Liu, W., & Wang, X. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.105596 · Cited by 84 · see citing works
Just-in-Time Adaptive Classifiers—Part II: Designing the Classifier
Alippi, C., & Roveri, M. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2003998 · Cited by 82 · see citing works
The Optimal Phenological Phase of Maize for Yield Prediction with High-Frequency UAV Remote Sensing
Yang, B. T., Zhu, W., Rezaei, E. E., Li, J., Sun, Z., & Zhang, J. (2022) · Remote Sensing · doi:10.3390/rs14071559 · Cited by 80 · see citing works
Identification of isometric contractions based on High Density EMG maps
Rojas-Martínez, M., Mañanas, M. Á., Alonso, J. F., & Merletti, R. (2012) · Journal of Electromyography and Kinesiology · doi:10.1016/j.jelekin.2012.06.009 · Cited by 80 · see citing works
Stability and Generalization of Bipartite Ranking Algorithms
Agarwal, S., & Niyogi, P. (2005) · Lecture notes in computer science · doi:10.1007/11503415_3 · Cited by 77 · see citing works
An adsorbed gas estimation model for shale gas reservoirs via statistical learning
Chen, Y., Jiang, S. Q., Zhang, D., & Liu, C. (2017) · Applied Energy · doi:10.1016/j.apenergy.2017.04.029 · Cited by 75 · see citing works
Wang, Y.-X., Lei, J., & Fienberg, S. E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.06309 · Cited by 74 · see citing works
Radiomics and radiogenomics in ovarian cancer: a literature review
Nougaret, S., McCague, C., Tibermacine, H., Vargas, H. A., Rizzo, S., & Sala, E. (2020) · Abdominal Radiology · doi:10.1007/s00261-020-02820-z · Cited by 73 · see citing works
Modeling multivariate distribution of multiple soil parameters using vine copula model
Lü, T.-J., Tang, X.-S., Li, D. Q., & Qi, X. (2019) · Computers and Geotechnics · doi:10.1016/j.compgeo.2019.103340 · Cited by 72 · see citing works
Information-theoretic analysis of stability and bias of learning algorithms
Raginsky, M., Rakhlin, A., Tsao, M., Wu, Y., & Xu, A. (2016) · 2016 IEEE Information Theory Workshop (ITW) · doi:10.1109/itw.2016.7606789 · Cited by 70 · see citing works
Prediction of GFP spectral properties using artificial neural network
Nantasenamat, C., Isarankura‐Na‐Ayudhya, C., Tansila, N., Naenna, T., & Prachayasittikul, V. (2007) · Journal of Computational Chemistry · doi:10.1002/jcc.20656 · Cited by 70 · see citing works
High probability generalization bounds for uniformly stable algorithms with nearly optimal ratePreprint
Feldman, V., & Vondrák, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10710 · Cited by 70 · see citing works
A Review of Matched-pairs Feature Selection Methods for Gene Expression Data Analysis
Liang, S., Ma, A., Yang, S., Wang, Y., & Ma, Q. (2018) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2018.02.005 · Cited by 68 · see citing works
Joseph, A. D., Nelson, B. A., Rubinstein, B. I. P., & Tygar, J. D. (2019) · Cambridge University Press eBooks · doi:10.1017/9781107338548 · Cited by 69 · see citing works
Lü, Z., Chen, X., Yao, S., Qin, H., Zhang, L., Yao, X., Yu, Z., & Lu, J. (2019) · Fuel · doi:10.1016/j.fuel.2019.116150 · Cited by 65 · see citing works
Lomasky, R., Brodley, C. E., Aernecke, M. J., Walt, D. R., & Friedl, M. A. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74958-5_63 · Cited by 64 · see citing works
The need for external validation in machine olfaction: emphasis on health-related applications
Marco, S. (2014) · Analytical and Bioanalytical Chemistry · doi:10.1007/s00216-014-7807-7 · Cited by 62 · see citing works
Detecting middle ear fluid using smartphones
Chan, J., Raju, S. C., Nandakumar, R., Bly, R. A., & Gollakota, S. (2019) · Science Translational Medicine · doi:10.1126/scitranslmed.aav1102 · Cited by 61 · see citing works
Yang, F., Wanik, D. W., Cerrai, D., Bhuiyan, M. A. E., & Anagnostou, E. N. (2020) · Sustainability · doi:10.3390/su12041525 · Cited by 60 · see citing works
The Pixels and Sounds of Emotion: General-Purpose Representations of Arousal in Games
Makantasis, K., Liapis, A., & Yannakakis, G. N. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3060877 · Cited by 59 · see citing works
An artificial neural network method for lumen and media-adventitia border detection in IVUS
Su, S., Hu, Z., Lin, Q., Hau, W. K., Gao, Z., & Zhang, H. (2016) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2016.11.003 · Cited by 58 · see citing works
Bachli, M. B., Sedeño, L., Ochab, J. K., Piguet, O., Kumfor, F., Reyes, P., Torralva, T., Roca, M., Cardona, J. F., Campo, C. G., et al. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2019.116456 · Cited by 57 · see citing works
Chang, J. P.-E., Goh, G. B., Leow, W. Q., Shen, L., Lim, K. H., & Tan, C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199166 · Cited by 57 · see citing works
Edge deep learning for neural implants: a case study of seizure detection and prediction
Liu, X., & Richardson, A. G. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/abf473 · Cited by 56 · see citing works
Altomare, D. F., Lena, M. D., Porcelli, F., Travaglio, E., Longobardi, F., Tutino, M., Depalma, N., Tedesco, G., Sardaro, A., Memeo, R., et al. (2015) · Annals of Surgery · doi:10.1097/sla.0000000000001471 · Cited by 55 · see citing works
Valladares‐Rodríguez, S., Perez-Rodriguez, R., Facal-Mayo, D., Iglesias, M. J. F., Anido, L., & Mouriño-García, M. (2017) · PeerJ · doi:10.7717/peerj.3508 · Cited by 55 · see citing works
What Can We Learn Privately?Preprint
Kasiviswanathan, S. P., Lee, H. K., Kobbi Nissim, B., Raskhodnikova, S., & Smith, A. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0803.0924 · Cited by 54 · see citing works
Automated Analysis of Optic Nerve Images for Detection and Staging of Papilledema
Echegaray, S., Zamora, G., Yu, H., Luo, W., Solíz, P., & Kardon, R. H. (2011) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.11-7484 · Cited by 53 · see citing works
Bias in Estimating the Variance of K-Fold Cross-Validation
Bengio, Y., & Grandvalet, Y. (2005) · Springer eBooks · doi:10.1007/0-387-24555-3_5 · Cited by 52 · see citing works
Ness, B. V., Ramos, C., Haznadar, M., Hoering, A., Haessler, J., Crowley, J. J., Jacobus, S. J., Oken, M. M., Rajkumar, S. V., Greipp, P. R., et al. (2008) · BMC Medicine · doi:10.1186/1741-7015-6-26 · Cited by 52 · see citing works
Abdizadeh, R., Hadizadeh, F., & Abdizadeh, T. (2019) · Journal of Molecular Structure · doi:10.1016/j.molstruc.2019.126961 · Cited by 51 · see citing works
Estimating the Confidence Interval for Prediction Errors of Support Vector Machine Classifiers
Jiang, B., Zhang, X. H.-F., & Cai, T. (2008) · Cited by 50 · see citing works
Zhang, H., Goll, D. S., Manzoni, S., Ciais, P., Guenet, B., & Huang, Y. (2018) · Geoscientific model development · doi:10.5194/gmd-11-4779-2018 · Cited by 47 · see citing works
On-Average KL-Privacy and Its Equivalence to Generalization for Max-Entropy Mechanisms
Wang, Y.-X., Lei, J., & Fienberg, S. E. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45381-1_10 · Cited by 48 · see citing works
Jegadeeshwaran, R., & Sugumaran, V. (2014) · Engineering Science and Technology an International Journal · doi:10.1016/j.jestch.2014.08.001 · Cited by 47 · see citing works
A new approach for the estimation of phytoplankton cell counts associated with algal blooms
Nazeer, M., Wong, M. S., & Nichol, J. E. (2017) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2017.02.182 · Cited by 46 · see citing works
Sensor Arrays, Virtual Multisensors, Data Fusion, and Gas Sensor Data Evaluation
Reimann, P., & Schütze, A. (2013) · Springer series on chemical sensors and biosensors · doi:10.1007/5346_2013_52 · Cited by 45 · see citing works
Selection of suitable hand gestures for reliable myoelectric human computer interface
Castro, M. C. F., Arjunan, S. P., & Kumar, D. (2015) · BioMedical Engineering OnLine · doi:10.1186/s12938-015-0025-5 · Cited by 43 · see citing works
Xia, J., Yokoya, N., & Pham, T. D. (2020) · Remote Sensing · doi:10.3390/rs12223834 · Cited by 42 · see citing works
- Authors
- Kearns, M., & Ron, D.
- Affiliations
- AT&T (United States) (US)
- Tel Aviv University (IL)
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 1999-08-01
- Volume
- 11
- Issue
- 6
- Pages
- 1427-1453
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2138263042
- Cited by
- 407
- References
- 18
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 11 · 2023: 18 · 2022: 22 · 2021: 26 · 2020: 35 · 2019: 46 · 2018: 23 · 2017: 22 · 2016: 27 · 2015: 18 · 2014: 17 · 2013: 24 · 2012: 14
- FWCI
- 4.55
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation
- Office of Naval Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- leave-one-out cross-validation, generalization error estimation, algorithm stability, VC dimension, lower bounds
- MeSH terms
- Algorithms, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
