The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index.
Neural Networks and the Bias/Variance Dilemma
Abstract
Feedforward neural networks trained by error backpropagation are examples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numerals. In way of conclusion, we suggest that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns.”
From Reconciling modern machine-learning practice and the classical bias–variance trade-off (2019) · cites this paper as [17]
In the citing paper
“The concept of bias in machine learning (ML) research and more generally in the field of predictive modeling is intrinsically tied to the concept of variance.14”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [14]
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,867
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., & White, H. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90020-8 · Cited by 21,728
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,030
Information Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer series in statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 18,636
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,501
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
(1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
Research works printed but not linked: 7
- 10.1016/S0364-0213(85)80012-4 · DOI · Paper page
- Barron A. R., 1989, Florida, 280
- Baum L. E., 1972, Inequalities, 3, 1
- Burr D. J., 1980, IEEE Transact. Pattern Anal. Machine Intell. PAMI-3, 6, 708
- Chauvin Y., 1990, Neural Information Processing Systems 11, D. S. Touretzky, ed., 642
- Denker J., 1987, Complex Syst., 1, 877
- Goldman L., 1982, J. Med., 307, 588
Cited by
Sorted by citations
Items 501–600 of 1,006
FadeNet: Deep Learning-Based mm-Wave Large-Scale Channel Fading Prediction and its Applications
Ratnam, V. V., Chen, H., Pawar, S., Zhang, B., Zhang, J. C., Kim, Y., Lee, S. Y., Cho, M., & Yoon, S.-R. (2020) · IEEE Access · doi:10.1109/access.2020.3048583 · Cited by 92
A Hybrid System Based on Dynamic Selection for Time Series Forecasting
de Oliveira, J. F. L., da Silva, E. G., & Neto, P. S. G. D. M. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3051384 · Cited by 89
Rathore, S. S., & Kumar, S. (2016) · Knowledge-Based Systems · doi:10.1016/j.knosys.2016.12.017 · Cited by 88
The simplicity principle in perception and cognition
Feldman, J. (2016) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1406 · Cited by 102
“In the modern literature on machine learning and statistical learning, the intimate connection between simplicity and probability is part of what is called the bias/variance trade-off, terminology introduced by Geman and coauthors.(25) Very broadly speaking, complex theories are inherently more flexible and thus more…”
Alsulaili, A. D., & Refaie, A. (2020) · Water Science & Technology Water Supply · doi:10.2166/ws.2020.199 · Cited by 85
Maximum Likelihood Estimation in Data-Driven Modeling and Control
Yin, M., Iannelli, A., & Smith, R. S. (2021) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2021.3137788 · Cited by 85
“However, it is well-known that when data are noisy, overparametrized models may lead to high variances and overfitting [18].”
BackgroundEstimation of heavy metal sorption in German soils using artificial neural networks
Anagu, I., Ingwersen, J., Utermann, J., & Streck, T. (2009) · Geoderma · doi:10.1016/j.geoderma.2009.06.004 · Cited by 85
Siou, L. K. A., Johannet, A., Estupina, V. B., & Pistre, S. (2011) · Journal of Hydrology · doi:10.1016/j.jhydrol.2011.04.015 · Cited by 102
“This problem is known as the bias-variance dilemma and has been mathematically defined in Geman et al. (1992).”
BackgroundSwarm intelligence for self-organized clustering
Thrun, M. C., & Ultsch, A. (2020) · Artificial Intelligence · doi:10.1016/j.artint.2020.103237 · Cited by 83
Oblique and rotation double random forest
Ganaie, M. A., Tanveer, M., Suganthan, P. N., & Snåšel, V. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.06.012 · Cited by 97
Dimoulas, C. A., Kalliris, G. M., Papanikolaou, G. V., Petridis, V., & Kalampakas, A. (2006) · Expert Systems with Applications · doi:10.1016/j.eswa.2006.08.014 · Cited by 84
Larder, B. A., Wang, D., Revell, A. D., Montaner, J., Harrigan, R., de Wolf, F., Lange, J. M. A., Wegner, S. A., Ruiz, L., Pérez-Elı́as, M. J., et al. (2007) · Antiviral Therapy · doi:10.1177/135965350701200112 · Cited by 84
“Previous studies have demonstrated that, as expected, the committee average performance is superior to the performance of individual models probably because of the biasvariance decomposition of ensemble learning schemes (data on file and [41]).”
BackgroundOn the effect of data set size on bias and variance in classification learning
Brain, D., & Webb, G. I. (1999) · Cited by 84
Neural networks in structural analysis and design: An overview
Hajela, P., & Berke, L. (1992) · Computing Systems in Engineering · doi:10.1016/0956-0521(92)90138-9 · Cited by 84
Sensing in the presence of strong noise by deep learning of dynamic multimode fiber interference
Nguyen, L. V., Nguyen, C. C., Carneiro, G., Ebendorff‐Heidepriem, H., & Warren-Smith, S. C. (2021) · Photonics Research · doi:10.1364/prj.415902 · Cited by 85
“While mitigating the generalization issue with advanced deep learning techniques is not within the scope of this paper, readers are referred to reviews of this active area of machine learning research [47 – 49].”
BackgroundModel Multiplicity: Opportunities, Concerns, and Solutions
Black, E., Raghavan, M., & Barocas, S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533149 · Cited by 85
“Conceptually, errors in machine learning systems come from three sources: bias, variance, and irreducible noise [25, 33].”
BackgroundMultiobjective analysis of chaotic dynamic systems with sparse learning machines
Khalil, A. F., McKee, M., Kemblowski, M., Asefa, T., & Bastidas, L. (2005) · Advances in Water Resources · doi:10.1016/j.advwatres.2005.05.011 · Cited by 85
Tetko, I. V. (2002) · Neural Processing Letters · doi:10.1023/a:1019903710291 · Cited by 83
“The MLP variance can also contribute to poor performance of this method [4].”
MethodArtificial neural networks for cancer research: Outcome prediction
BURKE, H. B. (1994) · Seminars in Surgical Oncology · doi:10.1002/ssu.2980100111 · Cited by 83
Gigerenzer, G. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199390076.003.0007 · Cited by 101
Mitigating Bias in Radiology Machine Learning: 2. Model Development
Zhang, K., Khosravi, B., Vahdati, S., Faghani, S., Nugen, F., Rassoulinejad-Mousavi, S. M., Moassefi, M., Jagtap, J. M. M., Singh, Y., Rouzrokh, P., et al. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.220010 · Cited by 87
Bias/Variance Analyses of Mixtures-of-Experts Architectures
Jacobs, R. A. (1997) · Neural Computation · doi:10.1162/neco.1997.9.2.369 · Cited by 82
“The equations for estimating the quantities of interest are closely related to the equations for estimating the bias and variance of neural networks presented by Geman et al. (1992) .”
BackgroundNetzer, M., Millonig, G., Osl, M., Pfeifer, B. E., Praun, S., Villinger, J., Vogel, W., & Baumgärtner, C. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp093 · Cited by 82
“Thus, the bias-variance decomposition as defined by Geman and authors (1992) distinguishes between three sources of errors: (i) The bias error is a systematic component of the error resulting from differences between the learning method and the domain (Putten et al.”
MethodStriking a Blow for Sanity in Theories of Rationality
Gigerenzer, G. (2004) · The MIT Press eBooks · doi:10.7551/mitpress/4709.003.0039 · Cited by 84
Piotrowski, A., Napiorkowski, J. J., & Piotrowska, A. E. (2019) · Earth-Science Reviews · doi:10.1016/j.earscirev.2019.103076 · Cited by 90
Dharmapala, K., Rajapakse, A. D., Narendra, K., & Zhang, Y. (2020) · IEEE Access · doi:10.1109/access.2020.3043935 · Cited by 83
“Analysis of the learning curves showed that these two trained ANNR models have high variance in testing error [43], although the training error was decreasing with each iteration.”
BackgroundJin, H., Shi, L., Chen, X., Qian, B., Yang, B., & Jin, H. (2021) · Renewable Energy · doi:10.1016/j.renene.2021.04.028 · Cited by 81
DIVACE: Diverse and Accurate Ensemble Learning Algorithm
Chandra, A., & Yao, X. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-28651-6_91 · Cited by 81
“[6] explained this dilemma very well.”
Assessing geometrical errors of multi-axis machines by three-dimensional length measurements
Florussen, G. H. J., Delbressine, F. L. M., van de Molengraft, R., & Schellekens, P. H. J. (2001) · Measurement · doi:10.1016/s0263-2241(01)00016-1 · Cited by 81
Generality versus Size in Genetic Programming
Rosca, J. P. (1996) · The MIT Press eBooks · doi:10.7551/mitpress/3242.003.0056 · Cited by 81
“A biased model has fewer de-grees of freedom and lower variance (Geman et al., 1992).”
BackgroundA DEEP CONVOLUTIONAL AUTO-ENCODER WITH POOLING – UNPOOLING LAYERS IN CAFFE
Turchenko, V., Chalmers, E., & Luczak, A. (2019) · International Journal of Computing · doi:10.47839/ijc.18.1.1270 · Cited by 93
Basic functional trade-offs in cognition: An integrative framework
Giudice, M. D., & Crespi, B. J. (2018) · Cognition · doi:10.1016/j.cognition.2018.06.008 · Cited by 86
“Trade-offs between performance and robustness include the ubiquitous phenomenon known as bias-variance trade-off (Geman et al., 1992; Hastie et al., 2009).”
Total Deep Variation for Linear Inverse Problems
Kobler, E., Effland, A., Kunisch, K. K., & Pock, T. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00757 · Cited by 80
A survey on epistemic (model) uncertainty in supervised learning: Recent advances and applications
Zhou, X., Liu, H., Pourpanah, F., Zeng, T., & Wang, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.10.119 · Cited by 87
“In a regression context, the generalization error can be decomposed through two well-known schemes, namely, ambiguity decomposition [98] and bias-variance decomposition [99].”
MethodGenetic Programming, Ensemble Methods and the Bias/Variance Tradeoff – Introductory Investigations
Keijzer, M., & Babovic, V. (2000) · Lecture notes in computer science · doi:10.1007/978-3-540-46239-2_6 · Cited by 79
“Geman et al. [ 5 ] go so far as to state that the dilemma can be circumvented only if one is willing to sacrifice generality, that is, purposefully introduce bias, in order to reduce the variance.”
BackgroundA dynamic classifier ensemble selection approach for noise data
Xiao, J., He, C., Jiang, X., & Liu, D. (2010) · Information Sciences · doi:10.1016/j.ins.2010.05.021 · Cited by 80
A novel radial basis function neural network for discriminant analysis
Yang, Z. R. (2006) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2006.873282 · Cited by 80
“Weight decay is such a method, where an additive cost term was added on the error function to penalize the weights with large values based on the assumption that the weights follow a single Gaussian [2], [3], [9], [21], [34].”
MethodWater quality assessment using diatom assemblages and advanced modelling techniques
Gevrey, M., Rimet, F., Park, Y., Tison‐Rosebery, J., Éctor, L., & Lek, S. (2004) · Freshwater Biology · doi:10.1046/j.1365-2426.2003.01174.x · Cited by 83
Subspace Information Criterion for Model Selection
Sugiyama, M., & Ogawa, H. (2001) · Neural Computation · doi:10.1162/08997660152469387 · Cited by 78
“The learning operator X θ is linear.”
Learning Analytics to identify dropout factors of Computer Science studies through Bayesian networks
Lacave, C., Molina, A. I., & Cruz-Lemus, J. A. (2018) · Behaviour and Information Technology · doi:10.1080/0144929x.2018.1485053 · Cited by 79
“It is recommended that k should take a value between 5 and 10 to address the ‘ bias/variance ’ dilemma (Geman, Bienenstock, and Doursat 1992), in which the minimisation of potential bias and prediction error created by an inappropriate data split competes with the minimisation of variance that is…”
BackgroundInterpreting neural-network results: a simulation study
Intrator, O. K., & Intrator, N. (2001) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(01)00016-0 · Cited by 78
“Simulated data from known models are used to demonstrate the interpretability results of the ANNs.”
MethodGlaze, C. M., Filipowicz, A. L. S., Kable, J. W., Balasubramanian, V., & Gold, J. I. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0297-4 · Cited by 84
Toda, A., Imamizu, H., Kawato, M., & Sato, M.-A. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.09.057 · Cited by 78
“This is because having too many parameters (e.g., weights for input signals in a multiple linear regression model) in relation to the number of training datasets is known to lead to poor generalization performance (over-fitting problem) (Akaike, 1974; Geman et al., 1992).”
Computer aided detection of clusters of microcalcifications on full field digital mammograms
Ge, J., Sahiner, B., Hadjiiski, L. M., Chan, H., Wei, J., Helvie, M. A., & Zhou, C. (2006) · Medical Physics · doi:10.1118/1.2211710 · Cited by 78
Reducing Variance of Committee Prediction with Resampling Techniques
Parmanto, B., Munro, P., & Doyle, H. R. (1996) · Connection Science · doi:10.1080/095400996116848 · Cited by 78
Predicting fish yield of African lakes using neural networks
Laë, R., Lek, S., & Moreau, J. (1999) · Ecological Modelling · doi:10.1016/s0304-3800(99)00112-x · Cited by 77
“a test for the number of different possible configurations and the selection of that which provides the best compromise between bias and variance (Geman et al., 1992; Kohavi, 1995), which is the training that gives a good generalization.”
Miller, K. J., Schalk, G., Hermes, D., Ojemann, J. G., & Rao, R. P. N. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004660 · Cited by 77
“This is because of what is known as the “bias-variance tradeoff” [42,43].”
Function Approximation with Neural Networks and Local Methods: Bias, Variance and Smoothness
Lawrence, S., Tsoi, A. C., & Back, A. D. (1996) · Cited by 77
Decision Making: Nonrational Theories
Gigerenzer, G., & Gaissmaier, W. (2015) · Elsevier eBooks · doi:10.1016/b978-0-08-097086-8.26017-0 · Cited by 89
“A general formulation for when and why one should simplify is the bias–variance dilemma (Geman et al., 1992).”
BackgroundRegularization approach to inductive genetic programming
Nikolaev, N. Y., & Iba, H. (2001) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.942530 · Cited by 76
“Such difficulties may be addressed by decomposing the error of the solution being an approximation of the true function with respect to the training data in two components [ 10 ]”
MethodColor Distribution Information for the Reduced-Reference Assessment of Perceived Image Quality
Redi, J., Gastaldo, P., Heynderickx, I. E. J., & Zunino, R. (2010) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2010.2087456 · Cited by 76
“An “ensemble” [46] gathers parallel networks [46], [47] that have been trained on the same problem.”
BackgroundDonate, J. P., Cortez, P. C., Gutiérrez, G., & Sanchis, A. (2013) · Neurocomputing · doi:10.1016/j.neucom.2012.02.053 · Cited by 76
“mance of classification or regression tasks, since aggregated responses tend to perform better than individual ones [21, 22].”
BackgroundA Best Practice Guide to Resource Forecasting for Computing Systems
Hoffmann, G. A., Trivedi, K. S., & Malek, M. (2007) · IEEE Transactions on Reliability · doi:10.1109/tr.2007.909764 · Cited by 79
“In fact, including unfavorable variable combinations can degrade model performance significantly [3], [21], [24].”
BackgroundClassifying Alzheimer's disease with brain imaging and genetic data using a neural network framework
Ning, K., Chen, B., Sun, F., Hobel, Z. B., Zhao, L., Matloff, W. J., & Toga, A. W. (2018) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2018.04.009 · Cited by 79
“Third, since NN models have more parameters than LR models, more data points are needed to train the NN models (Geman et al., 1992).”
MethodTime series prediction with ensemble models
Wichard, J., & OGORZAŁEK, M. J. (2005) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2004.1380203 · Cited by 75
“where the expectation ED[ ] is taken with respect to all possible realizations of training sets D with fixed sample size N . According to [ 8 ] the bias/variance decomposition is”
BackgroundHierarchical testing designs for pattern recognition
Blanchard, G., & Geman, D. (2005) · The Annals of Statistics · doi:10.1214/009053605000000174 · Cited by 76
“In these cases, efficient representations may be as important as statistical learning [18], and thinking about computation at the start of the day may be essential.”
BackgroundFlood-routing modeling with neural network optimized by social-based algorithm
Nikoo, M., Ramezani, F., Nyarko, E. K., Nyarko, E. K., & Nikoo, M. R. (2016) · Natural Hazards · doi:10.1007/s11069-016-2176-5 · Cited by 75
Estimating Semantic Networks of Groups and Individuals from Fluency Data
Zemla, J. C., & Austerweil, J. L. (2018) · Computational Brain & Behavior · doi:10.1007/s42113-018-0003-7 · Cited by 79
“This is an example of the bias-variance dilemma, where we have introduced bias to enable estimation from small sample sizes (Geman et al. 1992; Griffiths 2010).”
MethodLi, Z., Tate, D., Lane, C., & Adams, C. (2011) · Computer-Aided Design · doi:10.1016/j.cad.2011.12.006 · Cited by 75
Cross-validation aggregation for combining autoregressive neural network forecasts
Barrow, D. K., & Crone, S. F. (2016) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2015.12.011 · Cited by 77
Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models
Rocks, J. W., & Mehta, P. (2022) · Physical Review Research · doi:10.1103/physrevresearch.4.013201 · Cited by 74
Deep Learning for Massive MIMO Uplink Detectors
Albreem, M. A. M., Alhabbash, A., Shahabuddin, S., & Juntti, M. J. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3135542 · Cited by 81
“a DL based detection consists of variance, bias, and irreducible error [148], [149].”
BackgroundSoft computing-based modeling of flotation processes – A review
Jovanović, I., Miljanović, I., & Jovanović, T. (2015) · Minerals Engineering · doi:10.1016/j.mineng.2015.09.020 · Cited by 74
SAN: Learning Relationship Between Convolutional Features for Multi-scale Object Detection
Kim, Y., Kang, B.-N., & Kim, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01228-1_20 · Cited by 74
Evaluation in Music Information Retrieval
Urbano, J., Schedl, M., & Serra, X. (2013) · Journal of Intelligent Information Systems · doi:10.1007/s10844-013-0249-4 · Cited by 74
“In Machine Learning terms, validity refers to the bias of a learner, and reliability refers to its variance (Geman et al. 1992).”
Tuning Your Priors to the World
Feldman, J. (2013) · Topics in Cognitive Science · doi:10.1111/tops.12003 · Cited by 73
“Jaynes (2003) directly addresses the question of why Gaussian priors so often out-perform more detailed priors based on frequency tabulations.”
BackgroundJain, S., & Saha, A. (2021) · Science of Computer Programming · doi:10.1016/j.scico.2021.102713 · Cited by 73
The Morton-Massaro law of information integration: Implications for models of perception.
Movellan, J. R., & McClelland, J. L. (2001) · Psychological Review · doi:10.1037/0033-295x.108.1.113 · Cited by 71
Bayesian adaptive user profiling with explicit & implicit feedback
Zigoris, P., & Zhang, Y. (2006) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1183614.1183672 · Cited by 73
“However, there is a natural “biasvariance trade-off” for any learning algorithm[5].”
BackgroundCircular-ELM for the reduced-reference assessment of perceived image quality
Decherchi, S., Gastaldo, P., Zunino, R., Cambria, E., & Redi, J. (2012) · Neurocomputing · doi:10.1016/j.neucom.2011.12.050 · Cited by 71
“Such a structural enhancement eventually improves the ability of the resulting CI model to tackle complex, non-linear problems, such as modeling perceptual mechanisms.”
BackgroundIdentifying rice fields at risk from damage by the greater flamingo
Tourenq, C., Aulagnier, S., Durieux, L., Lek, S., Mesleárd, F., Johnson, A. R., & Martin, J. (2001) · Journal of Applied Ecology · doi:10.1046/j.1365-2664.2001.00581.x · Cited by 75
Towards a computational theory of experience
Fekete, T., & Edelman, S. (2011) · Consciousness and Cognition · doi:10.1016/j.concog.2011.02.010 · Cited by 72
Kadkhodaie, A., Rezaee, R., & Rahimpour‐Bonab, H. (2009) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2008.12.012 · Cited by 72
The Curse of Dimensionality for Local Kernel Machines
Bengio, Y., & Delalleau, O. (2005) · Cited by 75
Xue, H., Yang, Q., & Chen, S. (2009) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420089653-10 · Cited by 77
Optimizing agent behavior over long time scales by transporting value
Hung, C.-C., Lillicrap, T., Abramson, J., Wu, Y., Mirza, M., Carnevale, F., Ahuja, A., & Wayne, G. (2019) · Nature Communications · doi:10.1038/s41467-019-13073-w · Cited by 90
Estimation and decision fusion: A survey
Sinha, A., Chen, H., Danu, D. G., Kirubarajan, T., & Farooq, M. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.06.016 · Cited by 72
“The term ensemble ensembles is that of improving their generalization ability [19].”
BackgroundNeural Network Studies. 3. Variable Selection in the Cascade-Correlation Learning Architecture
Kovalishyn, V., Tetko, I. V., Luik, A. I., Kholodovych, V. V., Villa, A. E. P., & Livingstone, D. J. (1998) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci980325n · Cited by 71
Variational Bayes for continuous hidden Markov models and its application to active learning
Ji, S., Krishnapuram, B., & Carin, L. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.85 · Cited by 71
“The expectation inside the integration may be decomposed as [18], [19]”
BackgroundMethodGeneralization performance of regularized neural network models
Larsen, J. O., & Hansen, L. K. (2002) · Machine learning for signal processing ... · doi:10.1109/nnsp.1994.366065 · Cited by 71
The interpretation of uncertainty in ecological rationality
Kozyreva, A., & Hertwig, R. (2019) · Synthese · doi:10.1007/s11229-019-02140-w · Cited by 78
A theory of learning to infer.
Dasgupta, I., Schulz, E., Tenenbaum, J. B., & Gershman, S. J. (2020) · Psychological Review · doi:10.1037/rev0000178 · Cited by 70
Random Forests and the measurement of super-efficiency in the context of Free Disposal Hull
Esteve, M., Aparicio, J., Rodriguez-Sala, J. J., & Zhu, J. (2022) · European Journal of Operational Research · doi:10.1016/j.ejor.2022.04.024 · Cited by 70
Brosse, S., Arbuckle, C. J., & Townsend, C. R. (2003) · Biodiversity and Conservation · doi:10.1023/a:1024107915183 · Cited by 71
“This zone corresponds to the best compromise between bias and variance (i.e. the number of iterations yielding the lowest MSE in both training and testing; Geman et al. 1992; Lek et al. 1996).”
BackgroundGranitto, P. M., Franco Biasioli, A., Aprea, E., Mott, D., Furlanello, C., Märk, T. D., & Gasperi, F. (2006) · Sensors and Actuators B Chemical · doi:10.1016/j.snb.2006.03.047 · Cited by 70
“The statistical principle of the “bias-variance trade-off” is at the basis of the new modelling tools based on ensembles (or groups) of predictors: if a combined classifier is formed out of several low-bias-high-variance models, variance is reduced and the result is a classification method which is more accurate than…”
MethodA General Neural Network Model for Estimating Telecommunications Network Reliability
Altıparmak, F., Dengiz, B., & Smith, A. E. (2009) · IEEE Transactions on Reliability · doi:10.1109/tr.2008.2011854 · Cited by 70
Chokshi, P., Dashwood, R. J., & Hughes, D. J. (2017) · Computers & Structures · doi:10.1016/j.compstruc.2017.05.015 · Cited by 68
Cabrera, A. A., & Palsbøll, P. J. (2017) · Molecular Ecology Resources · doi:10.1111/1755-0998.12696 · Cited by 70
Combined Neural Networks for Time Series Analysis
Ginzburg, I., & Horn, D. (1993) · neural information processing systems · Cited by 70
Models with higher effective dimensions tend to produce more uncertain estimates
Puy, A., Beneventano, P., Levin, S. A., Piano, S. L., Portaluri, T., & Saltelli, A. (2022) · Science Advances · doi:10.1126/sciadv.abn9450 · Cited by 69
An Experimental Bias-Variance Analysis of SVM Ensembles Based on Resampling Techniques
Valentini, G. (2005) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2005.850183 · Cited by 68
Learning with Preknowledge: Clustering with Point and Graph Matching Distance Measures
Gold, S., Rangarajan, A., & Mjolsness, E. (1996) · Neural Computation · doi:10.1162/neco.1996.8.4.787 · Cited by 69
“Recently, the importance of such preknowledge for learning has been convincingly argued from a statistical framework [Geman et al., 1992].”
BackgroundNetwork approaches for expert decisions in sports
Glöckner, A., Heinen, T., Johnson, J. G., & Raab, M. (2011) · Human Movement Science · doi:10.1016/j.humov.2010.11.002 · Cited by 68
MLPNN Training via a Multiobjective Optimization of Training Error and Stochastic Sensitivity
Yeung, D., Li, J., Ng, W. W. Y., & Chan, P. P. K. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2431251 · Cited by 68
Atmospheric dispersion modeling using Artificial Neural Network based cellular automata
Lauret, P., Heymes, F., Aprin, L., & Johannet, A. (2016) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2016.08.001 · Cited by 68
“Variable selection is a critical task in neural network definition because it reduces the complexity of the model and avoids overfitting (Geman et al., 1992).”
BackgroundModelling the infiltration process with a multi-layer perceptron artificial neural network
SY, N. L. (2006) · Hydrological Sciences Journal · doi:10.1623/hysj.51.1.3 · Cited by 67
“Because the MLP is prone to overfitting (Geman et al ., 1992), some form of regularization is usually performed for faster learning and generalization improvement.”
BackgroundHinging hyperplane trees for approximation and identification
Ernst, S. (2002) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.1998.758452 · Cited by 67
Independent Components in Text
Kolenda, T., Hansen, L. K., & Sigurdsson, S. O. (2000) · Perspectives in neural computing · doi:10.1007/978-1-4471-0443-8_13 · Cited by 67
Local Overfitting Control via Leverages
Monari, G., & Dreyfus, G. (2002) · Neural Computation · doi:10.1162/089976602753713025 · Cited by 67
“The traditional view of overfitting refers mostly to the bias / variance tradeoff, introduced in (Geman & al., 1992): a family of parameterized functions with too few parameters, with respect to the complexity of a problem, is said to have too large a bias, because it cannot fit the deterministic…”
BackgroundQuartz, S. R. (2003) · Biology & Philosophy · doi:10.1023/a:1023395002672 · Cited by 67
Metadata
- Authors
- Geman, S., Bienenstock, E., & Doursat, R.
- Affiliations
- Brown University (US)
- ESPCI Paris (FR)
- ORCID
- Doursat, René
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 1992-01
- Volume
- 4
- Issue
- 1
- Pages
- 1-58
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2076118331
- Cited by
- 3,578
- References
- 96
- Citations by year
- 2026: 125 · 2025: 131 · 2024: 115 · 2023: 128 · 2022: 148 · 2021: 201 · 2020: 181 · 2019: 142 · 2018: 126 · 2017: 97 · 2016: 115 · 2015: 99 · 2014: 103 · 2013: 82 · 2012: 110
- FWCI
- 60.04
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- e-space (Manchester Metropolitan University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and ELM
- Keywords
- bias-variance trade-off, feedforward neural networks, backpropagation, nonparametric regression, nonparametric inference, handwritten digit recognition, machine perception, neural network representations, statistical learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
