The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index.
Neural Networks and the Bias/Variance Dilemma
Abstract
Feedforward neural networks trained by error backpropagation are examples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numerals. In way of conclusion, we suggest that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns.”
From Reconciling modern machine-learning practice and the classical bias–variance trade-off (2019) · cites this paper as [17]
In the citing paper
“The concept of bias in machine learning (ML) research and more generally in the field of predictive modeling is intrinsically tied to the concept of variance.14”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [14]
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,867
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., & White, H. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90020-8 · Cited by 21,728
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,030
Information Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer series in statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 18,636
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,501
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
(1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
Research works printed but not linked: 7
- 10.1016/S0364-0213(85)80012-4 · DOI · Paper page
- Barron A. R., 1989, Florida, 280
- Baum L. E., 1972, Inequalities, 3, 1
- Burr D. J., 1980, IEEE Transact. Pattern Anal. Machine Intell. PAMI-3, 6, 708
- Chauvin Y., 1990, Neural Information Processing Systems 11, D. S. Touretzky, ed., 642
- Denker J., 1987, Complex Syst., 1, 877
- Goldman L., 1982, J. Med., 307, 588
Cited by
Sorted by citations
Items 901–1,000 of 1,006
Power Transformer Fault Diagnosis Using Neural Network Optimization Techniques
Rokani, V., Kaminaris, S. D., Karaisas, P., & Kaminaris, D. (2023) · Mathematics · doi:10.3390/math11224693 · Cited by 38
“The bias–variance dilemma [31,32].”
Geigel, A. (2013) · Journal of Computer Security · doi:10.3233/jcs-2012-0460 · Cited by 38
“The neural Trojan embedding problem is directly related to the bias and variance of the neural network [13].”
Kolodny, O., Lotem, A., & Edelman, S. (2015) · Cognitive Sciences · doi:10.1111/cogs.12140 · Cited by 32
Social learning strategies for matters of taste
Analytis, P. P., Barkoczi, D., & Herzog, S. M. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0343-2 · Cited by 32
“From a bias–variance trade-off perspective [35, 39, 11, 37], this strategy suffers from potentially high bias, especially for people with alternative tastes, but it exhibits zero variance in its prediction error because it does not estimate any free parameters and makes the same prediction regardless of the past exper…”
BackgroundGigerenzer, G. (2002) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195153729.003.0009 · Cited by 39
Signal Processing and Classification Approaches for Brain-Computer Interface
Al-ani, T., & Trad, D. (2010) · InTech eBooks · doi:10.5772/7032 · Cited by 37
“To select the optimal model complexity, we must solve this bias-variance dilemma (Geman et al., 1992).”
MethodWang, Z., Chen, P. C., Yu, L. L., & Harrington, P. D. B. (2013) · Analytical Chemistry · doi:10.1021/ac303445v · Cited by 38
SUSTAIN: a model of human category learning
Love, B. C., & Medin, D. L. (1998) · UCL Discovery (University College London) · Cited by 39
The Big Data Paradox in Clinical Practice
Msaouel, P. (2022) · Cancer Investigation · doi:10.1080/07357907.2022.2084621 · Cited by 38
“Conversely, more complex models will have lower bias (lower “ systematic error ” ) but also higher variance because the effective sample size is reduced (16).”
BackgroundVisual Semantic Information Pursuit: A Survey
Liu, D., Bober, M., & Kittler, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2950025 · Cited by 39
A hybrid deep learning model with error correction for photovoltaic power forecasting
Zhang, R., Li, G., Bu, S., Kuang, G., He, W., Zhu, Y., & Aziz, S. (2022) · Frontiers in Energy Research · doi:10.3389/fenrg.2022.948308 · Cited by 42
“Although different advanced forecasting methods have been proposed to reduce the model bias, they always exist more significant model variance (Geman et al., 1992).”
BackgroundMethodViale, L., Daga, A. P., Fasana, A., & Garibaldi, L. (2023) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2023.110154 · Cited by 37
“This procedure can be observed with a different and useful interpretation thanks to the RMSE decomposition into bias 𝛽 and variance 𝜎 [43].”
Ensembles of Nearest Neighbor Forecasts
Yankov, D., DeCoste, D., & Keogh, E. (2006) · Lecture notes in computer science · doi:10.1007/11871842_51 · Cited by 37
“The effect is known as the bias-variance dilemma [5] and has been observed before in the context of k-NN forecasting [3, 10].”
BackgroundGonzález, D. S. (2008) · Applied Economics Letters · doi:10.1080/13504850600721973 · Cited by 37
Tool with Road-Level Crash Prediction for Transportation Safety Planning
Tarko, A. P., Inerowicz, M., Ramos, J. R., & Li, W. (2008) · Transportation Research Record Journal of the Transportation Research Board · doi:10.3141/2083-03 · Cited by 37
“A subset of variables should be judiciously selected to balance the needs to avoid both overfitting and underfitting (24).”
BackgroundFollow the money: using payment behaviour as predictor for future self-exclusion
Haeusler, J. (2016) · International Gambling Studies · doi:10.1080/14459795.2016.1158306 · Cited by 37
Chen, Q., Xue, B., Niu, B., & Zhang, M. (2016) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec.2016.7744270 · Cited by 37
“In many fields in machine learning, generalisation is treated as an important issue for a long time [3], [5], [6].”
BackgroundPredicting accuracy on large datasets from smaller pilot data
Johnson, M., Anderson, P., Dras, M., & Steedman, M. J. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2072 · Cited by 35
“The inverse square-root curve is what one would expect if the error is distributed according to a Bias-Variance decomposition (Geman et al., 1992) with a constant bias term a and a variance term that asymptotically follows the Central Limit Theorem.”
BackgroundMethodModelling the spatial distribution of three Portunidae crabs in Haizhou Bay, China
Luan, J., Zhang, C., Xu, B., Xue, Y., & Ren, Y. (2018) · PLoS ONE · doi:10.1371/journal.pone.0207457 · Cited by 37
Jin, H., Pan, B., Chen, X., & Qian, B. (2018) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2018.12.002 · Cited by 37
Colour and tropospheric cloud structure of Jupiter from MUSE/VLT: Retrieving a universal chromophore
Braude, A., Irwin, P. G. J., Orton, G. S., & Fletcher, L. N. (2019) · Icarus · doi:10.1016/j.icarus.2019.113589 · Cited by 37
“ue to spectral correlation, as well as systematic ‘bias’ errors from oversmoothing over regions with a high density of solar spectral and telluric lines, a consequence of the ‘bias-variance dilemma’ (Geman et al., 1992). This bias was quantified simply by taking the dierence in radiance at a given wavelength (at SpeX r…”
BackgroundA deep recurrent neural network approach to learn sequence similarities for user-identification
Vamosi, S., Reutterer, T., & Platzer, M. (2022) · Decision Support Systems · doi:10.1016/j.dss.2021.113718 · Cited by 39
A genetic-based subspace analysis method for improving Error-Correcting Output Coding
Bagheri, M. A., Gao, Q., & Escalera, S. (2013) · Pattern Recognition · doi:10.1016/j.patcog.2013.03.014 · Cited by 37
“This idea, usually called subspace approach, can effectively make use of the diversity of base learners to reduce the variance as well as the bias errors [23,24].”
BackgroundArtificial Neural Network Models of Watershed Nutrient Loading
Kim, R. J., Loucks, D. P., & Stedinger, J. R. (2012) · Water Resources Management · doi:10.1007/s11269-012-0045-x · Cited by 37
“Input and target output data dimensions of the ANN determine the number of nodes in the input and the output layers, respectively, but the number of hidden layers and their nodes are determined set by the user (Hajnayeb et al. 2011; Geman et al. 1992; Swingler 1996).”
BackgroundHu, M. Y., Shanker, M. S., & Hung, M. S. (1999) · International Journal of Research in Marketing · doi:10.1016/s0167-8116(99)00018-x · Cited by 37
“Theoretically, model selection should be based on the trade-off between model bias . and model ˝ariance Geman et al., 1992 . The bias of a model relates to the predictive accuracy of the model, whereas variance refers to the variability of the predictions.”
BackgroundMore evidence on size modification in spatial axes systems of varying complexity
Lange‐Küttner, C. (2004) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2004.02.003 · Cited by 37
Development of Object and Grasping Knowledge by Robot Exploration
Kraft, D., Detry, R. J., Pugeault, N., Başeski, E., Guérin, F., Piater, J., & Krüger, N. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2069098 · Cited by 37
“[3]) would inevitably fail because of the huge dimensionalities of the learning spaces of the actual problems [4].”
BackgroundGurumoorthy, K. S., Rajwade, A. V., Banerjee, A., & Rangarajan, A. (2009) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2034991 · Cited by 37
“This i s an instance of the well-known bias-variance tradeoff common in machine learning algorit hms [35].”
SOLOMATINE, D. P., & Wagener, T. (2011) · Elsevier eBooks · doi:10.1016/b978-0-444-53199-5.00044-0 · Cited by 37
Modeling the relationship between corporate strategy and wealth creation using neural networks
John, C. H. S., Balakrishnan, N., & Fiet, J. O. (2000) · Computers & Operations Research · doi:10.1016/s0305-0548(99)00143-4 · Cited by 37
Behavioral and computational aspects of language and its acquisition
Edelman, S., & Waterfall, H. (2007) · Physics of Life Reviews · doi:10.1016/j.plrev.2007.10.001 · Cited by 39
Intrator, N. (1993) · Neural Computation · doi:10.1162/neco.1993.5.3.443 · Cited by 37
“Since this method can approximate any continuous function, great care should be taken so that the variance of the estimator is not large, namely, that the model does not "overfit" the training data (Wahba 1990; Geman et al. 1992, for discussion).”
MethodA computational model for recognizing objects and planning hand shapes in grasping movements
Uno, Y., Fukumura, N., Suzuki, R., & Kawato, M. (1995) · Neural Networks · doi:10.1016/0893-6080(95)00002-h · Cited by 37
Measurement of the W mass in e+e− collisions at production threshold
Barate, R., Buskulic, D., Décamp, D., Ghez, P., Goy, C., Lees, J. P., Lucotte, A., Minard, M.-N., Nief, J. Y., Pietrzyk, B., et al. (1997) · Physics Letters B · doi:10.1016/s0370-2693(97)00460-7 · Cited by 37
Prediction of Type-2 Diabetes Based on Several Element Levels in Blood and Chemometrics
Chen, H., & Tan, C. (2011) · Biological Trace Element Research · doi:10.1007/s12011-011-9306-4 · Cited by 36
“the case of (binary) classificationmodels, the situation was not so clear because the classical bias-variance decomposition of the squared error loss in regression problems had to be extended to the zero–one loss function [31].”
BackgroundTaver, V., Johannet, A., Estupina, V. B., & Pistre, S. (2014) · Hydrological Sciences Journal · doi:10.1080/02626667.2014.967696 · Cited by 39
“The training step could lead to overfitting, which means poor generalization ability, due to the bias-variance dilemma (Geman et al .”
BackgroundUnrestricted Mixture Models for Class Identification in Growth Mixture Modeling
Liu, M., & Hancock, G. R. (2014) · Educational and Psychological Measurement · doi:10.1177/0013164413519798 · Cited by 40
De praeceptis ferendis : good practice in multi-model ensembles
Kioutsioukis, I., & Galmarini, S. (2014) · Atmospheric chemistry and physics · doi:10.5194/acp-14-11791-2014 · Cited by 37
“The dilemmais thus finding an optimal balance between bias and variance in order to make the error as small as possible (Geman et al., 1992; Bishop, 1995).”
BackgroundA nonparametric Bayesian framework for constructing flexible feature representations.
Austerweil, J. L., & Griffiths, T. L. (2013) · Psychological Review · doi:10.1037/a0034194 · Cited by 36
Miyagi, Y., Habara, T., Hirata, R., & Hayashi, N. (2019) · Reproductive Medicine and Biology · doi:10.1002/rmb2.12284 · Cited by 35
Goulier, L., Paas, B., Ehrnsperger, L., & Klemm, O. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17062025 · Cited by 36
“These two demands lead to a conflict and are referred to as the Bias and Variance Dilemma [49].”
BackgroundSomepalli, G., Fowl, L., Bansal, A., Yeh-Chiang, P., Dar, Y., Baraniuk, R. G., Goldblum, M., & Goldstein, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01333 · Cited by 47
“This is known as the Bias-Variance trade-off [14].”
BackgroundOthman, K., & Abdelwahab, H. (2023) · Ain Shams Engineering Journal · doi:10.1016/j.asej.2022.101988 · Cited by 42
On-line Supervised Adaptive Training Using Radial Basis Function Networks
Fung, C. F., Billings, S. A., & Luo, W. (1996) · Neural Networks · doi:10.1016/s0893-6080(96)00024-x · Cited by 36
Invariance and selectivity in the ventral visual pathway
Geman, S. (2006) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2007.01.001 · Cited by 36
Self-citationContext-dependent selection of visuomotor maps
Salinas, E. (2004) · BMC Neuroscience · doi:10.1186/1471-2202-5-47 · Cited by 37
“For each value of γ, the optimal algorithm considers both the mean and the variance of the output responses [32,33], striking a balance between them that, overall, minimizes the average squared difference between the driven and the desired output rates (Equation 11).”
BackgroundThe adaptive toolbox: toward a darwinian rationality.
Gigerenzer, G. (2001) · PubMed · Cited by 36
A robust neural network filter for electricity demand prediction
Connor, J. T. (1996) · Journal of Forecasting · doi:10.1002/(sici)1099-131x(199611)15:6 · Cited by 36
Predictive models of collembolan diversity and abundance in a riparian habitat
Lek‐Ang, S., Deharveng, L., & Lek, S. (1999) · Ecological Modelling · doi:10.1016/s0304-3800(99)00106-4 · Cited by 36
“testing various possibilities and selecting the one that provides the best compromise between bias and variance (Geman et al., 1992; Kohavi, 1995).”
BackgroundRandomness in generalization ability: a source to improve it
Sarkar, D. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.501725 · Cited by 36
Business Forecasting with Artificial Neural Networks
Zhang, G. P. (2011) · IGI Global eBooks · doi:10.4018/978-1-59140-176-6.ch001 · Cited by 36
Local Dimensionality Reduction
Stefan Schaal, A., Vijayakumar, S., & Atkeson, C. G. (1997) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 36
Haykin, S., Yee, P., & Derbez, E. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650104 · Cited by 36
“It is well-known that without such guidelines, the problems of overfitting (too large an ) and underfitting (too small an ) can lead to poor generalization performance, e.g., see [ 6 ].”
BackgroundFuzzy-Wavelet Method for Time Series Analysis
Popoola, A. (2006) · Surrey Research Insight Open Access (The University of Surrey) · Cited by 36
Generalized Thompson sampling for sequential decision-making and causal inferencePreprint
Ortega, P. A., & Braun, D. A. (2014) · Cited by 36
Bring digital twins back to Earth
Saltelli, A., Gigerenzer, G., Hulme, M., Katsikopoulos, K. V., Melsen, L. A., Peters, G. P., Pielke, R. A., Robertson, S., Stirling, A., Tavoni, M., et al. (2024) · Wiley Interdisciplinary Reviews Climate Change · doi:10.1002/wcc.915 · Cited by 35
“ENDNOTE 1 A formalism that can explain this phenomenon and delineate its boundary conditions is the bias-variance trade-off (Geman et al., 1992).”
Coevolutionary learning of neural network ensemble for complex classification tasks
Tian, J., Li, M., Chen, F., & Kou, J. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2011.09.012 · Cited by 35
Smoothing regularizers for projective basis function networks
Moody, J. E. (2018) · doi:10.6083/vx021f42c · Cited by 35
An overview on weight initialization methods for feedforward neural networks
de Sousa, C. A. R. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727180 · Cited by 35
“Unfortunately, these algorithms may have high variance to achieve this goal on some application domains [18].”
BackgroundMojid, M. A., Hossain, A. B. M. Z., & Ashraf, M. (2019) · Environmental Pollution · doi:10.1016/j.envpol.2019.113355 · Cited by 35
Zoveidavianpoor, M., Samsuri, A., & Shadizadeh, S. R. (2013) · Journal of Geophysics and Engineering · doi:10.1088/1742-2132/10/4/045014 · Cited by 38
Prediction of rate constants for the reaction of O3 with different organic compounds
Pompe, M., & Veber, M. (2001) · Atmospheric Environment · doi:10.1016/s1352-2310(01)00165-0 · Cited by 35
Prediction and Classification with Neural Network Models
Zeng, L. (1999) · Sociological Methods & Research · doi:10.1177/0049124199027004002 · Cited by 35
Setzkorn, C., & Paton, R. C. (2005) · Biosystems · doi:10.1016/j.biosystems.2005.02.003 · Cited by 35
Error Estimation and Model Selection
Scheffer, T. (1999) · Cited by 35
Competition among networks improves committee performance
Munro, P., & Parmanto, B. (1997) · Cited by 35
Use of bias term in projection pursuit learning improves approximation and convergence properties
Kwok, T.-Y., & Yeung, D. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.536312 · Cited by 35
“This is usually explained as a manifestation of the bias-variance dilemma [ 34 ].”
BackgroundRakhshan, K., Morel, J.-C., & Daneshkhah, A. (2021) · Journal of Building Engineering · doi:10.1016/j.jobe.2021.102791 · Cited by 53
“Therefore, in selecting the proper method for developing a predictive model, this trade-off between bias and variance should be considered [51,53,54].”
MethodA novel hybrid learning algorithm for full Bayesian approach of artificial neural networks
Kocadağlı, O. (2015) · Applied Soft Computing · doi:10.1016/j.asoc.2015.06.003 · Cited by 35
A novel decorrelated neural network ensemble algorithm for face recognition
Dai, K., Zhao, J., & Cao, F. (2015) · Knowledge-Based Systems · doi:10.1016/j.knosys.2015.09.002 · Cited by 34
Darras, T., Estupina, V. B., Kong-A-Siou, L., Vayssade, B., Johannet, A., & Pistre, S. (2015) · Hydrology and earth system sciences · doi:10.5194/hess-19-4397-2015 · Cited by 34
“This major issue of neural network modelling is called bias–variance trade-off (Geman et al., 1992).”
Surrogate modeling of deformable joint contact using artificial neural networks
Eskinazi, I., & Fregly, B. J. (2015) · Medical Engineering & Physics · doi:10.1016/j.medengphy.2015.06.006 · Cited by 34
Murchison, C. R., Zharikov, Y., & Nol, E. (2016) · Environmental Management · doi:10.1007/s00267-016-0727-x · Cited by 34
“However, as PRNPR is a popular recreational area, seasonal beach closures would undoubtedly be unpopular with local recreationists and tourists alike.”
MethodInfluentialMenezes, M. V. F., Torres, L. C. B., & Braga, A. P. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.08.001 · Cited by 36
Estimating the Required Training Dataset Size for Transmitter Classification Using Deep Learning
Oyedare, T., & Park, J.-M. J. (2019) · IEEE International Symposium on Dynamic Spectrum Access Networks · doi:10.1109/dyspan.2019.8935823 · Cited by 34
“2) Inverse Square Root Model: This model was motivated by Bias-Variance decomposition proposed by Geman et al. [20].”
MethodColoring Graph Neural Networks for Node Disambiguation
Dasoulas, G., Santos, L. D., Scaman, K., & Virmaux, A. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/294 · Cited by 34
Determining cross sections from transport coefficients using deep neural networks
Stokes, P. W., Cocks, D. G., Brunger, M. J., & White, R. D. (2020) · Plasma Sources Science and Technology · doi:10.1088/1361-6595/ab85b6 · Cited by 34
An Introduction to Machine Learning for Panel Data
Chen, J. M. (2021) · International Advances in Economic Research · doi:10.1007/s11294-021-09815-6 · Cited by 34
Classification of mental workload based on multiple features of ECG signals
Qu, H., Gao, X., & Pang, L. (2021) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2021.100575 · Cited by 34
Mojtahedi, F. F., Ahmadihosseini, A., & Sadeghi, H. (2022) · Geotechnical and Geological Engineering · doi:10.1007/s10706-022-02297-1 · Cited by 34
“The standard neural network architectures (e.g. fully connected multilayer perception) are subject to overfitting as they often have an extensive parameter space (Geman et al. 1992).”
BackgroundNeural networks for the no-reference assessment of perceived quality
Gastaldo, P., & Zunino, R. (2005) · Journal of Electronic Imaging · doi:10.1117/1.1988313 · Cited by 35
Analysis of JET charge exchange spectra using neural networks
Svensson, J., von Hellermann, M., & König, R. (1999) · Plasma Physics and Controlled Fusion · doi:10.1088/0741-3335/41/2/016 · Cited by 34
Financial markets: very noisy information processing
Magdon‐Ismail, M., Nicholson, A., & Abu‐Mostafa, Y. S. (1998) · Proceedings of the IEEE · doi:10.1109/5.726786 · Cited by 34
“One can compare this tradeoff to the bias-variance tradeoff [ 8 ].”
BackgroundLearning visual representations for perception-action systems
Piater, J., Jodogne, S., Detry, R. J., Kraft, D., Krüger, N., Kroemer, O., & Peters, J. (2010) · The International Journal of Robotics Research · doi:10.1177/0278364910382464 · Cited by 34
“Geman et al. (1992) compellingly demonstrated the inescapable tradeoff between flexibility of the learning methods and quantities of training data required for stable generalization, the bias/variance dilemma faced by any inductive learning method.”
BackgroundBescoby, D. J., Cawley, G. C., & Chroston, P. N. (2006) · Geophysics · doi:10.1190/1.2231110 · Cited by 34
Mhamdi, E. M. E., & Guerraoui, R. (2017) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2017.66 · Cited by 36
“This creates a dilemma that somehow resembles the famous bias/variance dilemma [35] in machine learning.”
BackgroundClustering-based selective neural network ensemble
Qiang, F., Hu, S., & Shengying, Z. (2005) · Journal of Zhejiang University. Science A · doi:10.1631/jzus.2005.a0387 · Cited by 51
Bias—Variance Analysis and Ensembles of SVM
Valentini, G., & Dietterich, T. G. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45428-4_22 · Cited by 34
“Historically, the bias–variance insight was borrowed from the field of regression, using squared–loss as the loss function [ 7 ].”
MethodLanguage and Literacy from a Cognitive Neuroscience Perspective
Petersson, K. M., Ingvar, M., & Reis, A. (2010) · Cambridge University Press eBooks · doi:10.1017/cbo9780511609664.010 · Cited by 35
Armeni, K., Güçlü, U., van Gerven, M. A. J., & Schoffelen, J. (2022) · Scientific Data · doi:10.1038/s41597-022-01382-7 · Cited by 41
“That is, because the number of optimized parameters is typically very large, the models need to be constrained by sufficient amounts of data points in order to reduce the error variance during training 45 .”
BackgroundGüneralp, İ., Filippi, A. M., & Hales, B. U. (2013) · GIScience & Remote Sensing · doi:10.1080/15481603.2013.778560 · Cited by 33
“Given the bias-variance tradeoff and overfitting problem (Geman, Bienenstock, and Doursat 1992; Montgomery and Peck 1992), SVM classifier aims to attain the best generalization by balancing the relationship between the accuracy achieved on a given finite quantity of training data and the machine…”
BackgroundChen, M., Fan, M.-H., Chen, Y.-L., & Wei, H.-M. (2013) · Neural Network World · doi:10.14311/nnw.2013.23.023 · Cited by 33
“The experimental frequencies of Montgomery’s 2 full factorial design will increase significantly according to the increasing factor number, K, such as 2(3) = 8, 2(4) = 16, and 2(6) = 64 [50].”
BackgroundMethodInfluentialSVD and Clustering for Unsupervised POS Tagging
LaMar, M. D., Maron, Y., Johnson, M., & Bienenstock, E. (2010) · Cited by 33
Self-citationA Neural Network Based Approach to Simulate Electrothermal Device Interaction in SPICE Environment
Chiozzi, D., Bernardoni, M., Delmonte, N., & Cova, P. (2018) · IEEE Transactions on Power Electronics · doi:10.1109/tpel.2018.2863186 · Cited by 33
“In order to model the drain current of a power MOSFET , MLP with bias, which are networks with constants neurons [25], and sigmoid activation functions were used.”
MethodArtificial intelligence in basic and clinical neuroscience: Opportunities and ethical challenges.
Kellmeyer, P. (2019) · Neuroforum · doi:10.1515/nf-2019-0018 · Cited by 45
Birattari, M., Ligot, A., & Francesca, G. (2021) · Natural Computing Series · doi:10.1007/978-3-030-72069-8_5 · Cited by 22
Bias in artificial intelligence systems
Rejmaniak, R. (2021) · Białostockie Studia Prawnicze · doi:10.15290/bsp.2021.26.03.02 · Cited by 22
“30 Th ere are also solutions that deliberately produce a given type of bias in an AI system (statistical bias) in order to counteract other biases.”
BackgroundA happiness degree predictor using the conceptual data structure for deep learning architectures
Pérez-Benito, F. J., Villacampa-Fernández, P., Conejero, J. A., García-Gómez, J. M., & Navarro-Pardo, E. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2017.11.004 · Cited by 20
“as many as incoming inputs), and b) the Bias/Variance Dilemma [55] (existence vs.”
BackgroundA bias/variance decomposition for models using collective inference
Neville, J., & Jensen, D. (2008) · Machine Learning · doi:10.1007/s10994-008-5066-6 · Cited by 21
“In conventional bias/variance analysis, loss is decomposed into three factors: bias, variance, and noise (Geman et al. 1992; Friedman 1997; Domingos 2000; James 2003).”
BackgroundMethodWhy we need biased AI: How including cognitive biases can enhance AI systems
Hagendorff, T., & Fabi, S. (2024) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2023.2178517 · Cited by 20
“Moreover, a paradigm known in psychology as well as in the machine learning community is the bias-variance dilemma (Geman et al., 1992; Gigerenzer & Brighton, 2009; Luxburg & Schoelkopf, 2008).”
Metadata
- Authors
- Geman, S., Bienenstock, E., & Doursat, R.
- Affiliations
- Brown University (US)
- ESPCI Paris (FR)
- ORCID
- Doursat, René
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 1992-01
- Volume
- 4
- Issue
- 1
- Pages
- 1-58
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2076118331
- Cited by
- 3,578
- References
- 96
- Citations by year
- 2026: 125 · 2025: 131 · 2024: 115 · 2023: 128 · 2022: 148 · 2021: 201 · 2020: 181 · 2019: 142 · 2018: 126 · 2017: 97 · 2016: 115 · 2015: 99 · 2014: 103 · 2013: 82 · 2012: 110
- FWCI
- 60.04
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- e-space (Manchester Metropolitan University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and ELM
- Keywords
- bias-variance trade-off, feedforward neural networks, backpropagation, nonparametric regression, nonparametric inference, handwritten digit recognition, machine perception, neural network representations, statistical learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
