The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index.
Neural Networks and the Bias/Variance Dilemma
Abstract
Feedforward neural networks trained by error backpropagation are examples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numerals. In way of conclusion, we suggest that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns.”
From Reconciling modern machine-learning practice and the classical bias–variance trade-off (2019) · cites this paper as [17]
In the citing paper
“The concept of bias in machine learning (ML) research and more generally in the field of predictive modeling is intrinsically tied to the concept of variance.14”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [14]
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,867
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., & White, H. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90020-8 · Cited by 21,728
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,030
Information Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer series in statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 18,636
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,501
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
(1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881
Multilayer feedforward networks are universal approximators
HornikK., StinchcombeM., & WhiteH. (1989) · Neural Networks · doi:10.5555/70405.70408 · Cited by 9,266
Multivariate Adaptive Regression Splines
Friedman, J. H. (1991) · The Annals of Statistics · doi:10.1214/aos/1176347963 · Cited by 8,108
An introduction to computing with neural nets
Lippmann, R. P. (1987) · IEEE ASSP Magazine · doi:10.1109/massp.1987.1165576 · Cited by 7,890
Hansen, L. K., & Salamon, P. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.58871 · Cited by 4,260
On the approximate realization of continuous mappings by neural networks
Funahashi, K.-I. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90003-8 · Cited by 4,206
Connectionism and cognitive architecture: A critical analysis
Fodor, J. A., & Pylyshyn, Z. W. (1988) · Cognition · doi:10.1016/0010-0277(88)90031-5 · Cited by 4,022
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
Plackett, R. L., & Cox, D. R. (1971) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344247 · Cited by 2,274
Estimation of Dependences Based on Empirical Data
Vapnik, V. N. (2006) · Information science and statistics · doi:10.1007/0-387-34239-7 · Cited by 2,252
Convergence of Stochastic Processes.
Robinson, P. M., & Pollard, D. (1985) · Economica · doi:10.2307/2553898 · Cited by 2,225
On the proper treatment of connectionism
Smolensky, P. (1988) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00052432 · Cited by 2,146
Friedman, J. H., & Stuetzle, W. (1981) · Journal of the American Statistical Association · doi:10.1080/01621459.1981.10477729 · Cited by 2,116
Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms
White, B. W., & Rosenblatt, F. (1963) · The American Journal of Psychology · doi:10.2307/1419730 · Cited by 1,818
Estimating Optimal Transformations for Multiple Regression and Correlation
Breiman, L., & Friedman, J. H. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10478157 · Cited by 1,713
What Size Net Gives Valid Generalization?
Baum, E. B., & Haussler, D. H. (1989) · Neural Computation · doi:10.1162/neco.1989.1.1.151 · Cited by 1,549
Baum, L. E. (1972) · Medical Entomology and Zoology · Cited by 1,405
Auto-association by multilayer perceptrons and singular value decomposition
Bourlard, H., & Kamp, Y. (1988) · Biological Cybernetics · doi:10.1007/bf00332918 · Cited by 1,391
Neural networks and principal component analysis: Learning from examples without local minima
Baldi, P., & Hornik, K. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90014-2 · Cited by 1,383
The Representation and Matching of Pictorial Structures
Fischler, M. A., & Elschlager, R. A. (1973) · IEEE Transactions on Computers · doi:10.1109/t-c.1973.223602 · Cited by 1,322
Huber, P. J. (1985) · The Annals of Statistics · doi:10.1214/aos/1176349519 · Cited by 1,277
Multiresolution elastic matching
Bajcsy, R., & Kovačič, S. (1989) · Computer Vision Graphics and Image Processing · doi:10.1016/s0734-189x(89)80014-3 · Cited by 1,118
The Correlation Theory of Brain Function
von der Malsburg, C. (1994) · Physics of neural networks · doi:10.1007/978-1-4612-4320-5_2 · Cited by 1,089
Learning and relearning in Boltzmann machines
Hinton, G. E., & Sejnowski, T. J. (2001) · Graphical Models · doi:10.7551/mitpress/3349.003.0005 · Cited by 1,037
Regularization Algorithms for Learning That Are Equivalent to Multilayer Networks
Poggio, T., & Girosi, F. (1990) · Science · doi:10.1126/science.247.4945.978 · Cited by 1,025
Bellman, R. (1964) · IEEE Transactions on Circuit Theory · doi:10.1109/tct.1964.1082221 · Cited by 1,003
Learning in Artificial Neural Networks: A Statistical Perspective
White, H. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.425 · Cited by 924
Stochastic Complexity and Modeling
Rissanen, J. (1986) · The Annals of Statistics · doi:10.1214/aos/1176350051 · Cited by 908
Anderson, J. A., Silverstein, J. W., Ritz, S. A., & Jones, R. S. (1977) · Psychological Review · doi:10.1037/0033-295x.84.5.413 · Cited by 891
A Statistical Model for Positron Emission Tomography
Vardi, Y., Shepp, L. A., & Kaufman, L. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10477119 · Cited by 863
Neurocomputing: Foundations of Research
Anderson, J. A., & Rosenfeld, E. (1988) · Cited by 851
Decision theoretic generalizations of the PAC model for neural net and other learning applications
Haussler, D. H. (1992) · Information and Computation · doi:10.1016/0890-5401(92)90010-d · Cited by 845
Layered Neural Networks with Gaussian Hidden Units as Universal Approximations
Hartman, E., Keeler, J. D., & Kowalski, J. M. (1990) · Neural Computation · doi:10.1162/neco.1990.2.2.210 · Cited by 785
The recent excitement about neural networks
Crick, F. (1989) · Nature · doi:10.1038/337129a0 · Cited by 740
Cross-Validatory Choice and Assessment of Statistical Predictions (With Discussion)
Stone, M. (1976) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1976.tb01573.x · Cited by 704
Connectionist nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings
White, H. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90004-5 · Cited by 678
A simple procedure for pruning back-propagation trained neural networks
Karnin, E. D. (1990) · IEEE Transactions on Neural Networks · doi:10.1109/72.80236 · Cited by 672
On the Consistency of Bayes Estimates
Diaconis, P. W., & Freedman, D. A. (1986) · The Annals of Statistics · doi:10.1214/aos/1176349830 · Cited by 614
A Theory of Adaptive Pattern Classifiers
Амари, Ш.-И. (1967) · IEEE Transactions on Electronic Computers · doi:10.1109/pgec.1967.264666 · Cited by 564
von der Malsburg, C. (1986) · Springer eBooks · doi:10.1007/978-3-642-70911-1_10 · Cited by 549
Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment
Mozer, M. C., & Smolensky, P. (1988) · neural information processing systems · Cited by 540
Wahba, G. (1985) · The Annals of Statistics · doi:10.1214/aos/1176349743 · Cited by 524
A Computer-Derived Protocol to Aid in the Diagnosis of Emergency Room Patients with Acute Chest Pain
Goldman, L., Weinberg, M. S., Weisberg, M. C., Olshen, R. A., Cook, E. F., Rk, S., Ga, L., Dennis, C. A., Wilson, C., Deckelbaum, L. I., et al. (1982) · New England Journal of Medicine · doi:10.1056/nejm198209023071004 · Cited by 502
Review of Neural Networks for Speech Recognition
Lippmann, R. P. (1989) · Neural Computation · doi:10.1162/neco.1989.1.1.1 · Cited by 483
Friedman, J. H., & Stuetzle, W. (1981) · Journal of the American Statistical Association · doi:10.2307/2287576 · Cited by 475
Biased and Unbiased Cross-Validation in Density Estimation
Scott, D. W., & Terrell, G. R. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478550 · Cited by 467
Estimating Optimal Transformations for Multiple Regression and Correlation
Breiman, L., & Friedman, J. H. (1985) · Journal of the American Statistical Association · doi:10.2307/2288473 · Cited by 452
How Far Are Automatically Chosen Regression Smoothing Parameters From Their Optimum?
Härdle, W. K., Hall, P. M., & Marron, J. S. (1988) · Journal of the American Statistical Association · doi:10.1080/01621459.1988.10478568 · Cited by 400
Nonparametric Maximum Likelihood Estimation by the Method of Sieves
Geman, S., & Hwang, C.-R. (1982) · The Annals of Statistics · doi:10.1214/aos/1176345782 · Cited by 370
Self-citationA completely automatic french curve: fitting spline functions by cross validation
Wahba, G., & Wold, S. (1975) · Communications in Statistics · doi:10.1080/03610927508827223 · Cited by 369
Links between Markov models and multilayer perceptrons
Bourlard, H., & Wellekens, C. J. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.62605 · Cited by 350
Structural Image Restoration through Deformable Templates
Amit, Y., Grenander, U., & Piccioni, M. (1991) · Journal of the American Statistical Association · doi:10.1080/01621459.1991.10475053 · Cited by 325
A Statistical Model for Positron Emission Tomography
Vardi, Y., Shepp, L. A., & Kaufman, L. (1985) · Journal of the American Statistical Association · doi:10.2307/2288030 · Cited by 299
Stopping Rule for the MLE Algorithm Based on Statistical Hypothesis Testing
Veklerov, E., & Llacer, J. (1987) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.1987.4307849 · Cited by 254
Generalization and Parameter Estimation in Feedforward Nets: Some Experiments
Morgan, N., & Bourlard, H. (1989) · neural information processing systems · Cited by 225
Generalized Deformable Models, Statistical Physics, and Matching Problems
Yuille, A. (1990) · Neural Computation · doi:10.1162/neco.1990.2.1.1 · Cited by 219
Automatic smoothing parameter selection: A survey
Marron, J. S. (1988) · Empirical Economics · doi:10.1007/bf01972448 · Cited by 195
Elastic Matching of Line Drawings
Burr, D. (1981) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1981.4767176 · Cited by 185
Consistent inference of probabilities in layered networks: predictions and generalizations
Tishby, Левин, & Solla (1989) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1989.118274 · Cited by 182
Complexity Regularization with Application to Artificial Neural Networks
Barron, A. R. (1991) · Springer eBooks · doi:10.1007/978-94-011-3222-0_42 · Cited by 160
How Far Are Automatically Chosen Regression Smoothing Parameters From Their Optimum?
Härdle, W. K., Hall, P., & Marron, J. S. (1988) · Journal of the American Statistical Association · doi:10.2307/2288922 · Cited by 152
Universal Donsker Classes and Metric Entropy
Dudley, R. M. (2010) · Springer eBooks · doi:10.1007/978-1-4419-5821-1_19 · Cited by 141
Bootstrap Choice of Bandwidth for Density Estimation
Faraway, J. J., & Jhun, M. (1990) · Journal of the American Statistical Association · doi:10.1080/01621459.1990.10474983 · Cited by 126
Collins, Ghosh, & Scofield (1988) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1988.23960 · Cited by 103
Ackley, D. H., Hinton, G. E., & B, T. J. S. A. (1987) · Elsevier eBooks · doi:10.1016/b978-0-08-051581-6.50053-2 · Cited by 102
Universal Donsker Classes and Metric Entropy
Dudley, R. M. (1987) · The Annals of Probability · doi:10.1214/aop/1176991978 · Cited by 101
Exhaustive Thermodynamical Analysis of Boolean Learning Networks
Carnevali, P., & Patarnello, S. (1987) · Europhysics Letters (EPL) · doi:10.1209/0295-5075/4/10/020 · Cited by 99
Recognizing Hand-Printed Letters and Digits Using Backpropagation Learning
Martin, G. L., & Pittman, J. A. (1991) · Neural Computation · doi:10.1162/neco.1991.3.2.258 · Cited by 88
Biased and Unbiased Cross-Validation in Density Estimation
Scott, D. W., & Terrell, G. R. (1987) · Journal of the American Statistical Association · doi:10.2307/2289391 · Cited by 86
The “rubber-mask” technique—I. Pattern measurement and analysis
Widrow, B. C. (1973) · Pattern Recognition · doi:10.1016/0031-3203(73)90042-3 · Cited by 84
Statistical properties of artificial neural networks
Barron, A. R. (2003) · Proceedings of the IEEE Conference on Decision & Control, including the Symposium on Adaptive Processes · doi:10.1109/cdc.1989.70117 · Cited by 81
On empirical spectral analysis of stochastic processes
Grenander, U. (1952) · Arkiv för matematik · doi:10.1007/bf02591360 · Cited by 76
A Proposal for More Powerful Learning Algorithms
Baum, E. B. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.201 · Cited by 73
Schwartz, D. B., Samalam, V. K., Solla, S. A., & Denker, J. S. (1990) · Neural Computation · doi:10.1162/neco.1990.2.3.374 · Cited by 73
von der Malsburg, C., & Bienenstock, E. (1986) · Springer eBooks · doi:10.1007/978-3-642-82657-3_26 · Cited by 72
Self-citationDualistic geometry of the manifold of higher-order neurons
Амари, Ш.-И. (1991) · Neural Networks · doi:10.1016/0893-6080(91)90040-c · Cited by 69
On the Nonconsistency of Maximum Likelihood Nonparametric Density Estimators
Schuster, E. F., & Gregory, G. G. (1981) · Springer eBooks · doi:10.1007/978-1-4613-9464-8_43 · Cited by 62
A cross validated bayesian retrieval algorithm for nonlinear remote sensing experiments
O’Sullivan, F., & Wahba, G. (1985) · Journal of Computational Physics · doi:10.1016/0021-9991(85)90121-4 · Cited by 62
Convergence rates of "thin plate" smoothing splines wihen the data are noisy
Wahba, G. (1979) · Lecture notes in mathematics · doi:10.1007/bfb0098499 · Cited by 61
The Perceptron Algorithm is Fast for Nonmalicious Distributions
Baum, E. B. (1990) · Neural Computation · doi:10.1162/neco.1990.2.2.248 · Cited by 61
Haussler, D. H. (1989) · Annual Symposium on Foundations of Computer Science · doi:10.1109/sfcs.1989.63453 · Cited by 61
The Bootstrap Widrow-Hoff Rule as a Cluster-Formation Algorithm
Hinton, G. E., & Nowlan, S. J. (1990) · Neural Computation · doi:10.1162/neco.1990.2.3.355 · Cited by 43
Structural Image Restoration Through Deformable Templates
Amit, Y., Grenander, U., & Piccioni, M. (1991) · Journal of the American Statistical Association · doi:10.2307/2290581 · Cited by 43
Projection pursuit learning networks for regression
Maechler, M., Martin, D., Schimert, J., Csoppenszky, M. A., & Hwang, J. N. (2002) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/tai.1990.130362 · Cited by 38
Dynamic behavior of constrained back propagation networks
Chauvin, Y. (1990) · neural information processing systems · Cited by 38
Bootstrap Choice of Bandwidth for Density Estimation
Faraway, J. J., & Jhun, M. (1990) · Journal of the American Statistical Association · doi:10.2307/2289609 · Cited by 35
When are k-nearest neighbor and back propagation accurate for feasible sized sets of examples?
Baum, E. B. (1990) · Lecture notes in computer science · doi:10.1007/3-540-52255-7_24 · Cited by 29
Synchronous Boltzmann Machines and Gibbs Fields: Learning Algorithms
Azencott, R. (1990) · Neurocomputing · doi:10.1007/978-3-642-76153-9_6 · Cited by 20
Experiments with Representation in Neural Networks: Object Motion, Speech, and Arithmetic
Anderson, J. A., Rossen, M. L., Viscuso, S. R., & Sereno, M. E. (1990) · Springer series in synergetics · doi:10.1007/978-3-642-48779-8_3 · Cited by 14
Speech recognition using time-delay neural networks
Waibel, A. H., Hanazawa, T., Shikano, K., Hinton, G. E., & Lang, K. (1988) · The Journal of the Acoustical Society of America · doi:10.1121/1.2025362 · Cited by 13
Elastic matching and pattern recognition in neural networks.
Bienenstock, E., & Doursat, R. (1989) · Cited by 11
Self-citationMultilayer perceptrons and data analysis.
Gallinari, P., Thiria, S., & Fogelman, F. S. (1988) · Cited by 6
Flagged first, then by citations
Research works printed but not linked: 7
- 10.1016/S0364-0213(85)80012-4 · DOI · Paper page
- Barron A. R., 1989, Florida, 280
- Baum L. E., 1972, Inequalities, 3, 1
- Burr D. J., 1980, IEEE Transact. Pattern Anal. Machine Intell. PAMI-3, 6, 708
- Chauvin Y., 1990, Neural Information Processing Systems 11, D. S. Touretzky, ed., 642
- Denker J., 1987, Complex Syst., 1, 877
- Goldman L., 1982, J. Med., 307, 588
Cited by
Sorted by citations
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
A Tutorial on Support Vector Machines for Pattern Recognition
Burges, C. J. C. (1998) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009715923555 · Cited by 16,616
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300
Wrappers for feature subset selection
Kohavi, R., & John, G. H. (1997) · Artificial Intelligence · doi:10.1016/s0004-3702(97)00043-x · Cited by 9,096
A training algorithm for optimal margin classifiers
Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130401 · Cited by 11,992
Metadata
- Authors
- Geman, S., Bienenstock, E., & Doursat, R.
- Affiliations
- Brown University (US)
- ESPCI Paris (FR)
- ORCID
- Doursat, René
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 1992-01
- Volume
- 4
- Issue
- 1
- Pages
- 1-58
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2076118331
- Cited by
- 3,578
- References
- 96
- Citations by year
- 2026: 125 · 2025: 131 · 2024: 115 · 2023: 128 · 2022: 148 · 2021: 201 · 2020: 181 · 2019: 142 · 2018: 126 · 2017: 97 · 2016: 115 · 2015: 99 · 2014: 103 · 2013: 82 · 2012: 110
- FWCI
- 60.04
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- e-space (Manchester Metropolitan University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and ELM
- Keywords
- bias-variance trade-off, feedforward neural networks, backpropagation, nonparametric regression, nonparametric inference, handwritten digit recognition, machine perception, neural network representations, statistical learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review