The DOI matches its Crossref record; 119 of the 119 references it cites were found in a registry or an index.
Regularization Theory and Neural Networks Architectures
Abstract
We had previously shown that regularization principles lead to approximation schemes that are equivalent to networks with one layer of hidden units, called regularization networks. In particular, standard smoothness functionals lead to a subclass of regularization networks, the well known radial basis functions approximation schemes. This paper shows that regularization networks encompass a much broader range of approximation schemes, including many of the popular general additive models and some of the neural networks. In particular, we introduce new classes of smoothness functionals that lead to different classes of basis functions. Additive splines as well as some tensor product splines can be obtained from appropriate classes of smoothness functionals. Furthermore, the same generalization that extends radial basis functions (RBF) to hyper basis functions (HBF) also leads from additive models to ridge approximation models, containing as special cases Breiman's hinge functions, some forms of projection pursuit regression, and several types of neural networks. We propose to use the term generalized regularization networks for this broad class of approximation schemes that follow from an extension of regularization. In the probabilistic interpretation of regularization, the different classes of basis functions correspond to different classes of prior probabilities on the approximating function spaces, and therefore to different types of smoothness assumptions. In summary, different multilayer networks with one hidden layer, which we collectively call generalized regularization networks, correspond to different classes of priors and associated smoothness functionals in a classical regularization principle. Three broad classes are (1) radial basis functions that can be generalized to hyper basis functions, (2) some tensor product splines, and (3) additive splines that can be generalized to schemes of the type of ridge approximation, hinge functions, and several perceptron-like neural networks with one hidden layer.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., & White, H. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90020-8 · Cited by 21,728
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
de Boor, C. (1978) · Applied Mathematical Sciences · doi:10.1007/978-1-4612-6333-3 · Cited by 11,944
On Estimation of a Probability Density Function and Mode
Parzen, E. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704472 · Cited by 10,585
Singular Integrals and Differentiability Properties of Functions.
Stein, E. M. (1971) · Cited by 10,183
Multilayer feedforward networks are universal approximators
HornikK., StinchcombeM., & WhiteH. (1989) · Neural Networks · doi:10.5555/70405.70408 · Cited by 9,081
Kohonen, T. (1990) · Proceedings of the IEEE · doi:10.1109/5.58325 · Cited by 8,414
Brown, R. A., Hastie, T. J., & Tibshirani, R. J. (1991) · Biometrics · doi:10.2307/2532174 · Cited by 8,176
Modeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011
Aronszajn, N. (1950) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1950-0051437-7 · Cited by 5,380
A generalized inverse for matrices
Penrose, R. (1955) · Mathematical Proceedings of the Cambridge Philosophical Society · doi:10.1017/s0305004100030401 · Cited by 4,551
A general regression neural network
Specht, D. F. (1991) · IEEE Transactions on Neural Networks · doi:10.1109/72.97934 · Cited by 4,492
On the approximate realization of continuous mappings by neural networks
Funahashi, K.-I. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90003-8 · Cited by 4,206
Fast Learning in Networks of Locally-Tuned Processing Units
Moody, J., & Darken, C. J. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.281 · Cited by 4,194
Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
Golub, G. H., Heath, M., & Wahba, G. (1979) · Technometrics · doi:10.1080/00401706.1979.10489751 · Cited by 3,934
Neural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578
Nadaraya, E. A. (1964) · Theory of Probability & Its Applications · doi:10.1137/1109020 · Cited by 3,462
Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks
Broomhead, D. S., & Lowe, D. (1988) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 3,443
Hastie, T., & Tibshirani, R. (1986) · Statistical Science · doi:10.1214/ss/1177013604 · Cited by 3,409
Craven, P., & Wahba, G. (1978) · Numerische Mathematik · doi:10.1007/bf01404567 · Cited by 3,388
Networks for approximation and learning
Poggio, T., & Girosi, F. (1990) · Proceedings of the IEEE · doi:10.1109/5.58326 · Cited by 3,252
Self-citationOn the Uniform Convergence of Relative Frequencies of Events to Their Probabilities
Vapnik, V. N., & Chervonenkis, A. Y. (2015) · Measures of Complexity · doi:10.1007/978-3-319-21852-6_3 · Cited by 3,105
Multiquadric equations of topography and other irregular surfaces
Hardy, R. L. (1971) · Journal of Geophysical Research Atmospheres · doi:10.1029/jb076i008p01905 · Cited by 2,811
Universal approximation bounds for superpositions of a sigmoidal function
Barron, A. R. (1993) · IEEE Transactions on Information Theory · doi:10.1109/18.256500 · Cited by 2,804
Watson, G. S. (1964) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 2,661
Applied nonparametric regression
(1991) · Choice Reviews Online · doi:10.5860/choice.29-0364 · Cited by 2,580
Spline Functions: Basic Theory.
Smith, P. W., & Schumaker, L. (1982) · Mathematics of Computation · doi:10.2307/2007301 · Cited by 2,463
On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities
Vapnik, V. N., & Chervonenkis, A. Y. (1971) · Theory of Probability & Its Applications · doi:10.1137/1116025 · Cited by 2,448
Solution of incorrectly formulated problems and the regularization method
Tikhonov, A. N. (1963) · Medical Entomology and Zoology · Cited by 2,359
Estimation of Dependences Based on Empirical Data
Vapnik, V. (2006) · Information Science and Statistics · doi:10.1007/0-387-34239-7 · Cited by 2,238
Friedman, J. H., & Stuetzle, W. (1981) · Journal of the American Statistical Association · doi:10.1080/01621459.1981.10477729 · Cited by 2,216
Convergence of Stochastic Processes.
Robinson, P. M., & Pollard, D. (1985) · Economica · doi:10.2307/2553898 · Cited by 2,184
Scattered data interpolation: tests of some methods
Franke, R. (1982) · Mathematics of Computation · doi:10.1090/s0025-5718-1982-0637296-4 · Cited by 1,895
What Size Net Gives Valid Generalization?
Baum, E. B., & Haussler, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.1.151 · Cited by 1,541
Parallel Networks that Learn to Pronounce English Text
B, T. J. S. A. (1987) · Cited by 1,514
The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction
Allen, D. M. (1974) · Technometrics · doi:10.1080/00401706.1974.10489157 · Cited by 1,469
Smoothing noisy data with spline functions
Wahba, G. (1975) · Numerische Mathematik · doi:10.1007/bf01437407 · Cited by 1,443
Computational vision and regularization theory
Poggio, T., Torre, V., & Koch, C. (1985) · Nature · doi:10.1038/317314a0 · Cited by 1,412
Self-citationSplines minimizing rotation-invariant semi-norms in Sobolev spaces
Duchon, J. (1977) · Lecture Notes in Mathematics · doi:10.1007/bfb0086566 · Cited by 1,380
Interpolation of Scattered Data: Distance Matrices and Conditionally Positive Definite Functions
Micchelli, C. A. (1984) · Approximation Theory and Spline Functions · doi:10.1007/978-94-009-6466-2_7 · Cited by 1,297
Huber, P. J. (1985) · The Annals of Statistics · doi:10.1214/aos/1176349519 · Cited by 1,277
Interpolation of scattered data: Distance matrices and conditionally positive definite functions
Micchelli, C. A. (1986) · Constructive Approximation · doi:10.1007/bf01893414 · Cited by 1,247
Additive Regression and Other Nonparametric Models
Stone, C. J. (1985) · The Annals of Statistics · doi:10.1214/aos/1176349548 · Cited by 1,232
Radial basis functions for multivariable interpolation: a review
Powell, M. J. D. (1987) · Clarendon Press eBooks · Cited by 1,197
(2017) · Approximation Techniques for Engineers · doi:10.1201/9781315205007-4 · Cited by 1,155
Spline Smoothing and Nonparametric Regression.
Jones, M. C., & Eubank, R. L. (1989) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982831 · Cited by 1,031
Self-citationRegularization Algorithms for Learning That Are Equivalent to Multilayer Networks
Poggio, T., & Girosi, F. (1990) · Science · doi:10.1126/science.247.4945.978 · Cited by 1,026
Self-citationBellman, R. (1964) · IEEE Transactions on Circuit Theory · doi:10.1109/tct.1964.1082221 · Cited by 991
Linear Smoothers and Additive Models
Buja, A., Hastie, T., & Tibshirani, R. (1989) · The Annals of Statistics · doi:10.1214/aos/1176347115 · Cited by 989
A Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines
Kimeldorf, G. S., & Wahba, G. (1970) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177697089 · Cited by 939
Learning in Artificial Neural Networks: A Statistical Perspective
White, H. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.425 · Cited by 922
Networks and the best approximation property
Girosi, F., & Poggio, T. (1990) · Biological Cybernetics · doi:10.1007/bf00195855 · Cited by 888
Self-citation(2018) · Physics Subject Headings (PhySH) · doi:10.29172/0660b3fe-654b-49c9-a7fa-79f18efe2aa0 · Cited by 882
Decision theoretic generalizations of the PAC model for neural net and other learning applications
Haussler, D. (1992) · Information and Computation · doi:10.1016/0890-5401(92)90010-d · Cited by 843
Interpolation using surface splines.
HARDER, R. L., & DESMARAIS, R. N. (1972) · Journal of Aircraft · doi:10.2514/3.44330 · Cited by 770
Probabilistic Solution of Ill-Posed Problems in Computational Vision
Marroquin, J., Mitter, S., & Poggio, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478393 · Cited by 766
Self-citationIll-posed problems in early vision
Bertero, M., Poggio, T. A., & Torre, V. (1988) · Proceedings of the IEEE · doi:10.1109/5.5962 · Cited by 723
Self-citationA Theory of Networks for Approximation and Learning
Poggio, T., & Girosi, F. (1989) · Cited by 703
Self-citationConnectionist nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings
White, H. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90004-5 · Cited by 679
Neural Networks and Related Methods for Classification
Ripley, B. D. (1994) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1994.tb01990.x · Cited by 627
Regression and the Moore-Penrose Pseudoinverse.
Troskie, C. G., & Albert, A. (1975) · Journal of the American Statistical Association · doi:10.2307/2285416 · Cited by 620
Asymptotics of Graphical Projection Pursuit
Diaconis, P., & Freedman, D. (1984) · The Annals of Statistics · doi:10.1214/aos/1176346703 · Cited by 547
Bottou, L., & Vapnik, V. (1992) · Neural Computation · doi:10.1162/neco.1992.4.6.888 · Cited by 542
Wahba, G. (1985) · The Annals of Statistics · doi:10.1214/aos/1176349743 · Cited by 523
Aronszajn, N. (1950) · Transactions of the American Mathematical Society · doi:10.2307/1990404 · Cited by 495
Review of Neural Networks for Speech Recognition
Lippmann, R. P. (1989) · Neural Computation · doi:10.1162/neco.1989.1.1.1 · Cited by 477
Jones, L. K. (1992) · The Annals of Statistics · doi:10.1214/aos/1176348546 · Cited by 474
(1994) · Control Engineering Practice · doi:10.1016/0967-0661(94)90338-7 · Cited by 472
Non-Parametric Function Fitting
Priestley, M. B., & Chao, M. T. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00916.x · Cited by 467
Hinging hyperplanes for regression, classification, and function approximation
Breiman, L. (1993) · IEEE Transactions on Information Theory · doi:10.1109/18.256506 · Cited by 445
Scattered Data Interpolation: Tests of Some Method
Franke, R. (1982) · Mathematics of Computation · doi:10.2307/2007474 · Cited by 443
Spline Smoothing: The Equivalent Variable Kernel Method
Silverman, B. W. (1984) · The Annals of Statistics · doi:10.1214/aos/1176346710 · Cited by 441
Moody, J. E. (1991) · Cited by 437
Estimating regression functions and their derivatives by the kernel method
Gasser, T., & Müller, H. (1984) · Scandinavian Journal of Statistics · Cited by 423
Multivariate interpolation at arbitrary points made simple
Meinguet, J. (1979) · Zeitschrift für angewandte Mathematik und Physik ZAMP · doi:10.1007/bf01601941 · Cited by 413
Generalized Additive Models; Some Applications
Hastie, T., & Tibshirani, R. (1985) · Lecture Notes in Statistics · doi:10.1007/978-1-4615-7070-7_8 · Cited by 407
Numerical Procedures for Surface Fitting of Scattered Data by Radial Functions
Dyn, N., Levin, D., & Rippa, S. (1986) · SIAM Journal on Scientific and Statistical Computing · doi:10.1137/0907043 · Cited by 402
Approximation and estimation bounds for artificial neural networks
Barron, A. R. (1994) · Machine Learning · doi:10.1007/bf00993164 · Cited by 399
Capabilities of three-layered perceptrons
Irie, & Miyake (1988) · IEEE International Conference on Neural Networks · doi:10.1109/icnn.1988.23901 · Cited by 398
Complexity-based induction systems: Comparisons and convergence theorems
Solomonoff, R. (1978) · IEEE Transactions on Information Theory · doi:10.1109/tit.1978.1055913 · Cited by 388
On the capabilities of multilayer perceptrons
Baum, E. B. (1988) · Journal of Complexity · doi:10.1016/0885-064x(88)90020-9 · Cited by 379
A completely automatic french curve: fitting spline functions by cross validation
Wahba, G., & Wold, S. (1975) · Communications in Statistics · doi:10.1080/03610927508827223 · Cited by 366
Poggio, T., & Beymer, D. (1996) · IEEE Spectrum · doi:10.1109/6.490058 · Cited by 332
Self-citationNeural Networks for Optimal Approximation of Smooth and Analytic Functions
Mhaskar, H. N. (1996) · Neural Computation · doi:10.1162/neco.1996.8.1.164 · Cited by 321
Cortical connections and parallel processing: Structure and function
Ballard, D. H. (1986) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00021555 · Cited by 299
Lapedes, A., & Farber, R. (1989) · Evolution, Learning and Cognition · doi:10.1142/9789814434102_0012 · Cited by 292
Dynamic Model of Visual Recognition Predicts Neural Response Properties in the Visual Cortex
Rao, R. P. N., & Ballard, D. H. (1997) · Neural Computation · doi:10.1162/neco.1997.9.4.721 · Cited by 290
Rosenblatt, M. (1971) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177693050 · Cited by 284
Approximation by superposition of sigmoidal and radial basis functions
Mhaskar, H. N., & Micchelli, C. A. (1992) · Advances in Applied Mathematics · doi:10.1016/0196-8858(92)90016-p · Cited by 257
A Theory of How the Brain Might Work
Poggio, T. (1990) · Cold Spring Harbor Symposia on Quantitative Biology · doi:10.1101/sqb.1990.055.01.084 · Cited by 252
Self-citationDevroye, L. P., & Wagner, T. J. (1980) · The Annals of Statistics · doi:10.1214/aos/1176344949 · Cited by 230
Optimal nonlinear approximation
DeVore, R. A., Howard, R., & Micchelli, C. (1989) · Manuscripta Mathematica · doi:10.1007/bf01171759 · Cited by 227
Synthesizing a color algorithm from examples
Hurlbert, A., & Poggio, T. (1988) · Science · doi:10.1126/science.239.4839.482 · Cited by 210
Self-citationNiyogi, P., & Girosi, F. (1996) · Neural Computation · doi:10.1162/neco.1996.8.4.819 · Cited by 172
Self-citationPositive definite functions and generalizations, an historical survey
Stewart, J. (1976) · Rocky Mountain Journal of Mathematics · doi:10.1216/rmj-1976-6-3-409 · Cited by 167
NMDA-Based Pattern Discrimination in a Modeled Cortical Neuron
Mel, B. W. (1992) · Neural Computation · doi:10.1162/neco.1992.4.4.502 · Cited by 159
Approximation properties of a multilayered feedforward artificial neural network
Mhaskar, H. N. (1993) · Advances in Computational Mathematics · doi:10.1007/bf02070821 · Cited by 157
Yuille, A., & Grzywacz, N. M. (2005) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/ccv.1988.590011 · Cited by 141
On optimal nonlinear associative recall
Poggio, T. (1975) · Biological Cybernetics · doi:10.1007/bf02281970 · Cited by 139
Self-citationNote on generalization, regularization and architecture selection in nonlinear learning systems
Moody, J. E. (2002) · Machine learning for signal processing ... · doi:10.1109/nnsp.1991.239541 · Cited by 138
A computational theory of visual surface interpolation
Grimson, W. E. L. (1982) · Philosophical transactions of the Royal Society of London. Series B, Biological sciences · doi:10.1098/rstb.1982.0088 · Cited by 136
A regularized solution to edge detection
Poggio, T., Voorhees, H. L., & Yuille, A. (1988) · Journal of Complexity · doi:10.1016/0885-064x(88)90024-6 · Cited by 131
Self-citationGeneralized Additive Models: Some Applications
Hastie, T., & Tibshirani, R. J. (1987) · Journal of the American Statistical Association · doi:10.2307/2289439 · Cited by 117
Multivariate Smoothing Spline Functions
Cox, D. D. (1984) · SIAM Journal on Numerical Analysis · doi:10.1137/0721053 · Cited by 110
Johansen, T. (1996) · at - Automatisierungstechnik · doi:10.1016/0005-1098(95)00146-8 · Cited by 100
Regularization methods for linear inverse problems
Bertero, M. (1986) · Lecture notes in mathematics · doi:10.1007/bfb0072660 · Cited by 87
Regularization Theory, Radial Basis Functions and Networks
Girosi, F. (1994) · Springer eBooks · doi:10.1007/978-3-642-79119-2_8 · Cited by 84
Self-citationNetwork Structuring and Training Using Rule-based Knowledge
Volker Tresp, C., Hollatz, J., & Ahmad, S. (1992) · Cited by 78
Some extensions of radial basis functions and their applications in artificial intelligence
Girosi, F. (1992) · Computers & Mathematics with Applications · doi:10.1016/0898-1221(92)90172-e · Cited by 76
Self-citationLimitations of the approximation capabilities of neural networks with one hidden layer
Chui, C. K., Li, X., & Mhaskar, H. N. (1996) · Advances in Computational Mathematics · doi:10.1007/bf02124745 · Cited by 73
Shakespeare vs. fletcher: A stylometric analysis by radial basis functions
Lowe, D. G., & Matthews, R. (1995) · Computers and the Humanities · doi:10.1007/bf01829876 · Cited by 66
A Smoothing Regularizer for Feedforward and Recurrent Neural Networks
Wu, L., & Moody, J. E. (1996) · Neural Computation · doi:10.1162/neco.1996.8.3.461 · Cited by 51
MURPHY: A Robot that Learns by Doing
Mel, B. W. (1987) · neural information processing systems · Cited by 48
Cross-validation techniques for smoothing spline functions in one or two dimensions
Utreras, F. I. (1979) · Lecture notes in mathematics · doi:10.1007/bfb0098498 · Cited by 42
Smoothing and Ill-Posed Problems
Wahba, G. (1979) · Springer eBooks · doi:10.1007/978-1-4757-1466-1_7 · Cited by 37
Neural networks for localized approximation of real functions
Mhaskar, H. N. (2002) · Machine learning for signal processing ... · doi:10.1109/nnsp.1993.471870 · Cited by 27
Models of Noise and Robust Estimation
Girosi, F. (1991) · Defense Technical Information Center (DTIC) · Cited by 23
Self-citationObservations on Cortical Mechanisms for Object Recognition and Learning
Poggio, T., & Hurlbert, A. C. (1993) · Cited by 16
Self-citationNonlinear dilation network for prediction applications
Chow, T. W. S., & Leung, C.-T. (1996) · Engineering Applications of Artificial Intelligence · doi:10.1016/0952-1976(96)00020-6 · Cited by 1
Cited by
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
Shape matching and object recognition using shape contexts
Belongie, S., Malik, J., & Puzicha, J. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.993558 · Cited by 6,306
An overview of statistical learning theory
Vapnik, V. N. (1999) · IEEE Transactions on Neural Networks · doi:10.1109/72.788640 · Cited by 6,266
Point Set Registration: Coherent Point Drift
Myronenko, A., & Song, X. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.46 · Cited by 2,762
Advances in Large-Margin Classifiers
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/1113.001.0001 · Cited by 1,869
Metadata
- Authors
- Girosi, F., Jones, M., & Poggio, T.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 1995-03
- Volume
- 7
- Issue
- 2
- Pages
- 219-269
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W1981025032
- Cited by
- 1,349
- References
- 119
- FWCI
- 41.17
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.