The DOI matches its Crossref record; 119 of the 119 references it cites were found in a registry or an index.
Regularization Theory and Neural Networks Architectures
Abstract
We had previously shown that regularization principles lead to approximation schemes that are equivalent to networks with one layer of hidden units, called regularization networks. In particular, standard smoothness functionals lead to a subclass of regularization networks, the well known radial basis functions approximation schemes. This paper shows that regularization networks encompass a much broader range of approximation schemes, including many of the popular general additive models and some of the neural networks. In particular, we introduce new classes of smoothness functionals that lead to different classes of basis functions. Additive splines as well as some tensor product splines can be obtained from appropriate classes of smoothness functionals. Furthermore, the same generalization that extends radial basis functions (RBF) to hyper basis functions (HBF) also leads from additive models to ridge approximation models, containing as special cases Breiman's hinge functions, some forms of projection pursuit regression, and several types of neural networks. We propose to use the term generalized regularization networks for this broad class of approximation schemes that follow from an extension of regularization. In the probabilistic interpretation of regularization, the different classes of basis functions correspond to different classes of prior probabilities on the approximating function spaces, and therefore to different types of smoothness assumptions. In summary, different multilayer networks with one hidden layer, which we collectively call generalized regularization networks, correspond to different classes of priors and associated smoothness functionals in a classical regularization principle. Three broad classes are (1) radial basis functions that can be generalized to hyper basis functions, (2) some tensor product splines, and (3) additive splines that can be generalized to schemes of the type of ridge approximation, hinge functions, and several perceptron-like neural networks with one hidden layer.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., & White, H. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90020-8 · Cited by 21,728
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
de Boor, C. (1978) · Applied Mathematical Sciences · doi:10.1007/978-1-4612-6333-3 · Cited by 11,944
On Estimation of a Probability Density Function and Mode
Parzen, E. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704472 · Cited by 10,585
Singular Integrals and Differentiability Properties of Functions.
Stein, E. M. (1971) · Cited by 10,183
Multilayer feedforward networks are universal approximators
HornikK., StinchcombeM., & WhiteH. (1989) · Neural Networks · doi:10.5555/70405.70408 · Cited by 9,081
Kohonen, T. (1990) · Proceedings of the IEEE · doi:10.1109/5.58325 · Cited by 8,414
Brown, R. A., Hastie, T. J., & Tibshirani, R. J. (1991) · Biometrics · doi:10.2307/2532174 · Cited by 8,176
Modeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011
Aronszajn, N. (1950) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1950-0051437-7 · Cited by 5,380
Cited by
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
Shape matching and object recognition using shape contexts
Belongie, S., Malik, J., & Puzicha, J. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.993558 · Cited by 6,306
An overview of statistical learning theory
Vapnik, V. N. (1999) · IEEE Transactions on Neural Networks · doi:10.1109/72.788640 · Cited by 6,266
Point Set Registration: Coherent Point Drift
Myronenko, A., & Song, X. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.46 · Cited by 2,762
Advances in Large-Margin Classifiers
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/1113.001.0001 · Cited by 1,869
A Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589
Regularization Networks and Support Vector Machines
Evgeniou, T., Pontil, M., & Poggio, T. (2000) · Advances in Computational Mathematics · doi:10.1023/a:1018946025316 · Cited by 1,197
Self-citationImproving support vector machine classifiers by modifying kernel functions
Amari, S., & Wu, S. (1999) · Neural Networks · doi:10.1016/s0893-6080(99)00032-5 · Cited by 936
Kernel partial least squares regression in reproducing kernel hilbert space
Rosipal, R., & Trejo, L. J. (2002) · doi:10.5555/944790.944806 · Cited by 873
Deep Learning Based Vulnerability Detection: Are We There Yet?
Chakraborty, S., Krishna, R., Ding, Y., & Ray, B. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3087402 · Cited by 790
Wagner, W., Ullrich, A., Ducic, V., Melzer, T., & Studnicka, N. (2006) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2005.12.001 · Cited by 762
Kernel methods in system identification, machine learning and function estimation: A survey
Pillonetto, G., Dinuzzo, F., Chen, T., De Nicolao, G., & Ljung, L. (2014) · Automatica · doi:10.1016/j.automatica.2014.01.001 · Cited by 741
A Survey on Deep Learning for Data-Driven Soft Sensors
Sun, Q., & Ge, Z. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3053128 · Cited by 665
Prediction with Gaussian Processes: From Linear Regression to Linear Prediction and Beyond
Williams, C. K. I. (1998) · Learning in Graphical Models · doi:10.1007/978-94-011-5014-9_23 · Cited by 659
Metaheuristic design of feedforward neural networks: A review of two decades of research
Ojha, V. K., Abraham, A., & Snášel, V. (2017) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2017.01.013 · Cited by 606
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244303768966085 · Cited by 583
New algorithms for 2D and 3D point matching
Gold, S., Rangarajan, A., Lu, C.-P., Pappu, S., & Mjolsness, E. (1998) · Pattern Recognition · doi:10.1016/s0031-3203(98)80010-1 · Cited by 580
A systematic review on overfitting control in shallow and deep neural networks
Bejani, M. M., & Ghatee, M. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09975-1 · Cited by 573
Kernel independent component analysis
Bach, F. R., & Jordan, M. I. (2004) · 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03). · doi:10.1109/icassp.2003.1202783 · Cited by 568
Three learning phases for radial-basis-function networks
Schwenker, F., Kestler, H. A., & Palm, G. (2001) · Neural Networks · doi:10.1016/s0893-6080(01)00027-2 · Cited by 533
Structural risk minimization over data-dependent hierarchies
Shawe-Taylor, J., Bartlett, P. L., Williamson, R. C., & Anthony, M. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.705570 · Cited by 527
An Equivalence Between Sparse Approximation and Support Vector Machines
Girosi, F. (1998) · Neural Computation · doi:10.1162/089976698300017269 · Cited by 506
Self-citationAre Loss Functions All the Same?
Rosasco, L., Vito, E. D., Caponnetto, A., Piana, M., & Verri, A. (2004) · Neural Computation · doi:10.1162/089976604773135104 · Cited by 501
Bregman Divergence-Based Regularization for Transfer Subspace Learning
Si, S., Tao, D., & Geng, B. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.126 · Cited by 477
Machine Learning in Additive Manufacturing: A Review
Meng, L., McWilliams, B., Jarosinski, W., Park, H.-Y., Jung, Y.-G., Lee, J., & Zhang, J. (2020) · JOM · doi:10.1007/s11837-020-04155-y · Cited by 470
Pareto-Based Multiobjective Machine Learning: An Overview and Case Studies
Jin, Y., & Sendhoff, B. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2008.919172 · Cited by 455
An Evolutionary Algorithm for Large-Scale Sparse Multiobjective Optimization Problems
Tian, Y., Zhang, X., Wang, C., & Jin, Y. (2020) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2918140 · Cited by 440
Spatial Transformations in the Parietal Cortex Using Basis Functions
Pouget, A., & Sejnowski, T. J. (1997) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1997.9.2.222 · Cited by 425
Gene networks inference using dynamic Bayesian networks
Perrin, B.-E., Ralaivola, L., Mazurie, A., Bottani, S., Mallet, J., & d’Alché–Buc, F. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg1071 · Cited by 425
Quantized Kernel Least Mean Square Algorithm
Chen, B., Zhao, S., Zhu, P., & Principe, J. C. (2012) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2011.2178446 · Cited by 407
Clustered Multi-Task Learning: A Convex Formulation
Jacob, L., Bach, F., & Vert, J. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0809.2085 · Cited by 400
The Mathematics of Learning: Dealing With Data
Smale, S. (2005) · 2005 International Conference on Neural Networks and Brain · doi:10.1109/icnnb.2005.1614860 · Cited by 398
Remaining useful life prediction using multi-scale deep convolutional neural network
Li, H., Zhao, W., Zhang, Y., & Zio, E. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106113 · Cited by 390
Research Review for Broad Learning System: Algorithms, Theory, and Applications
Gong, X., Zhang, T., Chen, C. L. P., & Liu, Z. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3061094 · Cited by 350
Ghahramani, Z. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-28650-9_5 · Cited by 346
Non-rigid point set registration: Coherent Point Drift
Myronenko, A., Song, X., & Carreira-Perpiñán, M. Á. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0131 · Cited by 339
Poggio, T., & Beymer, D. (1996) · IEEE Spectrum · doi:10.1109/6.490058 · Cited by 332
Self-citationNeural Networks for Optimal Approximation of Smooth and Analytic Functions
Mhaskar, H. N. (1996) · Neural Computation · doi:10.1162/neco.1996.8.1.164 · Cited by 321
Incorporating prior information in machine learning by creating virtual examples
Niyogi, P., Girosi, F., & Poggio, T. (1998) · Proceedings of the IEEE · doi:10.1109/5.726787 · Cited by 313
Self-citationNeural circuit policies enabling auditable autonomy
Lechner, M., Hasani, R., Amini, A., Henzinger, T. A., Rus, D., & Grosu, R. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00237-3 · Cited by 308
Three-dimensional convolutional neural network (3D-CNN) for heterogeneous material homogenization
Rao, C., & Liu, Y. (2020) · Computational Materials Science · doi:10.1016/j.commatsci.2020.109850 · Cited by 297
Dynamic Model of Visual Recognition Predicts Neural Response Properties in the Visual Cortex
Rao, R. P. N., & Ballard, D. H. (1997) · Neural Computation · doi:10.1162/neco.1997.9.4.721 · Cited by 290
“3 3The variance-related regularization parameters , , and may be chosen according to a variety of techniques (see [Girosi et al., 1995] for references) but for the experiments in this paper, we used appropriately small but arbitrary fixed values for these parameters.”
MethodImage Representations for Visual Learning
Beymer, D., & Poggio, T. (1996) · Science · doi:10.1126/science.272.5270.1905 · Cited by 271
Self-citationHigh-Order Distance-Based Multiview Stochastic Learning in Image Classification
Yu, J., Rui, Y., Tang, Y. Y., & Tao, D. (2014) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2307862 · Cited by 260
Development of convolutional neural network and its application in image classification: a survey
Wang, W., Yang, Y., Wang, X., Wang, W., & Li, J. (2019) · Optical Engineering · doi:10.1117/1.oe.58.4.040901 · Cited by 252
A new hybrid ant colony optimization algorithm for feature selection
Kabir, M. M., Shahjahan, M., & Murase, K. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2011.09.073 · Cited by 252
Solving nonlinear differential equations with differentiable quantum circuits
Kyriienko, O., Paine, A. E., & Elfving, V. E. (2021) · Physical Review A · doi:10.1103/physreva.103.052416 · Cited by 185
“We can achieve this by introducing a regularization procedure [167, 168], also helping the optimizer to avoid getting trapped in local minima.”
MethodSolving nonlinear differential equations with differentiable quantum circuits
Kyriienko, O., Paine, A. E., & Elfving, V. E. (2020) · arXiv · doi:10.48550/arxiv.2011.10395 · Cited by 185
Mixing up contrastive learning: Self-supervised representation learning for time series
Wickstrøm, K., Kampffmeyer, M., Mikalsen, K. Ø., & Jenssen, R. (2022) · Pattern Recognition Letters · doi:10.1016/j.patrec.2022.02.007 · Cited by 113
System Design Perspective for Human-Level Agents Using Deep Reinforcement Learning: A Survey
Nguyen, N. D., Nguyen, T., & Nahavandi, S. (2017) · IEEE Access · doi:10.1109/access.2017.2777827 · Cited by 100
Park, J., Kwak, I.-Y., & Lim, C. (2021) · Electronics · doi:10.3390/electronics10161996 · Cited by 24
“Regularization can improve the model performance and prevent overfitting by controlling trainable weights via model complexity [46].”
BackgroundTiny Machine Learning Battery State-of-Charge Estimation Hardware Accelerated
Pau, D. P., & Aniballi, A. (2024) · Applied Sciences · doi:10.3390/app14146240 · Cited by 22
Metadata
- Authors
- Girosi, F., Jones, M., & Poggio, T.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 1995-03
- Volume
- 7
- Issue
- 2
- Pages
- 219-269
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W1981025032
- Cited by
- 1,349
- References
- 119
- FWCI
- 41.17
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.