The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Links between perceptrons, MLPs and SVMs
Open accessAbstract
We propose to study links between three important classification algorithms: Perceptrons, Multi-Layer Perceptrons (MLPs) and Support Vector Machines (SVMs). We first study ways to control the capacity of Perceptrons (mainly regularization parameters and early stopping), using the margin idea introduced with SVMs. After showing that under simple conditions a Perceptron is equivalent to an SVM, we show it can be computationally expensive in time to train an SVM (and thus a Perceptron) with stochastic gradient descent, mainly because of the margin maximization term in the cost function. We then show that if we remove this margin maximization term, the learning rate or the use of early stopping can still control the margin. These ideas are extended afterward to the case of MLPs. Moreover, under some assumptions it also appears that MLPs are a kind of mixture of SVMs, maximizing the margin in the hidden layer space. Finally, we present a very simple MLP based on the previous findings, which yields better performances in generalization and speed than the other models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., & White, H. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90020-8 · Cited by 21,728
Neural networks for pattern recognition
(1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,538
(1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895
Multilayer feedforward networks are universal approximators
HornikK., StinchcombeM., & WhiteH. (1989) · Neural Networks · doi:10.5555/70405.70408 · Cited by 9,081
Practical Methods of Optimization
Fletcher, R. (2000) · Wiley eBooks · doi:10.1002/9781118723203 · Cited by 7,605
Practical Methods of Optimization.
Witzgall, C., & Fletcher, R. (1989) · Mathematics of Computation · doi:10.2307/2008742 · Cited by 3,591
Making Large-Scale Support Vector Machine Learning Practical
Joachims, T. (1998) · Advances in Kernel Methods · doi:10.7551/mitpress/1130.003.0015 · Cited by 1,613
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333
Learning internal representations by back-propagating errors
Rumelhart, D. E. (1986) · Medical Entomology and Zoology · Cited by 337
Cited by
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259
Deep Learning in Mobile and Wireless Networking: A Survey
Zhang, C., Patras, P., & Haddadi, H. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2904897 · Cited by 1,768
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498
“Between contiguous FCNN layers, the output of a neuron (cell), either the input or hidden cell, is directly passed to and activated by neurons belong to the next layer [80].”
BackgroundDeep learning classifiers for hyperspectral imaging: A review
Paoletti, M. E., Haut, J. M., Plaza, J., & Plaza, A. (2019) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2019.09.006 · Cited by 1,073
Metadata
- Authors
- Collobert, R., & Bengio, S.
- Affiliations
- Idiap Research Institute (CH)
- Venue
- Twenty-first international conference on Machine learning - ICML '04
- Publisher
- ACM Press
- Published
- 2004
- Pages
- 23
- OpenAlex
- W1978381081
- Cited by
- 302
- References
- 17
- Citations by year
- 2026: 1 · 2025: 9 · 2024: 16 · 2023: 21 · 2022: 23 · 2021: 30 · 2020: 37 · 2019: 23 · 2018: 28 · 2017: 24 · 2016: 13 · 2015: 6 · 2014: 15 · 2013: 9 · 2012: 7
- Type
- conference abstract
- Language
- en
- Open access
- green
- infoscience.epfl.ch
- Licence
- Locations
- International Conference on Machine Learning (publishedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- perceptron, multilayer perceptron, support vector machine, margin maximization, regularization, early stopping, stochastic gradient descent, generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Links between perceptrons, MLPs and SVMs. https://app.cochat.ai/papers/10.1145/1015330.1015415
- The paper
- Collobert, R., & Bengio, S. (2004). Links between perceptrons, MLPs and SVMs. Twenty-first international conference on Machine learning - ICML '04. https://doi.org/10.1145/1015330.1015415
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.