Using random weights to train multilayer networks of hard-limiting units
Abstract
A gradient descent algorithm suitable for training multilayer feedforward networks of processing units with hard-limiting output functions is presented. The conventional backpropagation algorithm cannot be applied in this case because the required derivatives are not available. However, if the network weights are random variables with smooth distribution functions, the probability of a hard-limiting unit taking one of its two possible values is a continuously differentiable function. In the paper, this is used to develop an algorithm similar to backpropagation, but for the hard-limiting case. It is shown that the computational framework of this algorithm is similar to standard backpropagation, but there is an additional computational expense involved in the estimation of gradients. Upper bounds on this estimation penalty are given. Two examples which indicate that, when this algorithm is used to train networks of hard-limiting units, its performance is similar to that of conventional backpropagation applied to networks of units with sigmoidal characteristics are presented.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Gradient descent has been achieved using Heaviside activation functions by taking the weights as Gaussian variables and taking partial derivatives with respect to the means and standard deviations.
“Nevertheless, gradient descent has also been achieved using Heaviside activation functions, by taking the weights as Gaussian variables and taking partial derivatives w.r.t. the means and standard deviations of the appropriate Gaussian distributions.33,34”
From Quantum generalisation of feedforward neural networks (2017) · cites this paper as [33]
Cites
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,501 · see citing works
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686 · see citing works
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369 · see citing works
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402 · see citing works
Advances in Neural Information Processing Systems 37
Touretzky, D. S. (2024) · doi:10.52202/079017 · Cited by 2,874 · see citing works
Convergence of Stochastic Processes.
Robinson, P. M., & Pollard, D. (1985) · Economica · doi:10.2307/2553898 · Cited by 2,239 · see citing works
Convergence of Stochastic Processes
Pollard, D. (1984) · Springer series in statistics · doi:10.1007/978-1-4612-5254-2 · Cited by 1,859 · see citing works
What Size Net Gives Valid Generalization?
Baum, E. B., & Haussler, D. H. (1989) · Neural Computation · doi:10.1162/neco.1989.1.1.151 · Cited by 1,549 · see citing works
Connectionist learning procedures
Hinton, G. E. (1989) · Artificial Intelligence · doi:10.1016/0004-3702(89)90049-0 · Cited by 1,487 · see citing works
Decision theoretic generalizations of the PAC model for neural net and other learning applications
Haussler, D. H. (1992) · Information and Computation · doi:10.1016/0890-5401(92)90010-d · Cited by 845 · see citing works
The recent excitement about neural networks
Crick, F. (1989) · Nature · doi:10.1038/337129a0 · Cited by 724 · see citing works
Hinton, G. E. (1990) · Elsevier eBooks · doi:10.1016/b978-0-08-051055-2.50029-8 · Cited by 523 · see citing works
Neural nets for adaptive filtering and adaptive pattern recognition
Widrow, B. C., & Winter, R. (1988) · Computer · doi:10.1109/2.29 · Cited by 470 · see citing works
Accelerated learning in layered neural networks
Solla, S. A., Levin, E., & Fleisher, M. (1988) · Complex Systems · Cited by 210 · see citing works
Multilayer perceptron structures applied to adaptive equalisers for data communications
Gibson, G. J., Siu, S., & Cowen, C. F. N. (2003) · International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1989.266645 · Cited by 106 · see citing works
Training binary node feedforward neural networks by back propagation of error
Toms, D. J. (1990) · Electronics Letters · doi:10.1049/el:19901121 · Cited by 34 · see citing works
Neural nets for adaptive filtering and adaptive pattern recognition
Widrow, B. C., & Winter, R. (1990) · Academic Press Professional, Inc. eBooks · Cited by 28 · see citing works
Reinforcement learning algorithms as function optimizers
Williams, & Peng, P. (1989) · doi:10.1109/ijcnn.1989.118683 · Cited by 23 · see citing works
Learning in Feedforward Networks with Nonsmooth Functions
Redding, N. J., & Downs, T. (1991) · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · see citing works
Quantum generalisation of feedforward neural networks
Wan, K. H., Dahlsten, O., Kristjánsson, H., Gardner, R., & Kim, M. S. (2017) · npj Quantum Information · doi:10.1038/s41534-017-0032-4 · Cited by 276 · see citing works
Downs, T., & Gaynier, R. J. (1995) · Mathematical and Computer Modelling · doi:10.1016/0895-7177(95)00180-a · Cited by 5 · see citing works
Self-citation
- Authors
- Barlett, P. L., & Downs, T.
- Affiliations
- The University of Queensland (AU)
- Venue
- IEEE Transactions on Neural Networks
- Published
- 1992-03
- Volume
- 3
- Issue
- 2
- Pages
- 202-210
- ISSN
- 1045-9227 (ISSN-L)
- OpenAlex
- W2125329357
- Cited by
- 40
- References
- 20
- Citations by year
- 2023: 1 · 2022: 2 · 2021: 4 · 2020: 1 · 2019: 1 · 2018: 2 · 2017: 3 · 2016: 3 · 2015: 1
- FWCI
- 3.12
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Neural Networks (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and ELM
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- random weights, gradient estimation, multilayer perceptron, training algorithms, backpropagation alternatives, gradient bounds, neural network training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
