The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
On the momentum term in gradient descent learning algorithms
Open accessAbstract
A momentum term is usually included in the simulations of connectionist learning algorithms. Although it is well known that such a term greatly improves the speed of learning, there have been few rigorous studies of its mechanisms. In this paper, I show that in the limit of continuous time, the momentum parameter is analogous to the mass of Newtonian particles that move through a viscous medium in a conservative force field. The behavior of the system near a local minimum is equivalent to a set of coupled and damped harmonic oscillators. The momentum term improves the speed of convergence by bringing some eigen components of the system closer to critical damping. Similar results can be obtained for the discrete time case used in computer simulations. In particular, I derive the bounds for convergence on learning-rate and momentum parameters, and demonstrate that the momentum term can increase the range of learning rate over which the system converges. The optimal condition for convergence is also analyzed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning internal representations by error propagation
RUMELHART, D. E., HINTON, G. E., & WILLIAMS, R. J. (1988) · Readings in Cognitive Science · doi:10.1016/b978-1-4832-1446-7.50035-2 · Cited by 3,002
Churchland, P. S., & Sejnowski, T. J. (1992) · doi:10.7551/mitpress/2010.001.0001 · Cited by 1,757
Cited by
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576
A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects
Li, Z., Liu, F., Yang, W., Peng, S., & Zhou, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3084827 · Cited by 5,273
“proposed the Momentum algorithm [80].”
Convolutional neural networks: an overview and application in radiology
Yamashita, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018) · Insights into Imaging · doi:10.1007/s13244-018-0639-9 · Cited by 4,682
“In addition, many improvements on the gradient descent algorithm have been proposed and widely used, such as SGD with momentum, RMSprop, and Adam [21 – 23], though the details of these algorithms are beyond the scope of this article.”
MethodDeep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503
A systematic study of the class imbalance problem in convolutional neural networks
Buda, M., Maki, A., & Mazurowski, M. A. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.011 · Cited by 3,088
“9 [56].”
Metadata
- Authors
- Qian, N.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 1999-01
- Volume
- 12
- Issue
- 1
- Pages
- 145-151
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W1980287119
- Cited by
- 2,312
- References
- 2
- FWCI
- 2.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: On the momentum term in gradient descent learning algorithms. https://app.cochat.ai/papers/10.1016/s0893-6080(98)00116-6
- The paper
- Qian, N. (1999). On the momentum term in gradient descent learning algorithms. Neural Networks. https://doi.org/10.1016/s0893-6080(98)00116-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
