Learning rate schedules for faster stochastic gradient search
The DOI matches its Crossref record.
Abstract
The authors propose a new methodology for creating the first automatically adapting learning rates that achieve the optimal rate of convergence for stochastic gradient descent. Empirical tests agree with theoretical expectations that drift can be used to determine whether the crucial parameter c is large enough. Using this statistic, it will be possible to produce the first adaptive learning rates which converge at optimal speed.>
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
An overview of gradient descent optimization algorithms
Ruder, S. (2016) · arXiv · doi:10.48550/arxiv.1609.04747 · Cited by 4,962
A Survey of Optimization Methods From a Machine Learning Perspective
Sun, S., Cao, Z., Zhu, H., & Zhao, J. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2950779 · Cited by 789
Linear Least-Squares algorithms for temporal difference learning
Bradtke, S. J., & Barto, A. G. (1996) · Machine Learning · doi:10.1007/bf00114723 · Cited by 599
Fast iterative alignment of pose graphs with poor initial estimates
Olson, E., Leonard, J., & Teller, S. (2006) · Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. · doi:10.1109/robot.2006.1642040 · Cited by 427
Representation and separation of signals using nonlinear PCA type learning
Karhunen, J., & Joutsensalo, J. (1994) · Neural Networks · doi:10.1016/0893-6080(94)90060-4 · Cited by 371
Kim, J., Calhoun, V. D., Shim, E., & Lee, J.-H. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2015.05.018 · Cited by 354
Reinforcement Learning: A Tutorial Survey and Recent Advances
Gosavi, A. (2009) · INFORMS Journal on Computing · doi:10.1287/ijoc.1080.0305 · Cited by 326
LSTM-Based Intrusion Detection System for In-Vehicle Can Bus Communications
Hossain, M. D., Inoue, H., Ochiai, H., Fall, D., & Kadobayashi, Y. (2020) · IEEE Access · doi:10.1109/access.2020.3029307 · Cited by 261
Solving Semi-Markov Decision Problems Using Average Reward Reinforcement Learning
Das, T. K., Gosavi, A., Mahadevan, S., & Marchalleck, N. (1999) · Management Science · doi:10.1287/mnsc.45.4.560 · Cited by 218
“The learning rate m and the exploration rate pm are both decayed slowly to 0 according to a Darken-Chang-Moody search-then-converge procedure ([9]) as follows.”
MethodA Reinforcement Learning Model to Assess Market Power Under Auction-Based Energy Pricing
Nanduri, V., & Das, T. K. (2007) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2006.888977 · Cited by 168
“One-line diagram of the sample power network. g) Update the learning parameters and and exploration parameter following the DCM [45] scheme given below: where (10) where denotes the initial value of a learning/exploration rate, and is a large value (e.g., ) chosen to obtain a suitable decay rate for the learning/explo…”
MethodA Study of the Optimization Algorithms in Deep Learning
Zaheer, R., & Shaziya, H. (2019) · 2019 Third International Conference on Inventive Systems and Control (ICISC) · doi:10.1109/icisc44355.2019.9036442 · Cited by 164
“Its variant is stochastic gradient descent [3] that performs the training on the individual data element.”
BackgroundA Stochastic Game Approach for Modeling Wholesale Energy Bidding in Deregulated Power Markets
Ragupathi, R., & Das, T. K. (2004) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2004.825910 · Cited by 31
“1) Initialize iteration or decision epoch count , reinforcement values for all players , state , and actions where is the action space of player in state . Set the average reward for the players for all . 2) Initialize the input parameters for choosing (learning rates) and (exploration rate), respectively, according t…”
Method
Metadata
- Authors
- Darken, C., Chang, J., & Moody, J.
- Venue
- Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop
- Publisher
- IEEE
- Published
- 2003-01-02
- Pages
- 3-12
- OpenAlex
- W2534068475
- Cited by
- 212
- References
- 0
- FWCI
- 3.93
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.