The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
An empirical study of learning rates in deep neural networks for speech recognition
Abstract
Recent deep neural network systems for large vocabulary speech recognition are trained with minibatch stochastic gradient descent but use a variety of learning rate scheduling schemes. We investigate several of these schemes, particularly AdaGrad. Based on our analysis of its limitations, we propose a new variant `AdaDec' that decouples long-term learning-rate scheduling from per-parameter learning rate variation. AdaDec was found to result in higher frame accuracies than other methods. Overall, careful choice of learning rate schemes leads to faster convergence and lower word error rates.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationA Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795
A direct adaptive method for faster backpropagation learning: the RPROP algorithm
Riedmiller, M., & Braun, H. (2002) · IEEE International Conference on Neural Networks · doi:10.1109/icnn.1993.298623 · Cited by 3,917
Acceleration of Stochastic Approximation by Averaging
Polyak, B. T., & Juditsky, A. B. (1992) · SIAM Journal on Control and Optimization · doi:10.1137/0330046 · Cited by 1,827 · cited 1×
“Finally we apply a model averaging [23] with a running average [15] which increases the frame accuracy slightly but has had no measurable effect on the final WER.”
Method
Cited by
Katzman, J. L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., & Kluger, Y. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0482-1 · Cited by 1,995
“These include: standardizing the input, Scaled Exponential Linear Units (SELU) [19] as the activation function, Adaptive Moment Estimation (Adam) [20] for the gradient descent algorithm, Nesterov momentum [21], and learning rate scheduling [22].”
MethodSpeech Recognition Using Deep Neural Networks: A Systematic Review
Nassif, A. B., Shahin, I., Attili, I., Azzeh, M., & Shaalan, K. (2019) · IEEE Access · doi:10.1109/access.2019.2896880 · Cited by 1,152
Seide, F., Fu, H., Droppo, J., Li, G., & Yu, D. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-274 · Cited by 443
Hanson, J., Yang, Y., Paliwal, K., & Zhou, Y. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btw678 · Cited by 292
Darwish, A., Ezzat, D., & Hassanien, A. E. (2020) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2019.100616 · Cited by 279
Metadata
- Authors
- Senior, A., Heigold, G., Ranzato, M., & Yang, K.
- Venue
- 2013 IEEE International Conference on Acoustics, Speech and Signal Processing
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 6724-6728
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2000200144
- Cited by
- 169
- References
- 5
- FWCI
- 11.49
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An empirical study of learning rates in deep neural networks for speech recognition. https://app.cochat.ai/papers/10.1109/icassp.2013.6638963
- The paper
- Senior, A., Heigold, G., Ranzato, M., & Yang, K. (2013). An empirical study of learning rates in deep neural networks for speech recognition. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. https://doi.org/10.1109/icassp.2013.6638963
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.