Why gradient clipping accelerates training: A theoretical justification for adaptivity
Open accessThe DOI matches its OpenAlex record; 12 of the 12 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 1×
“…(RNN) language model (LM) (Mikolov et al., 2010) training to alleviate the exploding gradient problem, and has been used in more sophisticated RNN models (Hochreiter and Schmidhuber, 1997) or seq2seq models for language modelling or other NLP applications (Sutskever et al., 2014; Cho et al., 2014).”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
Beck, A., & Teboulle, M. (2009) · SIAM Journal on Imaging Sciences · doi:10.1137/080716542 · Cited by 11,952
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 5×
“We run language modeling on the Penn Treebank (PTB) (Mikolov et al., 2010) dataset with AWDLSTM models (Merity et al., 2018).”
MethodInfluentialTransformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
Some methods of speeding up the convergence of iteration methods
Polyak, B. T. (1964) · USSR Computational Mathematics and Mathematical Physics · doi:10.1016/0041-5553(64)90137-5 · Cited by 3,069
Convolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850 · cited 1×
“Even though gradient clipping is a standard practice in tasks such as language models (e.g. Merity et al., 2018; Gehring et al., 2017; Peters et al., 2018), it lacks a firm theoretical grounding.”
MethodLSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963 · cited 1×
“In this work we experiment with LSTM LM (Sundermeyer et al., 2012), which has been an important building block for many popular NLP models (Young et al., 2017).”
Method
Cited by
Privacy for Free: Wireless Federated Learning via Uncoded Transmission With Adaptive Power Control
Liu, D., & Simeone, O. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3036948 · Cited by 234
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Karimireddy, S. P., Jaggi, M., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., & Suresh, A. T. (2020) · arXiv · doi:10.48550/arxiv.2008.03606 · Cited by 96
“Further, algorithmic innovations such as momentum [30, 4], adaptivity [17, 41, 42], and clipping [38, 39, 43] are critical for success in deep learning applications and need to be incorporated into the client updates, replacing the SGD update of FEDAVG.”
BackgroundTransformers enable accurate prediction of acute and chronic chemical toxicity in aquatic organisms
Gustavsson, M., Käll, S., Svedberg, P., Inda-Diaz, J. S., Molander, S., Coria, J., Backhaus, T., & Kristiansson, E. (2024) · Science Advances · doi:10.1126/sciadv.adk6669 · Cited by 82
PICO entity extraction for preclinical animal literature.
Wang, Q., Liao, J., Lapata, M., & Macleod, M. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02074-4 · Cited by 29
Optimal Adaptive Power Control for Over-The-Air Federated Edge Learning Under Fading Channels
Yu, X., Xiao, B., Ni, W., & Wang, X. (2023) · IEEE Transactions on Communications · doi:10.1109/tcomm.2023.3283790 · Cited by 19
Metadata
- Authors
- Sra, S., He, T., Sra, S., & Jadbabaie, A.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2019-05-28
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.11881
- OpenAlex
- W2994747431
- Cited by
- 79
- References
- 12
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Why gradient clipping accelerates training: A theoretical justification for adaptivity. https://app.cochat.ai/papers/10.48550/arxiv.1905.11881
- The paper
- Sra, S., He, T., Sra, S., & Jadbabaie, A. (2019). Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv (Cornell University). https://doi.org/10.48550/arxiv.1905.11881
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.