Decoupled Weight Decay Regularization
Open accessAbstract
L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L$_2$ regularization (often calling it "weight decay" in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by \emph{decoupling} the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is available at https://github.com/loshchil/AdamW-and-SGDW
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AdamW offers better segmentation performance for Prithvi than Adam.
“The original Prithvi model uses Adam as the optimizer, but we found through our experiments that AdamW [30], which uses a different strategy for weight decay, offers better segmentation performance for Prithvi.”
From Assessment of a new GeoAI foundation model for flood inundation mapping (2023) · cites this paper as [30]
Log2Vec could not effectively handle some domain-specific words, such as technical terms or entity names.
“Therefore, Log2Vec could not effectively handle some domain-specific words, such as technical terms or entity names [50], [51], [52].”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [52]
Section 4.2 (p. 6) examines weight decay in SGD and Adam, while Section 4.5 (p. 8) discusses applications of AdamW. Searches for Log2Vec, domain-specific words, technical terms, and entity names found no passage addressing whether Log2Vec handles those words effectively. (LLM, conf=0.76)
Experimental — accuracy not yet measuredIt is hard to extract contextual information from characters to form meaningful words.
“However, it is hard to extract contextual information from characters to form meaningful words [50], [51], [52].”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [52]
Section 4.3 (p. 7) discusses AdamW’s generalization, and Section 4.5 (p. 8) mentions its use in natural-language-understanding models. Neither addresses the difficulty of extracting context from characters to form words. Searches for the claim’s wording and for character-level word formation found no supporting passage. (LLM, conf=0.76)
Experimental — accuracy not yet measuredFinetuning large transformers is significantly expensive and often infeasible for practitioners due to limited computation resources.
“finetuning becomes significantly expensive (e.g., up to millions of A100-GPU-hours [31]) and is even not feasible for practitioners due to the limited computation resources in most real-world applications.”
From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [31]
Section D (p. 13) calls 1800-epoch runs of standard Adam computationally expensive, but does not discuss fine-tuning large transformers. Section 4.5 (p. 8) mentions training Transformer architectures without giving A100 GPU-hour costs or saying fine-tuning is infeasible for resource-limited practitioners. Searches for those terms found no supporting passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredFor most popular trackers, correlation plays a critical role in integrating the template or target information into the regions of interest.
“For most of the popular trackers (such as SiamFC [1], SiamRPN [22], and ATOM [9]), correlation plays a critical role in integrating the template or target information into the regions of interest (ROI).”
From Transformer Tracking (2021) · cites this paper as [22]
The Introduction (p. 2) describes improving the Adam optimizer, and Section 4.5 (p. 8) discusses applications of AdamW. Searches for correlation in tracking and for SiamFC, SiamRPN, ATOM, templates, and ROIs found no passage addressing the claim. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Prior work such as SiamRPN, SiamRPN++, and Ocean selects final tracking results using post-processing schemes including cosine window penalty, scale change penalty, and bounding box smoothing.
“In the prior work such as SiamRPN [22], SiamRPN++ [21] and Ocean [48], the final tracking results are selected by post-processing schemes including cosine window penalty, scale change penalty and bounding box smoothing.”
From Transformer Tracking (2021) · cites this paper as [22]
The passages on cosine annealing (Section B.2, p. 13) concern learning-rate schedules for Adam, not penalties used to select tracking results. Searches for SiamRPN, SiamRPN++, Ocean, and the named post-processing schemes found no passage in our copy addressing the claim. (LLM, conf=0.76)
Experimental — accuracy not yet measuredSeveral object detectors used the bipartite matching loss.
“Several object detectors [9,25,35] used the bipartite matching loss.”
From End-to-End Object Detection with Transformers (2020) · cites this paper as [25]
Section 4.5 (p. 8) mentions one face-detection architecture trained with AdamW, but says nothing about its matching loss. Searches for “bipartite matching loss” and “Hungarian assignment” found no passage saying that several object detectors used such a loss. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
Cited by 12,522
Research works printed but not linked: 26
- Laurence Aitchison. A unified theory of adaptive stochastic gradient descent as Bayesian filtering. arXiv:1507.02030 , 2018. · arXiv
- Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of ImageNet as an alternative to the CIFAR datasets. arXiv:1707.08819 , 2017. · arXiv
- Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 , 2018. · arXiv
- Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv:1703.04933 , 2017. · arXiv
- John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research , 12:2121-2159, 2011.
- Xavier Gastaldi. Shake-Shake regularization. arXiv preprint arXiv:1705.07485 , 2017. · arXiv
- Stephen Jos´ e Hanson and Lorien Y Pratt. Comparing biases for minimal network construction with back-propagation. In Proceedings of the 1st International Conference on Neural Information Processing Systems , pp. 177-185, 1988.
- Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get m for free. arXiv:1704.00109 , 2017. · arXiv
- Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv:1609.04836 , 2016. · arXiv
- Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 , 2014. · arXiv
- Hao Li, Zheng Xu, Gavin Taylor, and Tom Goldstein. Visualizing the loss landscape of neural nets. arXiv preprint arXiv:1712.09913 , 2017. · arXiv
- Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. arXiv:1608.03983 , 2016. · arXiv
- James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning , pp. 2408-2417, 2015.
- Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434 , 2015. · arXiv
- Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. International Conference on Learning Representations , 2018.
- Leslie N Smith. Cyclical learning rates for training neural networks. arXiv:1506.01186v3 , 2016. · arXiv
- Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 4(2):2631, 2012.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pp. 5998-6008, 2017.
- Martin V¨ olker, Jiˇ r´ ı Hammer, Robin T Schirrmeister, Joos Behncke, Lukas DJ Fiederer, Andreas Schulze-Bonhage, Petr Marusiˇ c, Wolfram Burgard, and Tonio Ball. Intracranial error detection via deep learning. arXiv preprint arXiv:1805.01667 , 2018. · arXiv
- Jianfeng Wang, Ye Yuan, Gang Yu, and Sun Jian. Sface: An efficient network for face detection in large scale variations. arXiv preprint arXiv:1804.06559 , 2018. · arXiv
- Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. arXiv:1705.08292 , 2017. · arXiv
- Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning , pp. 2048-2057, 2015.
- Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang. Wider face: A face detection benchmark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 5525-5533, 2016.
- Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281 , 2018. · arXiv
- Zijun Zhang, Lin Ma, Zongpeng Li, and Chuan Wu. Normalized direction-preserving adam. arXiv:1709.04546 , 2017. · arXiv
- Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. In arXiv:1707.07012 [cs.CV] , 2017. · arXiv
- Authors
- Loshchilov, I., & Hutter, F.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1711.05101
- OpenAlex
- W2908510526
- Cited by
- 12,522
- References
- 0
- Citations by year
- 2026: 324 · 2025: 1997 · 2024: 3409 · 2023: 3260 · 2022: 1606 · 2021: 1379 · 2020: 459 · 2019: 81 · 2018: 4 · 2017: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- weight decay regularization, Tikhonov regularization, Adam optimizer, AdamW, stochastic gradient descent, learning rate, generalization, image classification, momentum SGD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
