How Memory in Optimization Algorithms Implicitly Modifies the Loss

May combine two claims.

Lion does not have the kind of implicit anti-regularization induced by memory that AdamW does, providing a theory-based explanation for Lion's better generalization performance recently documented.

“As an application of our theory, we find that Lion does not have the kind of implicit anti-regularization induced by memory that AdamW does, providing a theory-based explanation for Lion's better generalization performance recently documented [13].”

• Also stated in: § 1 Introduction · p. 4 · boosted

  “As a substantive application, Section 4 studies AdamW (Example 1.3) and LionK (Example 1.5), and demonstrates that Lion does not suffer from the anti-regularization effect that AdamW's memory has, which predicts better generalization of (full-batch) Lion.”

• cites [13] [13] X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C.J. Hsieh, Y. Lu, and Q. V. Le. Symbolic discovery of optimization algorithms. In Advances in Neural Information Processing Systems , volume 36, pages 49205-49233, 2023. URL https://proceedings.neurips.cc/paper\_files/paper/2023/hash/ 9a39b4925e35cf447ccba8757137d84f-Abstract.html . - sourceSupported

May combine two claims.

There is a non-negligible effect of mini-batch noise on the picture described in Section J; in particular, Lion does not necessarily outperform Adam if batches are small.

“Additionally, there is a non-negligible effect of mini-batch noise on the picture we are describing in Section J; in particular, Lion does not necessarily outperform Adam if batches are small [13].”

• cites [13] [13] X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C.J. Hsieh, Y. Lu, and Q. V. Le. Symbolic discovery of optimization algorithms. In Advances in Neural Information Processing Systems , volume 36, pages 49205-49233, 2023. URL https://proceedings.neurips.cc/paper\_files/paper/2023/hash/ 9a39b4925e35cf447ccba8757137d84f-Abstract.html . - sourcePartly supported

  Section 6 (p. 14) says Lion is likely to perform no better than AdamW at batch sizes below 64, and Section 4.6 (p. 13) shows that Lion prefers larger batches. This supports the small-batch qualification, but the paper compares Lion with AdamW rather than Adam and discusses added noise from Lion’s sign operation rather than explicitly attributing the effect to mini-batch noise. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

May combine two claims.

Lemma I.3: For LionK (Example 1.5), under the stated conditions, Assumption 3.1 holds; in particular, by Corollary 3.3 the inequality max n∈[0:⌊T/h⌋] ||θ(n)− ̃θ(n)||∞ ≤ C2 h2 holds.

“Lemma I.3 (Application of Corollary 3.3 to Example 1.5) . Let { θ ( n ) } n ∈ Z ⩾ 0 be the sequence of vectors generated by the iteration in Equation (1) with initial condition θ (0) , where F ( n ) ( · ) is as defined in Example 1.5, the function K ( · ) : M→ R defined in a open bounded region M⊂ R d is three times continuously differentiable with bounded derivatives, and the loss function L ( ·”

• AssumptionThe authors state this as an assumption: The loss function L is three times continuously differentiable with bounded derivatives on an open convex bounded domain D.
• AssumptionThe authors state this as an assumption: The function K is three times continuously differentiable with bounded derivatives on an open bounded region M.

May combine two claims.

Adam with lower values of β2 can sometimes outperform Lion with default hyperparameters and thus close the generalization gap between these two algorithms.

“We see that Adam with lower values of β 2 can sometimes outperform Lion with default hyperparameters and thus close the generalization gap between these two algorithms, consistent with the theory.”

May combine two claims.

Lemma H.1: If F(n) is a function of 'momentum variables' as described in Assumption C.1, then for any n and k ≤ n, the memory decays exponentially fast, and similarly for any n and k1, k2 ≤ n.

“Lemma H.1 (Memory decays exponentially fast) . If F ( n ) is a function of 'momentum variables' as described in Assumption C.1, then for any n and k ⩽ n”

• AssumptionThe authors state this as an assumption: F(n) is a function of 'momentum variables' as described in Assumption C.1, with L two times continuously differentiable functions g(n)ℓ, fixed βℓ in [0,1), bounded nonnegative sequences b(n+1)ℓ, and a fixed two times continuously differentiable Φ.

Probably not a substantial claim: may not pass S3 (load-bearing).

We reproduce the result that preconditioned sharpness oscillates around the stability threshold in a CNN trained on CIFAR-10.

“We train a CNN on CIFAR-10 as well, and reproduce this result in Figure 5.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the case of LionK (Example 1.5), Assumption C.1 applies with L = 3.

“In the case of LionK (Example 1.5), the assumption applies with L = 3”

• AssumptionThe authors state this as an assumption: F(n) is a function of 'momentum variables' as described in Assumption C.1, with L two times continuously differentiable functions g(n)ℓ, fixed βℓ in [0,1), bounded nonnegative sequences b(n+1)ℓ, and a fixed two times continuously differentiable Φ.

The quote only partly states this claim.

For AdamW, taking ε to zero, the memoryless iteration can be written in the form of preconditioned gradient descent with decoupled weight decay and a modified loss.

“Taking ε to zero, we can write this in the form of preconditioned gradient descent (with decoupled weight decay):”

Probably not a substantial claim: may not pass S4 (based).

E ∇g(i)r(θ)T g(j)(θ) depends only on whether i = j or i ≠ j.

“Note that E ∇ g ( i ) r ( θ ) T g ( j ) ( θ ) depends only on whether i = j or i = j .”

May combine two claims.

In Transformer-XL on WikiText-2 with full-batch Adam, higher β2 increases the best validation perplexity (hurts generalization), and sometimes taking lower β2 can close the gap between Adam and Lion.

“We observe in Figure 2(b) the same trends as above (in large-batch training, higher β 2 increases the best validation perplexity, that is, hurts generalization; sometimes, taking lower β 2 can close the gap between Adam and Lion).”

Classified as probably not a substantial claim. The quote only partly states this claim.

The correction term introduced by memory (9) is given by the paper's expression.

“The correction term introduced by memory (9) is”

• AssumptionThe authors state this as an assumption: Memory decays exponentially fast uniformly in n, as formalized by Assumption 3.1.

Probably not a substantial claim: may not pass S3 (load-bearing).

Ordinary sharpness λ1(Ht) first increases and then decreases during training.

“We also plot ordinary sharpness λ 1 ( H t ) (top hessian eigenvalue), which first increases and then decreases.”

The quote only partly states this claim.

For AdamW, neglecting coefficients decaying to zero exponentially fast, the memoryless iteration has a correction term M(n)(θ) given by the paper's expression.

“We first study AdamW with memory by an application of Theorem 3.2 and Corollary 3.3. Neglecting coefficients decaying to zero exponentially fast, we have”

• Also stated in: § 4 Application: the Effect of Memory on AdamW, Lion and Signum · p. 7 · plain

  “where M ( n ) ( θ ) is given by”

• AssumptionThe authors state this as an assumption: Memory decays exponentially fast uniformly in n, as formalized by Assumption 3.1.

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

Lemma H.2: Let {ak}∞k=1 and {bk}∞k=1 be sequences of reals such that ∑∞k=1(|ak|+|bk|) < ∞. Then a certain limit identity holds.

“Lemma H.2. Let { a k } ∞ k =1 and { b k } ∞ k =1 be sequences of reals such that ∑ ∞ k =1 ( | a k | + | b k | ) &lt; ∞ . Then”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the case of AdamW (Example 1.3), Assumption C.1 applies with L = 3.

“For instance, in the case of AdamW (Example 1.3), Assumption C.1 applies with L = 3”

• AssumptionThe authors state this as an assumption: F(n) is a function of 'momentum variables' as described in Assumption C.1, with L two times continuously differentiable functions g(n)ℓ, fixed βℓ in [0,1), bounded nonnegative sequences b(n+1)ℓ, and a fixed two times continuously differentiable Φ.

May combine two claims.

Heavy-ball momentum gradient descent can be written in form (1), and its momentum variable exponentially averages past gradients.

“Example 1.1 (Heavy-ball momentum gradient descent; Polyak [54]) . This optimizer can be written in the form (1) with”

• Also stated in: § 1 Introduction · p. 2 · plain

  “The optimizer in Example 1.1 is often just referred to as GD with momentum, where the exponential sum m ( n +1) 1 in Equation (2) is the 'momentum variable': it exponentially averages past gradients.”

• cites [54] [54] B. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964. ISSN 0041-5553. doi: https://doi.org/10.1016/0041-5553(64)90137-5. URL https://www.sciencedirect. com/science/article/pii/0041555364901375 . -Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Gradient descent with momentum has exponentially decaying memory through exponentially averaged past gradients.

“For example, gradient descent with momentum has exponentially decaying memory through exponentially averaged past gradients.”

• cites [54] [54] B. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964. ISSN 0041-5553. doi: https://doi.org/10.1016/0041-5553(64)90137-5. URL https://www.sciencedirect. com/science/article/pii/0041555364901375 . -Not checked yet

  Not checked in the free run.

May combine two claims.

AdamW can be written in form (1), with momentum variables m1 and m2 being exponentially averaged gradients and exponentially averaged squared gradient components, and m3 a degenerate momentum variable.

“Example 1.3 (AdamW; Loshchilov and Hutter [46]) . The optimizer can be written in the form (1) with”

• Also stated in: § 1 Introduction · p. 2 · plain

  “In Example 1.3, m ( n +1) 1 and m ( n +1) 2 are also 'momentum variables': exponentially averaged gradients and exponentially averaged squared gradient components respectively, with coefficients in front of the sum, such as (1 -β 1 )(1 -β n +1 1 ) -1 , providing 'bias correction' [36]. The variable m ( n +1) 3 here is a degenerate 'momentum variable', with memory decaying infinitely fast.”

• cites [46] [46] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id= Bkg6RiCqY7 . -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

LionK can be written in form (1).

“Example 1.5 (LionK ; Chen et al. [12]) . The optimizer can be written in the form of (1) with”

• cites [12] [12] L. Chen, B. Liu, K. Liang, and Q. Liu. Lion secretly solves a constrained optimization: As lyapunov predicts. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=e4xS9ZarDr . -Not checked yet

  Not checked in the free run.

May combine two claims.

Modern optimization methods used in deep learning have memory: each update depends on the history of previous iterations, and this dependence decays fast as the iterates go further into the past.

“In modern optimization methods used in deep learning, each update depends on the history of previous iterations, often referred to as memory , and this dependence decays fast as the iterates go further into the past.”

May combine two claims.

The correction term can be interpreted as a perturbation of the loss, and its nature can inform how memory implicitly (anti-)regularizes the optimization dynamics.

“This correction term can be interpreted as a perturbation of the loss, and the nature of this perturbation can inform how memory implicitly (anti-)regularizes the optimization dynamics.”

May combine two claims.

In large-batch training on vision tasks, taking β2 < β1 = 0.99 strongly regularizes training, moving the model parameters to flatter regions of the loss space.

“This suggests that in large-batch training on vision tasks, taking β 2 &lt; β 1 = 0 . 99 strongly regularizes training, moving the model parameters to flatter regions of the loss space.”

May combine two claims.

Since Lion with ρ1=ρ2 is Signum, the paper provides a novel perspective on the similarity between Adam with β1≈β2 and Signum, a point discussed and verified empirically in [69].

“Since Lion with ρ 1 = ρ 2 is Signum [8], we provide a novel perspective on the similarity between Adam with β 1 ≈ β 2 and Signum, a point discussed and verified empirically in [69].”

• cites [8] [8] J. Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar. signSGD: Compressed optimisation for non-convex problems. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 560-569. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/bernstein18a. html . -Not checked yet

  Not checked in the free run.

• cites [69] [69] R. Zhao, D. Morwani, D. Brandfonbrener, N. Vyas, and S. M. Kakade. Deconstructing what makes a good optimizer for autoregressive language models. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum? id=zfeso8ceqr .Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The observation that the correction term is zero if and only if the point is stationary is true of simpler full-batch algorithms, including Adam with λ=0 and full-batch GD with momentum.

“This is a generalization of the observation that the correction term is zero if and only if the point is stationary, true of simpler full-batch algorithms (for Adam with λ = 0 it follows from the above; for full-batch GD with momentum it is clear from (6)).”

May combine two claims.

The theory only relies on memory decaying sufficiently fast, not necessarily in the form of momentum variables, and thus covers all listed examples and many others, while allowing for both full-batch and mini-batch training.

“Our theory only relies on memory decaying sufficiently fast, not necessarily in the form of momentum variables, and thus covers all the examples listed above and many others, while also allowing for both full-batch and mini-batch training.”

Probably not a substantial claim: may not pass S4 (based).

Practically relevant learning rates are often small enough, especially mid-training if a learning rate decay schedule is used.

“Fortunately, practically relevant learning rates are often small enough (especially mid-training, if a learning rate decay schedule is used).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Between 2023 and 2025, the mainstream paradigm in deep learning is best described as scaling-centric rather than generalization-centric, which decreased interest in implicit regularization.

“Between 2023 and 2025, the mainstream paradigm in deep learning (especially large language models) is best described as scaling-centric rather than generalizationcentric [65] which decreased interest in implicit regularization.”

• cites [65] [65] L. Xiao. Rethinking conventional wisdom in machine learning: From generalization to scaling. arXiv preprint arXiv:2409.15156 , 2024. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Bernstein and Newhouse [7] view Adam and Lion as smoothed-out versions of steepest descent.

“Bernstein and Newhouse [7] also view Adam and Lion as smoothed-out versions of steepest descent.”

• cites [7] [7] J. Bernstein and L. Newhouse. Old optimizer, new norm: An anthology. In OPT 2024: Optimization for Machine Learning . -Not checked yet

  Not checked in the free run.

May combine two claims.

The term added implicitly to the loss by momentum can be interpreted as a first-order approximation of l2 sharpness, which suggests that it moves the trajectory towards flatter minima, often thought to improve generalization.

“In fact, this term (up to coefficients) can be interpreted as a first-order approximation of ℓ 2 sharpness [26], which suggests that it moves the trajectory towards 'flatter' minima; this is often thought to improve generalization [25].”

• cites [26] [26] A. Ghosh, H. Lyu, X. Zhang, and R. Wang. Implicit regularization in heavy-ball momentum accelerated stochastic gradient descent. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=ZzdBhtEH9yB . -Not checked yet

  Not checked in the free run.

• cites [25] [25] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=6Tm1mposlrM . -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The approximation results in Section 3.2 hold for other algorithms like Shampoo or its versions, assuming a typical choice of momentum schemes.

“Finally, we discuss some of the most popular optimizers in recent years, but other important algorithms like Shampoo [30, 58] or its versions are also amenable to this analysis, and the approximation results in Section 3.2 hold for them (assuming a typical choice of momentum schemes).”

• cites [30] [30] V. Gupta, T. Koren, and Y. Singer. Shampoo: Preconditioned stochastic tensor optimization. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1842-1850. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/gupta18a.html . -Not checked yet

  Not checked in the free run.

• cites [58] [58] H.-J. M. Shi, T.-H. Lee, S. Iwasaki, J. Gallego-Posada, Z. Li, K. Rangadurai, D. Mudigere, and M. Rabbat. A distributed data-parallel pytorch implementation of the distributed Shampoo optimizer for training neural networks at-scale. arXiv preprint arXiv:2309.06497 , 2023. URL https://arxiv.org/abs/2309.06497 . -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The parameter controlling the exponential forgetting of gradients β1 corresponds to the ρ2 parameter of Lion, so the default ρ2 = 0.99 in Lion would match β1 = 0.99 rather than β1 = 0.9 in Adam.

“Note that the parameter controlling the exponential forgetting of gradients β 1 corresponds to the ρ 2 parameter of Lion, so the default ρ 2 = 0 . 99 in Lion would match β 1 = 0 . 99 rather than β 1 = 0 . 9 in Adam.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Cohen et al. [16] define 'preconditioned sharpness' as the top eigenvalue of the preconditioned Hessian λ1(P−1t Ht), and observe that this quantity often oscillates around the stability threshold 2+2β1(1−β1)η.

“They define 'preconditioned sharpness' to be the top eigenvalue of the preconditioned Hessian λ 1 ( P -1 t H t ) , where H t is the Hessian of the loss, and observe that this quantity often oscillates around the stability threshold 2+2 β 1 (1 -β 1 ) η , where η is the learning rate.”

• cites [16] [16] J. M. Cohen, B. Ghorbani, S. Krishnan, N. Agarwal, S. Medapati, M. Badura, D. Suo, D. Cardoze, Z. Nado, G. E. Dahl, et al. Adaptive gradient methods at the edge of stability. arXiv preprint arXiv:2207.14484 , 2022. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Ordinary Lion is recovered by setting K(x)=||x||_1.

“As mentioned above, ordinary Lion is recovered by setting K ( x ) = ∥ x ∥ 1 .”

• cites [12] [12] L. Chen, B. Liu, K. Liang, and Q. Liu. Lion secretly solves a constrained optimization: As lyapunov predicts. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=e4xS9ZarDr . -Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

The implementations of the optimizers used for comparing the trajectories in Figure 1 are custom and match exactly the analytical formulas.

“The implementations of the optimizers used for comparing the trajectories in Figure 1 of the paper are custom and match exactly our analytical formulas (in particular, Lion has bias correction and the soft-sign function x ↦→ x/ √ x 2 + ε instead of the sign function), given below.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Our approach works for a wide class of algorithms, and we recover their main results in Section 5.

“Our approach works for a wide class of algorithms, and we recover their main results in Section 5.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Ordinary Lion corresponds to K(x)=||x||_1 and ∇K(x)=sign(x) in Example 1.5, with the sign function understood component-wise.

“Ordinary Lion corresponds to K ( x ) = ∥ x ∥ 1 and ∇K ( x ) = sign( x ) in Example 1.5, where the sign function is understood component-wise.”

• cites [12] [12] L. Chen, B. Liu, K. Liang, and Q. Liu. Lion secretly solves a constrained optimization: As lyapunov predicts. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=e4xS9ZarDr . -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Assumption C.1 is a special case of Assumption 3.1 by Lemma H.1.

“The following assumption, which is a special case of Assumption 3.1 by Lemma H.1, may look a bit technical but allows for a simpler calculation of correction terms.”

• cites Lemma H.1Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: F(n) is a function of 'momentum variables' as described in Assumption C.1, with L two times continuously differentiable functions g(n)ℓ, fixed βℓ in [0,1), bounded nonnegative sequences b(n+1)ℓ, and a fixed two times continuously differentiable Φ.

Probably not a substantial claim: may not pass S3 (load-bearing).

Weight decay causes AdamW and Lion to solve an l∞-norm constrained optimization problem.

“Xie and Li [66] and Chen et al. [12] prove that weight decay causes AdamW and Lion to solve an ℓ ∞ -norm constrained optimization problem.”

• cites [66] [66] S. Xie and Z. Li. Implicit bias of AdamW: ℓ ∞ -norm constrained optimization. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 54488-54510. PMLR, 2024. URL https://proceedings.mlr.press/v235/xie24e.html . -Not checked yet

  Not checked in the free run.

• cites [12] [12] L. Chen, B. Liu, K. Liang, and Q. Liu. Lion secretly solves a constrained optimization: As lyapunov predicts. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=e4xS9ZarDr . -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

AdamW and Lion behave asymptotically like (normalized) steepest descent with respect to l∞-norm.

“In that, they behave asymptotically like (normalized) steepest descent with respect to ℓ ∞ -norm.”

• cites [66] [66] S. Xie and Z. Li. Implicit bias of AdamW: ℓ ∞ -norm constrained optimization. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 54488-54510. PMLR, 2024. URL https://proceedings.mlr.press/v235/xie24e.html . -Not checked yet

  Not checked in the free run.

• cites [12] [12] L. Chen, B. Liu, K. Liang, and Q. Liu. Lion secretly solves a constrained optimization: As lyapunov predicts. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=e4xS9ZarDr . -Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The approximation bounds in Section 3.2 are in terms of the learning rate h, which means h has to be sufficiently small for the approximations and optimization trajectories to be close.

“The approximation bounds in Section 3.2 are in terms of the learning rate h , which means that h has to be sufficiently small for (our approximations and hence) the optimization trajectories to be close.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Cohen et al. [16] view Adam as momentum gradient descent with evolving preconditioner.

“They view Adam as momentum gradient descent with evolving preconditioner”

• cites [16] [16] J. M. Cohen, B. Ghorbani, S. Krishnan, N. Agarwal, S. Medapati, M. Badura, D. Suo, D. Cardoze, Z. Nado, G. E. Dahl, et al. Adaptive gradient methods at the edge of stability. arXiv preprint arXiv:2207.14484 , 2022. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

The general technique introduced in this paper can be used to derive a memoryless discrete iteration which can then be approximated by a continuous trajectory.

“The general technique introduced in this paper can be used to derive a memoryless discrete iteration which can then be approximated by a continuous trajectory.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Moreau-Yosida approximation of the loss function provides a concrete way to write down popular optimizers (including SGD with momentum, RMSProp and Adam) as a sequence of optimization problems.

“the latter work provides a concrete way to write down popular optimizers (including SGD with momentum, RMSProp and Adam) as a sequence of optimization problems.”

• cites [9] [9] A. Betti, G. Ciravegna, M. Gori, S. Melacci, K. Mottin, and F. Precioso. A new perspective on optimizers: leveraging Moreau-Yosida approximation in gradient-based learning. Intelligenza Artificiale , 18(2):301-311, 2024. - sourceNot checked — no full text of the source

May combine two claims.

If ||θ||∞ ≤ 1/λ, then ||∇L(θ)||_1 dominates the term λ∇L(θ)^T θ in absolute value, so the main effect of memory is anti-penalizing the one-norm of the gradient.

“If ∥ θ ∥ ∞ ⩽ 1 /λ , then the norm ∥∇L ( θ ) ∥ 1 dominates the term λ ∇L ( θ ) T θ in absolute value, so the main effect of memory is anti-penalizing the one-norm of the gradient.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A much more fine-grained analysis of heavy-ball momentum gradient descent is provided in [10], where the modified loss is described to arbitrary precision rather than just to first order in h.

“A much more fine-grained analysis of this algorithm is provided in [10] where, in particular, the modified loss is described to arbitrary precision rather than just to first order in h .”

• cites [10] [10] M. D. Cattaneo and B. Shigida. Modified loss of momentum gradient descent: Fine-grained analysis. arXiv preprint arXiv:2509.08483 , 2025. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Li et al. [43] prove in a certain setting that a larger learning rate leads to better generalization.

“Li et al. [43] prove in a certain setting that a larger learning rate leads to better generalization.”

• cites [43] [43] Y. Li, C. Wei, and T. Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances in Neural Information Processing Systems , volume 32, 2019. URL https://proceedings.neurips.cc/paper\_files/paper/2019/ hash/bce9abf229ffd7e570818476ee5d7dde-Abstract.html . -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Cohen et al. [16] notice that in a sense Adam trains at the edge of stability.

“Cohen et al. [16] notice that in a sense Adam trains at the edge of stability.”

• cites [16] [16] J. M. Cohen, B. Ghorbani, S. Krishnan, N. Agarwal, S. Medapati, M. Badura, D. Suo, D. Cardoze, Z. Nado, G. E. Dahl, et al. Adaptive gradient methods at the edge of stability. arXiv preprint arXiv:2207.14484 , 2022. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Cattaneo et al. [11]'s arguments are highly specialized to Adam, and they do not incorporate weight decay.

“In addition, their arguments are highly specialized to Adam, and they do not incorporate weight decay.”

• cites [11] [11] M. D. Cattaneo, J. M. Klusowski, and B. Shigida. On the implicit bias of Adam. In Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 5862-5906. PMLR, 2024. URL https://proceedings. mlr.press/v235/cattaneo24a.html . -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LionK covers a few known algorithms as special cases, and it includes heavy-ball momentum gradient descent as a special case by taking K(x)=||x||^2/2, ρ1=ρ2, and λ=0.

“We consider the generalized LionK algorithm because it covers a few known algorithms as special cases: see Table 1 and Section 3.1 in Chen et al. [12]. In fact, it also includes Example 1.1 as a special case by taking K ( x ) = ∥ x ∥ 2 / 2 , ρ 1 = ρ 2 , and λ = 0”

• cites [12] [12] L. Chen, B. Liu, K. Liang, and Q. Liu. Lion secretly solves a constrained optimization: As lyapunov predicts. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=e4xS9ZarDr . -Not checked yet

  Not checked in the free run.