CoChat
Sign Up
PreprintCited by 303Evolutionary Algorithms and ApplicationsPolynomial and algebraic computationMetaheuristic Optimization Algorithms Research

Symbolic Discovery of Optimization Algorithms

Open access
Chen, Xiangning, Liang, Chen, Huang, Da, Real, Esteban, Wang, Kaiyuan, Liu, Yao, Pham, Hieu, Dong, Xuanyi +4 more·2023·arXiv
DOI 10.48550/arxiv.2302.06675arXiv 2302.06675
Open this paper in CoChat — free accountOpen PDF

Abstract

We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, $\textbf{Lion}$ ($\textit{Evo$\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}$tum}$). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3% $\textit{zero-shot}$ and 91.1% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant. Lion is also successfully deployed in production systems such as Google search ads CTR model.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper formulates optimizer discovery as program search and discovers Lion, a momentum-only optimizer that uses the sign operation for updates. Lion is evaluated across vision, vision-language, diffusion, and language tasks, often matching or outperforming AdamW/Adafactor while sometimes saving compute.

Key findings

  • Lion is more memory-efficient than Adam because it only tracks momentum, and its update has uniform magnitude per parameter via the sign operation.
  • On image classification, Lion boosts ViT accuracy by up to 2% on ImageNet and saves up to 5x pre-training compute on JFT; on vision-language contrastive learning, BASIC-L with Lion reaches 88.3% zero-shot and 91.1% fine-tuning ImageNet accuracy, surpassing previous best results by 2% and 0.1%.
  • On diffusion models, Lion achieves a better FID score and reduces training compute by up to 2.3x; for 256×256 generation, final FID is 4.1 (Lion) vs. 4.7 (AdamW).
  • On language modeling, Lion shows similar or better performance than Adam, with up to 2x compute savings; on GLUE fine-tuning, Lion beats AdamW on average across T5 Base, Large, and 11B, with 10, 12, and 10 wins out of 12 scores, respectively.

Methods

  • Algorithm discovery is cast as program search over imperative programs operating on n-dimensional arrays, using regularized evolution with warm-start and restart, abstract execution, funnel selection, and program simplification.
  • Search uses low-cost proxy tasks: for vision, a 3-layer ViT on 10% ImageNet for 30k steps; for language, a 2-layer Transformer on LM1B for 20K steps; evaluation can complete on one TPU V2 chip within 20 min.
  • Each search experiment uses 100 TPU V2 chips for ~72h, generating 200–300K programs with ~20–30K actually evaluated; total cost is ~3K TPU V2 days. Evaluation compares Lion with AdamW/Adafactor across image classification, vision-language contrastive learning, diffusion, language modeling, and fine-tuning, with tuned peak learning rate and decoupled weight decay.

Limitations

  • The paper states search limitations: the search space is inspired by first-order optimizers, lacks functions for advanced second-order algorithms, has large search cost, requires manual simplification, and uses a simplistic program structure without effective conditional/loop/function constructs.
  • The paper states Lion limitations: gains are small on ResNets, decrease with strong augmentations, and are similar to AdamW on the Imagen text-to-image base model, large-scale internal-dataset autoregressive perplexity, and C4 masked language modeling; Lion likely performs no better than AdamW with batch size <64, and bfloat16 momentum tracking can still be expensive for giant models.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2302.06675

Cites

100 of 100 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

6 of 27 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Findingown data
    § 4.3 Diffusion Model · p. 11

    The final FID scores for 256×256 image generation are 4.1 for Lion vs. 4.7 for AdamW.

    “The final FID scores are 4.1 (Lion) vs. 4.7 (AdamW), and for reference, the FID of ADM (Dhariwal and Nichol, 2021) is 10.94.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Dhariwal and Nichol, 2021 source
  • Findingcitation
    § 4.1 Image Classification · p. 9

    Lion outperforms AdamW by 0.42% on CoAtNet-3 with strong augmentations.

    “When strong augmentations are applied, the gain of Lion over AdamW shrinks, but it still outperforms AdamW by 0.42% on CoAtNet-3, despite the strong regularization during training (Dai et al., 2021).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Dai et al., 2021 source

      The paper reports CoAtNet-3 results in §4.2 (p. 8) and describes augmentation and strong regularization in §4.1 (p. 7). It does not report a Lion–AdamW comparison or the claimed 0.42% advantage, so the performance claim is unsupported by this source. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingown data
    § 2.2 Efficient Search Techniques · p. 4

    The search process can become overwhelmed with invalid programs and cannot make progress without filtering out invalid programs.

    “Preliminary experiments have shown that the search process can become overwhelmed with invalid programs and cannot make progress without filtering out invalid programs.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 1 Introduction · p. 3

    Lion is more robust to different hyperparameter choices compared to AdamW.

    “Additionally, our experiments show that the gain of Lion increases with the batch size and it is more robust to different hyperparameter choices compared to AdamW.”
    • Also stated in: § 5 Hyperparameter Tuning · p. 14 · hedged
      “Suggested by the heatmaps, Lion is more robust to different hyperparameter choices compared to AdamW.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 2.1 Program Search Space · p. 4

    High-performing programs are sparse in the program search space.

    “More importantly, the challenge comes from the sparsity of high-performing programs in the search space.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 25
  • May combine two claims. 50
  • The quote only partly states this claim. 7

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.2 Analysis · p. 7

    The sign operation has been explored in previous optimizers.

    “The sign operation has been explored in previous optimizers (Bernstein et al., 2018; Riedmiller and Braun, 1993).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Bernstein et al., 2018 source
    • cites Riedmiller and Braun, 1993

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.2 Analysis · p. 7

    signSGD's focus is to mitigate communication costs between agents in distributed training, and it observes inferior performance when training ConvNets on image classification tasks.

    “Their focus is to mitigate communication costs between agents in distributed training, and they observe inferior performance when training ConvNets on image classification tasks.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Bernstein et al., 2018 source

      The Abstract (p. 1) says signSGD aims to reduce communication costs in distributed training. Figure 3 (Section 5, p. 7) reports that its momentum variant, SIGNUM, performs about 2% worse than SGD with well-tuned weight decay on ImageNet. The claim overgeneralizes that result: Figure A.3 (p. 11) reports little difference in final test performance on CIFAR-10. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.2 Analysis · p. 7

    The closest previous optimizer to Lion is signSGD (and its momentum variant), which also uses the sign operation but has a different momentum update rule.

    “The closest to ours is the handcrafted optimizer signSGD (Bernstein et al., 2018) (and its momentum variant) that also utilizes the sign operation to calculate the update but has a different momentum update rule from Lion.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Bernstein et al., 2018 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 1 Introduction · p. 2

    AutoML-Zero attempts to search every component of a machine learning pipeline while evaluating on toy tasks.

    “AutoML-Zero (Real et al., 2020) is an ambitious effort that attempts to search every component of a machine learning pipeline while evaluating on toy tasks.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Real et al., 2020 source

  • May combine two claims.

    Findingown data
    § 1 Introduction · p. 3

    The advantage of Lion is smaller if using strong augmentations or a small batch size (<64) during training.

    “The advantage of Lion is smaller if using strong augmentations or a small batch size ( &lt; 64) during training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 6 Limitations · p. 14

    The performance gain brought by Lion decreases when strong augmentations are utilized.

    “The performance gain brought by Lion decreases when strong augmentations are utilized.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Diffusion Model · p. 11

    Lion achieves a higher CLIP score and a less noisy FID metric compared to AdamW on the text-conditional super-resolution model.

    “It achieves a higher CLIP score and has a less noisy FID metric compared to AdamW.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § 6 Limitations · p. 14

    Lion still requires momentum tracking in bfloat16, which can be expensive for training giant models.

    “Additional, Lion still requires momentum tracking in bfloat16 , which can be expensive for training giant models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.4 Language Modeling and Fine-tuning · p. 11

    Lion achieves 1.6x and 1.5x speedup when training the medium size model on Wiki-40B and PG-19, respectively.

    “It achieves 1.6x and 1.5x speedup when training the medium size model on Wiki-40B and PG-19, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.6 Ablations · p. 13

    The optimal batch size for AdamW is 256, while for Lion is 4,096.

    “As shown in Figure 8 (Left), the optimal batch size for AdamW is 256, while for Lion is 4,096.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.2 Analysis · p. 7

    ViT-B/16 trained by Lion on ImageNet has a higher training error compared to AdamW but a 2% higher accuracy on the validation set.

    “An evidence is shown in Figure 11 (Right) in the Appendix, where the ViT-B/16 trained by Lion on ImageNet has a higher training error compared to AdamW but a 2% higher accuracy on the validation set (as shown in Table 2).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.5 Comparison with Other Popular Optimizers · p. 12

    The learning curves of the five adaptive optimizers are pretty similar, whereas Lion has a unique one that learns faster.

    “Figure 10 (in the Appendix) further shows that the learning curves of the five adaptive optimizers are pretty similar, whereas Lion has a unique one that learns faster.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x.

    “On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x.”
    • Also stated in: § 1 Introduction · p. 3 · plain
      “Additionally, Lion reduces the pre-training compute on JFT by up to 5x, improves training efficiency on diffusion models by 2.3x and achieves a better FID score, and offers similar or better performance on language modeling with up to 2x compute savings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given
    § 3.2 Analysis · p. 7

    AdamW needs at least 16 TPU V4 chips to train a ViT-B/16 with image resolution 224 and batch size 4,096, while Lion only needs 8.

    “As an example, AdamW needs at least 16 TPU V4 chips to train a ViT-B/16 with image resolution 224 and batch size 4,096, while Lion only needs 8 (both with bfloat16 momentum).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.2 Analysis · p. 7

    Lion leads to convergence in smoother regions, which usually results in better generalization.

    “Additionally, the results in Appendix G demonstrate that Lion leads to the convergence in smoother regions, which usually results in better generalization.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 2.2 Efficient Search Techniques · p. 5

    Evolutionary search with warm-start and restart significantly outperforms hyperparameter tuning on AdamW and random search, both with 4x more compute.

    “The evolutionary search, with mean and standard error calculated from five runs, significantly outperforms both of them.”
    • Also stated in: § 2.2 Efficient Search Techniques · p. 5 · boosted
      “Our search significantly outperforms the best results achieved by both baselines, demonstrated as the two dashed lines in the figure.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 3 Derivation and Analysis of Lion · p. 7

    The search also discovers other existing or novel algorithms, e.g., some with better regularization and some resembling AdaBelief and AdaGrad.

    “Note that the search also discovers other existing or novel algorithms shown in Appendix D, e.g., some with better regularization and some resembling AdaBelief (Zhuang et al., 2020) and AdaGrad (Duchi et al., 2011).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Zhuang et al., 2020 source
    • cites Duchi et al., 2011

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data
    § 1 Introduction · p. 3

    Lion offers similar or better performance on language modeling with up to 2x compute savings.

    “Additionally, Lion reduces the pre-training compute on JFT by up to 5x, improves training efficiency on diffusion models by 2.3x and achieves a better FID score, and offers similar or better performance on language modeling with up to 2x compute savings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Diffusion Model · p. 11

    Lion enables both better quality and faster convergence on the FID score for diffusion models.

    “Illustrated by Figure 1 (Right) and 5 (Middle and Right), Lion enables both better quality and faster convergence on the FID score.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § 4.4 Language Modeling and Fine-tuning · p. 11

    On the massive internal dataset, there is no perplexity difference between Lion and Adafactor throughout training.

    “On this massive dataset, we observe no perplexity difference throughout training.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Image Classification · p. 9

    Lion's ViT-L/16 matches previous ViT-H/14 results trained by AdamW while being 2x smaller.

    “Our ViT-L/16 matches the previous ViT-H/14 results trained by AdamW, while being 2x smaller.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 2.1 Program Search Space · p. 4

    Preliminary experiments investigated inclusion of conditional and loop statements, defining and calling new functions, but these do not yield improved results.

    “Preliminary experiments have investigated the inclusion of more advanced features such as conditional and loop statements, and defining and calling new functions, but these do not yield improved results, so we leave them out.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Vision-Language Contrastive Learning · p. 9

    Lion's gain over AdamW in retrieval is larger in Recall@1 than Recall@10 on Flickr30K, with +1.70% vs. +0.60% for image→text and +2.14% vs. +0.20% for text→image.

    “Lion outperforms AdamW on both datasets, with a larger gain in Recall@1 than Recall@10 on Flicker30K, implying more accurate retrieval results: +1.70% vs. +0.60% for image → text and +2.14% vs. +0.20% for text → image.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.5 Comparison with Other Popular Optimizers · p. 12

    AMSGrad performs the best on ViT-S/16 but the worst on ViT-B/16.

    “AMSGrad performs the best on ViT-S/16 but the worst on ViT-B/16.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 2.2 Efficient Search Techniques · p. 5

    The cache hit rate and redundant statements percentage increase along with search progress to ~90% and ~70%.

    “The cache hit rate and the redundant statements percentage increase along with the search progress to ∼ 90% and ∼ 70%.”
    • Also stated in: § 2.2 Efficient Search Techniques · p. 4 · plain
      “As seen in Figure 2 (Right), the percentage of redundant statements and cache hit rate both increase as the search proceeds.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.4 Language Modeling and Fine-tuning · p. 12

    Lion achieves 10, 12, and 10 wins out of 12 scores for T5 Base, Large, and 11B models, respectively.

    “It achieves 10, 12, and 10 wins out of 12 scores for T5 Base, Large, and 11B models, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4 Language Modeling and Fine-tuning · p. 11

    Lion outperforms Adafactor on both NLG and NLU tasks, particularly on NLG, with exact match improvements of +1.0, +0.9, and +0.6 for 1.1B, 2.1B, and 7.5B models.

    “Lion outperforms Adafactor on both NLG and NLU tasks, particularly on the NLG tasks, with an exact match improvement of +1.0, +0.9, and +0.6 for the 1.1B, 2.1B, and 7.5B models, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 4.2 Vision-Language Contrastive Learning · p. 10

    Using Lion on both image encoder pre-training and contrastive learning stages in BASIC-L achieves a 2.6% gain over the baseline and 88.3% zero-shot ImageNet accuracy.

    “As illustrated in Table 1, we achieve a significant 2.6% gain over the baseline, striking a 88.3% accuracy on zero-shot ImageNet classification.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Image Classification · p. 9

    The gaps between Lion and AdamW on larger models are consistently bigger, with +0.52% vs. +0.33% (ImageNet), +0.57% vs. +0.23% (ReaL), and +0.74% vs. +0.25% (V2) for ViT-L/16 and ViT-B/16, respectively.

    “The gaps on larger models are consistently bigger, with +0.52% vs. +0.33% (ImageNet), +0.57% vs. +0.23% (ReaL), and +0.74% vs. +0.25% (V2) for ViT-L/16 and ViT-B/16, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4 Language Modeling and Fine-tuning · p. 12

    Lion performs slightly better than AdamW on masked language modeling validation perplexity: 4.18 vs. 4.25 (small), 3.42 vs. 3.54 (medium), and 3.18 vs. 3.25 (large).

    “Lion performs slightly better than AdamW regarding the validation perplexity: 4.18 vs. 4.25 (small), 3.42 vs. 3.54 (medium), and 3.18 vs. 3.25 (large).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Diffusion Model · p. 11

    While there is no clear improvement on the base 64×64 text-to-image model, Lion outperforms AdamW on the text-conditional super-resolution model.

    “While there is no clear improvement on the base 64 × 64 model, Lion outperforms AdamW on the text-conditional super-resolution model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Image Classification · p. 9

    Lion improves accuracy over AdamW by 1.96% for ViT-B/16 and 0.58% for ViT-S/16.

    “Empirically, the improvement is more substantial on models with larger capacity, with accuracy increases of 1.96% and 0.58% for ViT-B/16 and ViT-S/16, respectively.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § D Other Discovered Programs · p. 26

    Reducing the amount of data in the proxy task makes it more likely to discover algorithms with better regularization (Program 5).

    “if we reduce the amount of data in the proxy task, we are more likely to discover algorithms with better regularization (Program 5)”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § I Abstract Execution · p. 29

    Lion achieves higher average NLU scores than Adafactor at 1.1B, 2.1B, and 7.5B on one-shot English NLP tasks.

    “| Avg NLU | 53.2 | 53.9 | 56.8 | 57.4 | 61.3 | 61.7 | 58.5 | 59.4 |”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § I Abstract Execution · p. 28

    A large number of programs are invalid, functionally equivalent, or contain redundant statements that waste compute during evaluation.

    “Our approach is motivated by the fact that a large number of programs are invalid, functionally equivalent, or contain redundant statements that waste compute during evaluation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § I Abstract Execution · p. 28

    The cost of abstract execution is usually negligible compared to the actual execution of the program.

    “The cost of the abstract execution is usually negligible compared to the actual execution of the program.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § F Details of Proxy Tasks · p. 26

    The evaluation time for a program is typically within 20 minutes on one TPU V2 chip.

    “The evaluation time may vary for different programs, but typically a evaluation can be done on one TPU V2 chip within 20min.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § I Abstract Execution · p. 29

    Lion achieves higher average NLG scores than Adafactor at 1.1B, 2.1B, and 7.5B on one-shot English NLP tasks.

    “| Avg NLG | 11.1 | 12.1 | 15.6 | 16.5 | 24.1 | 24.7 | 23.1 | 23.9 |”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § References · p. 24

    When ViT-L/16 is pre-trained on JFT-300M and fine-tuned on ImageNet, the Lion optimizer attains 88.09% ImageNet top-1 accuracy, higher than AdamW's 87.29%.

    “| ViT-L/16 | 304.72M | 14 / 1,035,583 | AdamW | 87.29 | 90.11 | 78.91 | 42.56 | 64.34 |”
    • Also stated in: § References · p. 24 · plain
      “| ViT-L/16 | 304.72M | 14 / 1,035,583 | Lion | 88.09 | 90.62 | 80.48 | 51.55 | 70.72 |”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning
    § I Abstract Execution · p. 28

    There are run-time errors, such as division by zero, that cannot be detected by type and shape inference.

    “Note that there are still run-time errors, such as division by zero, that cannot be detected in this manner.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § 5 Hyperparameter Tuning · p. 14

    Reducing β2 results in shorter memorization of historical information and enhanced training stability.

    “In our experience, reducing β 2 results in shorter memorization of historical information and enhanced training stability .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 6 Limitations · p. 14

    On vision tasks, the discrepancies between Lion, AdamW, and momentum SGD are pretty small on ResNets.

    “On vision tasks, the discrepancies between Lion, AdamW, and momentum SGD are pretty small on ResNets, likely due to the fact that ConvNets are easier to optimize compared to Transformers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation
    § 3.2 Analysis · p. 7

    The sign operation adds noise to the updates, which acts as a form of regularization and helps with generalization.

    “Intuitively, the sign operation adds noise to the updates, which acts as a form of regularization and helps with generalization (Chen et al., 2022; Foret et al., 2021; Neelakantan et al., 2017).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Chen et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Foret et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Neelakantan et al., 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationnone given
    § 1 Introduction · p. 2

    Lion demonstrates outstanding performance across a range of models (Transformer, MLP, ResNet, U-Net, and Hybrid) and tasks (image classification, vision-language contrastive learning, diffusion, language modeling, and fine-tuning).

    “Despite its simplicity, Lion demonstrates outstanding performance across a range of models (Transformer, MLP, ResNet, U-Net, and Hybrid) and tasks (image classification, vision-language contrastive learning, diffusion, language modeling, and fine-tuning).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 6 Limitations · p. 14

    It is likely that Lion performs no better than AdamW if the batch size is small (<64).

    “it is likely that Lion performs no better than AdamW if the batch size is small ( &lt; 64).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 4.6 Ablations · p. 13

    Lion prefers a larger batch size, but its performance remains robust even with a small 64 batch size.

    “This indicates that Lion indeed prefers a larger batch size, but its performance remains robust even with a small 64 batch size.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § I Abstract Execution · p. 28

    Without filtering of invalid programs, the search would be overwhelmed with invalid programs, making it difficult to achieve meaningful progress.

    “Without such filtering of invalid programs, the search would be overwhelmed with invalid programs, making it difficult to achieve meaningful progress.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning
    § 3.1 Derivation · p. 7

    Bias correction is no longer needed, as it does not change the direction.

    “Note that the bias correction is no longer needed, as it does not change the direction.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.2 Analysis · p. 7

    Lion is simpler and has fewer hyperparameters compared to AdamW and Adafactor as it does not require ε and factorization-related ones.

    “Lion is simpler and has fewer hyperparameters compared to AdamW and Adafactor as it does not require ϵ and factorization-related ones.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 3.2 Analysis · p. 7

    NAdam combines the updated first moment and the gradient to compute the update, but Lion decouples momentum tracking and how it is applied to the update through β2.

    “On the other hand, NAdam (Dozat, 2016) combines the updated first moment and the gradient to compute the update, but Lion decouples the momentum tracking and how it is applied to the update through β 2 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Dozat, 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § 3.2 Analysis · p. 7

    The update of Lion is an element-wise binary ±1 if the weight decay term is omitted, with larger norm than those produced by SGD and adaptive algorithms.

    “The update is an element-wise binary ± 1 if we omit the weight decay term, with larger norm than those produced by other optimizers like SGD and adaptive algorithms.”
    • Also stated in: § 1 Introduction · p. 3 · hedged
      “Users should be aware that the uniform update calculated using the sign function usually yields a larger norm compared to those generated by SGD and adaptive methods.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.1 Derivation · p. 7

    Although both m and v are utilized in Program 4, v only changes how the momentum is updated and does not need to be separately tracked.

    “Although both m and v are utilized in Program 4, v only changes how the momentum is updated (two interp functions with constants ∼ 0.9 and ∼ 1.1 is equivalent to one with ∼ 0.99) and does not need to be separately tracked.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 3.2 Analysis · p. 7

    This choice of EMA factor and interpolation allows Lion to balance between remembering a ~10x longer history of the gradient in momentum and putting more weight on the current gradient in the update.

    “This choice of EMA factor and interpolation allows Lion to balance between remembering a ∼ 10x longer history of the gradient in momentum and putting more weight on the current gradient in the update.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 4.6 Ablations · p. 13

    The results validate the effectiveness and necessity of using two linear interpolation functions in Lion.

    “Those results validate the effectiveness and necessity of using two linear interpolation functions, letting Lion to remember longer gradient history meanwhile assign a higher weight to the current gradient.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 4.5 Comparison with Other Popular Optimizers · p. 12

    The inferior performance of PowerSign and AddSign compared to other optimizers is consistent with previous observations that automatically discovered optimizers have difficulty generalizing to real-world learning tasks.

    “The inferior performance of PowerSign and AddSign compared to other optimizers is consistent with previous observations that automatically discovered optimizers have difficulty generalizing to real-world learning tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § 2 Symbolic Discovery of Algorithms · p. 3

    Program length can be used to estimate the complexity of different programs, making it easier to select the simpler, often more generalizable ones.

    “program length can be used to estimate the complexity of different programs, making it easier to select the simpler, often more generalizable ones.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given
    § 2 Symbolic Discovery of Algorithms · p. 3

    Symbolic representations like programs are easier to analyze, comprehend and transfer to new tasks compared to parameterized models such as neural networks.

    “symbolic representations like programs are easier to analyze, comprehend and transfer to new tasks compared to parameterized models such as neural networks”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning
    § 1 Introduction · p. 2

    Previous RL or Monte Carlo Sampling methods limit the search space by using fixed operands and restricting tree size, thereby limiting potential for discovery, and the algorithms discovered have not yet reached state-of-the-art.

    “However, to make the search manageable, they often limit the search space by using fixed operands and restricting the size of the tree, thereby limiting the potential for discovery. For example, they are unable to modify the tracking of momentum or how it contributes to the update, which is an essential component of Lion. Consequently, the algorithms discovered have not yet reached the state-of-”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Bello et al., 2017 source
    • cites Wang et al., 2022 source

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § 1 Introduction · p. 2

    The previous SOTA results on zero-shot and fine-tuning ImageNet accuracy are 86.3% and 91.0% (Yu et al., 2022).

    “The previous SOTA results on zero-shot and fi ne-tuning ImageNet accuracy are 86.3% and 91.0% (Yu et al., 2022).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Yu et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 2 Symbolic Discovery of Algorithms · p. 3

    The method is generally applicable to other tasks beyond optimizers for deep neural network training.

    “This work focuses on optimizers for deep neural network training, but the method is generally applicable to other tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning
    § 5 Hyperparameter Tuning · p. 14

    The update generated by Lion is an element-wise binary ±1, as a result of the sign operation, therefore it has a larger norm than those generated by other optimizers.

    “The update generated by Lion is an element-wise binary ± 1 , as a result of the sign operation, therefore it has a larger norm than those generated by other optimizers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § G Analysis of Loss Landscape · p. 27

    Lion enables ViT to converge to flatter regions, as it helps the model retain comparably lower error against Gaussian perturbations.

    “However, Lion can enable ViT to converge to flatter regions, as it helps the model retain comparably lower error against Gaussian perturbations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § C NLP Tasks · p. 25

    Program 7 uses the difference between gradient and momentum to track the second moment, resembling AdaBelief.

    “Algorithm uses the difference between gradient and momentum to track the second moment, resembling AdaBelief.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationown data
    § References · p. 24

    For ViT models pre-trained on JFT-300M and fine-tuned on ImageNet, Lion achieves higher accuracy than AdamW on ImageNet and on the robustness benchmarks ReaL, V2, A, and R.

    “Table 8: Model performance when pre-trained on JFT-300M then fine-tuned on ImageNet. Those numbers correspond to Figure 1 (Left) and Figure 4.”
    • Also stated in: § References · p. 24 · plain
      “| ViT-L/16 | 304.72M | 14 / 1,035,583 | Lion | 88.09 | 90.62 | 80.48 | 51.55 | 70.72 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § C NLP Tasks · p. 25

    Program 6 tracks the second moment without EMA decay, which is the same as AdaGrad.

    “Algorithm that tracks the second moment without EMA decay, which is the same as AdaGrad.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 6 Limitations · p. 14

    These tasks where Lion performs similarly to AdamW have massive and high-quality datasets, which results in a reduced difference between optimizers.

    “These tasks have a common characteristic in that the datasets are massive and of high quality, which results in a reduced difference between optimizers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 8 Conclusion · p. 15

    The paper's method discovers a simple and effective optimizer, Lion, that is memory-efficient and achieves strong generalization across architectures, datasets and tasks.

    “Our method discovers a simple and effective optimizer, Lion, that is memory-efficient and achieves strong generalization across architectures, datasets and tasks.”
    Quote in the paperSubstantial claim
67 of 67

  • The authors state this as an assumption.

    Simpler programs are more likely to generalize.

    “Simpler programs are easier to understand and our intuition is that they are more likely to generalize, so we simplify the programs with the following steps.”

  • The paper relies on this without support.

    The meta-validation tasks used for selection are representative enough of the target tasks to predict generalization.

    “We use performance on metavalidation tasks that are larger than the proxy tasks by increasing the model size and training steps, to select the programs that generalize beyond proxy tasks then further simplify them.”
    “The discovered algorithms face a significant challenge due to the substantial gap between the proxy tasks during search and the target tasks.”

  • The paper relies on this without support.

    The program search space contains high-quality optimization algorithms that can be discovered.

    “We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training.”
    “Our method discovers a simple and effective optimization algorithm, Lion ( Evo L ved S i gn M o me n tum ).”

  • The paper relies on this without support.

    AdamW is the de facto standard optimizer on a majority of learning tasks.

    “as it is exceedingly popular and the de facto standard optimizer on a majority of learning tasks”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RMSProp, SignSGD and its momentum-based variants, NAdam, AMSGrad, RAdam, and Lion are instances of an associative memory that aims to compress the gradients.

    “we discuss other similar algorithms such as RMSProp (Hinton et al. 2012), SignSGD and its momentum-based variants (Bernstein et al. 2018), NAdam (Dozat 2016), AMSGrad (Reddi et al. 2016), RAdam (Liu et al. 2020), and Lion (Chen et al. 2023) are also instances of an associative memory that aims to compress the gradients.”

    From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Chen et al. 2023

    Section 3.2 (p. 7) says Lion tracks momentum to remember gradient history and is more memory-efficient than AdamW. It does not characterize Lion as an associative memory or say its aim is to compress gradients. Chen et al. is cited alongside Lion specifically, so the paper's comparisons with the other named optimizers do not establish that broader characterization. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • There is a non-negligible effect of mini-batch noise on the picture described in Section J; in particular, Lion does not necessarily outperform Adam if batches are small.

    “Additionally, there is a non-negligible effect of mini-batch noise on the picture we are describing in Section J; in particular, Lion does not necessarily outperform Adam if batches are small [13].”

    From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]

    Section 6 (p. 14) says Lion is likely to perform no better than AdamW at batch sizes below 64, and Section 4.6 (p. 13) shows that Lion prefers larger batches. This supports the small-batch qualification, but the paper compares Lion with AdamW rather than Adam and discusses added noise from Lion’s sign operation rather than explicitly attributing the effect to mini-batch noise. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Lion does not have the kind of implicit anti-regularization induced by memory that AdamW does, providing a theory-based explanation for Lion's better generalization performance recently documented.

    “As an application of our theory, we find that Lion does not have the kind of implicit anti-regularization induced by memory that AdamW does, providing a theory-based explanation for Lion's better generalization performance recently documented [13].”

    From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]

  • Better generalization of Lion on a number of tasks may be partially attributed to the fact that memory does not anti-regularize Lion.

    “Therefore, better generalization of Lion on a number of tasks [13] may be partially attributed to the fact that memory does not anti-regularize Lion.”

    From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting counts of available search functions

    Minor

    Section 2.1 says the search uses 45 math functions, whereas Appendix H says it includes 43 available functions.

  • An ablated optimizer exceeds a compared baseline

    Minor

    The text says both ablations perform worse than all five compared baselines, but on ViT-B/16 ImageNet, Ablation 0.99 scores 79.90 versus AMSGrad's 79.85.

2 of 2

Cites

Flagged first, then by citations

11 of the 111 references couldn’t be matched to a known paper.

Research works printed but not linked: 10

  1. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.
  2. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from questionanswer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
  3. Timothy Dozat. Incorporating Nesterov Momentum into Adam. In Proceedings of the 4th International Conference on Learning Representations , pages 1-4, 2016.
  4. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pages 394-398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics.
  5. Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. Wiki-40B: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference , pages 2440-2452, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4.
  6. John H Holland. Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence . MIT press, 1992.
  7. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning , KR'12, page 552-561. AAAI Press, 2012. ISBN 9781577355601.
  8. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.
  9. Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations , 2017.
  10. Yuanhao Xiong, Li-Cheng Lan, Xiangning Chen, Ruochen Wang, and Cho-Jui Hsieh. Learning to schedule learning rate with graph neural networks. In International Conference on Learning Representations , 2022.

Cited by

Sorted by citations

Related

Authors
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., & Le, Q. V.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
49205-49233
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2302.06675
arXiv
2302.06675
OpenAlex
W7133224204
Semantic Scholar
a7f59b2162ae0ea2520753b1b9b17277490a9458
Cited by
303
References
111
Citations by year
2026: 55 · 2025: 81 · 2024: 116 · 2023: 51
FWCI
53.56
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Polynomial and algebraic computation
  • Physical Sciences › Computer Science › Artificial Intelligence › Metaheuristic Optimization Algorithms Research
Keywords
symbolic computation, optimization algorithms, algorithm discovery

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review