The DOI matches its DataCite record; 100 of the 111 references it cites were found in a registry or an index; 6 of the 27 sources its claims cite were checked for support.
Symbolic Discovery of Optimization Algorithms
Open accessAbstract
We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, $\textbf{Lion}$ ($\textit{Evo$\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}$tum}$). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3% $\textit{zero-shot}$ and 91.1% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant. Lion is also successfully deployed in production systems such as Google search ads CTR model.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper formulates optimizer discovery as program search and discovers Lion, a momentum-only optimizer that uses the sign operation for updates. Lion is evaluated across vision, vision-language, diffusion, and language tasks, often matching or outperforming AdamW/Adafactor while sometimes saving compute.
Key findings
- Lion is more memory-efficient than Adam because it only tracks momentum, and its update has uniform magnitude per parameter via the sign operation.
- On image classification, Lion boosts ViT accuracy by up to 2% on ImageNet and saves up to 5x pre-training compute on JFT; on vision-language contrastive learning, BASIC-L with Lion reaches 88.3% zero-shot and 91.1% fine-tuning ImageNet accuracy, surpassing previous best results by 2% and 0.1%.
- On diffusion models, Lion achieves a better FID score and reduces training compute by up to 2.3x; for 256×256 generation, final FID is 4.1 (Lion) vs. 4.7 (AdamW).
- On language modeling, Lion shows similar or better performance than Adam, with up to 2x compute savings; on GLUE fine-tuning, Lion beats AdamW on average across T5 Base, Large, and 11B, with 10, 12, and 10 wins out of 12 scores, respectively.
Methods
- Algorithm discovery is cast as program search over imperative programs operating on n-dimensional arrays, using regularized evolution with warm-start and restart, abstract execution, funnel selection, and program simplification.
- Search uses low-cost proxy tasks: for vision, a 3-layer ViT on 10% ImageNet for 30k steps; for language, a 2-layer Transformer on LM1B for 20K steps; evaluation can complete on one TPU V2 chip within 20 min.
- Each search experiment uses 100 TPU V2 chips for ~72h, generating 200–300K programs with ~20–30K actually evaluated; total cost is ~3K TPU V2 days. Evaluation compares Lion with AdamW/Adafactor across image classification, vision-language contrastive learning, diffusion, language modeling, and fine-tuning, with tuned peak learning rate and decoupled weight decay.
Limitations
- The paper states search limitations: the search space is inspired by first-order optimizers, lacks functions for advanced second-order algorithms, has large search cost, requires manual simplification, and uses a simplistic program structure without effective conditional/loop/function constructs.
- The paper states Lion limitations: gains are small on ResNets, decrease with strong augmentations, and are similar to AdamW on the Imagen text-to-image base model, large-scale internal-dataset autoregressive perplexity, and C4 masked language modeling; Lion likely performs no better than AdamW with batch size <64, and bfloat16 momentum tracking can still be expensive for giant models.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
71 Own result · 31 Own interpretation · 17 No basis given · 7 Not checked yet · 4 Sources not found · 3 Supported · 3 Partly supported · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“The final FID scores are 4.1 (Lion) vs. 4.7 (AdamW), and for reference, the FID of ADM (Dhariwal and Nichol, 2021) is 10.94.”
What the sources say
- Supportedcites Dhariwal and Nichol, 2021 source
The citation is attached to the reference value for ADM, not the Lion–AdamW comparison. The Results table (p. 9) lists an FID of 10.94 for ADM on ImageNet 256×256. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When strong augmentations are applied, the gain of Lion over AdamW shrinks, but it still outperforms AdamW by 0.42% on CoAtNet-3, despite the strong regularization during training (Dai et al., 2021).”
What the sources say
- Partly supportedcites Dai et al., 2021 sourceExperimental — accuracy not yet measured
The paper reports CoAtNet-3 results in §4.2 (p. 8) and describes augmentation and strong regularization in §4.1 (p. 7). It does not report a Lion–AdamW comparison or the claimed 0.42% advantage, so the performance claim is unsupported by this source. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Preliminary experiments have shown that the search process can become overwhelmed with invalid programs and cannot make progress without filtering out invalid programs.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, our experiments show that the gain of Lion increases with the batch size and it is more robust to different hyperparameter choices compared to AdamW.”
- Also stated in: § 5 Hyperparameter Tuning · p. 14 · hedged
“Suggested by the heatmaps, Lion is more robust to different hyperparameter choices compared to AdamW.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 5 Hyperparameter Tuning · p. 14 · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More importantly, the challenge comes from the sparsity of high-performing programs in the search space.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For limitations, the difference between Lion and AdamW is not statistical significant on some large-scale language and image-text datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis of Lion reveals that its performance gain grows with the training batch size.”
- Also stated in: § 1 Introduction · p. 3 · plain
“Additionally, our experiments show that the gain of Lion increases with the batch size and it is more robust to different hyperparameter choices compared to AdamW.”
- Also stated in: § 3.2 Analysis · p. 7 · plain
“Additionally, the advantage of Lion over AdamW enlarges as the batch size increases, which fits the common practice of scaling up model training through data parallelism (Section 4.6).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 3 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT.”
- Also stated in: § 1 Introduction · p. 3 · plain
“Additionally, Lion reduces the pre-training compute on JFT by up to 5x, improves training efficiency on diffusion models by 2.3x and achieves a better FID score, and offers similar or better performance on language modeling with up to 2x compute savings.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 3 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our observation suggests that evolutionary search experiments that meta-overfit later tend to uncover optimization algorithms that generalize better.”
- Also stated in: § 2.2 Efficient Search Techniques · p. 6 · hedged
“Blue cross depicts the best meta-validation metric averaged within each bin, indicating that meta-overfitting happening later leads to programs that generalize better.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.2 Efficient Search Techniques · p. 6 · hedged
- Probably not a substantial claim. 25
- It may state two separate claims as one. 50
- The quote only partly states this claim. 7
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The sign operation has been explored in previous optimizers (Bernstein et al., 2018; Riedmiller and Braun, 1993).”
What the sources say
- Supportedcites Bernstein et al., 2018 source
The discussion following Algorithm 2 (p. 1) explicitly says that sign-based methods have been studied since RPROP and describes SIGNSGD as taking the sign of each stochastic-gradient coordinate. This supports the claim that previous optimizers explored the sign operation. (LLM, conf=0.94)
- Not matchedcites Riedmiller and Braun, 1993
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Their focus is to mitigate communication costs between agents in distributed training, and they observe inferior performance when training ConvNets on image classification tasks.”
What the sources say
- Partly supportedcites Bernstein et al., 2018 sourceExperimental — accuracy not yet measured
The Abstract (p. 1) says signSGD aims to reduce communication costs in distributed training. Figure 3 (Section 5, p. 7) reports that its momentum variant, SIGNUM, performs about 2% worse than SGD with well-tuned weight decay on ImageNet. The claim overgeneralizes that result: Figure A.3 (p. 11) reports little difference in final test performance on CIFAR-10. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The closest to ours is the handcrafted optimizer signSGD (Bernstein et al., 2018) (and its momentum variant) that also utilizes the sign operation to calculate the update but has a different momentum update rule from Lion.”
What the sources say
- Supportedcites Bernstein et al., 2018 source
Algorithm 2 (p. 1) defines SIGNUM with a momentum update followed by a parameter update using the sign of that momentum. Section 5 (p. 7) explicitly identifies SIGNUM as the momentum version of signSGD. The description of Lion as the closest optimizer and the comparison with Lion’s rule are the citing authors’ own framing, not claims this source must establish. (LLM, conf=0.90)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“AutoML-Zero (Real et al., 2020) is an ambitious effort that attempts to search every component of a machine learning pipeline while evaluating on toy tasks.”
What the sources say
- Supportedcites Real et al., 2020 source
The Introduction (p. 1) says AutoML-Zero aims to search whole ML algorithms, including the model, optimization procedure, and initialization. The Methods (p. 3) describe evaluating candidates on small proxy tasks, supporting the citing sentence’s characterization of toy tasks. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The advantage of Lion is smaller if using strong augmentations or a small batch size ( < 64) during training.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The performance gain brought by Lion decreases when strong augmentations are utilized.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves a higher CLIP score and has a less noisy FID metric compared to AdamW.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additional, Lion still requires momentum tracking in bfloat16 , which can be expensive for training giant models.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves 1.6x and 1.5x speedup when training the medium size model on Wiki-40B and PG-19, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Figure 8 (Left), the optimal batch size for AdamW is 256, while for Lion is 4,096.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An evidence is shown in Figure 11 (Right) in the Appendix, where the ViT-B/16 trained by Lion on ImageNet has a higher training error compared to AdamW but a 2% higher accuracy on the validation set (as shown in Table 2).”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 10 (in the Appendix) further shows that the learning curves of the five adaptive optimizers are pretty similar, whereas Lion has a unique one that learns faster.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x.”
- Also stated in: § 1 Introduction · p. 3 · plain
“Additionally, Lion reduces the pre-training compute on JFT by up to 5x, improves training efficiency on diffusion models by 2.3x and achieves a better FID score, and offers similar or better performance on language modeling with up to 2x compute savings.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · p. 3 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As an example, AdamW needs at least 16 TPU V4 chips to train a ViT-B/16 with image resolution 224 and batch size 4,096, while Lion only needs 8 (both with bfloat16 momentum).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, the results in Appendix G demonstrate that Lion leads to the convergence in smoother regions, which usually results in better generalization.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evolutionary search, with mean and standard error calculated from five runs, significantly outperforms both of them.”
- Also stated in: § 2.2 Efficient Search Techniques · p. 5 · boosted
“Our search significantly outperforms the best results achieved by both baselines, demonstrated as the two dashed lines in the figure.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § 2.2 Efficient Search Techniques · p. 5 · boosted
None of the sources cited for this claim have been checked yet.
In the paper
“Note that the search also discovers other existing or novel algorithms shown in Appendix D, e.g., some with better regularization and some resembling AdaBelief (Zhuang et al., 2020) and AdaGrad (Duchi et al., 2011).”
What the sources say
- Not checked — no full text of the sourcecites Zhuang et al., 2020 source
- Not matchedcites Duchi et al., 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, Lion reduces the pre-training compute on JFT by up to 5x, improves training efficiency on diffusion models by 2.3x and achieves a better FID score, and offers similar or better performance on language modeling with up to 2x compute savings.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Illustrated by Figure 1 (Right) and 5 (Middle and Right), Lion enables both better quality and faster convergence on the FID score.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On this massive dataset, we observe no perplexity difference throughout training.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our ViT-L/16 matches the previous ViT-H/14 results trained by AdamW, while being 2x smaller.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Preliminary experiments have investigated the inclusion of more advanced features such as conditional and loop statements, and defining and calling new functions, but these do not yield improved results, so we leave them out.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion outperforms AdamW on both datasets, with a larger gain in Recall@1 than Recall@10 on Flicker30K, implying more accurate retrieval results: +1.70% vs. +0.60% for image → text and +2.14% vs. +0.20% for text → image.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AMSGrad performs the best on ViT-S/16 but the worst on ViT-B/16.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The cache hit rate and the redundant statements percentage increase along with the search progress to ∼ 90% and ∼ 70%.”
- Also stated in: § 2.2 Efficient Search Techniques · p. 4 · plain
“As seen in Figure 2 (Right), the percentage of redundant statements and cache hit rate both increase as the search proceeds.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 2.2 Efficient Search Techniques · p. 4 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves 10, 12, and 10 wins out of 12 scores for T5 Base, Large, and 11B models, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion outperforms Adafactor on both NLG and NLU tasks, particularly on the NLG tasks, with an exact match improvement of +1.0, +0.9, and +0.6 for the 1.1B, 2.1B, and 7.5B models, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated in Table 1, we achieve a significant 2.6% gain over the baseline, striking a 88.3% accuracy on zero-shot ImageNet classification.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The gaps on larger models are consistently bigger, with +0.52% vs. +0.33% (ImageNet), +0.57% vs. +0.23% (ReaL), and +0.74% vs. +0.25% (V2) for ViT-L/16 and ViT-B/16, respectively.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion performs slightly better than AdamW regarding the validation perplexity: 4.18 vs. 4.25 (small), 3.42 vs. 3.54 (medium), and 3.18 vs. 3.25 (large).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While there is no clear improvement on the base 64 × 64 model, Lion outperforms AdamW on the text-conditional super-resolution model.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirically, the improvement is more substantial on models with larger capacity, with accuracy increases of 1.96% and 0.58% for ViT-B/16 and ViT-S/16, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“if we reduce the amount of data in the proxy task, we are more likely to discover algorithms with better regularization (Program 5)”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Avg NLU | 53.2 | 53.9 | 56.8 | 57.4 | 61.3 | 61.7 | 58.5 | 59.4 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our approach is motivated by the fact that a large number of programs are invalid, functionally equivalent, or contain redundant statements that waste compute during evaluation.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The cost of the abstract execution is usually negligible compared to the actual execution of the program.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The evaluation time may vary for different programs, but typically a evaluation can be done on one TPU V2 chip within 20min.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Avg NLG | 11.1 | 12.1 | 15.6 | 16.5 | 24.1 | 24.7 | 23.1 | 23.9 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| ViT-L/16 | 304.72M | 14 / 1,035,583 | AdamW | 87.29 | 90.11 | 78.91 | 42.56 | 64.34 |”
- Also stated in: § References · p. 24 · plain
“| ViT-L/16 | 304.72M | 14 / 1,035,583 | Lion | 88.09 | 90.62 | 80.48 | 51.55 | 70.72 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § References · p. 24 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that there are still run-time errors, such as division by zero, that cannot be detected in this manner.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In our experience, reducing β 2 results in shorter memorization of historical information and enhanced training stability .”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On vision tasks, the discrepancies between Lion, AdamW, and momentum SGD are pretty small on ResNets, likely due to the fact that ConvNets are easier to optimize compared to Transformers.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Intuitively, the sign operation adds noise to the updates, which acts as a form of regularization and helps with generalization (Chen et al., 2022; Foret et al., 2021; Neelakantan et al., 2017).”
What the sources say
- Not matchedcites Chen et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Foret et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Neelakantan et al., 2017
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite its simplicity, Lion demonstrates outstanding performance across a range of models (Transformer, MLP, ResNet, U-Net, and Hybrid) and tasks (image classification, vision-language contrastive learning, diffusion, language modeling, and fine-tuning).”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it is likely that Lion performs no better than AdamW if the batch size is small ( < 64).”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indicates that Lion indeed prefers a larger batch size, but its performance remains robust even with a small 64 batch size.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without such filtering of invalid programs, the search would be overwhelmed with invalid programs, making it difficult to achieve meaningful progress.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that the bias correction is no longer needed, as it does not change the direction.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion is simpler and has fewer hyperparameters compared to AdamW and Adafactor as it does not require ϵ and factorization-related ones.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“On the other hand, NAdam (Dozat, 2016) combines the updated first moment and the gradient to compute the update, but Lion decouples the momentum tracking and how it is applied to the update through β 2 .”
What the sources say
- Not matchedcites Dozat, 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The update is an element-wise binary ± 1 if we omit the weight decay term, with larger norm than those produced by other optimizers like SGD and adaptive algorithms.”
- Also stated in: § 1 Introduction · p. 3 · hedged
“Users should be aware that the uniform update calculated using the sign function usually yields a larger norm compared to those generated by SGD and adaptive methods.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · p. 3 · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although both m and v are utilized in Program 4, v only changes how the momentum is updated (two interp functions with constants ∼ 0.9 and ∼ 1.1 is equivalent to one with ∼ 0.99) and does not need to be separately tracked.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This choice of EMA factor and interpolation allows Lion to balance between remembering a ∼ 10x longer history of the gradient in momentum and putting more weight on the current gradient in the update.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Those results validate the effectiveness and necessity of using two linear interpolation functions, letting Lion to remember longer gradient history meanwhile assign a higher weight to the current gradient.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The inferior performance of PowerSign and AddSign compared to other optimizers is consistent with previous observations that automatically discovered optimizers have difficulty generalizing to real-world learning tasks.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“program length can be used to estimate the complexity of different programs, making it easier to select the simpler, often more generalizable ones.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“symbolic representations like programs are easier to analyze, comprehend and transfer to new tasks compared to parameterized models such as neural networks”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“However, to make the search manageable, they often limit the search space by using fixed operands and restricting the size of the tree, thereby limiting the potential for discovery. For example, they are unable to modify the tracking of momentum or how it contributes to the update, which is an essential component of Lion. Consequently, the algorithms discovered have not yet reached the state-of-”
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The previous SOTA results on zero-shot and fi ne-tuning ImageNet accuracy are 86.3% and 91.0% (Yu et al., 2022).”
What the sources say
- Not matchedcites Yu et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This work focuses on optimizers for deep neural network training, but the method is generally applicable to other tasks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The update generated by Lion is an element-wise binary ± 1 , as a result of the sign operation, therefore it has a larger norm than those generated by other optimizers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, Lion can enable ViT to converge to flatter regions, as it helps the model retain comparably lower error against Gaussian perturbations.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Algorithm uses the difference between gradient and momentum to track the second moment, resembling AdaBelief.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 8: Model performance when pre-trained on JFT-300M then fine-tuned on ImageNet. Those numbers correspond to Figure 1 (Left) and Figure 4.”
- Also stated in: § References · p. 24 · plain
“| ViT-L/16 | 304.72M | 14 / 1,035,583 | Lion | 88.09 | 90.62 | 80.48 | 51.55 | 70.72 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § References · p. 24 · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Algorithm that tracks the second moment without EMA decay, which is the same as AdaGrad.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tasks have a common characteristic in that the datasets are massive and of high quality, which results in a reduced difference between optimizers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our method discovers a simple and effective optimizer, Lion, that is memory-efficient and achieves strong generalization across architectures, datasets and tasks.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The authors state this as an assumption.
Simpler programs are more likely to generalize.
“Simpler programs are easier to understand and our intuition is that they are more likely to generalize, so we simplify the programs with the following steps.”
The paper relies on this without support.
The meta-validation tasks used for selection are representative enough of the target tasks to predict generalization.
“We use performance on metavalidation tasks that are larger than the proxy tasks by increasing the model size and training steps, to select the programs that generalize beyond proxy tasks then further simplify them.”
“The discovered algorithms face a significant challenge due to the substantial gap between the proxy tasks during search and the target tasks.”
The paper relies on this without support.
The program search space contains high-quality optimization algorithms that can be discovered.
“We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training.”
“Our method discovers a simple and effective optimization algorithm, Lion ( Evo L ved S i gn M o me n tum ).”
The paper relies on this without support.
AdamW is the de facto standard optimizer on a majority of learning tasks.
“as it is exceedingly popular and the de facto standard optimizer on a majority of learning tasks”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we discuss other similar algorithms such as RMSProp (Hinton et al. 2012), SignSGD and its momentum-based variants (Bernstein et al. 2018), NAdam (Dozat 2016), AMSGrad (Reddi et al. 2016), RAdam (Liu et al. 2020), and Lion (Chen et al. 2023) are also instances of an associative memory that aims to compress the gradients.”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Chen et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, there is a non-negligible effect of mini-batch noise on the picture we are describing in Section J; in particular, Lion does not necessarily outperform Adam if batches are small [13].”
From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As an application of our theory, we find that Lion does not have the kind of implicit anti-regularization induced by memory that AdamW does, providing a theory-based explanation for Lion's better generalization performance recently documented [13].”
From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“Therefore, better generalization of Lion on a number of tasks [13] may be partially attributed to the fact that memory does not anti-regularize Lion.”
From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 2.1 Program Search Space · p. 4
“For the program, we select 45 common math functions”
§ H Available Functions · p. 27
“We include 43 available functions that can be used in the program during search.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.6 Ablations · p. 13
“the two ablated optimization algorithms perform worse than all five compared baselines”
§ 4.5 Comparison with Other Popular Optimizers · p. 13
“| ViT-B/16 | ImageNet | 80.12 | 80.26 | 80.32 | 80.29 | 79.85 | 78.95 | 78.50 | 79.54 | 79.90 | 80.77 |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“Train from scratch on ImageNet Following previous works (Dosovitskiy et al., 2021; He et al., 2016), we train ResNet-50 for 90 epochs with a batch size of 1,024, and other models for 300 epochs with a batch size of 4,096.”
MethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“…weight decay [60], also referred to as AdamW, and Adafactor with factorized second moments [86], are still the de facto standard optimizers for training most deep neural networks, especially the recent state-of-the-art language [12, 23, 90], vision [21, 26, 102] and multimodal [75, 84, 100] models.”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“However, Adam (Kingma and Ba, 2014) with decoupled weight decay (Loshchilov and Hutter, 2019), also referred to as AdamW, and Adafactor with factorized second moments (Shazeer and Stern, 2018), are still the de facto standard optimizers for training most deep neural networks, especially the recent state-of-the-art lan…”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 3×
“Train from scratch on ImageNet Following previous works [26, 37], we train ResNet-50 for 90 epochs with a batch size of 1,024, and other models for 300 epochs with a batch size of 4,096.”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 3×
“The zero-shot image-text retrieval results on MSCOCO (Lin et al., 2014) and Flickr30K (Plummer et al.”
MethodHarris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826 · cited 1×
“They are similar to Python code using NumPy / JAX [10, 36] as well as pseudo code of optimization algorithms.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Following GPT-3 [12], we train three models, from 1.1B to 7.5B parameters, on 300B tokens with a batch size of 3M tokens and a context length of 1K.”
BackgroundMethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 2×
“This section focuses on the vision-language contrastive training [75].”
Background
Research works printed but not linked: 10
- Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from questionanswer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
- Timothy Dozat. Incorporating Nesterov Momentum into Adam. In Proceedings of the 4th International Conference on Learning Representations , pages 1-4, 2016.
- Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pages 394-398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics.
- Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. Wiki-40B: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference , pages 2440-2452, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4.
- John H Holland. Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence . MIT press, 1992.
- Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning , KR'12, page 552-561. AAAI Press, 2012. ISBN 9781577355601.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.
- Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations , 2017.
- Yuanhao Xiong, Li-Cheng Lan, Xiangning Chen, Ruochen Wang, and Cho-Jui Hsieh. Learning to schedule learning rate with graph neural networks. In International Conference on Learning Representations , 2022.
Cited by
Sorted by citations
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078
“1%), obtained by Chen et al. (2023a).”
Sigmoid Loss for Language Image Pre-Training
Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01100 · Cited by 1,127
Automated model building and protein identification in cryo-EM maps
Jamali, K., Käll, L., Zhang, R., Brown, A., Kimanius, D., & Scheres, S. H. W. (2023) · doi:10.1101/2023.05.16.541002 · Cited by 686
Mathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., et al. (2024) · Nature · doi:10.1038/s41586-023-06924-6 · Cited by 544
Refusion: Enabling Large-Size Realistic Image Restoration with Latent-Space Diffusion Models
Luo, Z., Gustafsson, F., Zhao, Z., Sjölund, J., & Schön, T. B. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00169 · Cited by 175
“We use the Lion optimizer [9] with β 1 = 0 .”
MethodLearning high-accuracy error decoding for quantum processors
Bausch, J., Senior, A. W., Heras, F. J. H., Edlich, T., Davies, A., Newman, M., Jones, C., Satzinger, K., Niu, M. Y., Blackwell, S., et al. (2024) · Nature · doi:10.1038/s41586-024-08148-8 · Cited by 106
Tunnel lining crack detection model based on improved YOLOv5
Duan, S., Zhang, M., Qiu, S., Xiong, J., Zhang, H., Li, C., Jiang, Q., & Kou, Y. (2024) · Tunnelling and Underground Space Technology · doi:10.1016/j.tust.2024.105713 · Cited by 85
On Efficient Training of Large-Scale Deep Learning Models
Shen, L., Sun, Y., Yu, Z., Ding, L., Tian, X., & Tao, D. (2024) · ACM Computing Surveys · doi:10.1145/3700439 · Cited by 156
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78
Fast Hyperspectral Image Classification Combining Transformers and SimAM-Based CNNs
Liang, L., Zhang, Y., Zhang, S., Li, J., Plaza, A., & Kang, X. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3309245 · Cited by 81
“Empirically, the Lion optimizer’s ideal LR is generally three to ten times lower than that of the Adam optimizer [60].”
MethodOmniVec: Learning robust representations with cross modal sharing
Srivastava, S., & Sharma, G. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00127 · Cited by 91
Hong, J., Bai, Y., Huang, Y., & Chen, Z.-R. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123325 · Cited by 75
Dai, W., Zhu, W., Zhou, G., Liu, G., Xu, J., Zhou, H., Hu, Y., Liu, Z., Li, J., & Li, L. (2024) · Plant Phenomics · doi:10.34133/plantphenomics.0218 · Cited by 72
“We conduct experiments on several advanced optimizer algorithms (Adam, Lion [47], RAdam [48], and RMSprop [49]) under the same experimental configuration and compare them with experiments that only add the SOA.”
MethodEvaluating the Utilities of Foundation Models in Single-cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2023) · doi:10.1101/2023.09.08.555192 · Cited by 62
Optimizing the optimizer for physics-informed neural networks and Kolmogorov-Arnold networks
Kiyani, E., Shukla, K., Urbán, J. F., et al. (2025) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2025.118308 · Cited by 54
“To further evaluate the optimizer’s performance, we also include comparison metrics for several niche first-order optimizers—Lion [95], AdaBelief [96], and the non-linear conjugate gradient method with the Polak–Ribiere update rule [90]—using the same time budget, also presented in Table 14.”
ChatEDA: A Large Language Model Powered Autonomous Agent for EDA
He, Z., Wu, H., Zhang, X., Yao, X., Zheng, S., Zheng, H., & Yu, B. (2023) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1109/mlcad58807.2023.10299852 · Cited by 58
“For efficient fine-tuning on Llama2 using QLoRA, we implement a linear learning rate schedule using paged lion [22] 8-bit optimizer [23], initiating with a learning rate of 1 × 10 − 4 , no weight decay, a batch size of 128, and a sequence length of 4096 tokens.”
MethodGao, H., Yu, X., Xu, Y., Kim, J., & Wang, Y. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3353530 · Cited by 54
“During the training process, we utilized the Lion optimizer [38] with an initial learning rate of 1 e − 4 .”
MethodHolste, G., Zhou, Y., Wang, S., Jaiswal, A. K., Lin, M., Zhuge, S., Yang, Y., Kim, D., Nguyen-Mau, T.-H., Tran, M., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103224 · Cited by 65
LSKA-YOLOv8: A lightweight steel surface defect detection algorithm based on YOLOv8 improvement
Tie, J., Zhu, C., Lu, Z., Wang, H., Ruan, C., Wu, M., Xu, K. H., & Liu, J. (2024) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.08.087 · Cited by 64
Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology
Kraus, O. Z., Kenyon-Dean, K., Saberian, S., Fallah, M., McLean, P., Leung, J., Sharma, V., Khan, A., Balakrishnan, J., Çelik, S., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01117 · Cited by 61
“We found that training dynamics and downstream performance benefited from large batch sizes of up to 16,384 image crops and using the Lion optimizer [16], versus the typical choices of batch size and AdamW optimizer [3].”
MethodInfluentialRyumin, D., Axyonov, A., Рюмина, Е. В., Ivanko, D., Kashevnik, A. M., & Karpov, A. A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124159 · Cited by 48
Transformers as neural operators for solutions of differential equations with finite regularity
Shih, B., Peyvan, A., Zhang, Z., & Karniadakis, G. E. (2024) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.117560 · Cited by 37
LGSleepNet: An Automatic Sleep Staging Model Based on Local and Global Representation Learning
Shen, Q., Xin, J., Liu, X., Wang, Z., Li, C., Huang, Z., & Wang, Z. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3298639 · Cited by 38
“During the training process, a novel Lion optimizer [51] is adopted to update and optimize the network weights.”
MethodResultCross-View Hierarchy Network for Stereo Image Super-Resolution
Zou, W., Gao, H., Chen, L., Zhang, Y., Jiang, M., Yu, Z., & Tan, M. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00144 · Cited by 31
“Our network was optimized using the Lion method [2] with β 1 =0.9, β 2 =0.999, and a batch size of 8.”
MethodInfluentialSelf-citationMulti-modal news event detection with external knowledge
Lin, Z., Xie, J., & Li, Q. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103697 · Cited by 33
Karl, F., Thomas, J., Elstner, J., Gross, R., & Bischl, B. (2024) · Springer eBooks · doi:10.1007/978-3-031-64832-8_1 · Cited by 31
Ferreira, L., Bickley, R. W., Ellison, S. L., Patton, D. R., Byrne-Mamahit, S., Wilkinson, S., Bottrell, C., Fabbro, S., Gwyn, S., & McConnachie, A. W. (2024) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stae1885 · Cited by 28
QADQN: Quantum Attention Deep Q-Network for Financial Market Prediction
Dutta, S., Innan, N., Marchisio, A., Yahia, S. B., & Shafique, M. (2024) · IEEE International Conference on Quantum Computing and Engineering (QCE) · doi:10.1109/qce60285.2024.10303 · Cited by 38
“Update priorities in PD based on TD error 16: if t mod target update frequency == 0 then end for 20: end for Lion (EvoLved Sign Momentum) optimizer [37] is used for the classical optimization step.”
Rethinking pose estimation in crowds: overcoming the detection information bottleneck and ambiguity
Zhou, M., Stoffl, L., Mathis, M. W., & Mathis, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01350 · Cited by 33
“It could potentially also be adapted into video-based architectures, multi-modal learning settings, and with new optimization algorithms [7, 3].”
BackgroundDiffusion-Based Point Cloud Super-Resolution for mmWave Radar Data
Luan, K., Shi, C., Wang, N., Cheng, Y., Lu, H., & Chen, X. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611026 · Cited by 33
“We train our model using the Lion optimizer [5], with an initial learning rate of 4 × 10 − 5 .”
MethodSelf-citationZhang, G., He, C., Zhang, J., et al. (2025) · Engineering Structures · doi:10.1016/j.engstruct.2025.121449 · Cited by 24
A Lorentz-equivariant transformer for all of the LHC
Brehmer, J., Bresó-Pla, V., de Haan, P., Plehn, T., Qu, H., Spinner, J., & Thaler, J. (2025) · SciPost Physics · doi:10.21468/scipostphys.19.4.108 · Cited by 24
Fjeldstad, J., Qi, W., Siddique, N., Mercuri, N. D., Nayot, D., & Taub-Krivoy, A. (2024) · Scientific Reports · doi:10.1038/s41598-024-60901-1 · Cited by 24
“The model was trained on 4453 images for 40 epochs using a batch size of 24 and Lion optimizer with an initial learning rate of 0.0001 45 .”
MethodInfluentialGenuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal
Yang, Y., Wu, H., Avilés-Rivero, A. I., Zhang, Y., Qin, J., & Zhu, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02419 · Cited by 29
Lens-to-Lens Bokeh Effect Transformation. NTIRE 2023 Challenge Report
Conde, M. V., Kolmet, M., Seizinger, T., Bishop, T. E., Timofte, R., Kong, X., Zhang, D., Wu, J., Wang, F., Peng, J., et al. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00166 · Cited by 21
“For both models, they use the Lion [10] optimizer with learning rate initialized with 3e-5 and decayed to 1e-7 by the Cosine scheduler.”
MethodA Review of Optimization Algorithms for Training Neural Networks
Srivastava, A., Rawat, B. S., Singh, G., et al. (2023) · doi:10.1109/icseiet58677.2023.10303287 · Cited by 21
“The behavior of an optimizer may be affected by hyperparameters such as the learning rate, which determines the optimizer's appriseinstruction [11].”
BackgroundTensor neural networks for high-dimensional Fokker–Planck equations
Wang, T., Hu, Z., Kawaguchi, K., et al. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107165 · Cited by 19
Fixing Overconfidence in Dynamic Neural Networks
Meronen, L., Trapp, M., Pilzer, A., Yang, L., & Solin, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00266 · Cited by 29
“The largest MSDNet model used for ImageNet here has 62 million parameters, while the current state-of-the-art model [6] has 2440 million parameters.”
MethodOrchard Vision Navigation Line Extraction Based on YOLOv8-Trunk Detection
Cao, Z., Gong, C., Meng, J., Liu, L., Rao, Y., & Hou, W. (2024) · IEEE Access · doi:10.1109/access.2024.3422422 · Cited by 19
“To optimize the detection speed of the YOLOv8 network, the EvoLved Sign Momentum (Lion) [26] optimizer is incorporated.”
MethodAn Optimized Ensemble Framework for Multi-Label Classification on Long-Tailed Chest X-ray Data
Jeong, J., Jeoun, B.-S., Park, Y., et al. (2023) · IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw60793.2023.00289 · Cited by 17
“On the other hand, the Lion optimizer [8] is more memory-efficient than AdamW as it only keeps the track of the momentum.”
BackgroundMitigating spectral bias in neural operators via high-frequency scaling for physical systems
Khodakarami, S., Oommen, V., Bora, A., & Karniadakis, G. E. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108027 · Cited by 42
Surveillance Face Presentation Attack Detection Challenge
Fang, H., Liu, A., Wan, J., et al. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00677 · Cited by 16
“In addition, Google’s newly proposed LION optimizer [4] is used for network training, which has faster convergence and higher accuracy compared to other optimizers.”
MethodAlmawgani, A. H. M., Sorathiya, V. P., Soni, U., et al. (2023) · Optical and Quantum Electronics · doi:10.1007/s11082-023-05622-x · Cited by 16
Wan, F., Xu, B., Yao, J., Zeng, L., Pan, W., & Liu, H. (2024) · IET Image Processing · doi:10.1049/ipr2.13259 · Cited by 16
A Deep Learning Model for Bacterial Classification Using Big Transfer (BiT)
Visitsattaponge, S., Bunkum, M., Pintavirooj, C., & Paing, M. P. (2024) · IEEE Access · doi:10.1109/access.2024.3358671 · Cited by 15
“Besidesthe aforementioned models, we also conduct a comparison of our proposed model with the most recent state-of-the-art (SOTA) image classification methods including vision transformer (ViT) [29], model soup [30], Lion Fine-tune CNN [31], UzADL [32] and CMSFL [33].”
MethodStatistical downscaling of coastal directional wave spectra using deep learning
Gao, T., & Jiang, H. (2024) · Coastal Engineering · doi:10.1016/j.coastaleng.2024.104557 · Cited by 15
AlphaEvolve: A coding agent for scientific and algorithmic discovery
Novikov, A., Vũ, N., Eisenberger, M., Dupont, E., Huang, P.-S., Wagner, A. Z., Shirobokov, S., Kozlovskii, B., Ruiz, F. J. R., Mehrabian, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.13131 · Cited by 15
“In particular, classical evolutionary techniques have succeeded in symbolic regression applications [66, 87], automated scientific [21] or algorithmic [16] discovery, and scheduling [118] problems.”
Analysis of DAS Seismic Noise Generation and Elimination Process Based on Mean-SDE Diffusion Model
Feng, Q., Wang, S., & Li, Y. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3362853 · Cited by 14
“The optimizer utilized the Lion [36] with β 1 = 0 .”
MethodXGrad: Boosting Gradient-Based Optimizers With Weight Prediction
Guan, L., Li, D., Shi, Y., & Meng, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387399 · Cited by 15
“We believe that other adaptive gradient methods such as AdaBound, RAdam, Yogi, Lion [29], etc can be easily incorporated into our proposed framework.”
MethodA Survey of Federated Learning for mmWave Massive MIMO
Nugroho, V. A., & Lee, B. M. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3397302 · Cited by 16
“The authors devise FEDLION, an FL-based of the EvoLved Sign Momentum (Lion) [116] optimization algorithm, to address the challenges of privacy and data heterogeneity in mobile networks.”
MethodImproving Bird's Eye View Semantic Segmentation by Task Decomposition
Zhao, T., Chen, Y., Wu, Y., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01469 · Cited by 14
“We employ the Lion optimizer [7], set the initial learning rates at 5 × 10 − 4 , 2 × 10 − 5 , and 2 × 10 − 4 for the first, second, and fine-tuning stages, respectively.”
Literature review on deep learning for the segmentation of seismic images
Monteiro, B. A. A., Canguçu, G. L., Jorge, L. M. S., Vareto, R. H., Oliveira, B. S., Silva, T. H., Lima, L. A., Machado, A. M. C., Schwartz, W. R., & de Melo, P. O. S. V. (2024) · Earth-Science Reviews · doi:10.1016/j.earscirev.2024.104955 · Cited by 17
On the Convergence of an Adaptive Momentum Method for Adversarial Attacks
Long, S., Tao, W., Shuohao, L. I., Lei, J. W., & Zhang, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29323 · Cited by 13
Introducing Finite Element Method Integrated Networks (FEMIN)
Thel, S., Greve, L., van de Weg, B. P., & van der Smagt, P. (2024) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.117073 · Cited by 17
AdaPlus: Integrating Nesterov Momentum and Precise Stepsize Adjustment on Adamw Basis
Guan, L. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447337 · Cited by 12
“We compare AdaPlus with eight state-of-the-art optimzers including SGDM [1], Adam [2], Nadam [5], RAdam [7], AdamW [3], AdaBelief [6], AdamW-Win [8], and Lion [9].”
MethodInfluentialAdjustable Visible and Infrared Image Fusion
Wu, B., Nie, J., Wei, W., Zhang, L., & Zhang, Y. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3449638 · Cited by 10
“Implementation Details: When conducting training for the AdjFusion, we utilize the Lion [49] optimizer and set the initial learning rate as 1 .”
CoRe optimizer: an all-in-one solution for machine learning
Eckhoff, M., & Reiher, M. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad1f76 · Cited by 12
Bai, W., Zhuang, G., Zhao, X., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3416684 · Cited by 9
“Unlike typical manual optimizers like Adam, Lion operates as a programmatic auto-discovery optimization algorithm [30].”
MethodFedLion: Faster Adaptive Federated Optimization with Fewer Communication
Tang, Z., & Chang, T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447045 · Cited by 15
“FedLion adapts the cutting-edge centralized adaptive algorithm Lion [1] to the federated learning setting.”
BackgroundMethodInfluentialmmWave Beam Selection in Analog Beamforming Using Personalized Federated Learning
Isaksson, M., Vannella, F., Sandberg, D., & Cöster, R. (2023) · IEEE Future Networks World Forum (FNWF) · doi:10.1109/fnwf58287.2023.10520606 · Cited by 8
“We devise FedLion , a new FL algorithm based on Evo L ved S i gn M o me n tum (Lion) [12] as the optimization algorithm on the parameter server which achieves higher sample efficiency and accuracy than F ED A VG .”
MethodInfluentialDong, H., Zheng, K. W., Wen, S., Zhang, Z., Li, Y., & Zhu, B. (2024) · Sensors · doi:10.3390/s24113691 · Cited by 7
Characterizing hip joint morphology using a multitask deep learning model
Khosravi, B., Bukowiec, L. G., Mickley, J. P., Oeding, J. F., Rouzrokh, P., Erickson, B. J., Sierra, R. J., Taunton, M. J., Grigoriou, E., & Wyles, C. C. (2024) · Journal of Hip Preservation Surgery · doi:10.1093/jhps/hnae041 · Cited by 7
“The Lion optimizer was used for faster convergence [18].”
Evaluating the Utilities of Large Language Models in Single-cell Data AnalysisPreprint
Zhao, H., Liu, T., Li, K., Wang, Y., & Li, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3376641/v1 · Cited by 7
Intelligent fault diagnosis of rolling bearing based on time–frequency processing and DLKA-YOLO
Chen, Z., Zhang, H., Tao, Z., & Wang, Y. (2025) · Franklin Open · doi:10.1016/j.fraope.2025.100228 · Cited by 8
Thel, S., Greve, L., Karl, M., & van der Smagt, P. (2025) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2025.118046 · Cited by 8
On English-Chinese Neural Machine Translation leveraging Transformer model
Mondal, S. K., Chen, Y., Cheng, Y., Dai, H., Alam, S. B., & Kabir, H. M. D. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100166 · Cited by 8
GFRnet: A grain fast recognition network for detecting unsound wheat kernels in grain storage
Zhang, Q., Li, H., Lv, P., et al. (2025) · Journal of Stored Products Research · doi:10.1016/j.jspr.2025.102830 · Cited by 6
nanoT5: Fast & Simple Pre-training and Fine-tuning of T5 Models with Limited Resources
Nawrot, P. (2023) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/2023.nlposs-1.11 · Cited by 7
Reinterpreting Usability of Semantic Segmentation Approach for Darknet Traffic Analysis
Mezina, A., Bürget, R., & Ometov, A. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110493 · Cited by 5
Internet of Things intrusion detection: Research and practice of NSENet and LSTM fusion models
Li, S., Wang, Z., Yang, S., et al. (2024) · Egyptian Informatics Journal · doi:10.1016/j.eij.2024.100476 · Cited by 6
Chun, X., Wang, M., & Zhu, S. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126417 · Cited by 7
UNICAD: A Unified Approach for Attack Detection, Noise Reduction and Novel Class Identification
Pellicer, A. L., Giatgong, K., Li, Y., Suri, N., & Angelov, P. P. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651159 · Cited by 7
“Although highly efficient in tasks such as image classification [3]–[5] object detection [6]– [8], and natural language processing [9]–[11] this technology is exposed to a number of security risks that can be exploited by malicious entities, potentially resulting in harmful outcomes.”
BackgroundLiu, Q., Yun, F., Dong, M., & Wang, Y. (2025) · International Journal of Production Research · doi:10.1080/00207543.2025.2466067 · Cited by 6
“The Lion optimiser, with its lower hyperparameters and higher efficiency compared to mainstream optimisers, quickly minimises the loss function (Chen et al. 2023).”
Optimizing Backbone Networks Through Hybrid–Modal Fusion: A New Strategy for Waste Classification
Zhou, H., Ding, Q., Chang, C., et al. (2025) · Sensors · doi:10.3390/s25103241 · Cited by 5
Guo, X., Cao, J., Gao, B., Jiang, C., & Wang, X. (2023) · Measurement Science and Technology · doi:10.1088/1361-6501/acfe2d · Cited by 6
“Even now, many top-performing models across different datasets are improvements upon ViT, such as Chen et al [47] enhancing its accuracy on ImageNet by 2% using the Lion optimizer.”
MethodMachine Learning Across Network-Connected FPGAs
Diaconu, D., Xie, Y., Gungor, M., Handagala, S., Lin, X., & Leeser, M. E. (2023) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec58863.2023.10363454 · Cited by 5
“Currently, on the ImageNet-1K dataset, the highest accuracy is achieved by vision trans-formers [24]; [25] provides an example of a transformer acceleration framework.”
BackgroundMethodInfluentialAttention-driven acoustic properties learning for underwater target ranging
Chu, X., Zhou, H., Zhang, Y., Zhang, Y., Hu, R., Duan, H., Huang, Y., Zheng, Y., & Ji, R. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111560 · Cited by 6
WMFusion: a W-shaped dual encoder and single decoder network for multimodal medical image fusion
Shao, Y. F., Yu, L., & Tang, H. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06477-3 · Cited by 4
“To achieve an optimized hybrid loss function with a reduced memory footprint, the Lion optimization algorithm [26] with the initial learning rate set to 10 − 4 is employed, and the network is trained for 100 epochs with a batch size of 4.”
A refined lion optimizer for deep learning
Rong, J., Ma, C., Zhang, Q., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-07112-4 · Cited by 4
Du, Y., Geng, X., Zhou, Q., et al. (2025) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2025.146515 · Cited by 4
Li, T., Wang, L., Li, J., et al. (2026) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.70123 · Cited by 4
Ahmed, A., Imran, A. S., Kastrati, Z., Daudpota, S. M., Ullah, M., & Noor, W. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109044 · Cited by 6
Pyramid Ensemble Structure for High Resolution Image Shadow Removal
Cui, S., Huang, J., Tian, S., Fan, M., Zhang, J., Zhu, L., Wei, X., & Wei, X. K. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00138 · Cited by 5
Gesture Recognition Model with Multi-Tracking Capture System for Human-Robot Interaction
Nguyen, K. H. V., Phạm, D. A., Minh, T. B., et al. (2023) · International Conference on System Science and Engineering · doi:10.1109/icsse58758.2023.10227183 · Cited by 4
“Then, the Deep Neural Network (DNN) is trained using the recently introduced Lion optimizer from Google Brain [18].”
MethodSykas, D., Zografakis, D., Demestichas, K. P., et al. (2023) · International Conference on Remote Sensing and Geoinformation of the Environment (RSCy) · doi:10.1117/12.2680777 · Cited by 4
“A modification was added to the optimizer, as it was trained using Lion [4] optimizer, which, given the latest advances, has shown great potential, as explained in Section 1.1.”
MethodInfluentialSorathiya, V. P., Soni, U., Vekariya, V., et al. (2023) · Plasmonics · doi:10.1007/s11468-023-02073-8 · Cited by 4
“We have used the Lion optimizer [53] for updating the weights during the backpropagation and calculating loss.”
Fuad, M. M., Faiyaz, A., Arnob, N. M. K., Mridha, M. F., Saha, A. K., & Aung, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3370674 · Cited by 6
Morrison, N., & Ma, Y. (2025) · APL Machine Learning · doi:10.1063/5.0238444 · Cited by 4
Stream Processing Architectures for Continuous ECG Monitoring Using Subsampling- Based Classifiers
Loh, J., & Gemmeke, T. (2023) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2023.3329360 · Cited by 5
“Although both target the classification of images, best-in-class classifiers differ by multiple orders of magnitude in terms of model parameters, i.e., 1.5M [51] for MNIST and 2440M [52] for ImageNet.”
BackgroundBiCCT: A Compact Convolutional Transformer for EEG Emotion Recognition
Cao, G., Yang, L., Tang, C., Zhang, Q., & He, H. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385574 · Cited by 5
“In our paper, we use a state-of-the-art optimizer named Lion [40].”
MethodBiCAE – A Bimodal Convolutional Autoencoder for Seed Purity Testing
Kukushkin, M. V., Bogdan, M., & Schmid, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70381-2_28 · Cited by 5
He, X., Yan, L., Jiang, L., et al. (2025) · Optics Express · doi:10.1364/oe.563482 · Cited by 3
Machine learning-based analysis of the impact of 5′ untranslated region on protein expression
Wang, L., Liu, S., Huang, J.-X., Zhu, H., Li, S., Li, Y., Chen, S., Han, J., Zhu, Y., Wu, J., et al. (2025) · Nucleic Acids Research · doi:10.1093/nar/gkaf861 · Cited by 3
“First, the original Adam optimizer was replaced with the Lion optimizer [ 17 ] to improve training stability and convergence, enabling better capture of subtle regulatory signals in 5 (cid:2) UTR sequences despite inherent gradient noise in TE data.”
A real time segmentation network for lithium battery surface defect detection
Xie, J., Wu, P., Chen, J., Pan, Y., Zhong, H., Deng, Z., & Sun, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-18315-0 · Cited by 4
Spectral-spatial feature fusion for real-time facial expression recognition
Ma, J., Lin, Y., Qian, L., You, H., & Gao, T. (2025) · Scientific Reports · doi:10.1038/s41598-025-27666-7 · Cited by 3
Multimodal dynamic fusion framework: Multilevel feature fusion guided by prompts
Pan, L., & Wu, H. (2024) · Expert Systems · doi:10.1111/exsy.13668 · Cited by 4
“We employ Lion 2 (Chen et al., 2023) for the optimizer to enhance memory usage rate and training speed, with parameters (0.9, 0.99) from NAS and a weight decay of 1.0.”
InfluentialDeep feature response discriminative calibration
Xu, W., Qiu, T., Zhou, L., Feng, Z., Song, M., & Wang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128848 · Cited by 3
Sliti, W., & Besbes, O. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131168 · Cited by 4
CoLLiE: Collaborative Training of Large Language Models in an Efficient Way
Lv, K., Zhang, S., Gu, T., Xing, S., Hong, J., Chen, K., Liu, X., Yang, Y., Guo, H., Liu, T., et al. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.48 · Cited by 4
Korchagin, V. D. (2023) · Программные системы и вычислительные методы · doi:10.7256/2454-0714.2023.4.69306 · Cited by 4
Metadata
- Authors
- Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., & Le, Q. V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 49205-49233
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2302.06675
- OpenAlex
- W7133224204
- Semantic Scholar
- a7f59b2162ae0ea2520753b1b9b17277490a9458
- Cited by
- 303
- References
- 111
- Citations by year
- 2026: 55 · 2025: 81 · 2024: 116 · 2023: 51
- FWCI
- 53.56
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Polynomial and algebraic computation
- Physical Sciences › Computer Science › Artificial Intelligence › Metaheuristic Optimization Algorithms Research
- Keywords
- symbolic computation, optimization algorithms, algorithm discovery
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
