The DOI matches its DataCite record; 100 of the 111 references it cites were found in a registry or an index; 6 of the 27 sources its claims cite were checked for support.
Symbolic Discovery of Optimization Algorithms
Open accessAbstract
We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, $\textbf{Lion}$ ($\textit{Evo$\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}$tum}$). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3% $\textit{zero-shot}$ and 91.1% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant. Lion is also successfully deployed in production systems such as Google search ads CTR model.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper formulates optimizer discovery as program search and discovers Lion, a momentum-only optimizer that uses the sign operation for updates. Lion is evaluated across vision, vision-language, diffusion, and language tasks, often matching or outperforming AdamW/Adafactor while sometimes saving compute.
Key findings
- Lion is more memory-efficient than Adam because it only tracks momentum, and its update has uniform magnitude per parameter via the sign operation.
- On image classification, Lion boosts ViT accuracy by up to 2% on ImageNet and saves up to 5x pre-training compute on JFT; on vision-language contrastive learning, BASIC-L with Lion reaches 88.3% zero-shot and 91.1% fine-tuning ImageNet accuracy, surpassing previous best results by 2% and 0.1%.
- On diffusion models, Lion achieves a better FID score and reduces training compute by up to 2.3x; for 256×256 generation, final FID is 4.1 (Lion) vs. 4.7 (AdamW).
- On language modeling, Lion shows similar or better performance than Adam, with up to 2x compute savings; on GLUE fine-tuning, Lion beats AdamW on average across T5 Base, Large, and 11B, with 10, 12, and 10 wins out of 12 scores, respectively.
Methods
- Algorithm discovery is cast as program search over imperative programs operating on n-dimensional arrays, using regularized evolution with warm-start and restart, abstract execution, funnel selection, and program simplification.
- Search uses low-cost proxy tasks: for vision, a 3-layer ViT on 10% ImageNet for 30k steps; for language, a 2-layer Transformer on LM1B for 20K steps; evaluation can complete on one TPU V2 chip within 20 min.
- Each search experiment uses 100 TPU V2 chips for ~72h, generating 200–300K programs with ~20–30K actually evaluated; total cost is ~3K TPU V2 days. Evaluation compares Lion with AdamW/Adafactor across image classification, vision-language contrastive learning, diffusion, language modeling, and fine-tuning, with tuned peak learning rate and decoupled weight decay.
Limitations
- The paper states search limitations: the search space is inspired by first-order optimizers, lacks functions for advanced second-order algorithms, has large search cost, requires manual simplification, and uses a simplistic program structure without effective conditional/loop/function constructs.
- The paper states Lion limitations: gains are small on ResNets, decrease with strong augmentations, and are similar to AdamW on the Imagen text-to-image base model, large-scale internal-dataset autoregressive perplexity, and C4 masked language modeling; Lion likely performs no better than AdamW with batch size <64, and bfloat16 momentum tracking can still be expensive for giant models.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
71 Own result · 31 Own interpretation · 17 No basis given · 7 Not checked yet · 4 Sources not found · 3 Supported · 3 Partly supported · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“The final FID scores are 4.1 (Lion) vs. 4.7 (AdamW), and for reference, the FID of ADM (Dhariwal and Nichol, 2021) is 10.94.”
What the sources say
- Supportedcites Dhariwal and Nichol, 2021 source
The citation is attached to the reference value for ADM, not the Lion–AdamW comparison. The Results table (p. 9) lists an FID of 10.94 for ADM on ImageNet 256×256. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When strong augmentations are applied, the gain of Lion over AdamW shrinks, but it still outperforms AdamW by 0.42% on CoAtNet-3, despite the strong regularization during training (Dai et al., 2021).”
What the sources say
- Partly supportedcites Dai et al., 2021 sourceExperimental — accuracy not yet measured
The paper reports CoAtNet-3 results in §4.2 (p. 8) and describes augmentation and strong regularization in §4.1 (p. 7). It does not report a Lion–AdamW comparison or the claimed 0.42% advantage, so the performance claim is unsupported by this source. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Preliminary experiments have shown that the search process can become overwhelmed with invalid programs and cannot make progress without filtering out invalid programs.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, our experiments show that the gain of Lion increases with the batch size and it is more robust to different hyperparameter choices compared to AdamW.”
- Also stated in: § 5 Hyperparameter Tuning · p. 14 · hedged
“Suggested by the heatmaps, Lion is more robust to different hyperparameter choices compared to AdamW.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 5 Hyperparameter Tuning · p. 14 · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More importantly, the challenge comes from the sparsity of high-performing programs in the search space.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For limitations, the difference between Lion and AdamW is not statistical significant on some large-scale language and image-text datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis of Lion reveals that its performance gain grows with the training batch size.”
- Also stated in: § 1 Introduction · p. 3 · plain
“Additionally, our experiments show that the gain of Lion increases with the batch size and it is more robust to different hyperparameter choices compared to AdamW.”
- Also stated in: § 3.2 Analysis · p. 7 · plain
“Additionally, the advantage of Lion over AdamW enlarges as the batch size increases, which fits the common practice of scaling up model training through data parallelism (Section 4.6).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 3 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT.”
- Also stated in: § 1 Introduction · p. 3 · plain
“Additionally, Lion reduces the pre-training compute on JFT by up to 5x, improves training efficiency on diffusion models by 2.3x and achieves a better FID score, and offers similar or better performance on language modeling with up to 2x compute savings.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 3 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our observation suggests that evolutionary search experiments that meta-overfit later tend to uncover optimization algorithms that generalize better.”
- Also stated in: § 2.2 Efficient Search Techniques · p. 6 · hedged
“Blue cross depicts the best meta-validation metric averaged within each bin, indicating that meta-overfitting happening later leads to programs that generalize better.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.2 Efficient Search Techniques · p. 6 · hedged
- Probably not a substantial claim. 25
- It may state two separate claims as one. 50
- The quote only partly states this claim. 7
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The sign operation has been explored in previous optimizers (Bernstein et al., 2018; Riedmiller and Braun, 1993).”
What the sources say
- Supportedcites Bernstein et al., 2018 source
The discussion following Algorithm 2 (p. 1) explicitly says that sign-based methods have been studied since RPROP and describes SIGNSGD as taking the sign of each stochastic-gradient coordinate. This supports the claim that previous optimizers explored the sign operation. (LLM, conf=0.94)
- Not matchedcites Riedmiller and Braun, 1993
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Their focus is to mitigate communication costs between agents in distributed training, and they observe inferior performance when training ConvNets on image classification tasks.”
What the sources say
- Partly supportedcites Bernstein et al., 2018 sourceExperimental — accuracy not yet measured
The Abstract (p. 1) says signSGD aims to reduce communication costs in distributed training. Figure 3 (Section 5, p. 7) reports that its momentum variant, SIGNUM, performs about 2% worse than SGD with well-tuned weight decay on ImageNet. The claim overgeneralizes that result: Figure A.3 (p. 11) reports little difference in final test performance on CIFAR-10. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The closest to ours is the handcrafted optimizer signSGD (Bernstein et al., 2018) (and its momentum variant) that also utilizes the sign operation to calculate the update but has a different momentum update rule from Lion.”
What the sources say
- Supportedcites Bernstein et al., 2018 source
Algorithm 2 (p. 1) defines SIGNUM with a momentum update followed by a parameter update using the sign of that momentum. Section 5 (p. 7) explicitly identifies SIGNUM as the momentum version of signSGD. The description of Lion as the closest optimizer and the comparison with Lion’s rule are the citing authors’ own framing, not claims this source must establish. (LLM, conf=0.90)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“AutoML-Zero (Real et al., 2020) is an ambitious effort that attempts to search every component of a machine learning pipeline while evaluating on toy tasks.”
What the sources say
- Supportedcites Real et al., 2020 source
The Introduction (p. 1) says AutoML-Zero aims to search whole ML algorithms, including the model, optimization procedure, and initialization. The Methods (p. 3) describe evaluating candidates on small proxy tasks, supporting the citing sentence’s characterization of toy tasks. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The advantage of Lion is smaller if using strong augmentations or a small batch size ( < 64) during training.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The performance gain brought by Lion decreases when strong augmentations are utilized.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves a higher CLIP score and has a less noisy FID metric compared to AdamW.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additional, Lion still requires momentum tracking in bfloat16 , which can be expensive for training giant models.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves 1.6x and 1.5x speedup when training the medium size model on Wiki-40B and PG-19, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Figure 8 (Left), the optimal batch size for AdamW is 256, while for Lion is 4,096.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An evidence is shown in Figure 11 (Right) in the Appendix, where the ViT-B/16 trained by Lion on ImageNet has a higher training error compared to AdamW but a 2% higher accuracy on the validation set (as shown in Table 2).”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 10 (in the Appendix) further shows that the learning curves of the five adaptive optimizers are pretty similar, whereas Lion has a unique one that learns faster.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x.”
- Also stated in: § 1 Introduction · p. 3 · plain
“Additionally, Lion reduces the pre-training compute on JFT by up to 5x, improves training efficiency on diffusion models by 2.3x and achieves a better FID score, and offers similar or better performance on language modeling with up to 2x compute savings.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · p. 3 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As an example, AdamW needs at least 16 TPU V4 chips to train a ViT-B/16 with image resolution 224 and batch size 4,096, while Lion only needs 8 (both with bfloat16 momentum).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, the results in Appendix G demonstrate that Lion leads to the convergence in smoother regions, which usually results in better generalization.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evolutionary search, with mean and standard error calculated from five runs, significantly outperforms both of them.”
- Also stated in: § 2.2 Efficient Search Techniques · p. 5 · boosted
“Our search significantly outperforms the best results achieved by both baselines, demonstrated as the two dashed lines in the figure.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § 2.2 Efficient Search Techniques · p. 5 · boosted
None of the sources cited for this claim have been checked yet.
In the paper
“Note that the search also discovers other existing or novel algorithms shown in Appendix D, e.g., some with better regularization and some resembling AdaBelief (Zhuang et al., 2020) and AdaGrad (Duchi et al., 2011).”
What the sources say
- Not checked — no full text of the sourcecites Zhuang et al., 2020 source
- Not matchedcites Duchi et al., 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, Lion reduces the pre-training compute on JFT by up to 5x, improves training efficiency on diffusion models by 2.3x and achieves a better FID score, and offers similar or better performance on language modeling with up to 2x compute savings.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Illustrated by Figure 1 (Right) and 5 (Middle and Right), Lion enables both better quality and faster convergence on the FID score.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On this massive dataset, we observe no perplexity difference throughout training.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our ViT-L/16 matches the previous ViT-H/14 results trained by AdamW, while being 2x smaller.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Preliminary experiments have investigated the inclusion of more advanced features such as conditional and loop statements, and defining and calling new functions, but these do not yield improved results, so we leave them out.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion outperforms AdamW on both datasets, with a larger gain in Recall@1 than Recall@10 on Flicker30K, implying more accurate retrieval results: +1.70% vs. +0.60% for image → text and +2.14% vs. +0.20% for text → image.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AMSGrad performs the best on ViT-S/16 but the worst on ViT-B/16.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The cache hit rate and the redundant statements percentage increase along with the search progress to ∼ 90% and ∼ 70%.”
- Also stated in: § 2.2 Efficient Search Techniques · p. 4 · plain
“As seen in Figure 2 (Right), the percentage of redundant statements and cache hit rate both increase as the search proceeds.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 2.2 Efficient Search Techniques · p. 4 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves 10, 12, and 10 wins out of 12 scores for T5 Base, Large, and 11B models, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion outperforms Adafactor on both NLG and NLU tasks, particularly on the NLG tasks, with an exact match improvement of +1.0, +0.9, and +0.6 for the 1.1B, 2.1B, and 7.5B models, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated in Table 1, we achieve a significant 2.6% gain over the baseline, striking a 88.3% accuracy on zero-shot ImageNet classification.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The gaps on larger models are consistently bigger, with +0.52% vs. +0.33% (ImageNet), +0.57% vs. +0.23% (ReaL), and +0.74% vs. +0.25% (V2) for ViT-L/16 and ViT-B/16, respectively.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion performs slightly better than AdamW regarding the validation perplexity: 4.18 vs. 4.25 (small), 3.42 vs. 3.54 (medium), and 3.18 vs. 3.25 (large).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While there is no clear improvement on the base 64 × 64 model, Lion outperforms AdamW on the text-conditional super-resolution model.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirically, the improvement is more substantial on models with larger capacity, with accuracy increases of 1.96% and 0.58% for ViT-B/16 and ViT-S/16, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“if we reduce the amount of data in the proxy task, we are more likely to discover algorithms with better regularization (Program 5)”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Avg NLU | 53.2 | 53.9 | 56.8 | 57.4 | 61.3 | 61.7 | 58.5 | 59.4 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our approach is motivated by the fact that a large number of programs are invalid, functionally equivalent, or contain redundant statements that waste compute during evaluation.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The cost of the abstract execution is usually negligible compared to the actual execution of the program.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The evaluation time may vary for different programs, but typically a evaluation can be done on one TPU V2 chip within 20min.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Avg NLG | 11.1 | 12.1 | 15.6 | 16.5 | 24.1 | 24.7 | 23.1 | 23.9 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| ViT-L/16 | 304.72M | 14 / 1,035,583 | AdamW | 87.29 | 90.11 | 78.91 | 42.56 | 64.34 |”
- Also stated in: § References · p. 24 · plain
“| ViT-L/16 | 304.72M | 14 / 1,035,583 | Lion | 88.09 | 90.62 | 80.48 | 51.55 | 70.72 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § References · p. 24 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that there are still run-time errors, such as division by zero, that cannot be detected in this manner.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In our experience, reducing β 2 results in shorter memorization of historical information and enhanced training stability .”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On vision tasks, the discrepancies between Lion, AdamW, and momentum SGD are pretty small on ResNets, likely due to the fact that ConvNets are easier to optimize compared to Transformers.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Intuitively, the sign operation adds noise to the updates, which acts as a form of regularization and helps with generalization (Chen et al., 2022; Foret et al., 2021; Neelakantan et al., 2017).”
What the sources say
- Not matchedcites Chen et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Foret et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Neelakantan et al., 2017
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite its simplicity, Lion demonstrates outstanding performance across a range of models (Transformer, MLP, ResNet, U-Net, and Hybrid) and tasks (image classification, vision-language contrastive learning, diffusion, language modeling, and fine-tuning).”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it is likely that Lion performs no better than AdamW if the batch size is small ( < 64).”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indicates that Lion indeed prefers a larger batch size, but its performance remains robust even with a small 64 batch size.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without such filtering of invalid programs, the search would be overwhelmed with invalid programs, making it difficult to achieve meaningful progress.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that the bias correction is no longer needed, as it does not change the direction.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion is simpler and has fewer hyperparameters compared to AdamW and Adafactor as it does not require ϵ and factorization-related ones.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“On the other hand, NAdam (Dozat, 2016) combines the updated first moment and the gradient to compute the update, but Lion decouples the momentum tracking and how it is applied to the update through β 2 .”
What the sources say
- Not matchedcites Dozat, 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The update is an element-wise binary ± 1 if we omit the weight decay term, with larger norm than those produced by other optimizers like SGD and adaptive algorithms.”
- Also stated in: § 1 Introduction · p. 3 · hedged
“Users should be aware that the uniform update calculated using the sign function usually yields a larger norm compared to those generated by SGD and adaptive methods.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · p. 3 · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although both m and v are utilized in Program 4, v only changes how the momentum is updated (two interp functions with constants ∼ 0.9 and ∼ 1.1 is equivalent to one with ∼ 0.99) and does not need to be separately tracked.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This choice of EMA factor and interpolation allows Lion to balance between remembering a ∼ 10x longer history of the gradient in momentum and putting more weight on the current gradient in the update.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Those results validate the effectiveness and necessity of using two linear interpolation functions, letting Lion to remember longer gradient history meanwhile assign a higher weight to the current gradient.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The inferior performance of PowerSign and AddSign compared to other optimizers is consistent with previous observations that automatically discovered optimizers have difficulty generalizing to real-world learning tasks.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“program length can be used to estimate the complexity of different programs, making it easier to select the simpler, often more generalizable ones.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“symbolic representations like programs are easier to analyze, comprehend and transfer to new tasks compared to parameterized models such as neural networks”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“However, to make the search manageable, they often limit the search space by using fixed operands and restricting the size of the tree, thereby limiting the potential for discovery. For example, they are unable to modify the tracking of momentum or how it contributes to the update, which is an essential component of Lion. Consequently, the algorithms discovered have not yet reached the state-of-”
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The previous SOTA results on zero-shot and fi ne-tuning ImageNet accuracy are 86.3% and 91.0% (Yu et al., 2022).”
What the sources say
- Not matchedcites Yu et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This work focuses on optimizers for deep neural network training, but the method is generally applicable to other tasks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The update generated by Lion is an element-wise binary ± 1 , as a result of the sign operation, therefore it has a larger norm than those generated by other optimizers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, Lion can enable ViT to converge to flatter regions, as it helps the model retain comparably lower error against Gaussian perturbations.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Algorithm uses the difference between gradient and momentum to track the second moment, resembling AdaBelief.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 8: Model performance when pre-trained on JFT-300M then fine-tuned on ImageNet. Those numbers correspond to Figure 1 (Left) and Figure 4.”
- Also stated in: § References · p. 24 · plain
“| ViT-L/16 | 304.72M | 14 / 1,035,583 | Lion | 88.09 | 90.62 | 80.48 | 51.55 | 70.72 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § References · p. 24 · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Algorithm that tracks the second moment without EMA decay, which is the same as AdaGrad.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tasks have a common characteristic in that the datasets are massive and of high quality, which results in a reduced difference between optimizers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our method discovers a simple and effective optimizer, Lion, that is memory-efficient and achieves strong generalization across architectures, datasets and tasks.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The authors state this as an assumption.
Simpler programs are more likely to generalize.
“Simpler programs are easier to understand and our intuition is that they are more likely to generalize, so we simplify the programs with the following steps.”
The paper relies on this without support.
The meta-validation tasks used for selection are representative enough of the target tasks to predict generalization.
“We use performance on metavalidation tasks that are larger than the proxy tasks by increasing the model size and training steps, to select the programs that generalize beyond proxy tasks then further simplify them.”
“The discovered algorithms face a significant challenge due to the substantial gap between the proxy tasks during search and the target tasks.”
The paper relies on this without support.
The program search space contains high-quality optimization algorithms that can be discovered.
“We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training.”
“Our method discovers a simple and effective optimization algorithm, Lion ( Evo L ved S i gn M o me n tum ).”
The paper relies on this without support.
AdamW is the de facto standard optimizer on a majority of learning tasks.
“as it is exceedingly popular and the de facto standard optimizer on a majority of learning tasks”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we discuss other similar algorithms such as RMSProp (Hinton et al. 2012), SignSGD and its momentum-based variants (Bernstein et al. 2018), NAdam (Dozat 2016), AMSGrad (Reddi et al. 2016), RAdam (Liu et al. 2020), and Lion (Chen et al. 2023) are also instances of an associative memory that aims to compress the gradients.”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Chen et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, there is a non-negligible effect of mini-batch noise on the picture we are describing in Section J; in particular, Lion does not necessarily outperform Adam if batches are small [13].”
From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As an application of our theory, we find that Lion does not have the kind of implicit anti-regularization induced by memory that AdamW does, providing a theory-based explanation for Lion's better generalization performance recently documented [13].”
From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“Therefore, better generalization of Lion on a number of tasks [13] may be partially attributed to the fact that memory does not anti-regularize Lion.”
From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 2.1 Program Search Space · p. 4
“For the program, we select 45 common math functions”
§ H Available Functions · p. 27
“We include 43 available functions that can be used in the program during search.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.6 Ablations · p. 13
“the two ablated optimization algorithms perform worse than all five compared baselines”
§ 4.5 Comparison with Other Popular Optimizers · p. 13
“| ViT-B/16 | ImageNet | 80.12 | 80.26 | 80.32 | 80.29 | 79.85 | 78.95 | 78.50 | 79.54 | 79.90 | 80.77 |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“Train from scratch on ImageNet Following previous works (Dosovitskiy et al., 2021; He et al., 2016), we train ResNet-50 for 90 epochs with a batch size of 1,024, and other models for 300 epochs with a batch size of 4,096.”
MethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“…weight decay [60], also referred to as AdamW, and Adafactor with factorized second moments [86], are still the de facto standard optimizers for training most deep neural networks, especially the recent state-of-the-art language [12, 23, 90], vision [21, 26, 102] and multimodal [75, 84, 100] models.”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“However, Adam (Kingma and Ba, 2014) with decoupled weight decay (Loshchilov and Hutter, 2019), also referred to as AdamW, and Adafactor with factorized second moments (Shazeer and Stern, 2018), are still the de facto standard optimizers for training most deep neural networks, especially the recent state-of-the-art lan…”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 3×
“Train from scratch on ImageNet Following previous works [26, 37], we train ResNet-50 for 90 epochs with a batch size of 1,024, and other models for 300 epochs with a batch size of 4,096.”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 3×
“The zero-shot image-text retrieval results on MSCOCO (Lin et al., 2014) and Flickr30K (Plummer et al.”
MethodHarris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826 · cited 1×
“They are similar to Python code using NumPy / JAX [10, 36] as well as pseudo code of optimization algorithms.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Following GPT-3 [12], we train three models, from 1.1B to 7.5B parameters, on 300B tokens with a batch size of 3M tokens and a context length of 1K.”
BackgroundMethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 2×
“This section focuses on the vision-language contrastive training [75].”
Background
Research works printed but not linked: 10
- Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from questionanswer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
- Timothy Dozat. Incorporating Nesterov Momentum into Adam. In Proceedings of the 4th International Conference on Learning Representations , pages 1-4, 2016.
- Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pages 394-398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics.
- Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. Wiki-40B: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference , pages 2440-2452, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4.
- John H Holland. Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence . MIT press, 1992.
- Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning , KR'12, page 552-561. AAAI Press, 2012. ISBN 9781577355601.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.
- Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations , 2017.
- Yuanhao Xiong, Li-Cheng Lan, Xiangning Chen, Ruochen Wang, and Cho-Jui Hsieh. Learning to schedule learning rate with graph neural networks. In International Conference on Learning Representations , 2022.
Cited by
Sorted by citations
Items 101–172 of 172
Multiple Gain Adaptations for Improved Neural Networks Training
Challagundla, J., Tyagi, K., Chugh, T., & MANRY, M. T. (2024) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc60891.2024.10427781 · Cited by 3
“Furthermore, the Symbolic Discovery of Optimization Algorithms (LION) [13] marks a pioneering attempt at exploring various optimization algorithms comprehensively.”
BackgroundSNIPPET: A Framework for Subjective Evaluation of Visual Explanations Applied to DeepFake Detection
Yang, Y., Joukovsky, B., Jose Oramas, M., Tuytelaars, T., & Deligiannis, N. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3665248 · Cited by 4
Fast&Focused-Net: Enhancing Small Object Encoding With VDP Layer in Deep Neural Networks
Ali, T., Roy, P. P., & Saini, R. (2024) · IEEE Access · doi:10.1109/access.2024.3447888 · Cited by 6
SEVEN: Pruning Transformer Model by Reserving Sentinels
Xiao, J., Li, P., Nie, J., & Tang, Z. C. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651013 · Cited by 5
“To further validate the generality of our pruning criterion, we conducted fine-tuning using the optimizer Lion [48], which is based on automated search.”
Separable Physics-Informed Neural Networks for Solving the Bgk Model of the Boltzmann EquationPreprint
Oh, J., Cho, S. Y., Yun, S.-B., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4858127 · Cited by 3
AODGCN: Adaptive object detection with attention-guided dynamic graph convolutional network
Zhang, M., Guo, Y., Wang, H., & Shangguan, H. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104386 · Cited by 3
Zhang, X., Dong, H., Yang, J., Lu, Z., Gong, L., & Zhang, L. (2025) · Journal of Phytopathology · doi:10.1111/jph.70106 · Cited by 2
Automated vulnerability evaluation with large language models and vulnerability ontologies
Ghosh, R., von Stockhausen, H., Schmitt, M., Marica, V. G., Karn, S. K., & Farri, O. (2025) · AI Magazine · doi:10.1002/aaai.70031 · Cited by 3
Controlling the Output of a Generative Model by Latent Feature Vector Shifting
Belanec, R., Lacko, P., & Malinovská, K. (2023) · World Symposium on Digital Intelligence for Systems and Machines (DISA) · doi:10.1109/disa59116.2023.10308936 · Cited by 2
“It is also worth mentioning that better classification performance was recently achieved by developing new optimization algorithms like the Lion [17].”
MethodAhmed, A., Imran, A. S., Ullah, M., Kastrati, Z., & Daudpota, S. M. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10648070 · Cited by 2
“We enhance the network’s performance further by integrating the Evo l ved S i gn M o me n tum (LION) optimizer [19], creating a more robust variant.”
Durve, M., Tucny, J., Orsini, S., et al. (2024) · Physics of Fluids · doi:10.1063/5.0232673 · Cited by 2
Imperfect physics-guided neural networks
Carter, A., Imtiaz, S. A., & Naterer, G. F. (2024) · Chemical Engineering Science · doi:10.1016/j.ces.2024.121153 · Cited by 3
Toward Mitigating Architecture Overfitting on Distilled Datasets
Zhong, X., & Liu, C. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3525062 · Cited by 9
“Finally, we adopt a periodical learning rate scheduler, a gradient symbol-based optimizer [67], and a stronger data augmentation scheme to improve the performance further.”
InfluentialZhou, X., Zeng, H., Wang, P., et al. (2025) · Remote Sensing Applications Society and Environment · doi:10.1016/j.rsase.2025.101530 · Cited by 2
Findings from the Bambara - French Machine Translation Competition (BFMT 2023)
Silva, N. A. D., Ajayi, T. O., Antonov, A., Kamate, P. A., Coulibaly, M., Rio, M. D., Diarra, Y., Diarra, S., Emezue, C. C., Hamilcaro, J., et al. (2023) · doi:10.18653/v1/2023.loresmt-1.9 · Cited by 2
“, 2020) LION optimizer (Chen et al., 2023) LoRA (Hu et al.”
BackgroundInformation Extraction from Rich Text Images with RoBERTa and LION Optimizer
Nguyen, T., Doan, T. T., Le, K., Do, T., Ngo, T. D., & Le, D.-D. (2023) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr59823.2023.10288832 · Cited by 2
“However, these optimizers, especially AdamW, demand significant computational resources and time due to their need to track a long history of the gradient in momentum, as highlighted in the LION study [6].”
BackgroundRadwan, A., & Shehata, M. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00031 · Cited by 2
SERS-3DPlace: Ensemble Reinforcement Learning for 3D Monolithic Placement
Mansoor, A., & Chrzanowska-Jeske, M. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10558181 · Cited by 2
“The state-of-the-art algorithms for the ImageNet challenge can now achieve near-human performance and are very fast [8].”
BackgroundFemtosecond pulse parameter estimation from photoelectron momenta using machine learning
Szołdra, T., Ciappina, M. F., Werby, N., Bucksbaum, P. H., Lewenstein, M., Zakrzewski, J., & Maxwell, A. (2023) · New Journal of Physics · doi:10.1088/1367-2630/acee19 · Cited by 1
“Althoughdeep CNNs achieve state-of-the-art in image recognition [52–54], sometimes this requires the use of prohibitively large training datasets and computing power.”
BackgroundLightweight detection method for deepfake face video
Yang, R., Xu, D., & Cheng, Y. (2023) · International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS) · doi:10.1109/iciibms60103.2023.10347675 · Cited by 1
“The Lion (Layerwise-learned Out-of-network-weights for Neural networks) optimizer[19] is an adaptive learning rate method based on the Adam optimizer.”
MethodUCSC-CGEC: A Unified Approach For Chinese Spelling Check And Grammatical Error Correction
Su, J., Xie, Y., & Mou, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651014 · Cited by 1
“In addition, during training, we utilize the Lion [34] optimizer with a warm-up and linear decay schedule.”
MethodJiang, F., Ma, Z., Gong, Z., et al. (2025) · Algorithms · doi:10.3390/a18100618 · Cited by 1
Galaxy mergers in UNIONS – II. Predicting time-scales in the post-merger regime
Ferreira, L., Ellison, S. L., Patton, D. R., Byrne-Mamahit, S., Wilkinson, S., & Bickley, R. W. (2026) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stag178 · Cited by 1
EfficientNet-Lion: A Deep Learning Approach for Spandex Defect Image Classification
Guangyuan, G., Yanqiu, C., & Jianghe, Q. (2023) · International Conference on Information and Network Technologies (ICINT) · doi:10.1109/icint58947.2023.00012 · Cited by 1
Wang, Y., Shi, T., Gao, F., Tian, S., & Yu, L. (2024) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ad25c1 · Cited by 1
Mitigation of Gradient Inversion Attacks in Federated Learning with Private Adaptive Optimization
Lewis, C., Varadharajan, V., Noman, N., Tupakula, U. K., & Li, N. (2024) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs60910.2024.00082 · Cited by 1
“With the representation inversion attack, we have performed 1000 iterations using the Lion optimizer [42] with a learning rate of 0 .”
MethodSAR-To-Optical Translation Using Conditional Diffusion Models for Wildfire-Burned Area Segmentation
Brune, E., & Yifang Ban, A. (2024) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss53475.2024.10641072 · Cited by 1
“Our training employs the Lion optimizer [14], chosen for its quick convergence, especially when the batch size is moderately high.”
MethodUnlocking Chain of Thought in Base Language Models by Heuristic Instruction
Ren, Z., Wang, B., Sun, S., Wang, Y., Ding, Z., & Liu, W. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650937 · Cited by 1
“To mitigate the issues of overfit and collinearity, the ElasticNet Regular-izer [35] and the Lion gradient strategy [36] were applied for optimal calibration of model parameters.”
MethodZhang, M., Song, B., Chen, C., et al. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05720-7 · Cited by 1
Abouzahir, S., Sabir, E., Sadik, M., Abouzahir, M., & Agramelal, F. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109577 · Cited by 1
Lee, D., Kim, J., & Kim, Y. (2024) · 대한원격탐사학회지 · doi:10.7780/kjrs.2024.40.5.1.11 · Cited by 1
AI-Enabled Placement for 2D and 3D ICs
Mansoor, A., Chrzanowska-Jeske, M., & Jeske, M. (2024) · Springer eBooks · doi:10.1007/978-3-031-71436-8_6 · Cited by 1
A probabilistic optimizer for binary neural networks
He, X., Geng, X., Zhang, T., et al. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130297 · Cited by 1
Quality‐Diversity Methods for the Modern Data Scientist
Stock, M., Hauwermeiren, D. V., Bernard De Baets, C., Taelman, S., Marzougui, D., & Haeverbeke, M. V. (2025) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.70047 · Cited by 3
BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?
Zhao, X., Wang, T., Osborn, S., & Rios, A. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.16 · Cited by 4
CoRe Optimizer: An All-in-One Solution for Machine LearningPreprint
Eckhoff, M., & Reiher, M. (2023) · arXiv (Cornell University) · doi:10.3929/ethz-b-000658265 · Cited by 1
Nested Learning: The Illusion of Deep Learning Architectures
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2512.24695 · Cited by 37
“…that aims to predict the variance of gradients, we discuss other similar algorithms such as RMSProp [109], SignSGD and its momentum-based variants [110], NAdam [111], AMSGrad [112], RAdam [113], and Lion [114] are also instances of an associative memory that aims to compress the gradients.”
EvoX: Meta-Evolution for Automated Discovery
Liu, S., Agarwal, S., Maheswaran, M., Cemri, M., Li, Z., Mang, Q., Naren, A., Boneh, E., Cheng, A., Pan, M. Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.23413 · Cited by 1
“Prior work explores learned optimizers, symbolic discovery of update rules [6], gradient-based meta-learning [2], and reinforcement learning-based optimizers [26].”
Can AI Read the Minds of Corporate Executives?Preprint
Chapados, N., Fan, Z., Goyenko, R. Y., Laradji, I., Liu, F., & Zhang, C. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4493166
Research on joint optimizer for point cloud classification
Tai, H., & Ma, X. (2023) · IEEE Conference on Industrial Electronics and Applications (ICIEA) · doi:10.1109/iciea58696.2023.10241869
NeuroSMPC: A Neural Network Guided Sampling Based MPC for On-Road Autonomous Driving
Pal, K., Sharma, A., Omama, M., Shah, P. N., & Krishna, K. M. (2023) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case56687.2023.10260513
New developed Methods used for Data Science OptimizationPreprint
Mahboub, M. A., & Krishna, T. G. (2023) · Research Square · doi:10.21203/rs.3.rs-3367194/v1
Two Proposed Solutions for Mitigating Blurred Output of Autoencoder
Kim, S., & Jo, K.-H. (2023) · International Workshop on Intelligent Systems (IWIS) · doi:10.1109/iwis58789.2023.10284593
“2) Optimizer: This study utilized the Lion Optimizer [7], which outperforms AdamW [8].”
MethodHolste, G., Zhou, Y., Wang, S., et al. (2023) · PubMed · doi:10.48550/arxiv.2310.16112
Convolutional Correlation Branch Network for Image ClassificationPreprint
Li, C., Luo, W., & Zhu, L. (2023) · Research Square · doi:10.21203/rs.3.rs-3706179/v1
Confusing Large Models by Confusing Small Models
Albiero, V., Mehta, R., Evtimov, I., Bell, S. J., Sagun, L., & Markosyan, A. H. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00465
“This progress has taken place across a range of computer vision tasks along a gradient of increasing complexity, from image classification [6, 9, 12], through object detection [24, 38], to image segmentation [22,39] and beyond.”
BackgroundAlmawgani, A. H. M., Sorathiya, V., Soni, U., et al. (2024) · Physica Scripta · doi:10.1088/1402-4896/ad28df
Pdcnet: A Lightweight Model for Plant Disease DetectionPreprint
Shi, J., Dai, W., Sun, H., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4749751
A Comparison Between CCTV and Industrial Cameras for Vehicle Attribute Recognition
Asadi, M., Fakhar, M. Y., Hashemi, S. S., Rezaei, S., Abari, M. K., & Abbaspour, S. A. (2024) · Iranian Conference on Machine Vision and Image Processing · doi:10.1109/mvip62238.2024.10491159
Zhou, Y., Zhao, S., Sonneck, J., & Chen, J. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.05.24.595656
Optical neural networks trained in situ with reinforcement learning
Neill, O., & Faccio, D. (2024) · doi:10.1117/12.3021870
Enhancing Metric Learning: Addressing Inseparability with Harder Metric Learning Loss Function
Xia, L., Huang, X., Wei, H., & Ni, H. (2024) · International Conference on Power Electronics and Artificial Intelligence (PEAI) · doi:10.1145/3674225.3674368
“In recent years, with the development of deep metric learning, it has been widely applied in various fields such as face recognition, visual tracking, and image classification [1–6, 17, 22].”
Survey on Advanced Metaheuristic Algorithms in Cybersecurity Problems Over Web3.0
(2024) · Advances in web technologies and engineering book series · doi:10.4018/978-1-6684-8686-3.ch007
Deep Detection Dreams: Enhancing Visualization Tools for Single Stage Object Detectors
Limberg, C., Harter, A., Melnik, A., et al. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-66743-5_6
Deep Explainable Ensemble for Medical Image Analysis
Syed, H. H., Aneja, N., Malik, O. A., & Minhaj, M. A. R. (2024) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon61640.2024.10903010
“The optimization algorithm utilized for the ensemble is Lion (Evo L ved S i gn M o me n tum), developed by Google in 2023 [13].”
Minghao, Z., Bifeng, S., Xiaojun, Y., & Wang, L. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110373
Kitano, H., Kimura, Y., & Shibata, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5271878
Lin, Q., Sheng, J., Zhou, C., et al. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.05.050
Novel Optimization Methods for Fuzzy Networks
Mikhailov, D. A., & Abramov, M. V. (2025) · doi:10.1109/scm66446.2025.11060426
“There are studies that show that for data with a large number of individuals and features, as well as on complex large models, it may perform worse [15, 16].”
Marion, B., Ortega, J., Tran, L. W., & Ricci, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5377268
Adaptive Dynamic Projection: A Novel Approach for Enhancing Multimodal Large Language Models
Du, P., Xu, C., Zhang, Q., Wu, J. Z., & Zhao, K. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.07.140 · Cited by 1
BiCrossNet: resource-efficient cross-view geolocalization with binary neural networks
Fontana, F., Jantos, T., Steinbrener, J., Cinque, L., Foresti, G. L., & Rinner, B. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/adfa67
Huang, Y.-J., & Li, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5405061
A SHDAViT-MCA Block-Based Network for Remote-Sensing Semantic Change Detection
Ren, W., Zhang, Z., Liu, S., Xu, H., Ma, Z., Gao, R., Kong, Q., Dong, S., & Su, Z. (2025) · Remote Sensing · doi:10.3390/rs17173026
Optimizing deep learning models: A review
Belhadri, A., & Benchennane, I. (2025) · Multiagent and Grid Systems · doi:10.1177/15741702251370052
“With the advent of large—scale language models (LLMs) and the transformer paradigm, a new generation of optimizers has emerged–most notably Lion (EvoLved Sign Momentum) 9 and Sophia ( Scalable One-Pass Hessian Approximation ).”
Vision-based multimodal energy expenditure estimation for aerobic exercise in adults
Jin, L., Zhang, S., Shi, M., Yu, L., Wang, M., Song, M., & Xu, W. (2025) · Frontiers in Physiology · doi:10.3389/fphys.2025.1666616
“Our model and comparison models were implemented by PyTorch and optimized by Lion optimizer (Chen et al., 2024) with a learning rate of 10 −4 , weight decay of 5 × 10 −4 , and cosine annealing as the learning rate decay schedule.”
Drainage Pipe Defect Detection and Effect Analysis Based on Improved YOLOv8_DGW
Wang, J., Cui, Z., Wang, H., et al. (2025) · Journal of Pipeline Systems Engineering and Practice · doi:10.1061/jpsea2.pseng-1907
Self-citationLi, X., Yuan, G., Yu, P., et al. (2025) · International Journal of Remote Sensing · doi:10.1080/01431161.2025.2589946
Adaptive fireworks swarm learning: a hybrid approach for non-convex optimization
Liu, Y., & Tan, Y. (2026) · Evolutionary Intelligence · doi:10.1007/s12065-026-01194-3
Self-citationHow Memory in Optimization Algorithms Implicitly Modifies the Loss
Cattaneo, M., & Shigida, B. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-5215
Metadata
- Authors
- Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., & Le, Q. V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 49205-49233
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2302.06675
- OpenAlex
- W7133224204
- Semantic Scholar
- a7f59b2162ae0ea2520753b1b9b17277490a9458
- Cited by
- 303
- References
- 111
- Citations by year
- 2026: 55 · 2025: 81 · 2024: 116 · 2023: 51
- FWCI
- 53.56
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Polynomial and algebraic computation
- Physical Sciences › Computer Science › Artificial Intelligence › Metaheuristic Optimization Algorithms Research
- Keywords
- symbolic computation, optimization algorithms, algorithm discovery
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
