The DOI matches its DataCite record; 100 of the 111 references it cites were found in a registry or an index; 6 of the 27 sources its claims cite were checked for support.
Symbolic Discovery of Optimization Algorithms
Open accessAbstract
We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, $\textbf{Lion}$ ($\textit{Evo$\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}$tum}$). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3% $\textit{zero-shot}$ and 91.1% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant. Lion is also successfully deployed in production systems such as Google search ads CTR model.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper formulates optimizer discovery as program search and discovers Lion, a momentum-only optimizer that uses the sign operation for updates. Lion is evaluated across vision, vision-language, diffusion, and language tasks, often matching or outperforming AdamW/Adafactor while sometimes saving compute.
Key findings
- Lion is more memory-efficient than Adam because it only tracks momentum, and its update has uniform magnitude per parameter via the sign operation.
- On image classification, Lion boosts ViT accuracy by up to 2% on ImageNet and saves up to 5x pre-training compute on JFT; on vision-language contrastive learning, BASIC-L with Lion reaches 88.3% zero-shot and 91.1% fine-tuning ImageNet accuracy, surpassing previous best results by 2% and 0.1%.
- On diffusion models, Lion achieves a better FID score and reduces training compute by up to 2.3x; for 256×256 generation, final FID is 4.1 (Lion) vs. 4.7 (AdamW).
- On language modeling, Lion shows similar or better performance than Adam, with up to 2x compute savings; on GLUE fine-tuning, Lion beats AdamW on average across T5 Base, Large, and 11B, with 10, 12, and 10 wins out of 12 scores, respectively.
Methods
- Algorithm discovery is cast as program search over imperative programs operating on n-dimensional arrays, using regularized evolution with warm-start and restart, abstract execution, funnel selection, and program simplification.
- Search uses low-cost proxy tasks: for vision, a 3-layer ViT on 10% ImageNet for 30k steps; for language, a 2-layer Transformer on LM1B for 20K steps; evaluation can complete on one TPU V2 chip within 20 min.
- Each search experiment uses 100 TPU V2 chips for ~72h, generating 200–300K programs with ~20–30K actually evaluated; total cost is ~3K TPU V2 days. Evaluation compares Lion with AdamW/Adafactor across image classification, vision-language contrastive learning, diffusion, language modeling, and fine-tuning, with tuned peak learning rate and decoupled weight decay.
Limitations
- The paper states search limitations: the search space is inspired by first-order optimizers, lacks functions for advanced second-order algorithms, has large search cost, requires manual simplification, and uses a simplistic program structure without effective conditional/loop/function constructs.
- The paper states Lion limitations: gains are small on ResNets, decrease with strong augmentations, and are similar to AdamW on the Imagen text-to-image base model, large-scale internal-dataset autoregressive perplexity, and C4 masked language modeling; Lion likely performs no better than AdamW with batch size <64, and bfloat16 momentum tracking can still be expensive for giant models.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
71 Own result · 31 Own interpretation · 17 No basis given · 7 Not checked yet · 4 Sources not found · 3 Supported · 3 Partly supported · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“The final FID scores are 4.1 (Lion) vs. 4.7 (AdamW), and for reference, the FID of ADM (Dhariwal and Nichol, 2021) is 10.94.”
What the sources say
- Supportedcites Dhariwal and Nichol, 2021 source
The citation is attached to the reference value for ADM, not the Lion–AdamW comparison. The Results table (p. 9) lists an FID of 10.94 for ADM on ImageNet 256×256. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When strong augmentations are applied, the gain of Lion over AdamW shrinks, but it still outperforms AdamW by 0.42% on CoAtNet-3, despite the strong regularization during training (Dai et al., 2021).”
What the sources say
- Partly supportedcites Dai et al., 2021 sourceExperimental — accuracy not yet measured
The paper reports CoAtNet-3 results in §4.2 (p. 8) and describes augmentation and strong regularization in §4.1 (p. 7). It does not report a Lion–AdamW comparison or the claimed 0.42% advantage, so the performance claim is unsupported by this source. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Preliminary experiments have shown that the search process can become overwhelmed with invalid programs and cannot make progress without filtering out invalid programs.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, our experiments show that the gain of Lion increases with the batch size and it is more robust to different hyperparameter choices compared to AdamW.”
- Also stated in: § 5 Hyperparameter Tuning · p. 14 · hedged
“Suggested by the heatmaps, Lion is more robust to different hyperparameter choices compared to AdamW.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 5 Hyperparameter Tuning · p. 14 · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More importantly, the challenge comes from the sparsity of high-performing programs in the search space.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For limitations, the difference between Lion and AdamW is not statistical significant on some large-scale language and image-text datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis of Lion reveals that its performance gain grows with the training batch size.”
- Also stated in: § 1 Introduction · p. 3 · plain
“Additionally, our experiments show that the gain of Lion increases with the batch size and it is more robust to different hyperparameter choices compared to AdamW.”
- Also stated in: § 3.2 Analysis · p. 7 · plain
“Additionally, the advantage of Lion over AdamW enlarges as the batch size increases, which fits the common practice of scaling up model training through data parallelism (Section 4.6).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 3 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT.”
- Also stated in: § 1 Introduction · p. 3 · plain
“Additionally, Lion reduces the pre-training compute on JFT by up to 5x, improves training efficiency on diffusion models by 2.3x and achieves a better FID score, and offers similar or better performance on language modeling with up to 2x compute savings.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 3 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our observation suggests that evolutionary search experiments that meta-overfit later tend to uncover optimization algorithms that generalize better.”
- Also stated in: § 2.2 Efficient Search Techniques · p. 6 · hedged
“Blue cross depicts the best meta-validation metric averaged within each bin, indicating that meta-overfitting happening later leads to programs that generalize better.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.2 Efficient Search Techniques · p. 6 · hedged
- Probably not a substantial claim. 25
- It may state two separate claims as one. 50
- The quote only partly states this claim. 7
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The sign operation has been explored in previous optimizers (Bernstein et al., 2018; Riedmiller and Braun, 1993).”
What the sources say
- Supportedcites Bernstein et al., 2018 source
The discussion following Algorithm 2 (p. 1) explicitly says that sign-based methods have been studied since RPROP and describes SIGNSGD as taking the sign of each stochastic-gradient coordinate. This supports the claim that previous optimizers explored the sign operation. (LLM, conf=0.94)
- Not matchedcites Riedmiller and Braun, 1993
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Their focus is to mitigate communication costs between agents in distributed training, and they observe inferior performance when training ConvNets on image classification tasks.”
What the sources say
- Partly supportedcites Bernstein et al., 2018 sourceExperimental — accuracy not yet measured
The Abstract (p. 1) says signSGD aims to reduce communication costs in distributed training. Figure 3 (Section 5, p. 7) reports that its momentum variant, SIGNUM, performs about 2% worse than SGD with well-tuned weight decay on ImageNet. The claim overgeneralizes that result: Figure A.3 (p. 11) reports little difference in final test performance on CIFAR-10. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The closest to ours is the handcrafted optimizer signSGD (Bernstein et al., 2018) (and its momentum variant) that also utilizes the sign operation to calculate the update but has a different momentum update rule from Lion.”
What the sources say
- Supportedcites Bernstein et al., 2018 source
Algorithm 2 (p. 1) defines SIGNUM with a momentum update followed by a parameter update using the sign of that momentum. Section 5 (p. 7) explicitly identifies SIGNUM as the momentum version of signSGD. The description of Lion as the closest optimizer and the comparison with Lion’s rule are the citing authors’ own framing, not claims this source must establish. (LLM, conf=0.90)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“AutoML-Zero (Real et al., 2020) is an ambitious effort that attempts to search every component of a machine learning pipeline while evaluating on toy tasks.”
What the sources say
- Supportedcites Real et al., 2020 source
The Introduction (p. 1) says AutoML-Zero aims to search whole ML algorithms, including the model, optimization procedure, and initialization. The Methods (p. 3) describe evaluating candidates on small proxy tasks, supporting the citing sentence’s characterization of toy tasks. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The advantage of Lion is smaller if using strong augmentations or a small batch size ( < 64) during training.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The performance gain brought by Lion decreases when strong augmentations are utilized.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves a higher CLIP score and has a less noisy FID metric compared to AdamW.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additional, Lion still requires momentum tracking in bfloat16 , which can be expensive for training giant models.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves 1.6x and 1.5x speedup when training the medium size model on Wiki-40B and PG-19, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Figure 8 (Left), the optimal batch size for AdamW is 256, while for Lion is 4,096.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An evidence is shown in Figure 11 (Right) in the Appendix, where the ViT-B/16 trained by Lion on ImageNet has a higher training error compared to AdamW but a 2% higher accuracy on the validation set (as shown in Table 2).”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 10 (in the Appendix) further shows that the learning curves of the five adaptive optimizers are pretty similar, whereas Lion has a unique one that learns faster.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x.”
- Also stated in: § 1 Introduction · p. 3 · plain
“Additionally, Lion reduces the pre-training compute on JFT by up to 5x, improves training efficiency on diffusion models by 2.3x and achieves a better FID score, and offers similar or better performance on language modeling with up to 2x compute savings.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · p. 3 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As an example, AdamW needs at least 16 TPU V4 chips to train a ViT-B/16 with image resolution 224 and batch size 4,096, while Lion only needs 8 (both with bfloat16 momentum).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, the results in Appendix G demonstrate that Lion leads to the convergence in smoother regions, which usually results in better generalization.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evolutionary search, with mean and standard error calculated from five runs, significantly outperforms both of them.”
- Also stated in: § 2.2 Efficient Search Techniques · p. 5 · boosted
“Our search significantly outperforms the best results achieved by both baselines, demonstrated as the two dashed lines in the figure.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § 2.2 Efficient Search Techniques · p. 5 · boosted
None of the sources cited for this claim have been checked yet.
In the paper
“Note that the search also discovers other existing or novel algorithms shown in Appendix D, e.g., some with better regularization and some resembling AdaBelief (Zhuang et al., 2020) and AdaGrad (Duchi et al., 2011).”
What the sources say
- Not checked — no full text of the sourcecites Zhuang et al., 2020 source
- Not matchedcites Duchi et al., 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, Lion reduces the pre-training compute on JFT by up to 5x, improves training efficiency on diffusion models by 2.3x and achieves a better FID score, and offers similar or better performance on language modeling with up to 2x compute savings.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Illustrated by Figure 1 (Right) and 5 (Middle and Right), Lion enables both better quality and faster convergence on the FID score.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On this massive dataset, we observe no perplexity difference throughout training.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our ViT-L/16 matches the previous ViT-H/14 results trained by AdamW, while being 2x smaller.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Preliminary experiments have investigated the inclusion of more advanced features such as conditional and loop statements, and defining and calling new functions, but these do not yield improved results, so we leave them out.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion outperforms AdamW on both datasets, with a larger gain in Recall@1 than Recall@10 on Flicker30K, implying more accurate retrieval results: +1.70% vs. +0.60% for image → text and +2.14% vs. +0.20% for text → image.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AMSGrad performs the best on ViT-S/16 but the worst on ViT-B/16.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The cache hit rate and the redundant statements percentage increase along with the search progress to ∼ 90% and ∼ 70%.”
- Also stated in: § 2.2 Efficient Search Techniques · p. 4 · plain
“As seen in Figure 2 (Right), the percentage of redundant statements and cache hit rate both increase as the search proceeds.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 2.2 Efficient Search Techniques · p. 4 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves 10, 12, and 10 wins out of 12 scores for T5 Base, Large, and 11B models, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion outperforms Adafactor on both NLG and NLU tasks, particularly on the NLG tasks, with an exact match improvement of +1.0, +0.9, and +0.6 for the 1.1B, 2.1B, and 7.5B models, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated in Table 1, we achieve a significant 2.6% gain over the baseline, striking a 88.3% accuracy on zero-shot ImageNet classification.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The gaps on larger models are consistently bigger, with +0.52% vs. +0.33% (ImageNet), +0.57% vs. +0.23% (ReaL), and +0.74% vs. +0.25% (V2) for ViT-L/16 and ViT-B/16, respectively.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion performs slightly better than AdamW regarding the validation perplexity: 4.18 vs. 4.25 (small), 3.42 vs. 3.54 (medium), and 3.18 vs. 3.25 (large).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While there is no clear improvement on the base 64 × 64 model, Lion outperforms AdamW on the text-conditional super-resolution model.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirically, the improvement is more substantial on models with larger capacity, with accuracy increases of 1.96% and 0.58% for ViT-B/16 and ViT-S/16, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“if we reduce the amount of data in the proxy task, we are more likely to discover algorithms with better regularization (Program 5)”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Avg NLU | 53.2 | 53.9 | 56.8 | 57.4 | 61.3 | 61.7 | 58.5 | 59.4 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our approach is motivated by the fact that a large number of programs are invalid, functionally equivalent, or contain redundant statements that waste compute during evaluation.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The cost of the abstract execution is usually negligible compared to the actual execution of the program.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The evaluation time may vary for different programs, but typically a evaluation can be done on one TPU V2 chip within 20min.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Avg NLG | 11.1 | 12.1 | 15.6 | 16.5 | 24.1 | 24.7 | 23.1 | 23.9 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| ViT-L/16 | 304.72M | 14 / 1,035,583 | AdamW | 87.29 | 90.11 | 78.91 | 42.56 | 64.34 |”
- Also stated in: § References · p. 24 · plain
“| ViT-L/16 | 304.72M | 14 / 1,035,583 | Lion | 88.09 | 90.62 | 80.48 | 51.55 | 70.72 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § References · p. 24 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that there are still run-time errors, such as division by zero, that cannot be detected in this manner.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In our experience, reducing β 2 results in shorter memorization of historical information and enhanced training stability .”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On vision tasks, the discrepancies between Lion, AdamW, and momentum SGD are pretty small on ResNets, likely due to the fact that ConvNets are easier to optimize compared to Transformers.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Intuitively, the sign operation adds noise to the updates, which acts as a form of regularization and helps with generalization (Chen et al., 2022; Foret et al., 2021; Neelakantan et al., 2017).”
What the sources say
- Not matchedcites Chen et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Foret et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Neelakantan et al., 2017
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite its simplicity, Lion demonstrates outstanding performance across a range of models (Transformer, MLP, ResNet, U-Net, and Hybrid) and tasks (image classification, vision-language contrastive learning, diffusion, language modeling, and fine-tuning).”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it is likely that Lion performs no better than AdamW if the batch size is small ( < 64).”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indicates that Lion indeed prefers a larger batch size, but its performance remains robust even with a small 64 batch size.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without such filtering of invalid programs, the search would be overwhelmed with invalid programs, making it difficult to achieve meaningful progress.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that the bias correction is no longer needed, as it does not change the direction.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lion is simpler and has fewer hyperparameters compared to AdamW and Adafactor as it does not require ϵ and factorization-related ones.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“On the other hand, NAdam (Dozat, 2016) combines the updated first moment and the gradient to compute the update, but Lion decouples the momentum tracking and how it is applied to the update through β 2 .”
What the sources say
- Not matchedcites Dozat, 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The update is an element-wise binary ± 1 if we omit the weight decay term, with larger norm than those produced by other optimizers like SGD and adaptive algorithms.”
- Also stated in: § 1 Introduction · p. 3 · hedged
“Users should be aware that the uniform update calculated using the sign function usually yields a larger norm compared to those generated by SGD and adaptive methods.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · p. 3 · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although both m and v are utilized in Program 4, v only changes how the momentum is updated (two interp functions with constants ∼ 0.9 and ∼ 1.1 is equivalent to one with ∼ 0.99) and does not need to be separately tracked.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This choice of EMA factor and interpolation allows Lion to balance between remembering a ∼ 10x longer history of the gradient in momentum and putting more weight on the current gradient in the update.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Those results validate the effectiveness and necessity of using two linear interpolation functions, letting Lion to remember longer gradient history meanwhile assign a higher weight to the current gradient.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The inferior performance of PowerSign and AddSign compared to other optimizers is consistent with previous observations that automatically discovered optimizers have difficulty generalizing to real-world learning tasks.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“program length can be used to estimate the complexity of different programs, making it easier to select the simpler, often more generalizable ones.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“symbolic representations like programs are easier to analyze, comprehend and transfer to new tasks compared to parameterized models such as neural networks”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“However, to make the search manageable, they often limit the search space by using fixed operands and restricting the size of the tree, thereby limiting the potential for discovery. For example, they are unable to modify the tracking of momentum or how it contributes to the update, which is an essential component of Lion. Consequently, the algorithms discovered have not yet reached the state-of-”
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The previous SOTA results on zero-shot and fi ne-tuning ImageNet accuracy are 86.3% and 91.0% (Yu et al., 2022).”
What the sources say
- Not matchedcites Yu et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This work focuses on optimizers for deep neural network training, but the method is generally applicable to other tasks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The update generated by Lion is an element-wise binary ± 1 , as a result of the sign operation, therefore it has a larger norm than those generated by other optimizers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, Lion can enable ViT to converge to flatter regions, as it helps the model retain comparably lower error against Gaussian perturbations.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Algorithm uses the difference between gradient and momentum to track the second moment, resembling AdaBelief.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 8: Model performance when pre-trained on JFT-300M then fine-tuned on ImageNet. Those numbers correspond to Figure 1 (Left) and Figure 4.”
- Also stated in: § References · p. 24 · plain
“| ViT-L/16 | 304.72M | 14 / 1,035,583 | Lion | 88.09 | 90.62 | 80.48 | 51.55 | 70.72 |”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § References · p. 24 · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Algorithm that tracks the second moment without EMA decay, which is the same as AdaGrad.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tasks have a common characteristic in that the datasets are massive and of high quality, which results in a reduced difference between optimizers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our method discovers a simple and effective optimizer, Lion, that is memory-efficient and achieves strong generalization across architectures, datasets and tasks.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The authors state this as an assumption.
Simpler programs are more likely to generalize.
“Simpler programs are easier to understand and our intuition is that they are more likely to generalize, so we simplify the programs with the following steps.”
The paper relies on this without support.
The meta-validation tasks used for selection are representative enough of the target tasks to predict generalization.
“We use performance on metavalidation tasks that are larger than the proxy tasks by increasing the model size and training steps, to select the programs that generalize beyond proxy tasks then further simplify them.”
“The discovered algorithms face a significant challenge due to the substantial gap between the proxy tasks during search and the target tasks.”
The paper relies on this without support.
The program search space contains high-quality optimization algorithms that can be discovered.
“We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training.”
“Our method discovers a simple and effective optimization algorithm, Lion ( Evo L ved S i gn M o me n tum ).”
The paper relies on this without support.
AdamW is the de facto standard optimizer on a majority of learning tasks.
“as it is exceedingly popular and the de facto standard optimizer on a majority of learning tasks”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we discuss other similar algorithms such as RMSProp (Hinton et al. 2012), SignSGD and its momentum-based variants (Bernstein et al. 2018), NAdam (Dozat 2016), AMSGrad (Reddi et al. 2016), RAdam (Liu et al. 2020), and Lion (Chen et al. 2023) are also instances of an associative memory that aims to compress the gradients.”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Chen et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, there is a non-negligible effect of mini-batch noise on the picture we are describing in Section J; in particular, Lion does not necessarily outperform Adam if batches are small [13].”
From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As an application of our theory, we find that Lion does not have the kind of implicit anti-regularization induced by memory that AdamW does, providing a theory-based explanation for Lion's better generalization performance recently documented [13].”
From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“Therefore, better generalization of Lion on a number of tasks [13] may be partially attributed to the fact that memory does not anti-regularize Lion.”
From How Memory in Optimization Algorithms Implicitly Modifies the Loss (2025) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 2.1 Program Search Space · p. 4
“For the program, we select 45 common math functions”
§ H Available Functions · p. 27
“We include 43 available functions that can be used in the program during search.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.6 Ablations · p. 13
“the two ablated optimization algorithms perform worse than all five compared baselines”
§ 4.5 Comparison with Other Popular Optimizers · p. 13
“| ViT-B/16 | ImageNet | 80.12 | 80.26 | 80.32 | 80.29 | 79.85 | 78.95 | 78.50 | 79.54 | 79.90 | 80.77 |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“Train from scratch on ImageNet Following previous works (Dosovitskiy et al., 2021; He et al., 2016), we train ResNet-50 for 90 epochs with a batch size of 1,024, and other models for 300 epochs with a batch size of 4,096.”
MethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“…weight decay [60], also referred to as AdamW, and Adafactor with factorized second moments [86], are still the de facto standard optimizers for training most deep neural networks, especially the recent state-of-the-art language [12, 23, 90], vision [21, 26, 102] and multimodal [75, 84, 100] models.”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“However, Adam (Kingma and Ba, 2014) with decoupled weight decay (Loshchilov and Hutter, 2019), also referred to as AdamW, and Adafactor with factorized second moments (Shazeer and Stern, 2018), are still the de facto standard optimizers for training most deep neural networks, especially the recent state-of-the-art lan…”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 3×
“Train from scratch on ImageNet Following previous works [26, 37], we train ResNet-50 for 90 epochs with a batch size of 1,024, and other models for 300 epochs with a batch size of 4,096.”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 3×
“The zero-shot image-text retrieval results on MSCOCO (Lin et al., 2014) and Flickr30K (Plummer et al.”
MethodHarris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826 · cited 1×
“They are similar to Python code using NumPy / JAX [10, 36] as well as pseudo code of optimization algorithms.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Following GPT-3 [12], we train three models, from 1.1B to 7.5B parameters, on 300B tokens with a batch size of 3M tokens and a context length of 1K.”
BackgroundMethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 2×
“This section focuses on the vision-language contrastive training [75].”
BackgroundAndrew, A. M. (1993) · Robotica (Cambridge. Print) · doi:10.1017/s0263574700017136 · Cited by 11,252
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · cited 1×
“Recently, diffusion models achieve a huge success on image generation [24, 40, 41, 84, 88], so we test the performance of Lion on unconditional image synthesis and multimodal text-to-image generation.”
BackgroundNeural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,705 · cited 1×
“, 2022), neural architecture search (Chen and Hsieh, 2020; Chen et al., 2021; Liu et al., 2019a; Pham et al., 2018; Real et al., 2019; So et al., 2019; Wang et al., 2021; Yang et al., 2022; Zoph and Le, 2017) and hyperparameter optimization (Dong et al.”
BackgroundSelf-citationGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
A direct adaptive method for faster backpropagation learning: the RPROP algorithm
Riedmiller, M., & Braun, H. (2002) · IEEE International Conference on Neural Networks · doi:10.1109/icnn.1993.298623 · Cited by 3,917 · cited 2×
“Other related works include numerous handcrafted optimizers [2, 7, 27, 29, 35, 48, 58, 61, 82, 83, 86, 105].”
BackgroundDiffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P. (2014) · arXiv · doi:10.48550/arxiv.1405.0312 · Cited by 2,579
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528
Denoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507 · cited 1×
“Recently, diffusion models achieve a huge success on image generation (Dhariwal and Nichol, 2021; Ho and Salimans, 2022; Ho et al., 2020; Saharia et al., 2022; Song et al., 2021).”
BackgroundRevisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484 · cited 5×
“larger well-established datasets, ImageNet-21K and JFT (Sun et al., 2017).”
InfluentialSequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
A Comparative Analysis of Selection Schemes Used in Genetic Algorithms
Goldberg, D. E., & Deb, K. (1991) · Foundations of genetic algorithms · doi:10.1016/b978-0-08-050684-5.50008-2 · Cited by 2,456 · cited 1×
“Each cycle picks T P algorithms at random and the best performer is chosen as the parent , i.e., tournament selection [32].”
BackgroundOn Large-Batch Training for Deep Learning: Generalization Gap and Sharp MinimaPreprint
Keskar, N. S., Mudigere, D., Nocedal, J. M., Smelyanskiy, M., & Tang, P. T. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.04836 · Cited by 2,270
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 1×
“• NLG: TriviaQA [45], Natural Questions [51], Web Questions [6].”
BackgroundSelf-citationOn the Variance of the Adaptive Learning Rate and BeyondPreprint
Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., & Han, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.03265 · Cited by 1,837 · cited 4×
“We also employ four popular handcrafted optimizers: RAdam [58], NAdam [27], AdaBelief [105], AMSGrad [82] and two optimizers discovered by AutoML: PowerSign [5] and AddSign [5] to train ViT-S/16 and ViT-B/16 on ImageNet (with RandAug and Mixup).”
BackgroundMethodInfluentialGooch, P. (1969) · doi:10.1136/bmj.4.5674.53-c · Cited by 1,834 · cited 1×
“Our evaluation covers various benchmarks: ImageNet, ImageNet ReaL [8], ImageNet V2 [81], ImageNet A [39], ImageNet R [38], ImageNet Sketch [92], ObjectNet [4], CIFAR-100 [50], and Oxford-IIIT Pet [69].”
MethodOn the Convergence of Adam and Beyond
Reddi, S. J., Kale, S., & Kumar, S. (2019) · arXiv · doi:10.48550/arxiv.1904.09237 · Cited by 1,771 · cited 2×
“We also employ four popular handcrafted optimizers: RAdam [58], NAdam [27], AdaBelief [105], AMSGrad [82] and two optimizers discovered by AutoML: PowerSign [5] and AddSign [5] to train ViT-S/16 and ViT-B/16 on ImageNet (with RandAug and Mixup).”
BackgroundMethodParkhi, O., Vedaldi, A., Zisserman, A., & Jawahar, C. V. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248092 · Cited by 1,702
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 1×
“• NLG: TriviaQA [45], Natural Questions [51], Web Questions [6].”
BackgroundGenetic programming as a means for programming computers by natural selection
Koza, J. (1994) · Statistics and Computing · doi:10.1007/bf00175355 · Cited by 1,685 · cited 2×
“There is also a long history of using evolutionary methods to search for programs, i.e., genetic programming [11, 42, 49].”
BackgroundMethodThe PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P. E., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1260 · Cited by 1,602 · cited 1×
“• NLU: HellaSwag [101], StoryCloze [66], Winograd [53], Winogrande [85], RACE [52], PIQA [9], ARC [19], OpenbookQA [65], BoolQ [18], Copa [33], RTE [20], WiC [73], Multirc [47], WSC [53], ReCoRD [104], CB [22], Adversarial NLI [68].”
BackgroundSemantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1160 · Cited by 1,593
Learning to learn by gradient descent by gradient descent
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., & de Freitas, N. (2016) · arXiv · doi:10.48550/arxiv.1606.04474 · Cited by 1,486 · cited 2×
“Our work lies in the area of AutoML and meta-learning that includes learning to learn [1, 5, 63, 64, 78, 96, 97], neural architecture search [14, 15, 57, 71, 79, 87, 93, 94, 98, 106] and hyperparameter optimization [25, 43, 44, 55], etc.”
MethodFlickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444
CoCa: Contrastive Captioners are Image-Text Foundation ModelsPreprint
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., & Wu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.01917 · Cited by 1,413 · cited 1×
“0% higher than the previous best result (Yu et al., 2022).”
BackgroundAdafactor: Adaptive Learning Rates with Sublinear Memory CostPreprint
Shazeer, N., & Stern, M. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1804.04235 · Cited by 1,410 · cited 2×
“Other related works include numerous handcrafted optimizers [2, 7, 27, 29, 35, 48, 58, 61, 82, 83, 86, 105].”
BackgroundHyperband: A Novel Bandit-Based Approach to Hyperparameter OptimizationPreprint
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06560 · Cited by 1,368 · cited 1×
“Our work lies in the area of AutoML and meta-learning that includes learning to learn [1, 5, 63, 64, 78, 96, 97], neural architecture search [14, 15, 57, 71, 79, 87, 93, 94, 98, 106] and hyperparameter optimization [25, 43, 44, 55], etc.”
MethodThe Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00823 · Cited by 1,250
Classifier-Free Diffusion Guidance
Ho, J., & Salimans, T. (2022) · arXiv · doi:10.48550/arxiv.2207.12598 · Cited by 1,126
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01501 · Cited by 1,090
Efficient Neural Architecture Search via Parameter SharingPreprint
Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., & Dean, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03268 · Cited by 1,082 · cited 1×
“Our work lies in the area of AutoML and meta-learning that includes learning to learn [1, 5, 63, 64, 78, 96, 97], neural architecture search [14, 15, 57, 71, 79, 87, 93, 94, 98, 106] and hyperparameter optimization [25, 43, 44, 55], etc.”
MethodSelf-citationOne Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2013) · arXiv · doi:10.48550/arxiv.1312.3005 · Cited by 1,051
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044
CoAtNet: Marrying Convolution and Attention for All Data Sizes
Dai, Z., Liu, H., Le, Q. V., & Tan, M. (2021) · arXiv · doi:10.48550/arxiv.2106.04803 · Cited by 925
Self-citationZhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830
Do ImageNet Classifiers Generalize to ImageNet?
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019) · arXiv · doi:10.48550/arxiv.1902.10811 · Cited by 790
Adversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656 · cited 1×
“• NLU: HellaSwag [101], StoryCloze [66], Winograd [53], Winogrande [85], RACE [52], PIQA [9], ARC [19], OpenbookQA [65], BoolQ [18], Copa [33], RTE [20], WiC [73], Multirc [47], WSC [53], ReCoRD [104], CB [22], Adversarial NLI [68].”
BackgroundBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 655
One billion word benchmark for measuring progress in statistical language modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-564 · Cited by 627 · cited 1×
“For language tasks, we train a Transformer with two layers, 128 hidden units and two heads on LM1B [13] for 20K steps with batch size 64, sequence length 32 and vocabulary size 3K.”
MethodA Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., & Allen, J. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1098 · Cited by 598 · cited 1×
“• NLU: HellaSwag [101], StoryCloze [66], Winograd [53], Winogrande [85], RACE [52], PIQA [9], ARC [19], OpenbookQA [65], BoolQ [18], Copa [33], RTE [20], WiC [73], Multirc [47], WSC [53], ReCoRD [104], CB [22], Adversarial NLI [68].”
BackgroundWinoGrande: An Adversarial Winograd Schema Challenge at Scale
Sakaguchi, K., Le Bras, R., Bhagavatula, C., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6399 · Cited by 579
Adding Gradient Noise Improves Learning for Very Deep NetworksPreprint
Neelakantan, A., Vilnis, L., Le, Q. V., Sutskever, I., Kaiser, Ł., Kurach, K., & Martens, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06807 · Cited by 570
Self-citationsignSGD: Compressed Optimisation for Non-Convex Problems
Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., & Anandkumar, A. (2018) · arXiv · doi:10.48550/arxiv.1802.04434 · Cited by 541 · cited 4×
“Other related works include numerous handcrafted optimizers [2, 7, 27, 29, 35, 48, 58, 61, 82, 83, 86, 105].”
BackgroundInfluentialSharpness-Aware Minimization for Efficiently Improving Generalization
Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. (2020) · arXiv · doi:10.48550/arxiv.2010.01412 · Cited by 521
Optimizing Neural Networks with Kronecker-factored Approximate CurvaturePreprint
Martens, J., & Grosse, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.05671 · Cited by 476 · cited 1×
“It also lacks the functions required to construct advanced second-order algorithms [2, 35, 62].”
BackgroundThe CommitmentBank: Investigating projection in naturally occurring discourse
de Marneffe, M.-C., Simons, M., & Tonhauser, J. (2019) · Digital Access to Libraries · doi:10.18148/sub/2019.v23i2.601 · Cited by 471
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0965-7 · Cited by 429 · cited 1×
“The zero-shot image-text retrieval results on MSCOCO [56] and Flickr30K [74] can be found in Figure 6 (in the Appendix).”
MethodBrameier, M., & Banzhaf, W. (2007) · Springer eBooks · doi:10.1007/978-0-387-31030-5 · Cited by 422
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., & Roth, D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1023 · Cited by 407
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Pilehvar, M. T., & Camacho-Collados, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1128 · Cited by 406
GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsPreprint
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Fırat, O., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.06905 · Cited by 404
Self-citationLiT: Zero-Shot Transfer with Locked-image text Tuning
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01759 · Cited by 402 · cited 9×
“Our method, Lion, demonstrates consistent improvement over AdamW with gains of +1.10%, +1.13%, and +0.66% on zero-shot ImageNet accuracy for LiT-B/32-”
MethodInfluentialAdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
Zhuang, J., Tang, T., Ding, Y., Tatikonda, S., Dvornek, N., Papademetris, X., & Duncan, J. S. (2020) · arXiv · doi:10.48550/arxiv.2010.07468 · Cited by 382 · cited 6×
“, 2020), NAdam (Dozat, 2016), AdaBelief (Zhuang et al., 2020) and AMSGrad (Reddi et al.”
BackgroundMethodInfluentialAdaptive Subgradient Methods for Online Learning and Stochastic Optimization
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2021068 · Cited by 346 · cited 2×
“Other related works include numerous handcrafted optimizers [2, 7, 27, 29, 35, 48, 58, 61, 82, 83, 86, 105].”
BackgroundMethodReCoRD: Bridging the Gap between Human and Machine Commonsense Reading ComprehensionPreprint
Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., & Durme, B. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12885 · Cited by 339
Learning Robust Global Representations by Penalizing Local Predictive Power
Wang, H., Ge, S., Xing, E. P., & Lipton, Z. C. (2019) · arXiv · doi:10.48550/arxiv.1905.13549 · Cited by 337 · cited 1×
“Our evaluation covers various benchmarks: ImageNet, ImageNet ReaL [8], ImageNet V2 [81], ImageNet A [39], ImageNet R [38], ImageNet Sketch [92], ObjectNet [4], CIFAR-100 [50], and Oxford-IIIT Pet [69].”
MethodRegularized Evolution for Image Classifier Architecture Search
Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014780 · Cited by 328
Self-citationNeural Optimizer Search with Reinforcement Learning
Bello, I., Zoph, B., Vasudevan, V., & Le, Q. V. (2017) · arXiv · doi:10.48550/arxiv.1709.07417 · Cited by 319 · cited 4×
“As opposed to previous optimizer search attempts [5, 95], our method allows discovering better ways of updating the extra variables.”
MethodInfluentialSelf-citationNon-stochastic Best Arm Identification and Hyperparameter OptimizationPreprint
Jamieson, K., & Talwalkar, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.07943 · Cited by 307 · cited 1×
“, 2022; Zoph and Le, 2017) and hyperparameter optimization (Dong et al., 2021; Hutter et al., 2011; Jamieson and Talwalkar, 2016; Li et al., 2017), etc.”
BackgroundNAS-bench-101: Towards reproducible neural architecture search
Ying, C., Klein, A., Real, E., Christiansen, E., Murphy, K., & Hutter, F. (2019) · arXiv · doi:10.48550/arxiv.1902.09635 · Cited by 297 · cited 1×
“Evolution with warm-start and restart We apply regularized evolution as it is simple, scalable, and has shown success on many AutoML search tasks (Holland, 1992; Real et al., 2019, 2020; So et al., 2019; Ying et al., 2019).”
MethodSelf-citationSo, D. R., Liang, C., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1901.11117 · Cited by 277 · cited 2×
“Our work lies in the area of AutoML and meta-learning that includes learning to learn [1, 5, 63, 64, 78, 96, 97], neural architecture search [14, 15, 57, 71, 79, 87, 93, 94, 98, 106] and hyperparameter optimization [25, 43, 44, 55], etc.”
MethodSelf-citationCompressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205 · cited 1×
“The masked language modeling and fine-tuning results are shown in Appendix J. Autoregressive language modeling We first experiment on two smaller-scale academic datasets Wiki-40B [34] and PG-19 [76].”
MethodScalable Second Order Optimization for Deep LearningPreprint
Anil, R., Gupta, V., Koren, T., Regan, K., & Singer, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09018 · Cited by 197
Beyer, L., Hénaff, O. J., Kolesnikov, A., Zhai, X., & Oord, A. V. D. (2020) · arXiv · doi:10.48550/arxiv.2006.07159 · Cited by 176
Learned Optimizers that Scale and GeneralizePreprint
Wichrowska, O., Maheswaranathan, N., Hoffman, M. W., Colmenarejo, S. G., Denil, M., de Freitas, N., & Sohl‐Dickstein, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.04813 · Cited by 174
Quasi-hyperbolic momentum and Adam for deep learningPreprint
Ma, J., & Yarats, D. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1810.06801 · Cited by 170 · cited 1×
“Other related works include numerous handcrafted optimizers [2, 7, 27, 29, 35, 48, 58, 61, 82, 83, 86, 105].”
BackgroundLearning to OptimizePreprint
Li, K., & Malik, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01885 · Cited by 159 · cited 1×
“, neural networks, to output the updates (Andrychowicz et al., 2016; Li and Malik, 2017; Metz et al., 2019, 2022).”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
Real, E., Liang, C., So, D. R., & Le, Q. V. (2020) · arXiv · doi:10.48550/arxiv.2003.03384 · Cited by 143 · cited 4×
“Our approach builds upon a symbolic search space similar to AutoML-Zero [70, 80].”
BackgroundMethodInfluentialSelf-citationTransformer Quality in Linear Time
Hua, W., Dai, Z., Liu, H., & Le, Q. V. (2022) · arXiv · doi:10.48550/arxiv.2202.10447 · Cited by 137
Self-citationCiesielski, V. (2007) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-007-9036-8 · Cited by 129
When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
Chen, X., Hsieh, C.-J., & Gong, B. (2021) · arXiv · doi:10.48550/arxiv.2106.01548 · Cited by 128 · cited 1×
“Intuitively, the sign operation adds noise to the updates, which acts as a form of regularization and helps with generalization (Chen et al., 2022; Foret et al., 2021; Neelakantan et al., 2017).”
BackgroundSelf-citationStabilizing Differentiable Architecture Search via Perturbation-based Regularization
Chen, X., & Hsieh, C.-J. (2020) · arXiv · doi:10.48550/arxiv.2002.05283 · Cited by 126 · cited 1×
“, 2022), neural architecture search (Chen and Hsieh, 2020; Chen et al., 2021; Liu et al., 2019a; Pham et al., 2018; Real et al., 2019; So et al., 2019; Wang et al., 2021; Yang et al., 2022; Zoph and Le, 2017) and hyperparameter optimization (Dong et al.”
BackgroundSelf-citationRethinking Architecture Selection in Differentiable NASPreprint
Wang, R., Cheng, M., Chen, X., Tang, X., & Hsieh, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04392 · Cited by 106
Self-citationDissecting Adam: The Sign, Magnitude and Variance of Stochastic GradientsPreprint
Balles, L., & Hennig, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07774 · Cited by 103 · cited 1×
“There are a large number of handcrafted optimizers, mostly adaptive ones, introduced in recent years [2, 3, 7, 27, 58, 105].”
BackgroundVeLO: Training Versatile Learned Optimizers by Scaling UpPreprint
Metz, L., Harrison, J., Freeman, C. D., Merchant, A., Beyer, L., Bradbury, J., Agrawal, N., Poole, B., Mordatch, I., Roberts, A., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.09760 · Cited by 95
DrNAS: Dirichlet Neural Architecture SearchPreprint
Chen, X., Wang, R., Cheng, M., Tang, X., & Hsieh, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10355 · Cited by 79 · cited 1×
“, 2022), neural architecture search (Chen and Hsieh, 2020; Chen et al., 2021; Liu et al., 2019a; Pham et al., 2018; Real et al., 2019; So et al., 2019; Wang et al., 2021; Yang et al., 2022; Zoph and Le, 2017) and hyperparameter optimization (Dong et al.”
BackgroundSelf-citationUnderstanding and correcting pathologies in the training of learned optimizersPreprint
Metz, L., Maheswaranathan, N., Nixon, J., Freeman, C. D., & Sohl‐Dickstein, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.10180 · Cited by 66 · cited 2×
“Our work lies in the area of AutoML and meta-learning that includes learning to learn [1, 5, 63, 64, 78, 96, 97], neural architecture search [14, 15, 57, 71, 79, 87, 93, 94, 98, 106] and hyperparameter optimization [25, 43, 44, 55], etc.”
MethodShampoo: Preconditioned Stochastic Tensor Optimization
Gupta, V., Koren, T., & Singer, Y. (2018) · arXiv · doi:10.48550/arxiv.1802.09568 · Cited by 62 · cited 2×
“Other related works include numerous handcrafted optimizers (Anil et al., 2020; Bernstein et al., 2018; Dozat, 2016; Duchi et al., 2011; Gupta et al., 2018; Kingma and Ba, 2014; Liu et al., 2020; Reddi et al., 2018; Shazeer and Stern, 2018; Zhuang et al., 2020), which we discuss in Section 3.”
BackgroundAutoHAS: Efficient Hyperparameter and Architecture SearchPreprint
Dong, X., Tan, M., Yu, A. W., Peng, D., Gabryś, B., & Le, Q. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03656 · Cited by 49
Self-citationPIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Le bras, R., Gao, J., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6239 · Cited by 37 · cited 1×
“• NLU: HellaSwag [101], StoryCloze [66], Winograd [53], Winogrande [85], RACE [52], PIQA [9], ARC [19], OpenbookQA [65], BoolQ [18], Copa [33], RTE [20], WiC [73], Multirc [47], WSC [53], ReCoRD [104], CB [22], Adversarial NLI [68].”
BackgroundPyGlove: Symbolic Programming for Automated Machine LearningPreprint
Peng, D., Dong, X., Real, E., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.08809 · Cited by 13 · cited 1×
“Our approach builds upon a symbolic search space similar to AutoML-Zero [70, 80].”
MethodSelf-citationTabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets
Yang, C., Bender, G., Liu, H., et al. (2022) · Neural Information Processing Systems · doi:10.52202/068431-0865 · Cited by 9 · cited 1×
“, 2022), neural architecture search (Chen and Hsieh, 2020; Chen et al., 2021; Liu et al., 2019a; Pham et al., 2018; Real et al., 2019; So et al., 2019; Wang et al., 2021; Yang et al., 2022; Zoph and Le, 2017) and hyperparameter optimization (Dong et al.”
BackgroundSelf-citationRANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving
Wang, R., Chen, X., Cheng, M., Tang, X., & Hsieh, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01021 · Cited by 7
Self-citationUnified Functional Hashing in Automatic Machine LearningPreprint
Gillard, R., Jonany, S., Miao, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.05433 · Cited by 1
Self-citationEfficient Non-Parametric Optimizer Search for Diverse Tasks
Wang, R., Xiong, Y., Cheng, M., & Hsieh, C.-J. (2022) · arXiv · doi:10.48550/arxiv.2209.13575 · cited 2×
“Another line of methods (Bello et al., 2017; Wang et al., 2022) apply reinforcement learning or Monte Carlo Sampling to discover new optimizers, where the search space is defined by trees composed from predefined operands (e.”
MethodSelf-citation
Research works printed but not linked: 10
- Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from questionanswer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
- Timothy Dozat. Incorporating Nesterov Momentum into Adam. In Proceedings of the 4th International Conference on Learning Representations , pages 1-4, 2016.
- Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pages 394-398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics.
- Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. Wiki-40B: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference , pages 2440-2452, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4.
- John H Holland. Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence . MIT press, 1992.
- Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning , KR'12, page 552-561. AAAI Press, 2012. ISBN 9781577355601.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.
- Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations , 2017.
- Yuanhao Xiong, Li-Cheng Lan, Xiangning Chen, Ruochen Wang, and Cho-Jui Hsieh. Learning to schedule learning rate with graph neural networks. In International Conference on Learning Representations , 2022.
Cited by
Sorted by citations
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078
“1%), obtained by Chen et al. (2023a).”
Sigmoid Loss for Language Image Pre-Training
Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01100 · Cited by 1,127
Automated model building and protein identification in cryo-EM maps
Jamali, K., Käll, L., Zhang, R., Brown, A., Kimanius, D., & Scheres, S. H. W. (2023) · doi:10.1101/2023.05.16.541002 · Cited by 686
Mathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., et al. (2024) · Nature · doi:10.1038/s41586-023-06924-6 · Cited by 544
Refusion: Enabling Large-Size Realistic Image Restoration with Latent-Space Diffusion Models
Luo, Z., Gustafsson, F., Zhao, Z., Sjölund, J., & Schön, T. B. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00169 · Cited by 175
“We use the Lion optimizer [9] with β 1 = 0 .”
Method
Metadata
- Authors
- Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., & Le, Q. V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 49205-49233
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2302.06675
- OpenAlex
- W7133224204
- Semantic Scholar
- a7f59b2162ae0ea2520753b1b9b17277490a9458
- Cited by
- 303
- References
- 111
- Citations by year
- 2026: 55 · 2025: 81 · 2024: 116 · 2023: 51
- FWCI
- 53.56
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Polynomial and algebraic computation
- Physical Sciences › Computer Science › Artificial Intelligence › Metaheuristic Optimization Algorithms Research
- Keywords
- symbolic computation, optimization algorithms, algorithm discovery
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
