Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice

May combine two claims.

Besiroglu et al. [2024] replication reported that the original confidence intervals appeared implausibly tight and that some fitted parameters differed once the optimization was redone, while broadly supporting the balanced-scaling conclusion.

“Besiroglu et al. [2024] attempted to replicate the third (parametric) estimation method of Hoffmann et al. [2022] from the published data and reported that the original confidence intervals appeared implausibly tight and that some fitted parameters differed once the optimization was redone, while broadly supporting the balanced-scaling conclusion.”

• cites Besiroglu et al., 2024 sourceSupported

May combine two claims.

Hernandez et al. [2021] quantified effective data transferred by pretraining as a function of model size and fine-tuning set size, showing that the benefit of pretraining itself scales predictably.

“Hernandez et al. [2021] quantified this through scaling laws for transfer, measuring the”

• cites Hernandez et al., 2021 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Sorscher et al. [2022] argued that the power-law dependence on data is not inevitable: with a good example-difficulty ranking, aggressive data pruning can in principle yield faster-than-power-law decay of error.

“Sorscher et al. [2022] argued that the power-law dependence on data is not inevitable: with a good exampledifficulty ranking, aggressive data pruning can in principle yield faster-than-power-law decay of error”

• cites Sorscher et al., 2022 sourceSupported

May combine two claims. The quote only partly states this claim.

Chinchilla, a 70B-parameter model trained on 1.4 trillion tokens, outperformed Gopher, GPT-3, and others across a broad benchmark suite under a matched compute budget.

“Chinchilla outperformed the much larger Gopher [Rae et al., 2021], GPT-3 [Brown et al., 2020], and others [Smith et al., 2022] across a broad benchmark suite”

• cites Hoffmann et al., 2022 sourcePartly supported

  Section 4 (p. 9) identifies Chinchilla as a 70B-parameter model trained on 1.4 trillion tokens and says it used the same training FLOPs as Gopher. Sections 4.2.4–4.2.5 (pp. 11–12) report strong results against Gopher and GPT-3 across multiple benchmarks. The matched-compute comparison is established for Gopher, not for GPT-3 or the other models. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Rae et al., 2021 sourceSupported
• cites Brown et al., 2020 sourceSupported

May combine two claims.

Gadre et al. [2024] demonstrated that loss remains predictable when models are trained far beyond their compute-optimal token budget, and that downstream task error can be tied to pretraining loss via a further fitted relationship.

“Gadre et al. [2024] studied the over-trained regime systematically, demonstrating that loss remains predictable when models are trained far beyond their compute-optimal token budget, and-importantly-that downstream task error can be tied to pretraining loss via a further fitted relationship.”

• cites Gadre et al., 2024 sourceSupported

May combine two claims.

Generalization error decreases as a power law in training-set size across several orders of magnitude, with task-dependent exponents and an irreducible error floor.

“Hestness et al. [2017] conducted a large empirical study across machine translation, language modeling, image, and speech tasks, showing that generalization error decreases as a power law in training-set size across several orders of magnitude, with task-dependent exponents and an irreducible error floor.”

• cites Hestness et al., 2017 sourcePartly supported

  The Introduction (p. 2) reports power-law improvement across machine translation, language modeling, image classification, and speech recognition, with different exponents by application. But §5.1 (p. 11) says the study had not reached the irreducible-error region in real applications; it describes that floor as likely and reports testing it only in toy problems. The claim overstates the floor as an empirical finding across the studied tasks. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Clark et al. [2022] derived unified scaling laws for routed models, characterizing loss as a joint function of the dense base size and the number of experts, and identifying how the benefit of routing diminishes as the base model grows.

“Clark et al. [2022] derived unified scaling laws for routed models, characterizing loss as a joint function of the dense base size and the number of experts, and identifying how the benefit of routing diminishes as the base model grows.”

• cites Clark et al., 2022 sourceSupported

May combine two claims.

Henighan et al. [2020] reported strikingly similar power-law behavior for images, video, and multimodal data, suggesting that the phenomenon is not specific to text.

“Extending the analysis to other generative modalities, Henighan et al. [2020] reported strikingly similar power-law behavior for images, video, and multimodal data, suggesting that the phenomenon is not specific to text.”

• cites Henighan et al., 2020 sourceSupported

May combine two claims.

Kaplan et al. [2020] analysis implied N* growing roughly as C^0.73 and the optimal number of training tokens growing only weakly.

“their analysis implied N ⋆ growing roughly as C 0.73 and the optimal number of training tokens growing only weakly.”

• cites Kaplan et al., 2020Supported

• AssumptionThe authors state this as an assumption: The Kaplan et al. [2020] laws assumed near-optimal optimization choices.

May combine two claims.

Scaling law coefficients are not universal: they depend on the data distribution, tokenizer, context length, optimizer and learning-rate schedule, and the evaluation metric.

“First, the coefficients E, A, B, α, β are not universal: they depend on the data distribution, tokenizer, context length, optimizer and learning-rate schedule, and the evaluation metric”

• cites Kaplan et al., 2020Partly supported

  The Summary of Scaling Laws (p. 5) ties specific fitted values to WebText2 and says some numerical values depend on vocabulary size and tokenization. The paper also studies context-dependent performance (D.5, p. 25), but does not establish that all the cited coefficients depend on context length, optimizer, or evaluation metric. It reports instead that learning-rate schedule is mostly irrelevant under the tested conditions (D.6, p. 25). (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Hoffmann et al., 2022 sourcePartly supported

  Section 3.3 (p. 6) ties E to the data distribution, and Related Work (p. 3) says the learning-rate schedule affects measured loss and scaling conclusions. But Appendix C (p. 22) finds similar scaling across the datasets tested; the cited passages do not establish that E, A, B, α, and β depend on the tokenizer, context length, optimizer, or evaluation metric. The full list of dependencies is therefore not supported. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Tay et al., 2023 sourceNot checked — no full text of the source

May combine two claims.

Bahri et al. [2024] developed a taxonomy distinguishing variance-limited regimes from resolution-limited regimes, deriving exponents in each and connecting them to properties of the learned kernel.

“Bahri et al. [2024] developed this into a taxonomy distinguishing variance-limited regimes (where one resource is abundant and error falls quickly) from resolution-limited regimes (where error is set by how finely the model resolves the data manifold), deriving exponents in each and connecting them to properties of the learned kernel.”

• cites Bahri et al., 2024 sourceSupported

May combine two claims.

Smooth scaling of pretraining loss does not imply smooth scaling of every downstream capability; some benchmark behaviors appear to change sharply with scale.

“Second, smooth scaling of pretraining loss does not imply smooth scaling of every downstream capability; some benchmark behaviors appear to change sharply with scale [Wei et al., 2022]”

• cites Wei et al., 2022 sourcePartly supported

  Section 5.1 (p. 7) and Appendix A.1 (p. 19) report that cross-entropy loss improved at model scales where downstream benchmark metrics remained near random, before those metrics rose above random. This supports the contrast between gradual improvement and apparently sharp benchmark changes. The loss measured here was on BIG-Bench tasks, however, not pretraining loss, so the source does not directly establish that specific comparison. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

May combine two claims.

Theoretical accounts of power-law scaling are partial: they recover power-law forms and rationalize the dependence on data geometry, but they do not yet predict the precise coefficients of large transformer runs from first principles.

“These theories are partial-they recover power-law forms and rationalize the dependence on data geometry, but they do not yet predict the precise coefficients of large transformer runs from first principles”

• cites Sharma and Kaplan, 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Bahri et al., 2024 sourcePartly supported

  Section 1.2 derives power-law exponents and relates them to data-manifold dimension. Section 4.1 says the theoretical results are asymptotic and identifies limits in applying them to finite models and datasets. The paper does not specifically assess whether theories can predict precise coefficients for large transformer runs from first principles. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

May combine two claims.

Krajewski et al. [2024] extended this to fine-grained MoE, introducing granularity as an explicit variable in the scaling law and deriving compute-optimal settings that jointly choose model size, number of tokens, and expert granularity.

“Krajewski et al. [2024] extended this to fine-grained MoE, introducing granularity as an explicit variable in the scaling law and deriving compute-optimal settings that jointly choose model size, number of tokens, and expert granularity.”

• cites Krajewski et al., 2024 sourceSupported

Classified as probably not a substantial claim.

Under a fixed budget, one should train very large models and stop well before convergence rather than train smaller models to completion.

“A practical corollary was that, under a fixed budget, one should train very large models and stop well before convergence rather than train smaller models to completion.”

• cites Kaplan et al., 2020Supported

• AssumptionThe authors state this as an assumption: The Kaplan et al. [2020] laws assumed near-optimal optimization choices.

May combine two claims.

Fedus et al. [2022] demonstrated that Switch Transformers can scale to trillions of parameters with simplified routing while holding per-token FLOPs roughly fixed.

“Fedus et al. [2022] demonstrated that Switch Transformers can scale to trillions of parameters with simplified routing while holding per-token FLOPs roughly fixed”

• cites Fedus et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Sharma and Kaplan [2022] argued that the scaling exponent is set by the dimensionality of the manifold from which data are drawn, predicting that higher-dimensional data scale more slowly.

“Sharma and Kaplan [2022] argued that the scaling exponent is set by the dimensionality of the manifold from which data are drawn, predicting that higher-dimensional data scale more slowly.”

• cites Sharma and Kaplan, 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable).

Compute-optimal training of transformers has matured from a qualitative intuition into a quantitative discipline.

“Compute-optimal training of transformers has matured from a qualitative intuition into a quantitative discipline.”

May combine two claims.

Scaling behavior is predictable enough to plan around, yet contingent enough that its coefficients must be re-measured in each new regime.

“The unifying thread is that scaling behavior is predictable enough to plan around, yet contingent enough that its coefficients must be re-measured in each new regime.”

Probably not a substantial claim: may not pass S2 (checkable).

Scaling laws are an empirical instrument for budget allocation and risk reduction rather than a law of nature.

“we emphasize that scaling laws are an empirical instrument for budget allocation and risk reduction rather than a law of nature”

May combine two claims.

Kendiukhov [2026] establishes scaling laws for masked-reconstruction transformers on single-cell transcriptomics, showing that the same power-law machinery and the same compute-allocation questions apply to a biological data modality.

“Kendiukhov [2026] establishes scaling laws for masked-reconstruction transformers on single-cell transcriptomics, showing that the same power-law machinery-and the same compute-allocation questions-apply to a biological data modality with its own irreducible-loss structure.”

• cites Kendiukhov, 2026Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Tay et al. [2023] found that while inductive bias shifts the offset of scaling curves, the scaling exponents are relatively stable across many transformer variants, and that architectures cannot be ranked reliably from a single scale.

“Tay et al. [2023] systematically compared architectures and found that while inductive bias shifts the offset of scaling curves-some architectures are simply better at a given scale-the scaling exponents are relatively stable across many transformer variants, and that architectures cannot be ranked reliably from a single scale.”

• cites Tay et al., 2023 sourceNot checked — no full text of the source