Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Open accessAbstract
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, Donald Metzler. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Smaller models might have just the right capacity to capture the essential patterns without being bogged down by excessive complexity.
“A possible reason can be that these smaller models might have just the right capacity to capture the essential patterns without being bogged down by excessive complexity [71, 72].”
From Exploring the Capabilities of LLMs for Code-Change-Related Tasks (2025) · cites this paper as [71]
Tay et al. [2023] found that while inductive bias shifts the offset of scaling curves, the scaling exponents are relatively stable across many transformer variants, and that architectures cannot be ranked reliably from a single scale.
“Tay et al. [2023] systematically compared architectures and found that while inductive bias shifts the offset of scaling curves-some architectures are simply better at a given scale-the scaling exponents are relatively stable across many transformer variants, and that architectures cannot be ranked reliably from a single scale.”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Tay et al., 2023
Scaling law coefficients are not universal: they depend on the data distribution, tokenizer, context length, optimizer and learning-rate schedule, and the evaluation metric.
“First, the coefficients E, A, B, α, β are not universal: they depend on the data distribution, tokenizer, context length, optimizer and learning-rate schedule, and the evaluation metric”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Tay et al., 2023
The citation lists haven’t been fetched yet
References 40 · Cited by 34
- Authors
- Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., & Metzler, D.
- Corresponding author
- Dehghani, Mostafa
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Published
- 2023
- Pages
- 12342-12364
- OpenAlex
- W4389518729
- Cited by
- 34
- References
- 40
- Citations by year
- 2026: 6 · 2025: 6 · 2024: 6 · 2023: 15 · 2022: 1
- FWCI
- 5.85
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- scaling laws, inductive bias, model architecture, language model scaling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
