Chinchilla Scaling: A replication attempt
Open accessAbstract
Hoffmann et al. (2022) propose three methods for estimating a compute-optimal scaling law. We attempt to replicate their third estimation procedure, which involves fitting a parametric loss function to a reconstruction of data from their plots. We find that the reported estimates are inconsistent with their first two estimation methods, fail at fitting the extracted data, and report implausibly narrow confidence intervals--intervals this narrow would require over 600,000 experiments, while they likely only ran fewer than 500. In contrast, our rederivation of the scaling law using the third approach yields results that are compatible with the findings from the first two estimation procedures described by Hoffmann et al.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Besiroglu et al. [2024] replication reported that the original confidence intervals appeared implausibly tight and that some fitted parameters differed once the optimization was redone, while broadly supporting the balanced-scaling conclusion.
“Besiroglu et al. [2024] attempted to replicate the third (parametric) estimation method of Hoffmann et al. [2022] from the published data and reported that the original confidence intervals appeared implausibly tight and that some fitted parameters differed once the optimization was redone, while broadly supporting the balanced-scaling conclusion.”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Besiroglu et al., 2024
The replication scrutiny of Besiroglu et al. [2024] hinged on fitting procedure and reported uncertainty.
“the replication scrutiny of Besiroglu et al. [2024] hinged on fitting procedure and reported uncertainty.”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Besiroglu et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 8
- Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023). Palm 2 technical report. arXiv preprint arXiv:2305.10403. · arXiv
- Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. (2024). Deepseek llm: Scaling open-source language models with longter- mism. arXiv preprint arXiv:2401.02954. · arXiv
- Bordelon, B., Atanasov, A., and Pehlevan, C. (2024). A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092. · arXiv
- Google, G. T. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. · arXiv
- Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. · arXiv
- Michaud, E., Liu, Z., Girit, U., and Tegmark, M. (2024). The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36.
- Sardana, N. and Frankle, J. (2023). Beyond chinchilla- optimal: Accounting for inference in language model scal- ing laws. arXiv preprint arXiv:2401.00448. · arXiv
- Wilks, S. S. (1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. The annals of mathematical statistics, 9(1):60-62.
Cited by
Sorted by citations
Will we run out of data? Limits of LLM scaling based on human-generated data
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022) · arXiv · doi:10.48550/arxiv.2211.04325 · Cited by 80 · see citing works
Self-citationTraining Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice
Memming, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7142719
- Authors
- Besiroglu, T., Erdil, E., Barnett, M., & You, J.
- ORCID
- Besiroglu, Tamay
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2404.10102
- OpenAlex
- W4394907038
- Cited by
- 1
- References
- 0
- Citations by year
- 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- model size scaling, confidence intervals, neural network training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
