Understanding Emergent Abilities of Language Models from the Loss Perspective
Open accessAbstract
Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) there is doubt on the discontinuous metrics used to measure these abilities. In this paper, we propose to study emergent abilities in the lens of pre-training loss, instead of model size or training compute. We demonstrate that the Transformer models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks, with a fixed data corpus, tokenization, and model architecture. We also discover that a model exhibits emergent abilities on certain tasks -- regardless of the continuity of metrics -- when its pre-training loss falls below a specific threshold. Before reaching this threshold, its performance remains at the level of random guessing. This inspires us to redefine emergent abilities as those that manifest in models with lower pre-training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The preregistered predictions accurately extrapolate to new models released after the initial paper release, including Llama-3.1-405B.
“Our preregistered predictions also accurately extrapolate to new models released after the initial paper release, including Llama-3.1-405B [24].”
From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [24]
Preregistered predictions accurately extrapolate to new models released after the initial paper release, including Llama-3.1-405B.
“Our preregistered predictions also accurately extrapolate to new models released after the initial paper release, including Llama-3.1-405B[24].”
From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [24]
The paper assessed preregistered predictions on new models released after the initial paper release, collected as of September 1st 2024, including Llama 3.1-405B and Qwen2-72B, resulting in an additional test set of 20 models.
“We have assessed these predictions on new models released after the initial paper release, collected as of September 1st 2024, including most capable open models to date such as Llama 3.1-405B [24] and Qwen2-72B [105] (see the full collected model list in Appx.B.1.1),resulting in an additional test set of 20 models for robustness checks.”
From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [24]
Prior work has shown that scaling procedures only work for some benchmarks and not others.
“While there is a large and ever-growing number of benchmarks, prior work has shown these scaling procedures only works for some benchmarks and not others [66, 56, 15, 50].”
From Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (2025) · cites this paper as [15]
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Cited by 5
- Authors
- Du, Z., Zeng, A., Dong, Y., & Tang, J.
- ORCID
- Du, Zhengxiao
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 53138-53167
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2403.15796
- OpenAlex
- W4415797507
- Cited by
- 5
- References
- 0
- Citations by year
- 2026: 3 · 2025: 2
- FWCI
- 1.44
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62276148)
- Tsinghua University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- emergent capabilities, language models, training loss, loss landscape, large language models, emergent behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
