The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; none of the 28 sources its claims cite is checked for support yet.
Understanding Emergent Abilities of Language Models from the Loss Perspective
Open accessAbstract
Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) there is doubt on the discontinuous metrics used to measure these abilities. In this paper, we propose to study emergent abilities in the lens of pre-training loss, instead of model size or training compute. We demonstrate that the Transformer models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks, with a fixed data corpus, tokenization, and model architecture. We also discover that a model exhibits emergent abilities on certain tasks -- regardless of the continuity of metrics -- when its pre-training loss falls below a specific threshold. Before reaching this threshold, its performance remains at the level of random guessing. This inspires us to redefine emergent abilities as those that manifest in models with lower pre-training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper argues that emergent abilities in language models are better understood as a function of pre-training loss than of model size or training compute. Across more than 30 models and 12 downstream datasets, models with the same pre-training loss show the same task performance, and several tasks remain at random-guess level until loss falls below a threshold around 2.2.
Key findings
- Pre-training loss predicts downstream performance regardless of model size or token count: performance-vs-loss points from different model sizes fall on the same curves, with strong correlations such as TriviaQA Spearman −0.996 / Pearson −0.994 and HellaSwag −0.996 / −0.994; correlations are weaker on MMLU (−0.804 / −0.903) and C-Eval (−0.831 / −0.884).
- On MMLU, C-Eval, GSM8K, and GSM8K-Chinese, performance stays near random guessing until pre-training loss drops below about 2.2, after which it improves; the thresholds are “surprisingly all around 2.2” despite different languages, tasks, prompting types, and answer forms.
- The emergent increase persists with continuous metrics: on MMLU and C-Eval, accuracy, CorrectChoiceProb, and Brier Score all show a tipping point when loss falls below the threshold, contradicting the claim that emergence is only a metric artifact.
- The authors redefine an emergent ability as one not present in models with higher pre-training loss but present in models with lower pre-training loss, and show this can account for previously observed model-size emergence via scaling laws.
Methods
- Pre-trained more than 30 LMs from scratch with fixed data corpus, tokenization, and architecture: sizes 300M, 540M, 1B, 1.5B, 3B, 6B, and 32B; token counts from 33B to 3T; English-to-Chinese corpus ratio 4:1; LLaMA-like architecture with grouped-query attention and rotary position embeddings on half the query/key dimensions; BPE vocabulary size 65k; AdamW optimizer.
- Evaluated on 12 English and Chinese datasets covering closed-book QA, commonsense NLI, reading comprehension, coreference resolution, examination, and math word problems, with few-shot, zero-shot, and few-shot CoT prompting; used intermediate checkpoints saved about every 43B tokens for the 1.5B/6B/32B models and final checkpoints for smaller models.
- Analyzed performance-vs-loss curves with Spearman and Pearson correlations, compared loss against compute as a predictor, and validated generality using publicly available LLaMA and Pythia models.
Limitations
- The paper states it does not consider model architectures and training algorithms beyond those tested; fundamentally different architectures such as routed Transformers and non-Transformer models are outside its scope, and both its models and LLaMA use AdamW while other optimizers exist.
- The paper states pre-training loss is affected by tokenizer and pre-training corpus distribution, so loss values from models trained on different corpora are not directly comparable; normalized perplexity on a public validation set is suggested as a possible solution.
- The paper states it should not be taken as a push to expand model and data sizes beyond current scales, since new tipping points at larger scales are not guaranteed, and instruction tuning can improve zero-shot performance on unseen tasks including MMLU and GSM8K.
- Evident from the text: the specific threshold around 2.2 is tied to the authors’ fixed corpus, tokenizer, and architecture; LLaMA-65B shows an early-stage exception when loss is above 1.8, possibly due to exponential smoothing in the original plots; and neither Pythia nor LLaMA can be used to analyze the emergent abilities because Pythia’s largest model fails to exceed random chance on MMLU/GSM8K and LLaMA lacks released intermediate checkpoints.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 28 cited sources checked · 0 supported · 0 flagged
- Findingown data
The emergent performance increase persists across both discontinuous and continuous metrics.
“To exclude the impact of discontinuous metrics [46,61], we evaluate the emergent performance increase using continuous metrics and show that the emergent abilities persist across both discontinuous and continuous metrics.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [46]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [61]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
Performance on certain downstream tasks only improves beyond the level of random guessing when the pre-training loss falls below a specific threshold.
“Over the course, we find that performance on certain downstream tasks only improves beyond the level of random guessing when the pre-training loss falls below a specific threshold, i.e., emergent abilities.”
Quote in the paperSubstantial claim - Findingown data
Most data points from LLaMA models with different sizes (7B, 13B, 33B, 65B) fall on the same upwards trend.
“Excitingly, most data points from the LLaMA models with different sizes (7B, 13B, 33B, 65B) fall on the same upwards trend.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
All three metrics-accuracy, correct choice probability, and Brier Score-show emergent performance improvements when the pre-training loss drops below a certain threshold on MMLU and C-Eval.
“All three metricsaccuracy, correct choice probability, and Brier Score-show emergent performance improvements (value increase for the first two and decrease for the third) when the pre-training loss drops below a certain threshold.”
Quote in the paperSubstantial claim - Findingown data
The generality of the conclusion that pre-training loss predicts downstream performance is verified by the open LLaMA and Pythia models.
“The generality of this conclusion is further verified by extracting and observing the performance and loss relationship of the open LLaMA [55] and Pythia [3] models.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
The results for Pythia support the conclusion that pre-training loss predicts performance.
“The results for Pythia are shown in Appendix F, which also support our conclusion.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
On three BIG-bench tasks, with pre-training loss decreasing, there is a clear tipping point in the performance curves.
“We evaluate the other three tasks in the same setting as Section 2.3 and the results are shown in Figure 8.With pretraining loss decreases along the x-axis, we can clearly observe the tipping point in the performance curves.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [51]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
The performance of emergent abilities exhibits emergent increase when the pre-training loss falls below a certain threshold, even when evaluated with continuous metrics.
“The performance of emergent abilities exhibits emergent increase when the pre-training loss falls below a certain threshold, even when evaluated with continuous metrics.”
Quote in the paperSubstantial claim - Findingreasoning
The performance in terms of Brier Score is no better than random guess before the loss reaches the threshold.
“Therefore, the performance in terms of Brier Score is no better than random guess before the loss reaches the threshold.”
Quote in the paperSubstantial claim - Findingown data
On MMLU, when the pre-training loss drops below 2.2, the accuracy increases as the loss decreases, similar to the trend observed in the first group of tasks.
“However, when the pre-training loss drops below 2.2, the accuracy increases as the loss decreases, similar to the trend observed in the first group of tasks.”
Quote in the paperSubstantial claim - Findingnone given
Transformer models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks, with a fixed data corpus, tokenization, and model architecture.
“We demonstrate that the Transformer models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks, with a fixed data corpus, tokenization, and model architecture.”
Quote in the paperSubstantial claim - Findingown data
The overall training loss is a good predictor of performance on both English and Chinese tasks, although it is computed on a mixture of English and Chinese tokens.
“Interestingly, we find that the overall training loss is a good predictor of performance on both English and Chinese tasks, although it is computed on a mixture of English and Chinese tokens.”
Quote in the paperSubstantial claim - Findingnone given
Despite differences in languages, tasks, prompting types, and answer forms among the four datasets, the thresholds for performance improvement are all around 2.2.
“Despite differences in languages, tasks, prompting types, and answer forms among the four datasets are different, the thresholds for performance improvement are surprisingly all around 2.2.”
Quote in the paperSubstantial claim - Findingnone given
Before reaching the threshold, a model's performance on emergent tasks remains at the level of random guessing.
“Before reaching this threshold, its performance remains at the level of random guessing.”
Quote in the paperSubstantial claim - Findingown data
The relationship between performance and pre-training loss is weaker on MMLU, C-Eval, GSM8K, and GSM8K-Chinese than on the first group.
“The relationship is weaker on MMLU, CEval, GSM8K, and GSM8K-Chinese, verifying the emergence of performance which we discuss in Section 3.”
Quote in the paperSubstantial claim - Findingown data
For Pythia, points from different models fall on the same curve on all the tasks.
“From the plot, we can observe that the points from different models fall on the same curve on all the tasks.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
Generally, task performance improves as the training loss goes down, regardless of the model sizes.
“Generally, the task performance improves as the training loss goes down, regardless of the model sizes.”
Quote in the paperSubstantial claim - Findingnone given
The Brier Score also decreases when the pre-training loss is above the threshold.
“The Brier Score also decreases when the pre-training loss is above the threshold.”
Quote in the paperNot a substantial claim - Findingown data
The relationship between pre-training loss and task performance holds for LLaMA and Pythia with slightly different architectures.
“We analyze the performance-loss curves of LLaMA and Pythia with slightly different architectures, and find that the relationship holds for all the models.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingnone given
The outliers in LLaMA-65B only constitute the initial 10% training tokens.
“Note that the outliers only constitute the initial 10% training tokens.”
Quote in the paperNot a substantial claim - Findingown data
On MMLU, C-Eval, GSM8K, and GSM8K-Chinese, all models of three sizes perform at the random level until the pre-training loss decreases to about 2.2, after which the performance gradually climbs as the loss decreases.
“On MMLU, C-Eval, GSM8K, and GSM8K-Chinese, all models of three sizes perform at the random level until the pre-training loss decreases to about 2.2, after which the performance gradually climbs as the loss decreases.”
Quote in the paperSubstantial claim - Findingown data
When the training loss is higher than 1.8, LLaMA-65B performs worse than smaller models with the same training loss.
“We can see that when the training loss is higher than 1.8, LLaMA-65B performs worse than smaller models with the same training loss.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
A model exhibits emergent abilities on certain tasks, regardless of the continuity of metrics, when its pre-training loss falls below a specific threshold.
“We also discover that a model exhibits emergent abilities on certain tasks-regardless of the continuity of metrics-when its pre-training loss falls below a specific threshold.”
Quote in the paperSubstantial claim - Findingnone given
When the loss is above the threshold, performance on emergent tasks remains at the level of random guessing, even though performance on other tasks continues to improve from the outset.
“When the loss is above this threshold, performance remains at the level of random guessing, even though performance on other tasks continues to improve from the outset.”
Quote in the paperSubstantial claim - Findingown data
LMs with the same pre-training loss, regardless of token count and model size, exhibit the same performance on the 12 downstream tasks.
“In other words, the LMs with the same pre-training loss regardless of token count and model size exhibit the same performance on the 12 downstream tasks.”
Quote in the paperSubstantial claim - Findingown data
Some tasks exhibit emergent performance improvements from the random guess level when the pre-training loss drops below a certain threshold regardless of model size, token count, and continuity of metrics.
“In previous sections, we show that 1) the pre-training loss is predictive of the performance of language modes on downstream tasks, and 2) some tasks exhibit emergent performance improvements from the random guess level when the pre-training loss drops below a certain threshold regardless of model size, token count, and continuity of metrics.”
Quote in the paperSubstantial claim - Findingcitation
Both Spearman and Pearson correlation coefficients show that performance is strongly related to pre-training loss on TriviaQA, HellaSwag, RACE, WinoGrande, etc.
“Both spearman and pearson correlation coefficients show that performance is strongly related to pre-training loss on TriviaQA, HellaSwag, RACE, WinoGrande, etc.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [50]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
Training loss is a good predictor of performance on 12 downstream tasks.
“From the curves and statistics, we can see that the training loss is a good predictor of the performance on 12 downstream tasks.”
Quote in the paperSubstantial claim - Findingown data
The pre-training loss of an LM is predictive of its performance on downstream tasks, regardless of its model size or data size.
“We demonstrate that the pre-training loss of an LM is predictive of its performance on downstream tasks, regardless of its model size or data size.”
Quote in the paperSubstantial claim - Findingnone given
The loss thresholds for the tasks that show emergent abilities are the same.
“Interestingly, the loss thresholds for these tasks are the same.”
Quote in the paperSubstantial claim - Findingnone given
The distribution of the correct answers is uniform in the four options in MMLU and C-Eval.
“We find that the distribution of the correct answers is uniform in the four options in MMLU and C-Eval.”
Quote in the paperNot a substantial claim - Findingnone given
At the end of training, the perplexity of correct and incorrect options starts to diverge.
“the end of training that the perplexity of correct and incorrect options starts to diverge.This supports the existence of emergent abilities.”
Quote in the paperNot a substantial claim - Interpretationcitation
The decrease of Brier Score does not always represent improvements on the task, since the Brier Score is related to not only the predicted probability of the correct answer but also the predicted probabilities of the incorrect answers.
“However, the decrease of Brier Score does not always represent improvements on the task, since the Brier Score is related to not only the predicted probability of the correct answer but also the predicted probabilities of the incorrect answers.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationown data
The pre-training loss is a good indicator of LMs' performance on downstream tasks, independent of model sizes, training tokens, languages, and pre-training frameworks.
“Observed from previous experiments and analysis, we can conclude that the pre-training loss is a good indicator of LMs' performance on downstream tasks, independent of model sizes, training tokens, languages, and pre-training frameworks.”
Quote in the paperSubstantial claim - Interpretationreasoning
Emergent abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.
“This inspires us to redefine emergent abilities as those that manifest in models with lower pre-training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [58]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationcitation
There is a monotonic relationship between model size and pre-training loss.
“In other words, there is a monotonic relationship between model size and pre-training loss.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [28]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
When the model size exceeds the threshold, the model can exhibit performance above the random chance level.
“When the model size exceeds this threshold, the model can exhibit performance above the random chance level.”
Quote in the paperSubstantial claim - Interpretationown data
The pre-training loss is a better metric to represent the scaling effect of language models than model size or training compute.
“Empirical results show that the pre-training loss is a better metric to represent the scaling effect of language models than model size or training compute.”
Quote in the paperSubstantial claim - Interpretationreasoning
The learning dynamics of English and Chinese tokens are likely very similar during multilingual pre-training.
“This implies that the learning dynamics of English and Chinese tokens are likely very similar during multilingual pre-training.”
Quote in the paperSubstantial claim - Interpretationreasoning
The perplexity of correct options is not the correct metric to evaluate the performance of multi-choice questions.
“We argue that the perplexity of correct options is not the correct metric to evaluate the performance of multi-choice questions.”
Quote in the paperSubstantial claim - Interpretationcitation
The perplexity of correct options and incorrect options may decrease simultaneously.
“The perplexity of correct options and incorrect options may decrease simultaneously.”
Quote in the paperSubstantial claim - Interpretationnone given
In this paper we prove the existence of emergent abilities from the perspective of pre-training loss, even with continuous metrics.
“In this paper we prove the existence of emergent abilities from the perspective of pre-training loss, even with continuous metrics.”
Quote in the paperNot a substantial claim - Interpretationreasoning
For the second group of tasks, there is a threshold of model sizes that corresponds to the tipping point in the pre-training loss.
“For the second group of tasks, there is a threshold of model sizes that corresponds to the tipping point in the pre-training loss.”
Quote in the paperSubstantial claim - Premisecitation
According to the scaling law, with the number of training tokens fixed, the pre-training loss follows a power law with respect to model sizes.
“According to the scaling law [28], with the number of training tokens fixed, the pre-training loss follows a power law with respect to model sizes.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [28]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
The new definition of emergent abilities from pre-training loss can account for the previously observed emergent abilities in terms of model size or training compute.
“Therefore, the new emergent abilities can also account for the previously-observed emergent abilities in terms of model size or training compute.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [28]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationcitation
The observations on the second group of datasets can be related to emergent abilities, that is, abilities that only present in large models.
“Certainly, the observations on the second groups of datasets can also be related to emergent abilities [58], that is, abilities that only present in large models.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [58]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
The new definition can explain the emergent abilities observed in Wei et al. [58].
“From this equation, we can explain the emergent abilities observed in Wei et al. [58]: when model sizes are smaller than N 0 • (η -L ∞ ) -1/α N , the normalized performance is zero.When model sizes exceed N 0 • (η -L ∞ ) -1/α N , the increase in model size leads to a decrease of pre-training loss and an improvement in normalized performance.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [58]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Probably not a substantial claim. 6
- It may state two separate claims as one. 13
- The quote only partly states this claim. 2
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown dataThe performance-vs-loss curves for all intermediate checkpoints show the same trend as Figure 2, but with larger variance.
“The performance-vs-loss curves for all the intermediate checkpoints are shown in Figure 5.The trend is the same as Figure 2, but with larger variance.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataAccuracy is discontinuous, while CorrectChoiceProb and BrierScore are continuous.
“Accuracy: discontinuous; CorrectChoiceProb and BrierScore: continuous.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataPoints from different models do not fall on the same curves on most tasks in performance-compute curves, proving pre-training loss is a better indicator of task performance than compute.
“Compared with Figure 1, we observe that points from different models do not fall on the same curves on most tasks.This proves that pre-training loss is a better indicator of task performance than compute.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
FindingcitationFour tasks have 2, 2, 4, 5 options in each question; the values of Brier Score for random guess baselines on the four tasks are 0.25, 0.25, 0.1875, and 0.16; only the largest model surpasses the random guess baseline.
“Four tasks have 2, 2, 4, 5 options in each question.The values of Brier Score for random guess baselines on the four tasks are 0.25, 0.25, 0.1875, and 0.16.Only the largest model surpasses the random guess baseline.This also supports the existence of emergent abilities.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [46]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
It may state two separate claims as one.
FindingcitationNeither Pythia nor LLaMA can be used to analyze the emergent abilities; the largest Pythia model fails to achieve performance above random chance on MMLU and GSM8K; LLaMA has no intermediate checkpoints released and performance curves on MMLU and GSM8K are not available.
“However, neither Pythia nor LLaMA can be used to analyze the emergent abilities.The largest Pythia model fails to achieve performance above random chance on MMLU and GSM8K [2].Instead, LLaMA has no intermediate checkpoints released and performance curves on MMLU and GSM8K are not available.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [2]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
It may state two separate claims as one.
FindingreasoningOn MMLU, when the pre-training loss is higher than 2.2, the accuracy remains around 25%, which is no better than random guessing because each question has four options.
“Take MMLU as an example of the second group, when the pre-training loss is higher than 2.2, the accuracy remains around 25%.Since each question in MMLU has four options, this means the model prediction is no better than random guessing.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataOn MMLU, C-Eval, GSM8K, and GSM8K-Chinese, performance remains flat when the loss is higher than a certain threshold; once the pre-training loss is lower than this threshold, performance starts to improve.
“Second, on MMLU, C-Eval, GSM8K, and GSM8K-Chinese, the performance remains flat when the loss is higher than a certain threshold.Once the pre-training loss is lower than this threshold, the performance starts to improve.”
Quote in the paperSubstantial claimIt may state two separate claims as one. The quote only partly states this claim.
Findingown dataThe performance-vs-loss data points of different model sizes fall on the same trending curve; data points of different models are indistinguishable ignoring model sizes.
“Importantly, the performance-vs-loss data points of different model sizes fall on the same trending curve.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
FindingreasoningRACE and MMLU have similar prompting format (multi-choice examination questions with in-context demonstrations) but their performance curves are quite different.
“RACE in the first group has a prompting format similar to MMLU: both consist of multi-choice examination questions with in-context demonstrations, but their performance curves are quite different.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataThe performance curves on MMLU, C-Eval, GSM8K, and GSM8K-Chinese do not yield an uptrend, meaning the performance of these models on these four tasks is close to random (with fewer than 500B tokens).
“Another similar observation is that the performance curves on MMLU, C-Eval, GSM8K, and GSM8K-Chinese do not yield an uptrend, meaning that the performance of these models on these four tasks are close to random (with fewer than 500B tokens).”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataThe performance trends of C-Eval, GSM8K, and GSM8K-Chinese follow a similar pattern to MMLU.
“The performance trends of C-Eval, GSM8K, and GSM8K-Chinese follow a similar pattern.”
Quote in the paperNot a substantial claimProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Interpretationnone givenThe analysis of previous performance trends do not necessarily apply to larger models.
“We want to emphasize that the analysis of previous performance trends do not necessarily apply to the larger models.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
PremisecitationThe emergent abilities proposed in [58] are mainly a mirage caused by nonlinear and discontinuous metrics.
“[46] claim that emergent abilities proposed in [58] are mainly a mirage caused by nonlinear and discontinuos metrics.”
- Also stated in: § Additional information · plain
“[61] also support the idea.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [46]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [58]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [61]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Also stated in: § Additional information · plain
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Interpretationnone givenThe pre-training loss is affected by the tokenizer and the distribution of pre-training corpus, so values of pre-training loss of language models trained on different corpus are not directly comparable.
“The disadvantage of studying emergent abilities in the lens of pre-training loss is that the pre-training loss is affected by the tokenizer and the distribution of pre-training corpus.The values of pre-training loss of language models trained on different corpus are not directly comparable.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Interpretationown dataEmergent abilities of language models occur when the pre-training loss reaches a certain tipping point, and continuous metrics cannot eliminate the observed tipping point.
“We conclude that emergent abilities of language models occur when the pre-training loss reaches a certain tipping point, and continuous metrics cannot eliminate the observed tipping point.”
Quote in the paperSubstantial claimProbably not a substantial claim.
PremisecitationUsing the perplexity of correct options as the metric for BIG-Bench, the metric improves smoothly on almost all tasks of BIG-Bench.
“[61] use the perplexity of correct options as the metric for BIG-Bench and find that the metric impproves smoothly on almost all the tasks of BIG-Bench.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [61]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
It may state two separate claims as one.
InterpretationreasoningThe best Brier Score for a context-free predictor is achieved by always giving uniform probability to all the options, and in this case the Brier Score is equal to 0.75.
“The best Brier Score for a context-free predictor is achieved by always giving uniform probability to all the options.In this case, the Brier Score is equal to 0.75.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
It may state two separate claims as one.
InterpretationreasoningCombining the loss scaling relation and the new definition, the normalized performance as a function of model size is zero when model sizes are smaller than N0*(η-L∞)^(-1/αN), and increases when model sizes exceed that threshold.
“From this equation, we can explain the emergent abilities observed in Wei et al. [58]: when model sizes are smaller than N 0 • (η -L ∞ ) -1/α N , the normalized performance is zero.When model sizes exceed N 0 • (η -L ∞ ) -1/α N , the increase in model size leads to a decrease of pre-training loss and an improvement in normalized performance.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. It may be too vague to check against evidence.
InterpretationreasoningThe correct metric of multi-choice questions should reflect the ability of distinguishing correct options from incorrect options.
“The correct metric of multi-choice questions should reflect the ability of distinguishing correct options from incorrect options.”
Quote in the paperNot a substantial claim
Citations
Cites
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Kudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-4009 · Cited by 2,475
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Sakaguchi, K., Le Bras, R., Bhagavatula, C., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6399 · Cited by 579
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575
Crowdsourcing Multiple Choice Science Questions
Welbl, J., Liu, N. F., & Gardner, M. (2017) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/w17-4413 · Cited by 574
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Self-citationThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., & Sanghai, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.298 · Cited by 454
CLUE: A Chinese Language Understanding Evaluation Benchmark
Xu, L. J., Hu, H., Zhang, X., Lü, L., Cao, C., Li, Y., Xu, Y., Sun, K., Yu, D., Yu, C., et al. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.419 · Cited by 393
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Lei, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.08322 · Cited by 373
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingPreprint
Liu, H., Li, Z., Hall, D., Liang, P., & Ma, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.14342 · Cited by 224
Predictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207
(2023) · doi:10.52202/075280-2425 · Cited by 200
Are Emergent Abilities of Large Language Models a Mirage?
Schaeffer, R., Miranda, B., & Koyejo, S. (2023) · arXiv · doi:10.48550/arxiv.2304.15004 · Cited by 200
Scaling Laws for Autoregressive Generative Modeling
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14701 · Cited by 150
Generalization Error Bounds of Gradient Descent for Learning Over-Parameterized Deep ReLU Networks
Cao, Y. J., & Gu, Q. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5736 · Cited by 143
Dar, Y., Muthukumar, V., & Baraniuk, R. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.02355 · Cited by 94
Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension
Sun, K., Yu, D., Yu, D., & Cardie, C. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00305 · Cited by 94
The LAMBADA dataset: Word prediction requiring a broad discourse context
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., & Fernandez, R. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1144 · Cited by 87
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. (2022) · arXiv · doi:10.48550/arxiv.2201.02177 · Cited by 83
Training Trajectories of Language Models Across Scales
Xia, M., Artetxe, M., Zhou, C., Lin, X. V., Pasunuru, R., Chen, D., Zettlemoyer, L., & Stoyanov, V. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.767 · Cited by 49
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Shin, S., Lee, S.-W., Ahn, H., Kim, S., Kim, H., Kim, B., Cho, K., Lee, G., Park, W., Ha, J.-W., et al. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.380 · Cited by 43
PIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Le bras, R., Gao, J., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6239 · Cited by 37
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., & Metzler, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.825 · Cited by 34
CUGE: A Chinese Language Understanding and Generation Evaluation BenchmarkPreprint
Yao, Y., Dong, Q., Guan, J., Cao, B., Zhang, Z., Xiao, C., Wang, X., Qi, F., Bao, J., Nie, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.13610 · Cited by 29
Roh, J., Oh, S.-H., & Lee, S.-Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13220 · Cited by 8
Language models scale reliably with over-training and on downstream tasks
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.08540 · Cited by 5
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Flagged first, then by citations
Research works printed but not linked: 24
- Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397-2430. PMLR, 2023.
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1-240:113, 2023. URL http://jmlr.org/papers/v24/22-1144.html.
- Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake A. Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew J. Johnson, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Marc'Aurelio Ranzato, Jack W. Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. Unified scaling laws for routed language models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 4057-4086. PMLR, 2022. URL https://proceedings.mlr.press/v162/clark22a.html.
- Nan Duan. Overview of the nlpcc-iccpol 2016 shared task: Open domain chinese question answering. In Natural Language Understanding and Intelligent Applications, pages 942-948. Springer International Publishing, 2016. ISBN 978-3-319-50496-4.
- William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1-120:39, 2022. URL http://jmlr.org/papers/v23/21-0998.html.
- Daniel Y. Fu, Tri Dao, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=COZDy0WYGg.
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
- OpenReview.net, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ.
- Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, et al. Predicting emergent abilities with infinite resolution evaluation. arXiv e-prints, pages arXiv-2310, 2023.
- Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22188-22214. PMLR, 2023. URL https://proceedings.mlr.press/v202/liu23ao.html.
- Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22631-22648. PMLR, 2023. URL https://proceedings.mlr.press/v202/longpre23a.html.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
- OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
- Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 28043-28078. PMLR, 2023. URL https://proceedings.mlr. press/v202/poli23a.html.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020. URL http://jmlr. org/papers/v21/20-074.html.
- Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4.
- Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4603-4611.
- PMLR, 2018. URL http://proceedings.mlr.press/v80/shazeer18a.html.
- C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-101, 1904. ISSN 00029556.
- Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pretraining and finetuning transformers. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
- Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openreview.net/forum?id=gEZrGCozdqR.
- Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022. URL https://openreview.net/forum?id= yzkSU5zdwD.
- Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130B: an open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=-Aw0rrrPUF.
Cited by
Sorted by citations
Scaling up Masked Diffusion Models on TextPreprint
Nie, S., Zhu, F., Du, C., Pang, T., Liu, Q., Zeng, G., Lin, M., & Li, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.18514 · Cited by 121
Observational Scaling Laws and the Predictability of Langauge Model Performance
Ruan, Y., Maddison, C. J., & Hashimoto, T. (2024) · neural information processing systems · doi:10.52202/079017-0506 · Cited by 94
Emergent Abilities in Large Language Models: A SurveyPreprint
Berti, L., Giorgi, F., & Kasneci, G. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.05788 · Cited by 93
QuRating: Selecting High-Quality Data for Training Language ModelsPreprint
Wettig, A., Gupta, A., Malik, S., & Chen, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.09739 · Cited by 88
Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
Hägele, A., Bakouch, E., Kosson, A., Allal, L. B., von Werra, L., & Jaggi, M. (2024) · neural information processing systems · doi:10.52202/079017-2427 · Cited by 76
International AI Safety Report 2026Preprint
Bengio, Y., Clare, S., Prunkl, C., Andriushchenko, M., Bucknall, B., Murray, M., Bommasani, R., Casper, S., Davidson, T., Douglas, R., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.21012 · Cited by 74
Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
Ren, R., Basart, S., Khoja, A., Pan, A., Gatti, A., Phan, L., Yin, X., Mazeika, M., Mukobi, G., Kim, R., et al. (2024) · neural information processing systems · doi:10.52202/079017-2190 · Cited by 68
T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference ScalingPreprint
Hou, Z., Lv, X., Lu, R., Zhang, J., Li, Y., Yao, Z., Li, J., Tang, J., & Dong, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.11651 · Cited by 64
Self-citationInternational Scientific Report on the Safety of Advanced AI (Interim Report)Preprint
Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Goldfarb, D. A., Heidari, H., Khalatbari, L., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.05282 · Cited by 59
Scaling Laws for Predicting Downstream Performance in LLMsPreprint
Chen, Y., Huang, B., Gao, Y., Wang, Z., Yang, J., & Ji, H. M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.08527 · Cited by 40
Li, M., Wang, Z., He, K., Ma, X., & Liang, Y. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.920 · Cited by 33
Scaling Law with Learning Rate Annealing
Tissue, H., Wang, V., & Wang, L. (2025) · neural information processing systems · doi:10.52202/085713-3044 · Cited by 32
Selecting Large Language Model to Fine-tune via Rectified Scaling LawPreprint
Lin, H., Huang, B., Ye, H., Chen, Q., Wang, Z., Li, S., Ma, J., Wan, X., Zou, J., & Liang, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.02314 · Cited by 29
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.06395 · Cited by 28
Scaling Laws For Dense Retrieval
Fang, Y., Zhan, J., Ai, Q., Mao, J., Su, W., Chen, J., & Liu, Y. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657743 · Cited by 28
Generative Large Recommendation Models: Emerging Trends in LLMs for Recommendation
Wang, H., Guo, W., Zhang, L., Chin, J. Y., Ye, Y., Guo, H., Liu, Y., Lian, D., Tang, R., & Chen, E. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715865 · Cited by 24
SOLO: A Single Transformer for Scalable Vision-Language ModelingPreprint
Chen, Y., Wang, X., Peng, H., & Ji, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.06438 · Cited by 24
Does RLHF Scale? Exploring the Impacts From Data, Model, and MethodPreprint
Hou, Z., Du, P., Niu, Y., Du, Z., Zeng, A., Liu, X., Huang, M., Wang, H., Tang, J., & Dong, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.06000 · Cited by 24
Self-citationPerformance Law of Large Language ModelsPreprint
Wu, C., & Tang, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.09895 · Cited by 21
A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal LanguagePreprint
Lubana, E. S., Kawaguchi, K., Dick, R. P., & Tanaka, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.12578 · Cited by 19
Loss-to-Loss Prediction: Scaling Laws for All DatasetsPreprint
Brandfonbrener, D., Anand, N., Vyas, N., Malach, E., & Kakade, S. M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.12925 · Cited by 18
Optimizing Sequential Recommendation Models with Scaling Laws and Approximate EntropyPreprint
Shen, T., Wang, H., Wu, C., Chin, J. Y., Guo, W., Liu, Y., Guo, H., Lian, D., Tang, R., & Chen, E. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.00430 · Cited by 17
International AI Safety Report
Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Fox, P., Garfinkel, B., Goldfarb, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.17805 · Cited by 16
CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models
Gu, J., Yang, Z., Ding, C., Zhao, R., & Tan, F. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.903 · Cited by 16
Why are LLMs' abilities emergent?Preprint
Havlík, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.04401 · Cited by 16
SmolLM2: When Smol Goes Big - Data-Centric Training of a Small Language Model
Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydlíček, H., Lajarín, A. P., Srivastav, V., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.02737 · Cited by 13
Transformers Struggle to Learn to SearchPreprint
Saparov, A., Pawar, S., Pimpalgaonkar, S., Joshi, N., Pang, R. Y., Padmakumar, V., Kazemi, S. M., Kim, N., & He, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.04703 · Cited by 13
EvoLM: In Search of Lost Language Model Training DynamicsPreprint
Qi, Z., Nie, F., Alahi, A., Zou, J., Lakkaraju, H., Du, Y., Xing, E., Kakade, S. M., & Zhang, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.16029 · Cited by 12
Emergent Abilities of Large Language Models under Continued Pre-training for Language Adaptation
Elhady, A., Agirre, E., & Artetxe, M. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1547 · Cited by 11
Need a Small Specialized Language Model? Plan Early!Preprint
Grangier, D., Katharopoulos, A., Ablin, P., & Hannun, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01093 · Cited by 11
Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI SystemsPreprint
Gipiškis, R., Joaquin, A. S., Chin, Z. S., Regenfuß, A., Gil, A., & Holtman, K. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.23472 · Cited by 11
CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language ModelsPreprint
Gu, J., Yang, Z., Ding, C., Zhao, R., & Tan, F. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.17467 · Cited by 10
Neuron-based Multifractal Analysis of Neuron Interaction Dynamics in Large ModelsPreprint
Xiao, X., Ping, H., Zhou, C., Cao, D., Li, Y., Zhou, Y., Li, S., Kanakaris, N., & Bogdan, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.09099 · Cited by 8
The Right Time Matters: Data Arrangement Affects Zero-Shot Generalization in Instruction TuningPreprint
He, B., Ding, N., Qian, C., Deng, J. H., Cui, G., Yuan, L., Hong, H., Gao, H.-A., Huang, L., Xue, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.11721 · Cited by 8
Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based PerspectivePreprint
Xu, C., Chen, K., Li, X., Shen, K., & Li, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.17262 · Cited by 8
Emergent Abilities in Reduced-Scale Generative Language Models
Muckatira, S., Deshpande, V., Lialin, V., & Rumshisky, A. (2024) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2024.findings-naacl.79 · Cited by 7
Lee, D., Lee, H., & Yi, J. (2025) · Physical Review Research · doi:10.1103/tf5z-7q29 · Cited by 6
The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model ServingPreprint
Zeng, P., Ning, Z., Zhao, J., Cui, W., Xu, M., Guo, L., Chen, X., & Shan, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.11299 · Cited by 6
Psychometric Personality Shaping Modulates Capabilities and Safety in Language ModelsPreprint
Fitz, S., Romero, P., Basart, S., Chen, S., & Hernández‐Orallo, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.16332 · Cited by 6
Revisiting the Scaling Properties of Downstream Metrics in Large Language Model TrainingPreprint
Krajewski, J., Shidani, A., Busbridge, D., Wiseman, S., & Ramapuram, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.08894 · Cited by 6
Wang, J., Zhang, H., Slaton, K., Wang, S., Serban, R., Wu, J., & Negruţ, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.04062 · Cited by 4
Revisiting Transformers through the Lens of Low Entropy and Dynamic SparsityPreprint
Ren, R., & Liu, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.18929 · Cited by 4
Zhao, F., Zhao, C., Wang, L., & Sang, H. (2025) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2025.3649289 · Cited by 3
A lifecycle-oriented survey of LLM-based agents: Workflow, graph augmentation, and future directions
Kang, C., & Li, K. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134838 · Cited by 3
Grandes modelos de lenguaje: de la predicción de palabras a la comprensión?Preprint
Gómez‐Rodríguez, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.18205 · Cited by 3
PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous ComputingPreprint
Deng, C., Sun, L., Jiang, J., Zeng, Y., Wu, X., Zhao, W., Xiao, Q., Wang, J., Li, H., Chen, L., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.12167 · Cited by 3
ChronoLLM: customizing language models for physics-based simulation code generation
Wang, J., Negrut, A., Zhang, H., Slaton, K., Wang, S., Serban, R., Wu, J., & Negruţ, D. (2026) · Multibody System Dynamics · doi:10.1007/s11044-026-10152-x · Cited by 2
Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You NeedPreprint
Zhang, B., Yan, Y., Yang, B., Xue, Y., & Liu, G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.07111 · Cited by 2
Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory
Niu, X., Bai, B., Deng, L., & Han, W. (2024) · arXiv · doi:10.48550/arxiv.2405.08707 · Cited by 1
Ge, Q., Xing, S., Gao, S., Zhou, Y., Zou, Y., Zhang, S., Chen, Z., Yan, H., Zhang, Q., Guo, Q., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1157 · Cited by 1
LLM GameLab: An Interactive Platform for Testing Large Language Models in Board Games
Morillo, P., Terreros, A., Ferri, C., & Hernández‐Orallo, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06129-4_36 · Cited by 1
Rocha, A., de Oliveira, P. C., Ferreira, J. P., Alves, M., & Sousa, A. J. (2026) · Applied Sciences · doi:10.3390/app16157745 · Cited by 1
EvoLM: In Search of Lost Training Dynamics for Language Model Reasoning
Qi, Z., Nie, F., Alahi, A., Zou, J., Lakkaraju, H., Du, Y., Xing, E., Kakade, S. M., & Zhang, H. (2025) · neural information processing systems · doi:10.52202/085713-2778 · Cited by 1
Soup-of-Experts: Pretraining Specialist Models via Parameters AveragingPreprint
Ablin, P., Katharopoulos, A., Seto, S., & Grangier, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.01804 · Cited by 1
Meta-Learning Transformers to Improve In-Context GeneralizationPreprint
Braccaioli, L., Vettoruzzo, A., Singh, P., Vanschoren, J., Bouguelia, M.-R., & Conci, N. (2025) · TU/e Research Portal · doi:10.48550/arxiv.2507.05019 · Cited by 1
Towards a Universal Scaling Law of LLM Training and InferencePreprint
Wu, C., & Tang, R. (2024) · doi:10.14293/pr2199.001074.v1 · Cited by 1
AMix-1: A Pathway to Test-Time Scalable Protein Foundation ModelPreprint
Lv, C., Zhou, J., Long, S., Wang, L., Feng, J., Xue, D., Pei, Y., Wang, H., Zhang, Z., Cai, Y., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.08920 · Cited by 1
Post-Training Science for Supervised Fine-TuningPreprint
O'Neill, C., Jayasekara, M., & Partridge, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.01244 · Cited by 1
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
Ai, Q., Zhan, J., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2501.02842
Advancing Neural Network Performance through Emergence-Promoting Initialization SchemePreprint
Li, J. J., George, V. K., & Silva, G. A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.19044
Quantifying Emergence in Neural Networks: Insights from Pruning and Training DynamicsPreprint
AlShinaifi, F., Almoaigel, Z., Li, J. J., Kuleib, A., & Silva, G. A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.01568
Enhancing Flight Deck Decision Support with Distributed GenAI: A Multi-Agent Approach
Prado, J. P. D. A., Zammit-Mangion, D., Zammit, B., Gauci, J., Muscat, A., Mizzi, S., & Manicolo, A. (2025) · IEEE Aerospace Conference (AERO) · doi:10.1109/aero63441.2025.11068478
The Right Time Matters: Data Arrangement Affects Zero-Shot Generalization in Instruction Tuning
He, B., Ding, N., Qian, C., Deng, J., Cui, G., Yuan, L., Hong, H., Gao, H.-A., Huang, L., Xue, H., et al. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.13
CTRL: Contrastive Traffic Recognition with Lightweight network
Song, Z. (2025) · International Conference on Intelligent Computing (ICIC) · doi:10.65286/icic.v21i1.68305
MACHINE LEARNING FOR ART CRITIQUE GENERATION
Viswanathan, R., Yadav, P., Pavithra, M. S., Sachdeva, A., Panda, S., & Deshmukh, S. (2025) · ShodhKosh Journal of Visual and Performing Arts · doi:10.29121/shodhkosh.v6.i5s.2025.6921
Ding, Y., Han, X., Yang, J., Wang, T., Bi, Z., Song, X., Hao, J., Song, J., Ge, E., Peng, B., et al. (2026) · doi:10.36227/techrxiv.176784493.33333564/v1
Biological risk and the AI capability shift
Turner, S. D. (2026) · doi:10.59350/zp9qy-3gg73
LaCy: What Small Language Models Can and Should Learn is Not Just a Question of LossPreprint
Ujváry, S., Béthune, L., Ablin, P., Monteiro, J., Cuturi, M., & Kirchhof, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.12005
ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language ModelsPreprint
Wang, T., Wang, P., Shi, W., & li, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.19515
THE EVOLUTION OF POST-PHOTOGRAPHY ART IN AN ERA DOMINATED BY AI IMAGE SYNTHESIS
Shriyal, S. S., Kumaran, E. S., Mishra, D., Gandhi, A., Adhyaru, Y. B. K., Saraswati, B., & Srimathi, N. (2026) · ShodhKosh Journal of Visual and Performing Arts · doi:10.29121/shodhkosh.v7.i4s.2026.7476
The Art That Poses Back: Assessing AI Pastiches After Contemporary Artworks
Dinu, A., Mihail, A., Florescu, A.-M., & Creanga, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-24350-8_5
Benchmarking Byte-Pair Encoding Tokenizers on Different Languages with Bits per Byte
Chowdhury, S., & Woolf, W. (2026) · Workshop on Multilinguality in the Era of Large Language Models (MeLLM) · doi:10.18653/v1/2026.mellm-1.27
Memming, A. (2026) · Research Square · doi:10.21203/rs.3.rs-10434935/v1
Foundation Models for AstrophysicsPreprint
Zhao, X., & Ting, Y.-S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.02573
Artificial Intelligence and Human Language Cognition: An Integrative Comparative Framework
Ishfaq, A., & Sultan, N. (2025) · Critical Review of Social Sciences and Humanities
Metadata
- Authors
- Du, Z., Zeng, A., Dong, Y., & Tang, J.
- ORCID
- Du, Zhengxiao
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 53138-53167
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W4415797507
- Cited by
- 76
- References
- 29
- Citations by year
- 2026: 14 · 2025: 32 · 2024: 30
- FWCI
- 12.00
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62276148)
- Tsinghua University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- emergent capabilities, language models, training loss, loss landscape, large language models, emergent behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
