CoChat
Sign Up
PreprintCited by 1Topic ModelingNatural Language Processing TechniquesAdversarial Robustness in Machine Learning

Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation

Open access
Heineman, David, Hofmann, Valentin, Magnusson, Ian, Gu, Yuling, Smith, Noah A., Hajishirzi, Hannaneh, Lo, Kyle, Dodge, Jesse·2025·arXiv
DOI 10.48550/arxiv.2508.13144arXiv 2508.13144
Open this paper in CoChat — free accountOpen PDF

Abstract

Developing large language models is expensive and involves making decisions with small experiments, typically by evaluating on large, multi-task evaluation suites. In this work, we analyze specific properties which make a benchmark more reliable for such decisions, and interventions to design higher-quality evaluation benchmarks. We introduce two key metrics that show differences in current benchmarks: signal, a benchmark's ability to separate better models from worse models, and noise, a benchmark's sensitivity to random variability between training steps. We demonstrate that benchmarks with a better signal-to-noise ratio are more reliable when making decisions at small scale, and those with less noise have lower scaling law prediction error. These results suggest that improving signal or noise will lead to more useful benchmarks, so we introduce three interventions designed to directly affect signal or noise. For example, we propose that switching to a metric that has better signal and noise (e.g., perplexity rather than accuracy) leads to better reliability and improved scaling law error. We also find that filtering noisy subtasks, to improve an aggregate signal-to-noise ratio, leads to more reliable multi-task evaluations. We also find that averaging the output of a model's intermediate checkpoints to reduce noise leads to consistent improvements. We conclude by recommending that those creating new benchmarks, or selecting which existing benchmarks to use, aim for high signal and low noise. We use 30 benchmarks for these experiments, and 375 open-weight language models from 60M to 32B parameters, resulting in a new, publicly available dataset of 900K evaluation benchmark results, totaling 200M instances.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces signal (a benchmark’s ability to separate better from worse models) and noise (its sensitivity to random variability between training steps), showing that their ratio predicts small-to-large decision accuracy and scaling-law prediction error. It proposes three interventions—filtering noisy subtasks, averaging checkpoint scores, and using bits-per-byte—that improve benchmark reliability.

Key findings

  • Signal-to-noise ratio (SNR) correlates with decision accuracy across OLMES benchmarks (\(R = 0.791\), \(R^2 = 0.626\)), while signal or noise alone do not; noise correlates with scaling-law prediction error for OLMo 2 13B (\(R = 0.653\), \(R^2 = 0.426\)).
  • Filtering subtasks by SNR improved decision accuracy by +2.6% for MMLU and +5% for AutoBencher; MMLU with only 16 subtasks and AutoBencher with only 6 subtasks had higher SNR than the full sets.
  • Averaging the final \(k\) checkpoints improved decision accuracy by +2.4% for the 30-task average, improved all but two tasks, and reduced scaling-law prediction error for 20 of 30 tasks.
  • Switching to bits-per-byte (BPB) increased SNR for most benchmarks—e.g., GSM8K from 1.2 to 7.0 and MBPP from 2.0 to 41.8—improved decision accuracy for 90.0% of benchmarks, and lowered scaling-law prediction error for 73.3%.

Methods

  • Evaluated 465 language models on 30 benchmarks across 14 model sizes, including scaling-law models from 190M to 3.2B with 7B/13B targets, 25 DataDecide pretraining-corpus models from 60M to 1.3B, OLMo 2 final checkpoints for 1B/7B/13B/32B, and 73 open-weight base models; also trained 20 1B models varying random seed and data order.
  • Defined signal as relative dispersion (maximum pairwise score difference divided by mean score) and noise as the relative standard deviation of the final \(n\) checkpoints, then tested SNR against two settings: decision accuracy (small-model ranking predicts large-model ranking) and scaling-law prediction error (small models predict 13B performance).
  • Tested three interventions: filtering subtasks by SNR, averaging checkpoint scores during training, and measuring bits-per-byte instead of primary metrics such as accuracy, exact match, or pass@1.

Limitations

  • The paper states its scope is limited to the two settings of decision accuracy and scaling-law prediction error, and it only studies noise of the model during training; future work may explore other small-to-large phenomena and additional noise sources such as evaluation configuration.
  • The paper states that noise estimation depends on choosing \(n\) checkpoints: \(n = 9\) gives 95% confidence that sample noise is within one standard deviation of population noise, while a tighter \(0.2\sigma\) bound requires \(n = 20\) for 34 of 39 benchmarks considered.
  • Evident from the text: external open-weight models’ training FLOPs are estimated from reported token counts, and the analysis focuses on specific benchmark suites and model families, so generalization beyond those settings is not established.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2508.13144

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

22 of 29 cited sources checked · 9 supported · 1 flagged · accuracy not yet measured

  • Premisecitation
    § 1 Introduction · p. 1

    Language model development decisions rely on measuring phenomena at smaller scales and assuming the measured trends hold at larger scale.

    “During the development process, researchers need to make decisions such as what architecture to use, what training methods to employ, and what data to train on. These decisions rely on measuring phenomena at smaller, more economical scales, then hoping the trends measured hold for large scale models.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [31] [31] Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems , 37:14200-14282, 2024. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [17] [17] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. - source

      Section 3.1.2 says the authors train small models on candidate data mixes to predict large-model performance, and Section 3.2.1 describes using scaling laws to forecast flagship-model performance. The Introduction also discusses architecture and training-method choices, but does not say those choices relied on trends measured at smaller scales. The claim extends the documented approach to all three kinds of decision. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [38] [38] Ian Magnusson, Tai Nguyen, David Heineman, Jena D. Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A. Smith, Pang Wei Koh, Ben Bogin, and Jesse Dodge. Datadecide: How to predict best pretraining data with small experiments. under submission , 2025. - source
  • Premisecitation
    § 1 Introduction · p. 1

    Prior work has shown that scaling procedures only work for some benchmarks and not others.

    “While there is a large and ever-growing number of benchmarks, prior work has shown these scaling procedures only works for some benchmarks and not others [66, 56, 15, 50].”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [66] [66] Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation. arXiv preprint arXiv:2502.10341 , 2025. - source

      The Results (p. 6) report that combining topic and format mixtures improved performance on 8 of 9 tasks, showing that outcomes differ by benchmark. Experimental Details (p. 27) also note that some benchmark tasks were near random performance at the 1b-1x scale. Neither passage establishes the broader claim that scaling procedures work for some benchmarks but not others. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [56] [56] Kashun Shum, Yuzhen Huang, Hongjian Zou, Ding Qi, Yixuan Liao, Xiaoxin Chen, Qian Liu, and Junxian He. Predictive data selection: The data that predicts is the data that teaches. arXiv preprint arXiv:2503.00808 , 2025. - source

      Table 13 (Appendix C.2, p. 24) shows that PRESELECT improves performance over random selection on some benchmarks but not others. Section 3.4 (p. 5) likewise says it performs best on most tasks, not all. These passages support benchmark-dependent results for this data-selection method, but do not establish that scaling procedures generally work only for some benchmarks. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [15] [15] Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796 , 2024. - source
  • Premisecitation
    § 2 Predicting Large Model Phenomena with Small Models · p. 2

    Results from small scale experiments are not always reliable.

    “However, the results from small scale experiments are not always reliable.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [65] [65] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022. - source

      Section 5.1 (p. 7) says evaluations at small language-model scales can mask improvements, and Appendix A.1 (p. 19) finds that the measured performance of small models did not reveal those improvements. This supports caution about using small-model results to predict larger-model behavior, but not the broader claim that small-scale experiments in general are unreliable. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 3.1 Measuring Noise · p. 4

    Multiple training runs under the same configuration can lead to different performance as a result of different initialization or data order.

    “Previous work has shown multiple training runs under the same configuration can lead to different performance as a result of a different initialization or data order [14, 13].”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [14] [14] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305 , 2020. - source
    • cites [13] [13] Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research , 23(226):1-61, 2022. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisereasoning
    § A.2 Decision Accuracy Details · p. 16

    Decision accuracy is equivalent to Kendall's τ modulo a scale and shift.

    “Therefore, the decision accuracy measure in Magnusson et al. [38] is equivalent to Kendall's τ modulo a scale and shift.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [38] [38] Ian Magnusson, Tai Nguyen, David Heineman, Jena D. Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A. Smith, Pang Wei Koh, Ben Bogin, and Jesse Dodge. Datadecide: How to predict best pretraining data with small experiments. under submission , 2025. - source

      Section 2.3 (Prediction Metrics) defines decision accuracy as the fraction of data-recipe pairs whose predicted and observed rankings agree and says it is “nearly equivalent to Kendall’s τ, but ranges from 0 to 1.” This supports the relationship, but the source qualifies it as nearly equivalent rather than stating an exact equivalence under a scale and shift. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 18
  • May combine two claims. 21

  • Classified as probably not a substantial claim.

    Premisecitation
    § 5.3 Measuring bits-per-byte improves benchmark signal -tonoise ratio · p. 9

    Recent work has begun to evaluate by using the test set as a perplexity set, with the intuition that discontinuous metrics like accuracy or exact match erode the relationship between language modeling perplexity and the downstream metric.

    “Recent work has begun to evaluate by using the test set as a perplexity set, with the intuition that the discontinuous metrics like accuracy or exact match erode the relationship between the language modeling perplexity and the downstream metric [54, 25].”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [54] [54] Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting downstream capabilities of frontier ai models with scale remained elusive? arXiv preprint arXiv:2406.04391 , 2024. - source

      Section 5 (p. 6) finds that negative log likelihood of the correct choice corresponds cleanly with compute, while downstream metrics including accuracy lose that predictability. This supports the general concern about moving from likelihood-based evaluation to accuracy, but our copy does not describe using the test set as a perplexity set or analyze exact match; a search for “perplexity” found no such passage. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [25] [25] Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. arXiv preprint arXiv:2404.09937 , 2024. - source

      The Metrics section says the paper correlates benchmark scores with bits per character (BPC), rather than using the benchmark test set as a perplexity set. The figure descriptions in “E Better Alignment between MMLU and Textbooks” specify separate Common Crawl, GitHub, and arXiv corpora for BPC. Appendix A.3 uses option perplexity to select answers, but does not give the claimed rationale about discontinuous metrics. (LLM, conf=0.83)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2 Predicting Large Model Phenomena with Small Models · p. 2

    For some benchmarks, language model performance only rises above random chance for models trained at large compute budgets.

    “Work on so-called emergent capabilities [65] shows that for some benchmarks, language model performance only rises above random chance for models trained at large compute budgets.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [65] [65] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Tasks with higher noise also have higher scaling law error · p. 6

    Gadre et al. [19] find that the error from individual tasks in their work is too difficult to predict accurately.

    “In particular, Gadre et al. [19] find that the error from the individual tasks in their work to be too difficult to predict accurately.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [19] [19] Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540 , 2024. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 6 Related Work and Discussion · p. 10

    Interventions to improve evaluation have been well explored, such as constructing higher quality benchmarks by identifying errors, expanding test sets, selecting high quality instances, or generating new synthetic benchmarks.

    “Interventions to improve evaluation have been well explored, such as constructing higher quality benchmarks by identifying errors [62, 21], expanding test sets [64], selecting high quality instances from benchmarks [45], or generating entirely new synthetic benchmarks from a model [32].”
    Quote in the paperNot a substantial claim2 of 4 sources checked
    • cites [62] [62] Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.03461 , 2025. - source
    • cites [21] [21] Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? arXiv preprint arXiv:2406.04127 , 2024. - source
    • cites [64] [64] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2024. - source
    • cites [45] [45] Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating llms with fewer examples. arXiv preprint arXiv:2402.14992 , 2024. - source

      The Introduction and §5 describe selecting subsets of existing benchmark examples to estimate LLM performance accurately and efficiently. This supports the cited point about selecting instances from benchmarks, but the paper does not establish that the selected instances are individually higher quality; its selection criterion is how well they support evaluation of the full benchmark. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation
    § 6 Related Work and Discussion · p. 10

    Noise within evaluation benchmarks is frequently studied as the intrinsic noise of the dataset, rather than the noise as a result of differences in the model during training.

    “Noise within evaluation benchmarks is frequently studied as the intrinsic noise of the dataset [2, 7, 40, 6], rather than the noise as a result of differences in the model during training.”
    Quote in the paperSubstantial claim3 of 4 sources checked
    • cites [2] [2] Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. An empirical investigation of statistical significance in NLP. In Jun'ichi Tsujii, James Henderson, and Marius Pa¸ sca, editors, Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning , pages 995-1005, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/D12-1091/ . -

      Not checked in the free run.

    • cites [7] [7] Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. With little power comes great responsibility. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 9263-9274, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.745. URL https:// aclanthology.org/2020.emnlp-main.745/ . - source

      Section 3 (p. 3) discusses how small benchmark test sets can make model comparisons underpowered and notes that significance testing is widely emphasized. This supports attention to uncertainty arising from evaluation data, but the paper does not characterize that uncertainty as intrinsic dataset noise or establish that it is studied more often than variation from model training. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [40] [40] Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640 , 2024. - source

      Section 3 (p. 5) analyzes evaluation-score variance from sampled questions and variation in answers to those questions, supporting the dataset-related part of the claim. The paper does not compare that work with noise caused by differences between model training runs or establish that one is studied more frequently than the other. (LLM, conf=0.77)

      Experimental — accuracy not yet measured
    • cites [6] [6] Sam Bowyer, Laurence Aitchison, and Desi R Ivanova. Position: Don't use the clt in llm evals with fewer than a few hundred datapoints. arXiv preprint arXiv:2503.01747 , 2025. - source

      The Introduction (p. 1) discusses statistical uncertainty in benchmark evaluations, and §3.2 (p. 5) models variation across tasks and questions. This supports the dataset-side focus, but the paper does not establish that benchmark noise is frequently studied *rather than* variation arising during model training. (LLM, conf=0.80)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § A.1 Scaling Law Details · p. 16

    Recent work has begun using the downstream task as the prediction target.

    “Recent work has begun using the downstream task as the prediction target [17, 19]”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [17] [17] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. - source
    • cites [19] [19] Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540 , 2024. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § A.1 Scaling Law Details · p. 16

    Hoffmann et al. [24] models the improvement for larger model training budgets as a power function, proportional to the model parameters N and training tokens D.

    “Hoffmann et al. [24] models the improvement for larger model training budgets as a power function, proportional to the model parameters N and training tokens D , with the exact functional form and prediction setup varying between work [44].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [24] [24] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022. - source
    • cites [44] [44] Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws. arXiv preprint arXiv:2406.12907 , 2024. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 6 Related Work and Discussion · p. 10

    Madaan et al. [36] report a measure of SNR using the benchmark score of a single model and noise using 10 seed models, rather than a population of models.

    “Closest to our work is Madaan et al. [36], which report a measure of SNR using the benchmark score of a single model and noise using 10 seed models, rather than a population of models.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [36] [36] Lovish Madaan, Aaditya K Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation benchmarks. arXiv preprint arXiv:2406.10229 , 2024. - source

      The Table 2 caption (3.2 Results; page unavailable) says the SNR uses both the mean benchmark score and the standard deviation across the final checkpoints of 10 seed models. This supports the claim about noise from 10 seeds, but contradicts its description of the score as coming from a single model. Section 2.1 also says the paper uses a larger, diverse set of models for other analyses. (LLM, conf=0.89)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § 4.2 Tasks with higher noise also have higher scaling law error · p. 6

    Four tasks (MBPP+, SocialIQA, MMLU and TriviaQA) exhibit similar error (around 2-3%) but different amounts of noise around the prediction target.

    “we observe four tasks (MBPP+, SocialIQA, MMLU and TriviaQA) which exhibit similar error (around 2-3%), but exhibit different amounts of noise around the prediction target.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Assuming a scaling law with no bias, tasks with a lower standard deviation of the prediction target will also have a lower prediction error.

  • May combine two claims.

    Findingown data
    § 4.2 Tasks with higher noise also have higher scaling law error · p. 7

    Some individual tasks are easier to predict than multi-task averages, and have lower noise around the prediction target.

    “We find that some individual tasks are easier to predict than multi-task averages, and have lower noise around the prediction target.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Tasks with higher noise also have higher scaling law error · p. 7

    Generative tasks like TriviaQA or Jeopardy which evaluate exact match of short-form generation exhibit lower error than multi-task averages, and exhibit lower noise around the prediction target.

    “In particular, generative tasks like TriviaQA or Jeopardy which evaluate the exact match of a short-form generation exhibit lower error than the multi-task averages, and exhibit lower noise around the prediction target.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.1 Measuring Noise · p. 4

    Initialization noise, data order noise, and checkpoint-to-checkpoint noise across the whole training run all correlate highly with the relative standard deviation of the final n checkpoints (R^2 of 0.82, 0.86, and 0.95, respectively).

    “We find that the initialization noise, data order noise, and checkpoint-to-checkpoint noise across the whole training run all correlate highly with the relative standard deviation of the final n checkpoints ( R 2 of 0.82, 0.86, and 0.95, respectively, see Figure 7; and see the training curves in Figure 19).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.2 Measuring Signal · p. 5

    Signal alone and noise alone have low correlation with decision accuracy, while the signal-to-noise ratio is correlated with decision accuracy.

    “The signal alone (left) and noise alone (center) have low correlation with decision accuracy, while the signal -tonoise ratio (right) is correlated with decision accuracy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3 Quantifying Signal and Noise · p. 4

    Some tasks exhibit low noise between training checkpoints but low signal between models, and others exhibit high noise and high signal.

    “Some tasks (left, HellaSwag) exhibit low noise between training checkpoints but low signal between models, and others (center, ARC-Challenge) exhibit high noise and high signal .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.3 Measuring Signal -tonoise Ratio · p. 5

    The framework is designed to be independent of a particular metric: many other measures of signal produce similar results and measures of noise have high correlation.

    “we find many other measures of signal produce similar results in Appendix A.4 and measures of noise have high correlation in Appendix A.3.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § B.3 Signal-to-Noise Ratio at Large (>32B) Scales · p. 22

    ARC Easy and SocialIQA have high SNR at 1.5B-4T but low SNR at 32B-6T: 7.89 to 5.10 and 8.73 to 1.95 respectively.

    “for example ARC Easy and SocialIQA have high SNR at 1.5B-4T, but low SNR at 32B-6T: 7.89 to 5.10 and 8.73 to 1.95 respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Averaging checkpoint-to-checkpoint noise leads to better predictions · p. 9

    The BPB training curve is smoother, corresponding to a higher SNR and a lower error in the prediction settings.

    “Visually, the BPB training curve is smoother, corresponding to a higher SNR and a lower error in the prediction settings reported in the table”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.1 Filtering noisy sub-tasks improves signal -tonoise ratio · p. 8

    On small models used to make predictions, or to the large target models, or both, decision accuracy improves when averaging final k checkpoint scores.

    “Left : On small models used to make predictions ('Avg. Pred.'), or to the large target models ('Avg. Target'), or both ('Avg. Both'), decision accuracy improves.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § B.3 Signal-to-Noise Ratio at Large (>32B) Scales · p. 22

    Minerva MATH 500 has the lowest SNR of all tasks at 1.5B-4T (SNR = 0.91) but much higher SNR already at 7B-4T (SNR = 4.45).

    “For example, Minerva MATH 500 has the lowest SNR of all tasks at 1.5B-4T (SNR = 0 . 91 ) but much higher SNR already at 7B-4T (SNR = 4 . 45 ).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.1 Filtering noisy sub-tasks improves signal -tonoise ratio · p. 7

    Using the high SNR subset compared to the full benchmark led to higher decision accuracy: +2.6% for MMLU and +5% for AutoBencher.

    “The lower signal -tonoise ratio also led to a higher decision accuracy: +2.6% for MMLU and +5% for AutoBencher by using the high SNR subset compared to the full benchmark.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Many tasks are a macro-average of subtasks.

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § A.5.2 Benchmarks · p. 20

    Small (≤1B parameter) models show random-chance performance on MCQA benchmarks.

    “since our small ( ≤ 1B parameter) models show random-chance performance on MCQA benchmarks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § B.1 Noise measures the reliability of decision accuracy. · p. 22

    For tasks with a higher signal-to-noise ratio, the sampled decision accuracy distribution has a higher mean and lower variance.

    “For tasks with a higher signal -tonoise ratio, the sampled decision accuracy distribution has a higher mean and lower variance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 3.2 Measuring Signal · p. 5

    The signal-to-noise ratio gives information about whether a benchmark is useful during development, as high decision accuracy and high signal-to-noise ratio mean development decisions made at a small scale generalize to large scale models.

    “The signal -tonoise ratio gives us information about wether a benchmark is useful during development, as high decision accuracy (and signal -tonoise ratio) means development decisions made at a small scale generalize to large scale models.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: A benchmark is most useful during language model development if it can detect a true difference between a good model and a poor model.

  • May combine two claims.

    Interpretationown data
    § 4.1 Higher signal -tonoise ratio indicates higher decision accuracy · p. 6

    Benchmarks which have higher SNR at small scales exhibit higher decision accuracy, and are more likely for their results to hold at a larger scale.

    “We conclude that benchmarks which have higher SNR at small scales exhibit higher decision accuracy, and are more likely for their results to hold at a larger scale.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § Abstract · p. 1

    Switching to a metric that has better signal and noise (e.g., perplexity rather than accuracy) leads to better reliability and improved scaling law error.

    “For example, we propose that switching to a metric that has better signal and noise (e.g., perplexity rather than accuracy) leads to better reliability and improved scaling law error.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data
    § 4.2 Tasks with higher noise also have higher scaling law error · p. 6

    Noise measures the reliability of scaling law prediction errors.

    “Noise measures the reliability of scaling law prediction errors.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Assuming a scaling law with no bias, tasks with a lower standard deviation of the prediction target will also have a lower prediction error.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data
    § 4.2 Tasks with higher noise also have higher scaling law error · p. 7

    Using individual tasks may be a better decision in some cases than the multi-task average, if that task better represents the ability than a multi-task average.

    “For practitioners, we argue using individual tasks may be a better decision in some cases than the multi-task average, if that task better represents the ability than a multi-task average.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data
    § 3.1 Measuring Noise · p. 4

    Noise can be defined as the relative standard deviation of the final n checkpoints.

    “These results lead us to define noise as the relative standard deviation of the final n checkpoints, as this requires no additional training cost and only uses the final n checkpoints rather than the full training curve.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 6 Related Work and Discussion · p. 10

    EvalArena [63] reports a measure of SNR using the final checkpoints of a small/large model pair.

    “Finally, EvalArena [63] also reports a measure of SNR using the final checkpoints of a small/large model pair (e.g., Llama 3 7B vs. 70B).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [63] [63] Sida I. Wang, Alex Gu, Lovish Madaan, Dieuwke Hupkes, Jiawei Liu, Yuxiang Wei, Naman Jain, Yuhang Lai, Sten Sootla, Ofir Press, Baptiste Rozière, and Gabriel Synnaeve. Eval-Arena: noise and errors on llm evaluations. https://github.com/crux-eval/eval-arena , 2024. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data
    § 5.3 Measuring bits-per-byte improves benchmark signal -tonoise ratio · p. 10

    BPB is a useful metric and a higher quality development benchmark, particularly for challenging tasks at small scales that do not show above random-chance signal.

    “Our results confirm that BPB is a useful metric is both a higher quality development benchmark, particularly for challenging tasks at small scales that do not show above random-chance signal .”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § 4.2 Tasks with higher noise also have higher scaling law error · p. 7

    SNR can be calculated at any model size.

    “However, SNR can be calculated at any model size, so we show how the signal -tonoise ratio changes for tasks at larger 1B, 7B, 13B and 32B scales in Appendix B.3.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § 4.2 Tasks with higher noise also have higher scaling law error · p. 6

    Thirty checkpoints were found to be an adequate trade-off between sample size and compute cost.

    “We found 30 checkpoints to be an adequate trade-off between sample size and compute cost.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § A.3 Measures of Modeling Noise · p. 17

    Calculating the above sources of noise are either too expensive to estimate at large scales or difficult to run.

    “Calculating the above sources of noise are either too expensive to estimate at large scales (e.g., training LLMs by varying the random seed) or difficult to run (e.g., evaluating every checkpoint on an LLM training curve).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 6 Related Work and Discussion · p. 10

    The scope of the connection between SNR and predicting large scale phenomena is limited to the two decision accuracy and prediction error settings, and only studies the noise of the model during training.

    “Notably, this scope of our connection between the signal -tonoise ratio and predicting large scale phenomena is limited to the two decision accuracy and prediction error settings, and only studies the noise of the model during training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning
    § 5.2 Averaging checkpoint-to-checkpoint noise leads to better predictions · p. 8

    The measure of noise is between intermediate training checkpoints, so we are only reducing one of many sources of modeling noise.

    “Note, as our measure of noise is between intermediate training checkpoints, we are only reducing one of many sources of modeling noise .”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Typically, models are only compared using the evaluation of the final checkpoint.

  • May combine two claims.

    Premisereasoning
    § B.1 Noise measures the reliability of decision accuracy. · p. 21

    Checkpoint-to-checkpoint noise can change the ranking of models, which may affect the decision accuracy observed by only evaluating the final DataDecide model.

    “As discussed in §3.1, the checkpoint-to-checkpoint noise can change the ranking of models, which may effect the decision accuracy we observe by only evaluating the final DataDecide model.”
    Quote in the paperSubstantial claim
36 of 36

  • The paper relies on this without support.

    Many tasks are a macro-average of subtasks.

    “Many tasks are a macro-average of subtasks.”

  • The authors state this as an assumption.

    Assuming a scaling law with no bias, tasks with a lower standard deviation of the prediction target will also have a lower prediction error.

    “Assuming a scaling law with no bias, we expect tasks with a lower standard deviation of the prediction target to also have a lower prediction error.”

  • The paper relies on this without support.

    Training multiple large models is too expensive for most development scenarios.

    “Training multiple large models is too expensive for most development scenarios, but is necessary to evaluate how accurate this process is.”

  • The paper relies on this without support.

    A benchmark is most useful during language model development if it can detect a true difference between a good model and a poor model.

    “A benchmark is most useful during language model development if it can detect a true difference between a good model and a poor model, assuming a true difference exists between the models in the ability that the benchmark aims to measure.”

  • The paper relies on this without support.

    Typically, models are only compared using the evaluation of the final checkpoint.

    “Typically, models are only compared using the evaluation of the final checkpoint.”

  • The authors state this as an assumption.

    Checkpoint-to-checkpoint scores are independent and normally distributed.

    “We first assume the checkpoint to checkpoint scores are independent and normally distributed (which we observe when computing decision accuracy on intermediate checkpoints in Figure 7).”

  • The authors state this as an assumption.

    The 30 intermediate checkpoints provide a reasonable estimate of the population standard deviation.

    “If we assume the 30 intermediate checkpoints provide a reasonable estimate of the population standard deviation”
7 of 7

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Subtask filtering is described as both raising and lowering SNR

    Minor

    The results say the selected subsets have higher SNR than the full benchmarks, but the next sentence attributes their higher decision accuracy to a lower SNR.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 3

  1. [2][2] Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. An empirical investigation of statistical significance in NLP. In Jun'ichi Tsujii, James Henderson, and Marius Pa¸ sca, editors, Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning , pages 995-1005, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/D12-1091/ . -
  2. [13][13] Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research , 23(226):1-61, 2022. -
  3. [31][31] Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems , 37:14200-14282, 2024. -

Cited by

Sorted by citations

Authors
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J.
Affiliations
  • Allen Institute for Artificial Intelligence (US)
  • University of Washington (US)
  • New York University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
19286-19327
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2508.13144
arXiv
2508.13144
OpenAlex
W4414528631
Semantic Scholar
3f0c849509de91ddf09835c9a09c5887830e03e6
Cited by
0
References
0
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
LLM evaluation, signal-to-noise ratio, multi-task evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review