Quantifying Variance in Evaluation Benchmarks

“Polo et al. (2024) show that methods from human standardised testing (specifically, item response theory; Lord & Novick, 1968) can be combined with clustering to subselect evaluation benchmarks without incurring too much deviation from the mean.”

“…we also experimented with various techniques used in human standardised testing (item analysis; University of Washington (2024), item response theory; Cai et al. (2016)), but generally found these methods to be ineffective on the models and benchmarks we considered, in terms of reducing variance.”

“…al., 2011), GSM8k (Cobbe et al., 2021), Hellaswag (Zellers et al., 2019), HumanEval (Chen et al., 2021), MATH (Hendrycks et al., 2021), MMLU (Hendrycks et al., 2020), Natural Questions (Kwiatkowski et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), and TriviaQA (Joshi et al., 2017).”

“However, for some benchmarks, such as HellaSwag and TriviaQA, the difference is minimal, likely since these benchmarks saturate earlier in training.”

“We also show the comparison of model performance development for two different continuous metrics, char-length normalized NLL and token-length normalized NLL for GSM8k and HumanEval in Figure 8.”

“Motivated by prior work which suggests a move to continuous metrics (Srivastava et al., 2022; Schaeffer et al., 2023; Du et al., 2024; Schaeffer et al., 2024), we show a comparison of discrete and continuous metrics along with their signal to noise ratios (SNR = µ ( S , M 210 B ) σ ( S , M 210 B )…”

“In Tables 3 and 4, we report various metrics on the discrete performance measure for GSM8k, Hellaswag, and ARC-C.”

“In Tables 3 and 4, we report various metrics on the discrete performance measure for GSM8k, Hellaswag, and ARC-C.”

“…large-scale well-established NLP benchmarks: AGIEval (Zhong et al., 2023), AI2 Reasoning Challenge (ARC-C) (Clark et al., 2018), BIG Bench (Hard) (Srivastava et al., 2022; Suzgun et al., 2022), COPA (Roemmele et al., 2011), GSM8k (Cobbe et al., 2021), Hellaswag (Zellers et al., 2019), HumanEval…”

“Finally, we use 32 publicly available models from Huggingface (Wolf et al., 2020): Meta-Llama-3 {8, 70}B (Dubey et al., 2024), Gemma {2, 7}B (Mesnard et al., 2024), DBRX-Base (Databricks, 2024), Mistral 7B (Jiang et al., 2023), Mixtral 8x{7, 22}B (Jiang et al., 2024), Qwen-1.”

“We find that simply using the performance on the 100 datapoints selected by Polo et al. (2024) for each benchmark can lead to quite large deviations in the mean (an overestimation by 7% for ARC-C).”

“Finally, we use 32 publicly available models from Huggingface (Wolf et al., 2020): Meta-Llama-3 {8, 70}B (Dubey et al., 2024), Gemma {2, 7}B (Mesnard et al., 2024), DBRX-Base (Databricks, 2024), Mistral 7B (Jiang et al., 2023), Mixtral 8x{7, 22}B (Jiang et al., 2024), Qwen-1.”

“…et al., 2022), COPA (Roemmele et al., 2011), GSM8k (Cobbe et al., 2021), Hellaswag (Zellers et al., 2019), HumanEval (Chen et al., 2021), MATH (Hendrycks et al., 2021), MMLU (Hendrycks et al., 2020), Natural Questions (Kwiatkowski et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al.,…”

“Virtually any new model release is accompanied by a range of scores on common evaluation benchmarks, illustrating how the model tallies up against previous releases (Mesnard et al., 2024; AI@Meta, 2024; Achiam et al., 2023; Reid et al., 2024).”

“We make suggestions of how variance can be reduced for smaller scale models on choice tasks of important value (MMLU).”

“Finally, we use 32 publicly available models from Huggingface (Wolf et al., 2020): Meta-Llama-3 {8, 70}B (Dubey et al., 2024), Gemma {2, 7}B (Mesnard et al., 2024), DBRX-Base (Databricks, 2024), Mistral 7B (Jiang et al., 2023), Mixtral 8x{7, 22}B (Jiang et al., 2024), Qwen-1.”

“For our initial variance analysis, we use both benchmark-level scores (to compute variance and monotonicity) and sample level scores (to estimate 95% confidence intervals).”

“More recently, with accelerating progress in NLP, researchers have begun to study benchmarks in earnest to understand their properties and limitations (Gehrmann et al., 2023).”

“Kocmi et al. (2021) further leveraged large-scale human experiments to evaluate benchmarks with automated metrics and concluded that commonly used metrics such as BLEU score had led to poor deployment decisions.”

“5 {0.5, 1.8, 4, 7, 14, 32, 72, 110}B (Bai et al., 2023), Pythia {1, 1.4, 2.8, 6.9, 12}B (Biderman et al., 2023), Falcon {7, 40}B (Almazrouei et al., 2023), DeepSeek {7, 67}B (Bi et al., 2024), DeepSeek-MoE 16B (Bi et al., 2024), DeepSeek V2 (DeepSeek-AI, 2024), StableLM {1.6, 3, 7}B (StabilityAI,…”

“…noisier (and less monotonic), showing the increased difficulty pracitioners may have in interpreting results if using IRT-based methods. from human standardised testing literature (item analysis; Livingston, 2011), as well as showing that such methods generally do not meaningfully reduce variance.”

“…2023), Pythia {1, 1.4, 2.8, 6.9, 12}B (Biderman et al., 2023), Falcon {7, 40}B (Almazrouei et al., 2023), DeepSeek {7, 67}B (Bi et al., 2024), DeepSeek-MoE 16B (Bi et al., 2024), DeepSeek V2 (DeepSeek-AI, 2024), StableLM {1.6, 3, 7}B (StabilityAI, 2024), and MPT {7, 30}B (MosaicML NLP Team, 2023).”

“In a similar category to item analysis, item response theory (Cai et al., 2016; van der Linden, 2018; Brzezi ´ nska, 2020; Lord & Novick, 1968) describes a set of statistical models used to analyse human abilities on standardised test data.”

“Before the era of chat large language models, Marie et al. (2021) conducted a large scale meta-analysis of 769 research papers published from 2010 to 2020 and identified troubling trends, including one that partially motivates our work: models are frequently declared superior to competitors based…”

“In the recent past, the method has become popular as a means of understanding model performance on a set of evaluation samples (Lalor et al., 2016; Vania et al., 2021; Rodriguez et al., 2021).”

“If we instead consider item discriminations on a random split of models (Figure 3, third column), we see stronger correlations, indicating that the low correlation is in fact due to the difference in item discrimination on weaker and stronger models.”

“While it is well known that benchmarks scores can be heavily influenced by the choice of prompt (Sclar et al., 2023), the distributions of labels in the provided few-shots (Weber et al., 2023) or even the symbols that are used for the different options in a multiple choice setup (Zheng et al.,…”

“Spiritually similar claims were made by Dehghani et al. (2021) in their provocatively titled paper “The Benchmark Lottery”.”

“In the recent past, the method has become popular as a means of understanding model performance on a set of evaluation samples (Lalor et al., 2016; Vania et al., 2021; Rodriguez et al., 2021).”

“…NLP benchmarks: AGIEval (Zhong et al., 2023), AI2 Reasoning Challenge (ARC-C) (Clark et al., 2018), BIG Bench (Hard) (Srivastava et al., 2022; Suzgun et al., 2022), COPA (Roemmele et al., 2011), GSM8k (Cobbe et al., 2021), Hellaswag (Zellers et al., 2019), HumanEval (Chen et al., 2021), MATH…”

“…benchmarks scores can be heavily influenced by the choice of prompt (Sclar et al., 2023), the distributions of labels in the provided few-shots (Weber et al., 2023) or even the symbols that are used for the different options in a multiple choice setup (Zheng et al., 2023; Alzahrani et al.,…”

“Von Werra et al. (2022) proposed a framework to evaluate benchmarks themselves and provided a mechanism for researchers to share their benchmarking analyses.”

“…al., 2011), GSM8k (Cobbe et al., 2021), Hellaswag (Zellers et al., 2019), HumanEval (Chen et al., 2021), MATH (Hendrycks et al., 2021), MMLU (Hendrycks et al., 2020), Natural Questions (Kwiatkowski et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), and TriviaQA (Joshi et al., 2017).”

“On the other hand, in line with recent work advocating for a teleological approach to measuring capabilities (McCoy et al., 2023), we demonstrated LLM-specific techniques (e.g. the use of continuous metrics or cloze-formatted tasks) can improve the signal-to-noise ratio in our evals.”

“…scores can be heavily influenced by the choice of prompt (Sclar et al., 2023), the distributions of labels in the provided few-shots (Weber et al., 2023) or even the symbols that are used for the different options in a multiple choice setup (Zheng et al., 2023; Alzahrani et al., 2024), papers…”

“In the recent past, the method has become popular as a means of understanding model performance on a set of evaluation samples (Lalor et al., 2016; Vania et al., 2021; Rodriguez et al., 2021).”

“Item difficulty refers to the average score on an item across models; Item discrimination refers to the correlation between models’ performances on a single data point and models’ overall performances.”

“Motivated by prior work considering the inconsistency of multiple choice benchmarks (Wang et al., 2024; Alzahrani et al., 2024), we examined two formulations of MMLU: (Standard) MMLU and MMLU-Cloze.”

“To understand why MMLU-Cloze works better in the earlier stages (§ 3.3) whereas the final MMLU performance is higher during the later stages, we train a Llama-2-13 B-like model from scratch on our pre-training mix.”

“In this section, we present our comprehensive analysis for two scenarios.”

“Perhaps most similar to ours is the work of Xiang et al. (2022), who study different sources of variance in NLP benchmarks and offers cautionary advice about when one should (not) be confident in benchmark scores.”

“With the aims of improving efficiency in model development cycles, recent work proposes reducing the size of evaluation benchmarks by picking representative samples (Vivek et al., 2023; Polo et al., 2024).”

“…of labels in the provided few-shots (Weber et al., 2023) or even the symbols that are used for the different options in a multiple choice setup (Zheng et al., 2023; Alzahrani et al., 2024), papers rarely report more than a single number per benchmark, or specifics on how each number was computed.”

“Motivated by prior work which suggests a move to continuous metrics (Srivastava et al., 2022; Schaeffer et al., 2023; Du et al., 2024; Schaeffer et al., 2024), we show a comparison of discrete and continuous metrics along with their signal to noise ratios (SNR = µ ( S , M 210 B ) σ ( S , M 210 B )…”

“These results may thus help in building better scaling laws for downstream evaluation tasks (Achiam et al., 2023), along with accurate comparisons between two models that have performances lying within the confidence interval for the discrete metric.”

“[15] Chatbot Arena[4] has popularized the use of confidence intervals in its Elo scores, but error bars remain noticeably absent from traditional question-and-answer evals.”

“0-point pass@1 swings on SWE-bench Verified, often the magnitude of a single accepted edit, motivating the statistical reporting and consistency metrics this regime demands [30, 32, 52] and echoing earlier corrections in reinforcement learning and architecture search [1, 26].”

“Methods that also account for which tasks were chosen give wider intervals [Madaan et al., 2024].”

“Madaan et al. (2024) propose methods for quantifying and understanding variance in evaluation benchmarks.”