CoChat
Sign Up
PreprintCited by 17Topic ModelingNatural Language Processing TechniquesAdversarial Robustness in Machine Learning

tinyBenchmarks: evaluating LLMs with fewer examples

Open access
Polo, Felipe Maia, Weber, Lucas, Choshen, Leshem, Sun, Yuekai, Xu, Gongjun, Yurochkin, Mikhail·2024·arXiv
DOI 10.48550/arxiv.2402.14992arXiv 2402.14992
Open this paper in CoChat — free accountOpen PDF

Abstract

The versatility of large language models (LLMs) led to the creation of diverse benchmarks that thoroughly test a variety of language models' abilities. These benchmarks consist of tens of thousands of examples making evaluation of LLMs very expensive. In this paper, we investigate strategies to reduce the number of evaluations needed to assess the performance of an LLM on several key benchmarks. For example, we show that to accurately estimate the performance of an LLM on MMLU, a popular multiple-choice QA benchmark consisting of 14K examples, it is sufficient to evaluate this LLM on 100 curated examples. We release evaluation tools and tiny versions of popular benchmarks: Open LLM Leaderboard, MMLU, HELM, and AlpacaEval 2.0. Our empirical analysis demonstrates that these tools and tiny benchmarks are sufficient to reliably and efficiently reproduce the original evaluation results.

AI snapshot

Written from the paper’s full text.

TL;DR

tinyBenchmarks shows that evaluating LLMs on 100 curated examples per scenario can estimate full benchmark performance within about 2% average error, reducing costs by up to two orders of magnitude. It releases tiny versions of Open LLM Leaderboard, MMLU, HELM, and AlpacaEval 2.0 plus IRT-based tools.

Key findings

  • On MMLU (14K examples), 100 curated examples suffice for average estimation error under 2%, reducing evaluation cost by a factor of 140; for Open LLM Leaderboard, the text states even 30 examples per scenario are enough, reducing cost by a factor of 160 (from 29K to 180).
  • Across benchmarks, 100 examples per scenario achieves ≈2% average error: 600/29K for Open LLM Leaderboard, 100/14K for MMLU, 1000/10K for HELM, and 100/800 for AlpacaEval 2.0.
  • IRT-based methods (“IRT” and “IRT++”) perform consistently well across benchmarks and train-test splits; gp-IRT (“++”) always improves or matches its vanilla counterpart and adds only a few seconds.
  • Correctness-based anchors deteriorate on specialized LLMs, while IRT-based anchors are only slightly affected; on MMLU with 100 examples, estimation error never exceeds 4% except for one extremely low-performing LLM.

Methods

  • Evaluated strategies for selecting small example subsets: stratified random sampling, clustering by model correctness, and clustering IRT example representations; each with vanilla and “++” IRT-adjusted estimates.
  • Used publicly available correctness data from previously evaluated LLMs: Open LLM Leaderboard (395 LLMs, ~29K examples, 6 scenarios), MMLU (57 subjects, ~14K examples), HELM Lite v1.0.0 (30 models, ~10K examples, 10 scenarios), and AlpacaEval 2.0 (100 LLMs, 805 examples).
  • Train/test model splits were random or by date (HELM by organization with 11-fold cross-validation; AlpacaEval 4-fold cross-validation); IRT fit via variational inference with dimensions {2, 5, 10, 15}; results averaged over 5 restarts.

Limitations

  • Stated: severe distribution shifts can increase errors, e.g., models that fail simple questions but answer complicated ones correctly, architecture or pre-training changes, or rapid capability increases causing extrapolation errors; authors recommend periodically updating curated examples and IRT parameter estimates.
  • Stated: p-IRT does not promptly allow sample weighting and can deteriorate if predicted probabilities are inaccurate due to model misspecification; adaptive testing implementation takes over 5 minutes.
  • Evident from text: benchmark-specific results and released tiny versions depend on the available historical LLM correctness data and train-test splits, so performance may vary for future models or benchmarks not covered.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.14992

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

10 of 20 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    On MMLU, comparing accuracy estimates from evaluating LLMs on a curated subset of 100 examples to accuracy on all of MMLU achieves average estimation error under 2%.

    “In Figure 1 we demonstrate the efficacy of our best evaluation strategy on MMLU, where we compare accuracy estimates obtained from evaluating LLMs on a curated subset of 100 examples (less than 1% of the examples) to accuracy on all of MMLU, achieving average estimation error under 2%.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: We have access to correctness evaluations for a set of LLMs that have been previously evaluated on the entirety of the benchmark.
  • Findingreasoning

    For MMLU, using 100 examples reduces the evaluation cost by a factor of 140 (from 14k to 100).

    “For example, for MMLU this reduces the evaluation cost by a factor of 140 (from 14k to 100).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: We have access to correctness evaluations for a set of LLMs that have been previously evaluated on the entirety of the benchmark.
  • Findingreasoning

    Clustering IRT representations of examples allows finding a more robust evaluation set.

    “Clustering these representations allows us to find a more robust evaluation set.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The released evaluation tools and tiny versions of Open LLM Leaderboard, MMLU, HELM, and AlpacaEval 2.0 are sufficient to reliably and efficiently reproduce the original evaluation results.

    “Our empirical analysis demonstrates that these tools and tiny benchmarks are sufficient to reliably and efficiently reproduce the original evaluation results 1 .”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: We have access to correctness evaluations for a set of LLMs that have been previously evaluated on the entirety of the benchmark.
  • Findingown data

    To accurately estimate the performance of an LLM on MMLU, it is sufficient to evaluate this LLM on 100 curated examples.

    “For example, we show that to accurately estimate the performance of an LLM on MMLU, a popular multiple-choice QA benchmark consisting of 14K examples, it is sufficient to evaluate this LLM on 100 curated examples.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: We have access to correctness evaluations for a set of LLMs that have been previously evaluated on the entirety of the benchmark.
  • Classified as probably not a substantial claim. 19
  • May combine two claims. 23
  • The quote only partly states this claim. 2
  • Not yet classified. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Vivek et al. proposed clustering evaluation examples based on models' confidence in the correct class for faster evaluation on classification tasks.

    “Vivek et al. (2023) proposed clustering evaluation examples based on models' confidence in the correct class for faster evaluation on classification tasks.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Vivek et al. (2023) source

  • May combine two claims.

    Findingcitation

    Stratified random sampling is the simplest to use but can result in a large estimation error.

    “Stratified random sampling as proposed by Perlitz et al. (2023) for HELM.This approach is the simplest to use but can result in a large estimation error.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Perlitz et al. (2023) source

      Section 5.4 describes uniform sampling of example–prompt pairs for HELM, and Section 5.3 assesses errors when using fewer examples. Neither identifies a method as stratified random sampling or assesses whether that method is the simplest to use and can produce large estimation errors. Searches for “stratified” and “estimation error” found no passage settling the claim. (LLM, conf=0.72)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingcitation

    Clustering examples based on LLMs that have already been evaluated performs well in some settings but can be unreliable when correctness patterns are spurious.

    “Clustering examples based on LLMs that have already been evaluated.The key idea is to find examples where (in)correct prediction of an LLM implies that it will also be (in)correct on a subset of other examples.This method performs well in some settings but can be unreliable when such correctness patterns are spurious, e.g., when predicting the accuracy of an LLM specialized to a domain.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Vivek et al. (2023) source

      The Introduction (page unavailable) describes selecting examples using correlations in previously evaluated models’ predictions, and the Multitask Language Understanding section reports good results on several datasets but poor results on Global Facts, where learned correlations may be spurious. The paper does not identify predicting the accuracy of a domain-specialized LLM as an example of this failure. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Lalor et al. propose using IRT's latent variables to measure language model abilities.

    “For instance, Lalor et al. (2016) propose using IRT's latent variables to measure language model abilities”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Lalor et al. (2016) source

  • May combine two claims.

    Premisecitation

    Evaluation of a single model is often performed many times to monitor checkpoints during pre-training and to explore different prompting strategies or a wider range of hyperparameters.

    “Furthermore, evaluation of a single model is often performed many times to monitor checkpoints during pre-training (Biderman et al., 2023a;Liu et al., 2023) and to explore different prompting strategies or a wider range of hyperparameters (Weber et al., 2023b;Mizrahi et al., 2023;Sclar et al., 2023;Voronov et al., 2024).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Biderman et al., 2023a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Liu et al., 2023 source
    • cites Weber et al., 2023b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Vania et al. employs IRT models in language model benchmarking to study saturation of commonly used benchmarks.

    “Vania et al. (2021) employs IRT models in the context of language models benchmarking to study saturation (un-discriminability) of commonly used benchmarks”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Vania et al. (2021) source

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Perlitz et al. found that evaluation on HELM relies on diversity across datasets, but the number of examples currently used is excessive.

    “Perlitz et al. (2023) found that evaluation on HELM (Liang et al., 2022) relies on diversity across datasets, but the number of examples currently used is excessive.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Perlitz et al. (2023) source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Rodriguez et al. suggest IRT models can be reliably used to predict responses of LLMs in unseen items, categorize items, and rank models.

    “Rodriguez et al. (2021) study several applications of IRT in the context of language models, suggesting that IRT models can be reliably used to: predict responses of LLMs in unseen items, categorize items (e.g., according to their difficulty/discriminability), and rank models.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Rodriguez et al. (2021) source

      Sections 4.2.2–4.2.3 report accurate prediction of held-out responses from SQuAD models, §5.1 groups items by difficulty and discriminability, and §4.3.1 finds IRT rankings more reliable than accuracy rankings. But the prediction experiment holds out responses, not entirely unseen items, and the paper does not establish reliability for LLMs generally; §8 says its IRT models do not incorporate item content to predict responses. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Zhuang et al. used IRT for adaptive testing, making testing more efficient, but do not propose a performance estimator for LLMs, only rank models based on ability parameters.

    “More recently, Zhuang et al. (2023) used IRT for adaptive testing, making testing more efficient.However, the authors do not propose a performance estimator for LLMs but only rank models based on their ability parameters.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zhuang et al. (2023) source

      In “Comparison of Rankings with Full Dataset” (page unavailable), the paper reports that IRT-based adaptive testing achieves strong ranking consistency using about 60% of the items, supporting the efficiency and ranking portions of the claim. But “Hard Easy” (page unavailable) specifies a procedure for estimating each LLM’s ability and provides the final estimated ability as the assessment result; the work does not only rank models. The cited passages do not show a separate estimator of overall benchmark performance. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MMLU is a multiple choice QA scenario consisting of 57 subjects comprising approximately 14K examples.

    “MMLU (Hendrycks et al., 2020) is a multiple choice QA scenario consisting of 57 subjects (subscenarios) comprising approx.14K examples.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Hendrycks et al., 2020 source

      Section 3 (p. 3) describes a multiple-choice test with 57 tasks. It reports 14,079 test questions, but 15,908 questions in total; approximately 14K describes the test set, not the entire dataset. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingreasoning

    This corresponds to 600 out of 29K examples for Open LLM Leaderboard, 100 out of 14K examples for MMLU, 1000 out of 10K examples for HELM, and 100 out of 800 examples for AlpacaEval 2.0.

    “This corresponds to 600 out of 29K examples for Open LLM Leaderboard, 100 out of 14K examples for MMLU, 1000 out of 10K examples for HELM, and 100 out of 800 examples for AlpacaEval 2.0.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The IRT++ strategy can estimate the performance of more capable LLMs slightly better, although there is no strong dependency.

    “Our strategy can estimate the performance of more capable LLMs slightly better, although there is no strong dependency.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: We have access to correctness evaluations for a set of LLMs that have been previously evaluated on the entirety of the benchmark.

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    The error tends to be slightly lower for more capable models on MMLU.

    “The error tends to be slightly lower for more capable models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The current adaptive IRT implementation takes over 5 minutes to run.

    “Although the estimation performance has improved, our current implementation takes over 5 minutes to run, which might not be as appealing practically.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    There are scenarios in which our methods shine more than others.

    “It is clear from the following plots that there are scenarios in which our methods shine more than others.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The weight of the subjects in tinyMMLU is fairly different from its correctness-based counterpart.

    “Interestingly the weight of the subjects is fairly different from its correctness-based counterpart.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    For Open LLM Leaderboard, 30 examples per scenario is enough, reducing the evaluation cost by a factor of 160 (from 29K to 180).

    “For Open LLM Leaderboard even 30 examples per scenario is enough, reducing the evaluation cost by a factor of 160 (from 29K to 180).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: We have access to correctness evaluations for a set of LLMs that have been previously evaluated on the entirety of the benchmark.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Assessing the performance of LLMs on a randomly sampled subset of examples suffers from extra uncertainty in the sampling process, especially when the number of sampled examples is small.

    “Assessing the performance of LLM's on a randomly sampled subset of examples suffers from extra uncertainty in the sampling process, especially when the number of sampled examples is small.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Benchmarks consist of tens of thousands of examples, making evaluation of LLMs very expensive.

    “These benchmarks consist of tens of thousands of examples making evaluation of LLMs very expensive.”
    • Also stated in: § 1. Introduction · plain
      “These benchmarks are comprised of hundreds or thousands of examples, making the evaluation of modern LLMs with billions of parameters computationally, environmentally, and financially very costly.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Liang et al. (2022) source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    IRT is a well-established set of statistical models used in psychometrics to measure the latent abilities of individuals through standardized testing.

    “IRT (Cai et al., 2016;Van der Linden, 2018;Brzezińska, 2020;Lord et al., 1968) is a wellestablished set of statistical models used in psychometrics to measure the latent abilities of individuals through standardized testing (An & Yung, 2014;Kingston & Dorans, 1982;Petersen et al., 1982), e.g., in GRE, SAT, etc..”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lord et al., 1968

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Conditional on the training set, the variance of p-IRT is small, especially when θ_l is fitted with data from many scenarios, but its bias can be high when the IRT model is misspecified and does not vanish with growing sample size.

    “On the other hand, conditional on the training set, the variance of Ẑp-IRT jl is small, especially when θ l is fitted with data from many scenarios, but its bias can be high when the IRT model is misspecified and does not vanish with the growing sample size.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The IRT model is reasonably specified.

  • May combine two claims.

    Interpretationreasoning

    Conditional on the training set, p-IRT has low variance when θ_l is obtained from a large dataset and a small bias if the IRT model is reasonably specified.

    “Conditional on the training set, the estimator p-IRT has low variance when θ l is obtained from a large dataset and a small bias if the IRT model is reasonably specified.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The IRT model is reasonably specified.

  • May combine two claims.

    Interpretationreasoning

    The correctness-based clustering method can suffer from distribution shifts and from the curse of dimensionality when |L_tr| is big.

    “This method is compelling and simple in detecting anchor points.Still, it can suffer from distribution shifts since correctness patterns can vary, e.g., in time, and from the curse of dimensionality when |L tr | is big.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Interpretationreasoning

    Under assumptions that θ_l converges in probability to θ_l as |I|→∞ and that true (α_i, β_i) are known and sup_i ||α_i||_2 ≤ c, p-IRT converges in probability to the best approximation of Z_jl.

    “Proposition 4.1.Assuming that (i) θl → θ l in probability as | I| → ∞ and that (ii) the true values of (α i , β i )'s for all i ∈ I are known and sup i∈I ∥α i ∥ 2 ≤ c for a universal constant c, we have that | E[Z jl | Y i0l , • • • , Y i k l ] -E[Z jl | Y i0l , • • • , Y i k l ]| → 0 in probability as | I| → ∞.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: θ_l converges in probability to θ_l as |I| → ∞.
    • The authors state this as an assumption: The true values of (α_i, β_i) for all i ∈ I are known.
    • The authors state this as an assumption: sup_i∈I ||α_i||_2 ≤ c for a universal constant c.

  • May combine two claims.

    Interpretationreasoning

    Using random sampling or anchor points implies low bias but potentially high variance when I' is small; as I' grows, its variance decreases.

    “To choose λ, we first note that using random sampling (or anchor points) implies low bias but potentially high variance (when I j is small) for i∈ Ij w i Y il .As I j grows, its variance decreases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Using IRT representation of examples solves the dimensionality problem and potentially alleviates the distribution shift problem if the IRT model reasonably describes reality and the example representations are stable.

    “This approach immediately solves the dimensionality problem, since E i is relatively low-dimensional 3 , and potentially alleviates the distribution shift problem if the IRT model reasonably describes the reality and the example representations are stable.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The IRT model reasonably describes reality and the example representations are stable.

  • May combine two claims.

    Interpretationnone given

    In the gp-IRT construction, IRT bias and sample size |Ij| are both positively related to the weight given to the raw data in performance estimation.

    “Understanding the effect of IRT bias and sample size | Ij| in the gp-IRT construction: both quantities are positively related to the weight we give to the raw data in performance estimation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The use of tinyBenchmarks leads directly to savings in monetary costs associated with evaluating LLMs, as well as computational and environmental costs.

    “This leads directly to savings in terms of the monetary costs associated with evaluating LLMs, but also the computational and environmental costs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The performance of model l can be represented as a weighted average instead of a simple average.

    “This tells us that we can represent the performance of model l as a weighted average instead of a simple average.”
    Quote in the paperNot a substantial claim

  • Not yet classified.

    Interpretation

    The IRT model creates a meaningful representation for each example based on its difficulty and the abilities required to respond correctly.

    “The IRT model creates a meaningful representation for each example i based on their difficulty and the abilities required to respond to those examples correctly.”
    Quote in the paperNot yet classified
    • The authors state this as an assumption: The two-parameter multidimensional IRT model correctly specifies the probability of an LLM getting an example correctly as a logistic function of θ_l, α_i, and β_i.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    For an effective evaluation, sub-scenarios should be equally represented.

    “The rationale behind this method is that, for an effective evaluation, sub-scenarios should be equally represented.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The approaches for reducing evaluation costs remain practical when evaluating newer, more capable LLMs and can help save GPU hours when evaluating future LLMs and/or checkpoints during pre-training.

    “Thus our approaches for reducing evaluation costs remain practical when evaluating the performance of newer, more capable LLMs and can help save GPU hours when evaluating future LLMs and/or checkpoints during pre-training.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Open LLM Leaderboard consists of 6 scenarios, approximately 29K examples in total.

    “HuggingFace's Open LLM Leaderboard (Beeching et al., 2023) consists of 6 scenarios, approx.29K examples in total.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Beeching et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    AlpacaEval 2.0 consists of 100 LLMs evaluated on 805 examples.

    “AlpacaEval 2.0 (Li et al., 2023) consists of 100 LLMs evaluated on 805 examples.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Li et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    The latter train-test split (by date) better represents practical use cases and is more challenging as it likely results in distribution shift between train and test models due to improving model capabilities over time.

    “The latter split better represents practical use cases, while also being more challenging as it is likely to result in a distribution shift between the train and test models due to improving model capabilities over time that might affect the effectiveness of anchor points and the IRT model.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    HELM Lite v1.0.0 has 10 core scenarios (total of approximately 10K evaluation examples) and 30 models.

    “For HELM (Liang et al., 2022), we use HELM Lite v1.0.0, which has the 10 core scenarios (total of approx.10K evaluation examples) and 30 models that have their performances registered for all scenarios.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Liang et al., 2022 source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    Using our methods for efficient prompt-based model evaluation is a promising application.

    “The results demonstrate that using our methods for efficient prompt-based model evaluation is a promising application.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Specialized LLMs are likely to have unique strengths and perform well in specific MMLU categories while relatively underperforming on others, so their correctness patterns might differ from those in the train set.

    “Such models are likely to have unique strengths and perform well in specific MMLU categories while relatively underperforming on others.Thus, their correctness patterns might be different from those in the train set, posing a challenge for our evaluation strategies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    For AlpacaEval 2.0, evaluation requires GPT-4 as a judge; for each input, GPT-4 compares the responses of a candidate LLM and a baseline LLM and declares a winner.

    “For each input, GPT-4 compares the responses of a candidate LLM and a baseline LLM (currently also GPT-4) and declares a winner.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Li et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The main limitations of the methods described in this paper are related to potential severe distribution shifts.

    “The main limitations of the methods described in this paper are related to potential severe distribution shifts.”
    Quote in the paperNot a substantial claim
40 of 40

  • The authors state this as an assumption.

    The score for LLM l in scenario j is the simple average of the correctness of all items in that scenario.

    “To simplify the exposition in the text, we assume that the score for LLM l in scenario j is just the simple average of the correctness of all items in that scenario, that is,”

  • The authors state this as an assumption.

    We have access to correctness evaluations for a set of LLMs that have been previously evaluated on the entirety of the benchmark.

    “To intelligently choose I j we assume access to correctness evaluations for a set of LLMs that have been previously evaluated on the entirety of the benchmark.”

  • The authors state this as an assumption.

    θ_l converges in probability to θ_l as |I| → ∞.

    “Assuming that (i) θl → θ l in probability as | I| → ∞”

  • The authors state this as an assumption.

    The true values of (α_i, β_i) for all i ∈ I are known.

    “and that (ii) the true values of (α i , β i )'s for all i ∈ I are known”

  • The authors state this as an assumption.

    sup_i∈I ||α_i||_2 ≤ c for a universal constant c.

    “and sup i∈I ∥α i ∥ 2 ≤ c for a universal constant c”

  • The authors state this as an assumption.

    The main factors that might prevent gp-IRT from being a good estimator are the variance of the first estimator and the bias of the second one.

    “To apply this result, we assume that the main factors that might prevent gp-IRT from being a good estimator are the variance of the first estimator and the bias of the second one.”

  • The authors state this as an assumption.

    The first estimator's bias and the second estimator's variance can be approximated by zero.

    “Then we approximate the first estimator's bias and the second estimator's variance by zero.”

  • The authors state this as an assumption.

    The two-parameter multidimensional IRT model correctly specifies the probability of an LLM getting an example correctly as a logistic function of θ_l, α_i, and β_i.

    “The two-parameter multidimensional IRT model assumes that the probability of the LLM j getting example i correctly is given by”

  • The paper relies on this without support.

    Correctness data for previously evaluated LLMs is freely available for many popular benchmarks.

    “Such correctness data is freely available for many popular benchmarks.”

  • The paper relies on this without support.

    When scenarios have sub-scenarios, the scenario performance is given by a simple average of sub-scenarios performances.

    “When scenarios have sub-scenarios, it is usually the case that the scenario performance is given by a simple average of sub-scenarios performances.”

  • The authors state this as an assumption.

    The IRT model reasonably describes reality and the example representations are stable.

    “if the IRT model reasonably describes the reality and the example representations are stable”

  • The authors state this as an assumption.

    The IRT model is reasonably specified.

    “if the IRT model is reasonably specified”
12 of 12

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Interventions to improve evaluation have been well explored, such as constructing higher quality benchmarks by identifying errors, expanding test sets, selecting high quality instances, or generating new synthetic benchmarks.

    “Interventions to improve evaluation have been well explored, such as constructing higher quality benchmarks by identifying errors [62, 21], expanding test sets [64], selecting high quality instances from benchmarks [45], or generating entirely new synthetic benchmarks from a model [32].”

    From Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (2025) · cites this paper as [45]

    The Introduction and §5 describe selecting subsets of existing benchmark examples to estimate LLM performance accurately and efficiently. This supports the cited point about selecting instances from benchmarks, but the paper does not establish that the selected instances are individually higher quality; its selection criterion is how well they support evaluation of the full benchmark. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Subset construction has precedent in SWE-bench Lite and tinyBenchmarks.

    “Subset construction has precedent in SWE-bench Lite [28] and in the more general anchor-set methodology of tinyBenchmarks [25]; our 80-instance Lite subset extends this lineage with a rank-aware ILP.”

    From Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks (2026) · cites this paper as [25]

  • The V-optimality selection method significantly outperforms random selection and quickly converges to the prediction performance of using the full set of models.

    “We find that our V-optimality selection method significantly outperforms random selection and quickly converges to the prediction performance of using the full set of models.”

    From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [70]

    The paper’s Evaluation strategies section compares random sampling with correctness- and IRT-based selection of benchmark examples, not V-optimality selection of models. Searches for “V-optimality” and convergence to the performance of a full set of models found no passage addressing the claimed result. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.14992
arXiv
2402.14992
OpenAlex
W4392181972
Cited by
8
References
62
Citations by year
2026: 2 · 2025: 5 · 2024: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (2027737, 2113373)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
large language models, LLM evaluation, MMLU, multiple-choice question answering, Helm, AlpacaEval, evaluation efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review