Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data

Probably not a substantial claim: may not pass S3 (load-bearing).

The closest work uses sequential testing to stabilize performance estimates in reinforcement learning algorithms, where multiple runs with different seeds are required for statistical significance.

“The closest work to ours uses sequential testing to stabilize performance estimates in reinforcement learning (RL) algorithms, where multiple runs with different seeds are required for statistical significance (Mathieu et al., 2023).”

• cites Mathieu et al., 2023 sourceSupported

Classified as probably not a substantial claim.

Best-arm identification algorithms are commonly used in model selection and prompt selection, where the goal is to iteratively compare candidates to identify the best option.

“BAI algorithms are commonly used in model selection (Fu et al., 2024), and been applied to prompt selection (Polo et al., 2024b; Shi et al., 2024), where the goal is to iteratively compare candidates to identify the best option.”

• cites Fu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Polo et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Shi et al., 2024 sourceSupported

May combine two claims.

Practitioners often reduce costs by using fewer samples, but these heuristic approaches lack statistical guarantees.

“While practitioners often reduce costs by using fewer samples (Perlitz et al., 2024; Polo et al., 2024a; Fogliato et al., 2024; Zhao et al., 2024), these heuristic approaches lack statistical guarantees.”

• cites Perlitz et al., 2024 sourceSupported
• cites Polo et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fogliato et al., 2024 sourceSupported

May combine two claims.

Previous efficient evaluation methods rely on past statistics, require full benchmark runs on multiple models, and assume future models follow similar distributions, which limits their applicability.

“However, these methods rely on past statistics, require full benchmark runs on multiple models, and assume future models to follow similar distributions, which limits their applicability.”

• cites Zhang et al., 2024 sourcePartly supported

  Section 3 (p. 4) compares full-benchmark and lite-set scores across six LLaVA versions, supporting the narrower point that evaluating an efficient subset involved results from multiple models. The paper does not say that previous efficient methods generally require full benchmark runs or past statistics, or that they assume future models follow similar distributions. (LLM, conf=0.77)

  Experimental — accuracy not yet measured
• cites Vivek et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Full benchmarks should be larger and include large datasets to allow for reliable results.

“A secondary finding is that full benchmarks should be larger (echoing (Perlitz et al., 2024)) and include large datasets to allow for reliable results.”

• cites Perlitz et al., 2024 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The Open VLM Leaderboard, at the time of writing, includes 206 VLMs evaluated across 31 multimodal benchmarks.

“We conduct our experiments using evaluation data from the Open VLM Leaderboard (Duan et al., 2024) 2 , which, at the time of writing, includes 206 VLMsevaluated across 31 multimodal benchmarks.”

• cites Duan et al., 2024 sourcePartly supported

  Section 3.1 says the authors publish evaluation results on the OpenVLM Leaderboard, so the citation identifies the leaderboard. The Introduction says the toolkit supports more than 200 models and over 80 benchmarks, while Section 3.1 describes eight core leaderboard benchmarks. Neither passage establishes that the leaderboard included exactly 206 evaluated VLMs across 31 benchmarks at the citing paper’s time of writing. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

The framework adaptively manages the efficiency-reliability trade-off on the Open VLM Leaderboard, achieving an 80% reduction in computational cost compared to fixed-size evaluation with a 2.5-point CI width allowance while maintaining statistical significance.

“We demonstrate its ability to adaptively manage the efficiency-reliability trade-off on the Open VLM Leaderboard, including, for example, a 80% reduction in computational cost compared to fixed-size evaluation (with a 2.5-point CI width allowance) while maintaining statistical significance.”

• Also stated in: § 1 Introduction · p. 1 · plain

  “stopping when the CI reaches ± 2 . 5 , saving 80% of the evaluation cost”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In practice, the choice of t is not an issue as long as the batch is reasonably sized.

“In practice, we find it not to be an issue as long as the batch is reasonably sized.”

• AssumptionThe authors state this as an assumption: The observations are independent.
• AssumptionThe authors state this as an assumption: The test statistic Z_AB_k follows an approximately normal distribution under the null hypothesis, as justified by the Central Limit Theorem when T_k is sufficiently large.
• AssumptionThe authors state this as an assumption: The number of stages t is pre-specified.

May combine two claims.

Stopping when diminishing returns plateau reduces cost by 44% while sacrificing only 0.132 points in precision.

“stopping when diminishing returns plateau, reducing cost by 44% while sacrificing only 0.132 points in precision.”

May combine two claims.

In the model-selection case study, 86 models were filtered out early as underperforming, while the rest stopped upon reaching the target confidence interval.

“86 models were filtered out early as underperforming, while the rest stopped upon reaching the target CI.”

May combine two claims.

As sample size increases, the confidence interval narrows, reducing uncertainty in model performance estimates.

“As sample size increases, the CI narrows, reducing uncertainty in model performance estimates.”

May combine two claims.

In the statistical-guarantees case study, the method used only 60% of the examples, though 2.4 comparisons out of 20 per run were indistinguishable and required the full benchmark.

“Nonetheless, our method used only 60% of the examples, though 2.4 comparisons (out of 20) per run were indistinguishable and required the full benchmark.”

• AssumptionRests on an assumption: A 2-point performance difference is a practically meaningful equivalence margin for model comparison.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Our framework supports a broader range of evaluation needs by enabling practitioners to define flexible stopping conditions, ensuring more reliable and efficient evaluation.

“Moreover, our framework supports a broader range of evaluation needs by enabling practitioners to define flexible stopping conditions, ensuring more reliable and efficient evaluation.”

Probably not a substantial claim: may not pass S4 (based).

Best-arm identification algorithms focus primarily on minimizing regret while operating within a fixed budget.

“While these algorithms utilize sequential decision-making, their focus is primarily on minimizing regret while operating within a fixed budget.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Our work builds on sequential testing techniques, applying them to adaptive model evaluation frameworks to enable more efficient and transparent decision-making.

“Our work builds on these techniques, applying them to adaptive model evaluation frameworks to enable more efficient and transparent decision-making.”

Classified as probably not a substantial claim.

The results of hypothesis testing are influenced by sample size, effect size, and significance level.

“The results of hypothesis testing are influenced by sample size, effect size, and the significance level ( α ).”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Adaptive evaluation is efficient and transparent.

“This makes adaptive evaluation not only efficient but also transparent .”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The number of available examples in a benchmark might be excessive or not enough, creating a fundamental challenge of controlling the reliability-efficiency trade-off.

“However, the number of available examples in a benchmark might be excessive, or not enough. This creates a fundamental challenge: how to control the trade-off between reliability and efficiency in model evaluation.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Adaptive evaluation ensures comparisons stop as soon as the observed difference is statistically and practically significant, preventing over-evaluation and premature conclusions.

“Adaptive evaluation ensures that comparisons stop as soon as the observed difference is statistically and practically significant, preventing both over-evaluation and premature conclusions.”

Classified as probably not a substantial claim.

Adaptive evaluation stops when practical and statistical needs are met.

“Rather than enforcing a fixed sample size, adaptive evaluation stops when the practical and statistical needs are met.”

May combine two claims.

Our method offers strict statistical guarantees without needing to run models on the full benchmark to gather past statistics.

“In contrast, our method offers strict statistical guarantees without needing to run models on the full benchmark to gather past statistics.”

Probably not a substantial claim: may not pass S4 (based).

The more examples used, the more confident we can be in conclusions.

“Broadly speaking, the more examples we use, the more confident we can be in our conclusions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sequential testing methods such as Group Sequential Testing and the Sequential Probability Ratio Test have long been used to optimize sample size and decision-making in various fields by evaluating data at interim points.

“Sequential testing methods, such as Group Sequential Testing (GST) (Jennison and Turnbull, 1999a; Lan and DeMets, 1983; Pocock, 1977; O'Brien and Fleming, 1979) and the Sequential Probability Ratio Test (SPRT) (Wald, 1945c), have long been used to optimize sample size and decision-making in various fields (e.g. clinical trials), by evaluating data at interim points.”

• cites Jennison and Turnbull, 1999aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lan and DeMets, 1983 sourceNot checked — no full text of the source
• cites Pocock, 1977 sourceNot checked — no full text of the source

May combine two claims.

Unlike the RL-specific focus, our framework prioritizes sample efficiency and generalizes to diverse tasks, enabling adaptive evaluation with user-defined stopping criteria for model comparison, ranking, and benchmarking.

“Unlike their RL-specific focus, our framework prioritizes sample efficiency and generalizes to diverse tasks, enabling adaptive evaluation with user-defined stopping criteria for model comparison, ranking, and benchmarking.”

Probably not a substantial claim: may not pass S1 (asserted), S4 (based). May combine two claims.

Our framework aims to provide statistical guarantees while minimizing sample usage.

“In contrast, our framework aims to provide statistical guarantees while minimizing sample usage.”

Probably not a substantial claim: may not pass S4 (based).

When efficiency is prioritized, users are fully aware of what was lost in terms of statistical power, allowing informed decisions rather than guesswork.

“In addition, when efficiency is prioritized, users are fully aware of what was lost in terms of statistical power, allowing them to make informed decisions rather than rely on guesswork.”

May combine two claims.

Adaptive evaluation enables early detection of promising and faulty checkpoints, terminating evaluations only when necessary, reducing computational waste and saving time.

“Adaptive evaluation enables early detection of both promising and faulty checkpoints, terminating evaluations only when necessary, dramatically reducing computational waste and saving time (see §7.3).”

Probably not a substantial claim: may not pass S4 (based).

In practice, best-arm identification algorithms often exhaust their fixed budget before achieving desired reliability, potentially leading to suboptimal conclusions.

“In practice, BAI algorithms often exhaust their fixed budget before achieving the desired reliability, potentially leading to suboptimal conclusions.”

May combine two claims.

Debugging a model may only require an inexpensive coarse approximation, while definitively determining a superior model among close contenders demands sufficient data to achieve statistical significance.

“Debugging a model may only require a inexpensive coarse approximation, while definitively determining a superior model among close contenders demands sufficient data to achieve statistical significance.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The method ensures that evaluations are neither underpowered nor excessive, effectively balancing reliability and efficiency.

“Thus, users explicitly define their needs, and the method ensures that evaluations are neither underpowered nor excessive, effectively balancing reliability and efficiency.”

May combine two claims.

The framework can detect and stop evaluation based on stopping rules, ensuring both statistical rigor and evaluation efficiency.

“Our framework can detect and stop evaluation based on such rules, ensuring both statistical rigor and evaluation efficiency.”

• AssumptionThe authors state this as an assumption: The observations are independent.
• AssumptionThe authors state this as an assumption: The test statistic Z_AB_k follows an approximately normal distribution under the null hypothesis, as justified by the Central Limit Theorem when T_k is sufficiently large.
• AssumptionThe authors state this as an assumption: The number of stages t is pre-specified.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Because stopping points depend on observed performance, results may vary across runs, potentially reducing reproducibility unless randomness is tightly controlled.

“Because stopping points depend on observed performance, results may vary across runs, potentially reducing reproducibility unless randomness is tightly controlled.”

Probably not a substantial claim: may not pass S4 (based).

The smallest and most varying datasets included in the benchmark are the ones most important to increase in size.

“Specifically, the smallest and most varying datasets included in the benchmark are the ones most important to increase in size.”

May combine two claims.

The approach improves the resource efficiency of large-scale model evaluations and enhances the transparency of the evaluation process, enabling users to make informed decisions based on clear trade-offs.

“This approach not only improves the resource efficiency of large-scale model evaluations but also enhances the transparency of the evaluation process, enabling users to make informed decisions based on clear trade-offs.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Pairwise comparisons introduce the challenge of multiple hypothesis testing and require additional correction methods, but this aspect is simplified for the purpose of this study.

“While pairwise comparisons introduce the challenge of multiple hypothesis testing and require additional correction methods, we simplify this aspect for the purpose of this study.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Naively checking performance at multiple stages can lead to inflated error rates and reduce reliability, since the probability of a false positive increases.

“Naively checking the performance at multiple stages of the evaluation can lead to inflated error rates and reduce reliability, since the probability of obtaining a false positive increases.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Sequential methods introduce computational overhead due to repeated statistical checks, but this cost is often outweighed by savings in evaluation time and sample usage.

“Sequential methods introduce some computational overhead due to repeated statistical checks, but this cost is often outweighed by the savings in evaluation time and sample usage.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The Pocock spending function provides a well-balanced framework particularly suited for batch-based model evaluation.

“we choose the Pocock spending function for demonstration purposes, as it provides a well-balanced framework particularly suited for batch-based model evaluation.”

• AssumptionThe authors state this as an assumption: The observations are independent.
• AssumptionThe authors state this as an assumption: The test statistic Z_AB_k follows an approximately normal distribution under the null hypothesis, as justified by the Central Limit Theorem when T_k is sufficiently large.
• AssumptionThe authors state this as an assumption: The number of stages t is pre-specified.

Probably not a substantial claim: may not pass S4 (based).

Repeatedly checking and acting upon results during data collection, known as peeking, invalidates the standard interpretation of p-values and confidence intervals.

“However, repeatedly checking (and acting upon) results during data collection, a practice known as 'peeking,' invalidates the standard interpretation of p-values and CIs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A larger effect size makes the difference between models more detectable.

“A larger effect size, or true difference between models, also makes the difference more detectable.”

May combine two claims.

The Pocock approach uses a constant critical value c across all stages, such that the overall Type I error rate equals alpha.

“The Pocock approach addresses this by using a constant critical value c across all stages, such that the overall Type I error rate equals α .”

• cites Pocock, 1977 sourceNot checked — no full text of the source

• AssumptionThe authors state this as an assumption: The observations are independent.
• AssumptionThe authors state this as an assumption: The test statistic Z_AB_k follows an approximately normal distribution under the null hypothesis, as justified by the Central Limit Theorem when T_k is sufficiently large.
• AssumptionThe authors state this as an assumption: The number of stages t is pre-specified.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Users must define evaluation criteria such as equivalence margins or acceptable performance gaps, which often demands statistical expertise or domain-specific knowledge.

“Finally, users must define evaluation criteria-such as equivalence margins or acceptable performance gaps-which often demands statistical expertise or domain-specific knowledge.”

May combine two claims.

In heterogeneous benchmarks, smaller datasets with higher variance dominate the unweighted mean of means, which caps overall uncertainty and amplifies the effect of diminishing returns.

“A strong contributor to this effect is that in heterogeneous benchmarks, smaller datasets with higher variance dominate the unweighted mean of means. This caps overall uncertainty, and thus amplifies the effect of diminishing returns.”

• AssumptionRests on an assumption: The unweighted mean of means is an appropriate aggregation for overall uncertainty in heterogeneous benchmarks.

Probably not a substantial claim: may not pass S3 (load-bearing).

The significance level determines the threshold for rejecting the null hypothesis, impacting test power and reliability of conclusions.

“The significance level determines the threshold for rejecting H 0 , which directly impacts the test's power and the reliability of conclusions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The leaderboard provides detailed metadata on models, including architectural components, underlying vision and language models, and model sizes.

“The leaderboard provides detailed metadata on the models, including their architectural components, underlying vision and language models, and model sizes.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

A larger number of stages t enables more frequent interim analyses, potentially allowing earlier stopping decisions, but requires more samples to account for multiple testing.

“A larger t enables more frequent interim analyses, potentially allowing earlier stopping decisions, but it requires a more samples to account for multiple testing (i.e., repeated 'looks' at the data).”

• AssumptionThe authors state this as an assumption: The observations are independent.
• AssumptionThe authors state this as an assumption: The test statistic Z_AB_k follows an approximately normal distribution under the null hypothesis, as justified by the Central Limit Theorem when T_k is sufficiently large.
• AssumptionThe authors state this as an assumption: The number of stages t is pre-specified.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

In many real-world applications, users are often willing to accept a degree of uncertainty in exchange for faster evaluations and reduced costs.

“In many real-world applications, users are often willing to accept a degree of uncertainty in exchange for faster evaluations and reduced costs.”

May combine two claims.

Beyond a certain point, the confidence interval stabilizes due to diminishing returns: precision improves as n^-1/2, yielding rapid early gains but minimal impact from further sampling.

“Beyond this point, CI stabilizes due to diminishing returns: precision improves as n -1 / 2 yielding rapid early gains but minimal impact from further sampling.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Any sequential analysis method introduces a modest power reduction and slight computational overhead, but its efficiency gains far outweigh these costs.

“Though this and any sequential analysis method introduces a modest power reduction and slight computational overhead, its efficiency gains far outweigh these costs.”

• AssumptionThe authors state this as an assumption: The observations are independent.
• AssumptionThe authors state this as an assumption: The test statistic Z_AB_k follows an approximately normal distribution under the null hypothesis, as justified by the Central Limit Theorem when T_k is sufficiently large.
• AssumptionThe authors state this as an assumption: The number of stages t is pre-specified.

Classified as probably not a substantial claim.

Larger sample sizes help reduce noise, making it easier to detect true differences.

“Larger sample sizes help reduce noise, making it easier to detect true differences.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

In principle, a user can stop for any reason, and the statistical guarantees will hold.

“We note that in principle, a user can stop for any reason, and the statistical guarantees will hold.”

• AssumptionThe authors state this as an assumption: The observations are independent.
• AssumptionThe authors state this as an assumption: The test statistic Z_AB_k follows an approximately normal distribution under the null hypothesis, as justified by the Central Limit Theorem when T_k is sufficiently large.
• AssumptionThe authors state this as an assumption: The number of stages t is pre-specified.