With Little Power Comes Great Responsibility

If only significant findings are published, especially if based on underpowered experiments, this can lead to systematic distortions in the literature.

“This problem can lead to systematic distortions in the literature if only significant findings are published, especially if these results are based on underpowered experiments (Scargle, 1999).”

• cites Scargle, 1999 sourcePartly supported

  Section 4.1 models a setting in which studies meeting a significance threshold are published and those that do not are excluded. The Conclusions state that publication bias can produce apparently significant but spurious results and bias analyses of the published literature. The cited passages do not establish the added point that this distortion is especially pronounced for underpowered experiments. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

Many key details of experimental settings were not reported in the EMNLP 2019 sample.

“But, as similarly found by van der Lee et al. (2019), many key details were not reported in this sample of experiments.”

• cites van der Lee et al. (2019) sourceSupported

Underpowered experiments have contributed, in part, to the replication crisis in other fields.

“and has contributed, in part, to the replication crisis in other fields (Button et al., 2013; Szucs and Ioannidis, 2017; Ioannidis et al., 2017).”

• cites Button et al., 2013 sourceSupported
• cites Szucs and Ioannidis, 2017 sourceSupported
• cites Ioannidis et al., 2017 sourceNot checked — no full text of the source

Past work has reported great variation in how human evaluations are done.

“Past work has reported great variation in how human evaluations are done (van der Lee et al., 2019).”

• cites van der Lee et al., 2019 sourceSupported

Approximately 51% of reported improvements would have been significant.

“and approximately 51% would have been significant (based on estimated agreement and reported improvement).”

Using GLUE tasks which measure accuracy, regressing Pa on baseline accuracy and Δacc achieves R2 = 0.97.

“Using GLUE tasks which measure accuracy, we regress P a on baseline accuracy and acc , and achieve an R 2 of 0 . 97 .”

For typical MT test sets of around 2,000 examples, an improvement of 1 BLEU point can likely be detected with approximately 75% power.

“Based on this estimate, we conclude that for typical MT test sets of around 2,000 examples, an improvement of 1 BLEU point can likely be detected with approximately 75% power.”

Typical improvements on popular tasks tend to be small.

“Typical improvements on popular tasks tend to be small (see mean improvements in Table 2).”

The mean reported effect size (|Δacc|) is well below the estimated MDE for the three smallest test sets - WNLI, MRPC, and SST-2.

“As can be seen in Table 2, the mean reported effect size ( | acc | ) is well below the estimated MDE for the three smallest test sets - WNLI, MRPC, and SST-2.”

There is a very strong correlation between validation and test for both pairwise accuracy differences (Δacc) and agreement rates (Pa) (r = 0.99 for both).

“We find a very strong correlation between validation and test for both pairwise accuracy differences ( acc ) and agreement rates ( P a ) ( r = 0 . 99 for both, as shown in Figure 9 in Appendix D, with results on validation data included in the accompanying online materials)”

Statistical power has largely been ignored by the NLP community.

“Despite its importance to experimental design, statistical power (the probability that, given a real effect, an experiment will reject the null hypothesis) has largely been ignored by the NLP community.”

• Also stated in: § 1 Introduction · p. 1 · plain

  “Despite its importance to empirical evaluation, relatively little attention has been paid to statistical power in NLP.”

For machine translation, typical test sets of 2000 sentences have approximately 75% power to detect differences of 1 BLEU point.

“For machine translation, we find that typical test sets of 2000 sentences have approximately 75% power to detect differences of 1 BLEU point.”

In the EMNLP 2019 meta-analysis, reported effect sizes vary widely (s.d. = .12 on a [0,1] scale).

“One takeaway from this meta-analysis is that the reported effect sizes (that is, difference between the novel model and the best-performing baseline) vary widely (s.d. = . 12 on a [0, 1] scale).”

Repeating the Pa regression for SQuAD 2.0 gives R2 = 0.94.

“Repeating this for SQuAD 2.0, we get an R 2 of 0 . 94 .”

The most typical experimental design for human rating studies will be underpowered to detect small model differences, of the sort that are frequently studied.

“Similarly, based on reasonable assumptions, we find that the most typical experimental design for human rating studies will be underpowered to detect small model differences, of the sort that are frequently studied.”

• Also stated in: § 1 Introduction · p. 2 · plain

  “Among human evaluations, we find most experimental designs involve too few items and/or raters to detect small effects ( § 5).”

Power increases dramatically with sample size and effect size.

“As shown in Figure 4 this power level increases dramatically with sample size and effect size.”

Most of the papers from which we collected results did not report a significance test on the test set.

“Most of the papers from which we collected results did not report a significance test on the test set.”

In the EMNLP 2019 survey, there is greater variance in effects with smaller samples.

“As expected, we see greater variance in effects with smaller samples since, with smaller samples, we expect greater noise.”

• AssumptionRests on an assumption: The number of items rated is a valid proxy for sample size in human evaluations.

A survey of recent papers reveals a dearth of detailed reporting.

“Finally, a survey of recent papers reveals a general lack of statistical evaluation and a dearth of detailed reporting ( § 5.1).”

Underpowered experiments are common in the NLP literature.

“By metaanalyzing a set of existing NLP papers and datasets, we characterize typical power for a variety of settings and conclude that underpowered experiments are common in the NLP literature.”

• Also stated in: § 1 Introduction · p. 2 · plain

  “we present evidence that underpowered experiments are widespread in NLP research.”

There is legitimate reason to be concerned about underpowered comparisons in the case of certain widely used benchmarks.

“As we show in § 3.3, there is legitimate reason to be concerned about this in the case of certain widely used benchmarks.”

The typical distribution of individual effects from swapping individual output pairs across several models and datasets can be approximated by a mixture of a Delta distribution at zero and a Laplace distribution.

“Based on analyzing several models and datasets, we find the typical distribution of these individual effects can be approximated using a mixture of a Delta distribution at zero, and a Laplace distribution”

A survey of recent papers reveals a general lack of statistical evaluation.

“Finally, a survey of recent papers reveals a general lack of statistical evaluation and a dearth of detailed reporting ( § 5.1).”

The four MT comparisons produce similar estimates of P0 and b0.

“While far from identical, the four comparisons, each representing different stages of model evolution, all produce similar estimates.”

Only 46% of reported improvements would have predicted adequate power.

“This procedure reveals that only 46% would have predicted adequate power (using estimates for expected improvement and agreement)”

In regressing Δacc on baseline accuracy and task for GLUE, R2 = 0.69.

“In regressing acc on baseline accuracy and task, we achieve an R 2 = 0 . 69 , which is not a perfect fit, but still provides a prior on likely effect size.”

MDEs using Lachenbruch's suggested default prior are very similar to those found by using the regression-based approach.

“Lachenbruch (1992) provides a suggested default prior, and we find that MDEs using this method are very similar to those found by using the regression based approach.”

• cites Lachenbruch, 1992 sourceNot checked — no full text of the source

Alternative analyses give even more pessimistic estimates of likely improvements relative to MDE.

“Moreover, alternative analyses give even more pessimistic estimates of likely improvements relative to MDE, as described in Appendix E.4.”

In fitting a regression to SOTA improvements on the SQuAD 2.0 leaderboard, R2 = 0.67.

“Similarly, we achieve an R 2 = 0 . 67 when fitting a regression to SOTA improvements on the SQuAD 2.0 leaderboard”

Power for comparing two classifiers on accuracy using paired data depends on the size of the test set, the expected agreement, and the expected difference in accuracy.

“Power for comparing two classifiers on accuracy using paired data depends on the size of the test set ( n ), the expected agreement ( P a ), and the expected difference in accuracy ( acc ).”

The sum of individual effects closely approximates the net effect of swapping entire subsets.

“While these individual effects are not independent of each other due to the corpus-level nature of the metric, in practice, the sum of individual effects closely approximates the net effect of swapping entire subsets (see Figure 15 in Appendix G).”

Using high variance parameters typical of most datasets, the most common design at EMNLP 2019 (3 workers, 100 items) is underpowered unless the effect size is quite large (0.2 or higher on the [0,1] scale).

“Using the 'high variance' parameters (which are typical of most of the datasets we used), the most common design at EMNLP 2019 (3 workers, 100 items) is underpowered unless the effect size is quite large (0.2 or higher on the [0, 1] scale).”

• AssumptionRests on an assumption: The convenience sample of several large datasets is a valid proxy for typical variation in human evaluations in practice.
• AssumptionRests on an assumption: The number of items rated is a valid proxy for sample size in human evaluations.

The large datasets used for parameter estimation all use a similar methodology: many Likert-scale annotations per instance from many annotators and models.

“Although focused on different tasks, all use a similar methodology, namely, getting many Likert-scale annotations per instance from many annotators and models (in some cases as many as 20 ratings per item).”

In this paper, we have presented evidence that underpowered experiments are widespread in NLP.

“In this paper, we have presented evidence that underpowered experiments are widespread in NLP.”

In the EMNLP 2019 survey, the negative correlation between effect size and sample size is not significant (Kendall's tau = -.07, p = .32).

“However, in our case, this correlation is not significant (Kendall's ⌧ = . 07 , p = . 32 ) and so it is difficult to draw strong conclusions.”

• AssumptionRests on an assumption: The number of items rated is a valid proxy for sample size in human evaluations.

Even with low variance parameters, typical designs are underpowered to detect small effects (0.05 on the [0,1] scale) unless an unusually large number of ratings per item are collected (10+ for 100 items).

“Using our estimated parameters for the low variance setting, experiments will be underpowered to detect small effects (0.05 on the [0, 1] scale), unless an unusually large number of ratings per item are collected (10+ for 100 items).”

• AssumptionRests on an assumption: The convenience sample of several large datasets is a valid proxy for typical variation in human evaluations in practice.
• AssumptionRests on an assumption: The number of items rated is a valid proxy for sample size in human evaluations.

Because the mean reported effect size is based on models comparing to even weaker baselines, most future improvements are expected to be even smaller.

“Because this mean is based on models comparing to even weaker baselines, we would expect most future improvements to be even smaller.”

Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements.

“Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements, and increase the chances of exaggerated findings.”

Proceeding with an underpowered test means one is less likely to be able to draw any useful statistical conclusion from the experiment.

“Proceeding with a test that is underpowered (i.e., too few subjects or items; often taken to mean less than 80% power; Cohen, 1962) means that one is less likely to be able to draw any useful statistical conclusion from the experiment”

Evaluation data for MT tends to be scarce because MT depends on parallel text, most of which is used in training.

“Unfortunately, because MT depends on parallel text (most of which is used in training), evaluation data tends to be scarce.”

Significant findings from underpowered experiments are more likely to exaggerate or reverse the true effect (Type-M and Type-S errors).

“Moreover, significant findings from underpowered experiments are more likely to exaggerate or reverse the true effect - so-called Type-M (magnitude) and Type-S (sign) errors, respectively (Gelman and Carlin, 2014).”

• cites Gelman and Carlin, 2014 sourceNot checked — no full text of the source

Reliance on evidence from underpowered experiments risks leading to over-confidence about the relative ranking of various models.

“Reliance on such evidence risks leading to over-confidence about the relative ranking of various models.”

If an experiment shows significant improvement on a dataset such as MRPC, the potential for Type-M error should make us skeptical that this improvement will generalize to new data from the same domain.

“If an experiment does show significant improvement on a dataset such as MRPC, the potential for Type-M error should make us skeptical that this improvement will generalize to new data from the same domain.”

BLEU is a corpus-level metric, rather than being averaged across instances.

“This setting is more complicated because (1) BLEU is a corpus-level metric, rather than being averaged across instances”

• cites Papineni et al., 2002Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Most future experiments involving WNLI, MRPC, and SST-2 will not have adequate power to test for improvements over the current SOTA in the way that they are routinely used.

“Thus, most future experiments involving these three datasets will not have adequate power to test for improvements over the current SOTA in the way that they are routinely used.”

Underpowered experiments increase the chances of exaggerated findings.

“Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements, and increase the chances of exaggerated findings.”

• cites Gelman and Carlin, 2014 sourceNot checked — no full text of the source

Paired predictions on validation data can be used for power calculations when predictions from both models are available.

“suggesting we can use paired predictions on validation data for power calculations when we have access to the predictions from both models.”

• AssumptionThe authors state this as an assumption: For using validation data to estimate power, dev and test data are drawn from the same distribution and dev performance has not been artificially inflated.

If an experiment is badly underpowered, it cannot provide useful evidence that one model achieves slightly better performance than another for the underlying data distribution.

“If an experiment is badly underpowered, it cannot provide useful evidence that one model achieves slightly better performance than another for the underlying data distribution.”

For several tasks in the popular GLUE benchmark, small test sets mean that most attempted comparisons to state of the art models will not be adequately powered.

“In particular, for several tasks in the popular GLUE benchmark, small test sets mean that most attempted comparisons to state of the art models will not be adequately powered.”

• Also stated in: § 1 Introduction · p. 2 · plain

  “For comparing models in terms of accuracy, we find that some widely used benchmark datasets, including MRPC and SST-2, are now too small to be able to properly measure future progress against top performing models ( § 3).”

It is often difficult to ascertain, for any particular experiment, the details of the experimental setting necessary to evaluate the validity of the results.

“Thus, it is often difficult to ascertain, for any particular experiment, the details of the experimental setting that are necessary to evaluate the validity of the results.”

In aggregate, many human evaluations are underpowered and would benefit from larger sample sizes, particularly by using more workers per item.

“Given our model estimates and simulations, we conclude that, in aggregate, many human evaluations are underpowered and would benefit from larger sample sizes, particularly by using more workers per item.”

• AssumptionRests on an assumption: The convenience sample of several large datasets is a valid proxy for typical variation in human evaluations in practice.
• AssumptionRests on an assumption: The number of items rated is a valid proxy for sample size in human evaluations.