Power Analysis and Effect Size in Mixed Effects Models: A Tutorial

“Table 2: Outcome of the lmer analysis (Bates et al., 2015) of the Adelman et al.”

“What must be avoided, is that authors try out the various options and only report the one that produces a “significant” effect, a phenomenon known as p-hacking (Simmons et al., 2011).”

“First, p-values show wide sample-to-sample variability, particularly when they are based on studies with small sample sizes (Cumming, 2014; Halsey, Curran-Everett, Vowler, & Drummond, 2015).”

“A particularly interesting R package is simR (Green & MacLeod, 2016; Green, MacLeod, & Alday, 2016).2 2 The authors thank Jason Geller for pointing them to this package.”

“Some authors demand the use of transformations to get normal distributions (e.g., Baayen, 2008).”

“Even so, Cohen (1962) showed that many experiments published in the established psychology journals have a power considerably below 80% (often as low as 30–40%).”

“…problem combined with low power not only results in published papers with effects that do not exist at the population level, but also leads to an unacceptably high number of publications reporting and interpreting effects opposite to the one present at the population level (Gelman & Carlin, 2014).”

“Second, there is a file drawer problem (Fanelli, 2011; Franco, Malhotra, & Simonovits, 2014).”

“Others warn that transformations may alter the nature of the data (e.g., a logarithmic transformation may alter the interactions between independent variables; Lo & Andrews, 2015).”

“First, Westfall et al. (2014) showed how you can calculate the effect size (measured as d) for a design with random participants and random items.”

“Unless these are based on a proper power analysis, it is to be expected that many of these “reasons” will be fallacies aimed at decreasing the work load rather than cherishing the quality of the research done (Smaldino & McElreath, 2016).”

“As is general practice, RTs of error trials and outliers were excluded (outliers were detected using an adjusted boxplot for skewed distributions, Hubert & Vandervieren, 2008).”

“The consequences of underpowered studies are becoming increasingly clear (Loken & Gelman, 2017).”

“They are not more difficult to run anymore and better fit the data (Judd et al., 2017).”

“Because such p-values require large observed effect sizes in small designs, the effects sizes reported in the literature tend to be inflated, also those from studies that can be replicated (Kuhberger et al., 2014; Open Science Collaboration, 2015; Vasishth & Gelman, 2017).”

“Keuleers et al. (2012) observed that with a group of 40 participants, one needs 160 words and 160 nonwords per condition to find a word frequency effect of 20 ms with a power of .80 (this is a total of 40 * 160 * 4 = 25,600 observations, of which half are not used in the analysis, namely the data…”

“At the same time, it is becoming clear that the definition of replication success may be less straightforward than assumed at first (Amrhein, Korner-Nievergelt, & Roth, 2017; Etz & Vandekerckhove, 2016).”

“As a result, chances of publishing type I errors are considerably higher than the 5% expected when one uses the p < .05 criterion (Francis, 2012; Leggett, Thomas, Loetscher, & Nicholls, 2013; Simmons, Nelson, & Simonsohn, 2011).”

“Keuleers et al. (2012) observed that with a group of 40 participants, one needs 160 words and 160 nonwords per condition to find a word frequency effect of 20 ms with a power of .”

“…et al. (2015) Lexical decision 8 191 50 1194 Ussishkin et al. (2015) Exp 2a Lexical decision 3 66 36 792 Exp 2b Lexical decision 3 70 36 840 Kgolo & Eisenbeiss (2015) Exp 1 Lexical decision 7 70 55 550 Exp 2 Lexical decision 8 63 55 433 Exp 3 Lexical decision 8 65 55 446 Sulpizio & Job…”

“It tells us that the estimates of Westfall et al. (2014) are too optimistic.”

“For the Adelman et al. (2014) database, one requires some 6,000 observations for a properly powered experiment (e.g., 60 participants and 100 stimuli, or 80 participants and 80 stimuli, or 100 participants and 60 stimuli).”

“Bell et al. (2015) Exp 1 Semantic classification 6 23 120 460 Exp 2 Semantic classification 6 41 120 820 Exp 3 Semantic classification 6 39 120 780 Beyersmann et al. (2015) Lexical decision 8 191 50 1194 Ussishkin et al. (2015) Exp 2a Lexical decision 3 66 36 792 Exp 2b Lexical decision 3 70 36…”

“…460 Exp 2 Semantic classification 6 41 120 820 Exp 3 Semantic classification 6 39 120 780 Beyersmann et al. (2015) Lexical decision 8 191 50 1194 Ussishkin et al. (2015) Exp 2a Lexical decision 3 66 36 792 Exp 2b Lexical decision 3 70 36 840 Kgolo & Eisenbeiss (2015) Exp 1 Lexical decision 7 70…”

“What must be avoided, is that authors try out the various options and only report the one that produces a “significant” effect, a phenomenon known as p-hacking (Simmons et al., 2011).”

“…192 Exp 2 Word naming 6 24 48 192 Exp 3 Word naming 6 24 48 192 Exp 4 Word naming 9 20 56 124 Dasgupta et al. (2015) Lexical decision 10 28 490 1372 Guldenpenning et al. (2015) Exp 1 Action decision 24 44 408 748 Exp 2 Action decision 48 50 576 600 Atas et al. (2015) Direction decision 32 29 2560…”

“…408 748 Exp 2 Action decision 48 50 576 600 Atas et al. (2015) Direction decision 32 29 2560 2320 Perea et al. (2015) Lexical decision 4 40 120 1200 Kiefer et al. (2015) Exp 1 Evaluative decision 8 24 384 1152 Exp 2 Evaluative decision 16 24 768 1152 Exp 3 Evaluative decision 2 20 150 1500 should…”

“In contrast, Bradley and Russell (1998), on the basis of simulations, strongly warned against the introduction of an extra, in-between level to a repeated measure, because it decreases the power of the study.”

“Because such p-values require large observed effect sizes in small designs, the effects sizes reported in the literature tend to be inflated, also those from studies that can be replicated (Kuhberger et al., 2014; Open Science Collaboration, 2015; Vasishth & Gelman, 2017).”

“Bell et al. (2015) Exp 1 Semantic classification 6 23 120 460 Exp 2 Semantic classification 6 41 120 820 Exp 3 Semantic classification 6 39 120 780 Beyersmann et al. (2015) Lexical decision 8 191 50 1194 Ussishkin et al. (2015) Exp 2a Lexical decision 3 66 36 792 Exp 2b Lexical decision 3 70 36…”

“…3 70 36 840 Kgolo & Eisenbeiss (2015) Exp 1 Lexical decision 7 70 55 550 Exp 2 Lexical decision 8 63 55 433 Exp 3 Lexical decision 8 65 55 446 Sulpizio & Job (2015) Exp 1 Word naming 6 24 48 192 Exp 2 Word naming 6 24 48 192 Exp 3 Word naming 6 24 48 192 Exp 4 Word naming 9 20 56…”

“…et al. (2015) Lexical decision 10 28 490 1372 Guldenpenning et al. (2015) Exp 1 Action decision 24 44 408 748 Exp 2 Action decision 48 50 576 600 Atas et al. (2015) Direction decision 32 29 2560 2320 Perea et al. (2015) Lexical decision 4 40 120 1200 Kiefer et al. (2015) Exp 1 Evaluative…”

“…55 446 Sulpizio & Job (2015) Exp 1 Word naming 6 24 48 192 Exp 2 Word naming 6 24 48 192 Exp 3 Word naming 6 24 48 192 Exp 4 Word naming 9 20 56 124 Dasgupta et al. (2015) Lexical decision 10 28 490 1372 Guldenpenning et al. (2015) Exp 1 Action decision 24 44 408 748 Exp 2 Action decision 48 50…”

“Brysbaert and Stevens (2018) recommended 40 participants and 40 stimuli per condition as good practice in reaction time studies with two repeated-measures conditions.”