Power Analysis and Effect Size in Mixed Effects Models: A Tutorial

“Table 2: Outcome of the lmer analysis (Bates et al., 2015) of the Adelman et al.”

“What must be avoided, is that authors try out the various options and only report the one that produces a “significant” effect, a phenomenon known as p-hacking (Simmons et al., 2011).”

“First, p-values show wide sample-to-sample variability, particularly when they are based on studies with small sample sizes (Cumming, 2014; Halsey, Curran-Everett, Vowler, & Drummond, 2015).”

“A particularly interesting R package is simR (Green & MacLeod, 2016; Green, MacLeod, & Alday, 2016).2 2 The authors thank Jason Geller for pointing them to this package.”

“Some authors demand the use of transformations to get normal distributions (e.g., Baayen, 2008).”

“…level of 0.05, and an average effect size ( d = 0.4), commonly used in organizational psychology studies and recommended by recent works such as Brysbaert & stevens, 2018. in survey research, it is common practice to use a sample size determined by the g*Power software to ensure adequate…”

“To detect small effects using linear mixed-effects regression in designs with crossed participants and items, Brysbaert and Stevens (2018) recommend having at least 1,600 trials per condition for each pairwise comparison.”

“We chose to use linear mixed-effects models over traditional statistical approaches because these models are more appropriate given the data structure, are more flexibly and have increased statistical power (Brysbaert & Stevens, 2018).”

“Brysbaert and Stevens (2018) suggest that 1,600 readings per condition are necessary for studies of this design to be sufficiently powered to detect a small effect size, that is, d < 0.2; Costa (2013) reported d = 0.14.”

“The sample size was guided by the studies of Marcet et al. (2019) and Brysbaert and Stevens (2018).”

“Consistent with Exp1, we used data from the top 10 participants in Exp2 as the pilot data and calculated power curves for different sample sizes using Monte-Carlo simulations (Brysbaert & Stevens, 2018; DeBruine & Barr, 2021; Leroy et al., 2024).”

“Based on Brysbaert and Steven’s (2018) suggestion, it should ideally have a minimum of 1,600 word observations per condition for a reaction time experiment with repeated measures to have proper power.”

“…study was based on approximately 1200 observations, and a reasonable sample of people with dyslexia, but when the metaphor trials were separated based on response type, those analyses were based on approximately 700 observations, which is far below the suggestions of Brysbaert and Stevens (2018).”

“Recently, Brysbaert and Stevens (2018) recommended at least 1600 trials per condition across all participants for studies using linguistic stimulus material.”

“Indeed, Brysbaert and Stevens (2018) for instance recommend a minimum of 1600 observations per condition in designs with repeated measures (i.e. all designs including measures of the Stroop effect and of its reduction).”

“As suggested by Brysbaert and Stevens (2018), a minimum of 1,600 observations per condition is recommended for a well-powered repeated-measure study.”

“According to Brysbaert & Stevens (2018) , a general effect size commonly employed in psychological research ( d = [0.3, 0.4]) was utilized as the prior effect size for the analysis, double-tailed test, α = 0 .”

“Sample size was determined a priori following the same guidelines (Brysbaert & Stevens, 2018) adopted in the previous two experiments.”

“There is no consensus on the most appropriate e ff ect size measures appropriate for LMM analyses, and potential e ff ect size parameters must be interpreted with context and nuance (e.g., Brysbaert & Stevens, 2018; Westfall, Kenny, & Judd, 2014).”

“This observation count aligns with the recommendations of Brysbaert (2018) and Brysbaert and Stevens (2018), which for (g)LMM suggest 1600 observations per condition for within-group designs and twice as many for between-group designs, as used in our study.”

“Our sample size was defined based on guidelines for a simple two factors within-subjects (power at 0.8 and alpha at 0.05) design by Brysbaert (2019), which yielded a conservative minimum of 52 participants for an effect size of d = 0.4 (a typical effect in psychology; see Brysbaert & Stevens, 2018).”

“Based on Brysbaert and Steven’s (2018) recommendation to collect a minimum of 1,600 observations to ensure a well-powered design, we initially settled on a target n size of 36 participants, which when multiplied by the 75 questions in the delayed posttests (as only delayed posttest data were to be…”

“Recommendations for power analysis for a mixed effects model indicated that a sample size of at least 52 participants was needed to result in power of 0.8 (α -level = 0.05) to report medium effect sizes [12], [13].”

“However, based on simulations of RT experiments with repeated measures, Brysbaert and Stevens (2018) recommend a minimum of 1600 observations per condition for experiments with repeated measures.”

“This effect size has consistently been shown to be the average effect size in psychological science and is therefore considered to be a reliable guide when setting up new research (Brysbaert & Stevens, 2018).”

“According to the mental files theory, the observer has to decide which one of two vicarious files is to be deleted to represent the agent’s lack of access to one of the two identities of a dual-identity object in an intensionality situation.”

“Following Brysbaert and Stevens’ (2018) recommendation for at least 1,600 trials per condition, in both experiments presented in this article, we aimed to initially recruit at least 67 participants.”

“Scientists (Alyami & Mohsen, 2019; Brysbaert & Stevens, 2018; Rezaei & Mousanezhad Jeddi, 2020) offer two possible approaches to do such analysis.”

“The remaining dataset was composed of 28,760 observations, with at least 3153 observations per condition (see below for the exact number of observations for each condition), which exceeds the recommendation of Brysbaert and Stevens (2018) for having su ffi cient power.”

“…sizes for expectedness in DMPFC (d = 0.21), LTPJ (d = 0.11), and RTPJ (d = 0.15) ranged from more than 1.5× − 3× the standardized effect size for expectedness in PC (d = 0.07) (All d’s were calculated using variance estimates, as described in Brysbaert & Stevens, 2018; Westfall et al., 2014).”

“Because reaction times violate the assumption of normally distributed variables, they were converted to inverse reaction times, according to Brysbaert and Stevens (2018).”

“We report a measure of effect size d for LMMs, analogue to Cohen’s d , for the main effect of drug (Brysbaert and Stevens, 2018), and Cohen’s d and 95% CI for statistically significant pairwise comparisons.”

“265 We estimate effect sizes following [30, 31].”

“The remaining dataset was composed of 11,371 observations, a number that substantially exceeds the recommendation of Brysbaert and Stevens (2018) for having sufficient power.”

“This may seem small in comparison to the suggestion of sample sizes of forty advocated for by Brysbaert and Stevens (2018).”

“The expected power of fixed-effects was calculated a priori using the simr package in R (Brysbaert and Stevens, 2018).”

“To determine the size of the participant sample, we conducted a power analysis by simulation (see, e.g., Brysbaert & Stevens, 2018), based on Slovic and Fischhoff’s (1977) results with the same materials, assuming a conservative difference between foresight and hindsight (i.e., p = .10 compared…”

“This sample size was deemed sufficient because it ensured that we obtained approximately 3000 observations per condition which satisfies recently proposed criterion for properly powered experiments of this kind (Brysbaert & Stevens, 2018; Brysbaert, 2019).”

“Although there is still no complete agreement on the best practice to run a power analysis in a mixed effects model, the most reliable method to our knowledge is simulation-based power analysis (Brysbaert and Stevens 2018).”

“In response to new information regarding power in linear mixed model analyses (Brysbaert & Stevens, 2018), the target sample size was increased from 60 participants to 120 in order to achieve a sample of 1600 observations (1600 observations / 14 target words = 114 participants), allowing for some…”

“Our data collection stopping rule was to test participants until the end of the semester in which we reached 125 participants (exceeding the sample size needed to achieve 80% power with an alpha = 0.05; Brysbaert & Stevens, 2018).”

“This is despite having 40 participants and 48 items each per Childhood Frequency condition, above the minimum suggested for a properly powered contrast in mixed-effects models (Brysbaert & Stevens, 2018).”

“Brysbaert and Stevens (2018) stated that a properly powered experiment with repeated measures should have at least 1,600 observations per condition.”

“The sample size was sufficient to detect a small effect with participant as the highest level of analysis (Brysbaert & Stevens, 2018).”

“First, our sample size per cell was smaller than 40, which is recommended for obtaining sufficient power in a liner mixed-effect model analysis (Brysbaert & Stevens, 2018).”

“We therefore conducted a power analysis using the effect sizes obtained in Experiment 1 to estimate the sample size necessary to detect a reliable interaction (Brysbaert & Stevens, 2018).”

“Following recommendations (Brysbaert & Stevens, 2018) for analyzing continuous data, RTs were log-transformed prior to analysis to better approximate a normal distribution.”

“Instead, we followed Brysbaert and Stevens (2018) recommendation of including at least 1,600 observations per condition in a factorial design.”

“In accordance with the guideline provided by Brysbaert and Stevens (2018), which suggests a minimum of 1,600 observations per condition in designs involving repeated measures, and estimating a loss of 20% of them after removing errors and RT outliers (resulting in around 199,000 responses), we…”

“For RQ1–RQ3, we use mixed-effects models [5, 19] with random intercepts for students and exercises to account for repeated observations of each student and exercise.”

“Sample size was determined a priori following guidelines for linear mixed models (Brysbaert & Stevens, 2018), which recommend at least 1,600 observations per experimental condition, resulting in a minimum of approximately 27 participants.”

“Although there is still no complete agreement on the best practice to run a power analysis in a mixed-effects model, the most reliable method to our knowledge is simulation-based power analysis [48].”

“Note, however, that underlying SD s reflect total-trial variance which is dominated by residual (within-subject) variance and thus directly comparable only to effects in single-trial models rather than Cohen’s d based on means aggregated at the subject level (Brysbaert & Stevens, 2018).”

“…a target of 200 participants (50 per group) was set based on sample size recommendations for linear mixed models to achieve 80% power (Brysbaert and Stevens 2018), consistent with a G*Power calculation for classic linear models (effect size = 0.20, α = 0.05, power = 0.80; Faul et al.…”

“Neither assumption holds for the present designs, in which participants contributed many trials per condition (Brysbaert & Stevens, 2018).”

“Based on the best practices recommended by [39], 150 participants was determined to yield sufficient power for detecting a small to medium effect size for fixed https://doi.org/10.3390/jemr19050095 effects, with 2 levels and 40 observations at each level.”

“With 30 participants and 80 word trials in each priming condition per participant, this design yielded approximately 2,400 word observations per condition, providing a useful benchmark for stable estimation in masked priming experiments (Brysbaert & Stevens, 2018).”

“In addition, the design took into account the recommendations of Brysbaert and Stevens (2018) regarding statistical power in repeated-measures designs.”

“This sample size yielded 2,000 observations per condition, exceeding the recommendations for detect - ing small effects in mixed-effects models (Brysbaert & Stevens, 2018).”