Models of accuracy in repeated-measures designs

“In Simpson’s paradox (Simpson, 1951), for example, the pattern of overall means may not reflect the pattern observed in individual conditions or with individual subjects.”

“Data analysis was carried out using separate linear mixed-effects (LME) models for the accuracy (Dixon, 2008; Jaeger, 2008) and latency data (Tremblay, Derwing, Libben, & Westbury, 2011) in each separate learning and reading phase using R (R Development Core Team, 2014).”

“…for trial-level responses, nested within participants, and to model these responses with a more suitable distributional form (e.g., a binomial distribution to reflect the possibility of a correct or incorrect response on each trial) than the normal distribution assumed by ANOVA (e.g., Dixon, 2008).”

“All statistical analyses on these two dependent variables used logit mixed effect modeling (Dixon, 2008; Jaeger, 2008), as implemented in the lmer functions (lme4 package, Bates and Sarkar, 2009) of the R statistical program (Version 2.8.1; R Development Core Team, 2007).”

“Note that using ANOVA on data from an experimental design in which the dependent variable is an average over binary values (like proportion deleted in the current study) can be shown to be potentially misleading, and logistic regression cannot be used for repeatedmeasures designs (Dixon, 2008).”

“The point estimate thereby accounts for the distribution of the (experimental) population, which reduces the risk that it is unrepresentative of any given individual (i.e., Simpsons paradox; Dixon, 2008; Lee & Nelder, 2004).”

“The estimated predictor coefficients can be thought of as strengths of the predictors (expressed in a logit scale), and the logistic regression procedure can be regarded as a technique for estimating those strengths from the dichotomous responses (Dixon, 2008).”

“Reaction time data were analyzed using repeated measures ANOVA in SPSS version 20 (IBM Corporation, Armonk NY, USA).”

“Generalized Estimated Equations (GEE) with Gamma function1 was used instead of standard analysis of variance since rating values were discrete rather than continuous variables (Dixon, 2008).”

“Dixon (2008) demonstrated that for analyzing accuracy data, it is therefore desirable to use a logistic regression with binomial distribution in a GLMM.”

“The data were fitted into a generalized linear-mixed (logit) model using the maximum likelihood method (Laplace Approximation) (Baayen, 2008; Dixon, 2008; Matuschek et al., 2017).”

“Furthermore, generalized linear mixed effects models (an extension of LMM) allow for the analysis of non-normally distributed data, for instance, binary outcome variables composed into error rates (Bolker et al., 2009; e.g., Dixon, 2008; Jaeger, 2008).”

“Following Jaeger (2008) and Dixon (2008), a repeated-measure logistic regression analysis was carried out in order to analyze accuracy data, using the statistical software R (R Development Core Team, 2008). A group effect was found. G1 did not perform significantly different from G2 (Wald Z = –1.309, p = .191). The di…”

“Although our results showed that the dependent variables did not pass the Shapiro-Wilk test (alpha = 0.05) in some cases, it is known that ANOVA is robust against violations of the normality test assumptions [13, 36], and thus, we consistently ran repeated-measures ANOVAs.”

“Data were analysed by means of linear mixed-effect models (Dixon 2008; Jaeger 2008; an extensive description of themethod can be found inBaayen 2008; for a user-friendly overview tailored towards researcherswithout a strong computational background, seeCunnings 2012).”

“…analysis, since it has recently been argued that this analysis approach is the most powerful and appropriate way to control for between-participant variability, while also accounting for nonlinearities present in proportional data (i.e., 0–100% accuracy data) (Dixon, 2008; T. Florian Jaeger, 2008).”

“The logistic function removes the boundaries and compression that is experienced when error rates are very high or low (Dixon, 2008).”

“We estimated generalized linear mixed effect models with a logit-link function (Dixon, 2008) and linear mixed effect models with the R package lme4 (Version 1.1-21; Bates, Mächler, Bolker, & Walker, 2015).”

“Therefore, error rates will not be analyzed further (see Dixon, 2008 for comments regarding floor and ceiling effects in the analysis of error data).”

“We used generalised linear mixed-effect models (GLMMs) with a logit-link function (Dixon, 2008) to analyze correctness of provided answers in the criterial test.”

“We used a mixed factors analysis as suggest by Dixon (2008) for the analysis of proportion data.”

“For the statistical analysis, the results were analyzed separately with a linear mixed-effects model with a binomial linking function to account for the categorical nature of the dependent variable (c.f. Dixon, 2008).”

“This approach is more appropriate for dichotomous (0 or 1) accuracy data than analysis of variance and is analogous to a repeated measures nonlinear regression analysis (Dixon, 2008; Young et al., 2009).”

“1 The use of t tests on percentage correct scores was possible for our data because the scores fell within the 30%–70% range, as suggested by Jaeger (2008) and Dixon (2008).”

“the recommended approaches for accuracy data (Dixon, 2008; Jaeger, 2008).”

“In this example, the gestures are congruent with the speech without loss in power, as well as non-independence in the data (Dixon, 2008; Jaeger, 2008; Meteyard & Davies, 2020).”

“Using ANOVA models when the dependent variable is categorical (e.g., yes/no, counts, percentages) can lead to spurious significance values 40,43,44 .”

“54,55 Group (OFC, control), Emotional Valence (threat, neutral), and Emotion Relevance (relevant, irrelevant) were used as fixed effect predictors and Subject as a random effect predictor.”

“Comprehension accuracy and letter detection performance were analyzed using generalized linear models using the binomial family and a logistic link function as suggested by Dixon (2008).”

“This approach has several key advantages over traditional ANOVA approaches (for more details, see Dixon 2008; Jaeger 2008; Quené & van den Bergh 2008) applied to (transformed) average accuracy scores.”

“To better illustrate the source of the significant interactions, we derived the learning curve characteristics from the training data, which are usually referred to as threshold and slope in item response theory (see Baker, 2001; see also Dixon, 2008).”

“Although some of the data did not satisfy the assumptions, it is known that ANOVA is robust against violations of the test assumptions [12, 52].”

“Accuracy was analyzed with generalized mixed-effects models for binomial distributions (Dixon, 2008) via a logit transformation (Jaeger, 2008), using the same R function.”

“Second, for categorical dependent variables, logit models have been shown in simulations to possess greater statistical power (Dixon, 2008; Jaeger, 2008, p. 443).”

“A log odds ratio transform was applied on error rates (Dixon, 2008; Jaeger, 2008).”

“Accuracy was analyzed with generalized mixed-effects models for binomial distributions (Dixon, 2008) via a logit transformation (Jaeger, 2008), using the same model formula in function glmer of the lme4 package.”

“This analysis approach accounts for interdependence of the data arising from repeated measurements of the same participants and adjusts for non-normal distributions (Baayen, Davidson, & Bates, 2008; Dixon, 2008).”

“Since the dependent variable is a dichotomous variable (0 = errors; 1 = correct responses), it can be easily modeled using a GLMM for a logistic distribution [35]].”

“Because of this, the random effect was considered to be the effect of observer, of amplitude and observer, and of hemifield and observer (Baayen et al. 2008; Dixon 2008).”

“Dixon P (2008). Models of accuracy in repeated-measures designs. Journal of Memory and Language, 59, 447–456. Elston-Güttler KE, Gunter TC, & Kotz SA (2005). Zooming into L2: Global language context and adjustment affect processing of interlingual homographs in sentences. Cognitive Brain Research, 25, 57–70. [PubMed:…”

“However, this also means that errors are not approaching the upper bound and thus likely avoid underadditive or overadditive artefacts that would prompt the need for logistic regression analysis (Dixon, 2008).”

“While we plot data on its observed scale (i.e., percentage of judgments across all seven response categories) the modelling is done on log odds, considered a more appropriate latent scale (see Dixon, 2008).”

“Prior to analysis, response time data was inversely transformed to avoid potential problems associated with skewed data (Dixon, 2008) and response times for invalid trials were removed.”

“In contrast, trial-based analyses like those pursued here (and in Dixon, 2008) avoid averaging over items and model each binary score using a Bernoulli distribution.”

“Errors were analyzed using generalized binary logistic regression as proposed by Jaeger (2008) and Dixon (2008) so that Subject was used as a random effect predictor and Test and Distractor valence as fixed effect predictors.”

“We used logistic mixed-effects models to account for fixed and random effects arising from the repeated measures study design (e.g., OPEN MIND: Discoveries in Cognitive Science 95 Effects of Bilingualism on Cognition Dal Ben et al. Barr, 2008; Dixon, 2008; Humphrey & Swingley, 2018).”

“The alternative would be using a crossed-classified design with random effects of participants and items in a Generalized Linear Mixed Model (GLMM, Dixon, 2008).”

“The use of logistic regression allows examining accuracy independent of closed-scale limitations (Dixon, 2008) and might represent a promising approach for understanding profiles of reading errors in light of the presence of overadditive effects.”

“Likelihood ratio tests (Baayen et al., 2008; Dixon, 2008) were used to test whether the inclusion of the fixed and random factors significantly improved the overall fit of the model.”

“160 161 2.3.1 Behavioural Analysis 162 The proportion of trials with successful recall was modelled with a generalised linear mixed effects regression 163 model (Dixon, 2008) with the Laplace approximation as well as posterior simulation, using eye closure as a factor 164 (i.e. open and closed).”

“Data were analyzed using R (R Development Core Team, 2014) with the lme4 software package (Bates, Maechler, Bolker, & Walker, 2015; for the advantages of using linear mixed-e ff ects regression models for accuracy measures, see Dixon, 2008).”

“Therefore, to provide additional converging evidence, mixed logit models (a type of generalized linear mixed models; Dixon, 2008; Jaeger, 2008) with a maximal random effects structure (Barr, 2013; Barr et al.”

“In addition, we test models for nonlinearity, as prior research has shown that un-modeled nonlinear relationships may create a spurious interaction in linear regression analysis [16, 17].”

“To more accurately characterize both within- and acrosssubject error variance (Singmann & Kellen, 2019; Baayen, Davidson, & Bates, 2008; Dixon, 2008; Jaeger, 2008; Mumford & Poldrack, 2007; Clark, 1973), we implemented a mixed-modeling approach to evaluate statistical significance with the lme4 packing in R (Bates, Mä…”

“Modeling the log odds (or logit) of accuracy as a linear function of the factors in the design avoids issues such as averaging artifacts, which may occur when calculating proportion data for use in an ANOVA (see Dixon, 2008 for a comprehensive discussion of this problem).”

“To evaluate accuracy data, logistic regression has been proposed rather than standard analysis of variance because accuracy data are discrete rather than continuous (Dixon, 2008).”

“…models (LMEM) with crossed random effects for participants and items (Baayen, 2008; Baayen, Davidson, & Bates, 2008), using a binomial model (Dixon, 2008) via a logit link (Jaeger, 2008), fitted using restricted maximum likelihood by package lme4 (Bates, Maechler, Bolker, & Walker, 2014) in…”

“…using these methods can be misleading, among other things by underestimating standard errors and causing false-positive significance tests (for a discussion for continuous outcome variables see Richter, 2006; Baayen et al., 2008; for categorial outcome variables see Dixon, 2008; Jaeger, 2008).”

“The results were analyzed with a linear mixed-effect regression model, with a logistic linking function because of the categorical dependent variable (cf. Dixon, 2008).”

“Therefore, it should be noted that while we plotted data on its observed scale, the modelling is done on log odds, a more appropriate latent scale (see Dixon, 2008, for further details on the appropriateness of this scale).”

“In addition, LMEs reduce Type 1 error (Clark, 1973) and perform better in handling of unbalanced data sets and categorical data (Dixon, 2008; Jaeger, 2008). separately was motivated by two considerations.”

“Stress assignment proportions were compared using generalized linear mixed-effects models for binomial distributions (Dixon, 2008)”