Models of accuracy in repeated-measures designs

“In Simpson’s paradox (Simpson, 1951), for example, the pattern of overall means may not reflect the pattern observed in individual conditions or with individual subjects.”

“Furthermore, none of the LMEM articles in the JML special issue (Baayen et al., 2008; Barr, 2008; Dixon, 2008; Jaeger, 2008; Mirman, Dixon, & Magnuson, 2008; Quené & van den Bergh, 2008) set a”

“Specifically, measures of effect size for continuous data (such as Cohen’s d) are suboptimal for the analysis of accuracy data that are based on categorical veracity judgments (true vs. false; Dixon, 2008).”

“…for continuous response variables from a normally distributed population, mixed-effects models can also be used for categorical response variables (Dixon, 2008; Jaeger, 2008). lme4 includes the function glmer which can be used for count data (Poisson distribution), binary/proportion data…”

“Because our data were categorical, we used mixed logit models, which are designed for binomially distributed data (Dixon, 2008; Jaeger, 2008).”

“…switch position predescription or mid-description; verb language English or Dutch), logistic regression makes it possible to perform the analyses directly on the participants’ actual responses; it does not require aggregation to a mean response per condition (see also Dixon, 2008; Jaeger, 2008).”

“Because of recent concerns in the literature about the appropriateness of analyses of variance in psycholinguistics (Baayen, 2004; Baayen, Davidson, & Bates, 2008; Dixon, 2008; Jaeger, 2008), we performed generalized linear mixed-effects regression (Breslow & Clayton, 1993) as well as ANOVAs.”

“…traditional measures of sensitivity (Macmillan & Creelman, 2005); however, a major benefit of fitting the data to a generalized linear mixed model for signal detection is that such modeling is robust to finding differences between values at extreme ends of the binomial distribution (Dixon, 2008).”

“We used a logistic linking function to deal with the categorical nature of the dependent variable ([0,1]; cf., Dixon, 2008).”

“…(2) binomial/logistic mixed-effects regression permits a more appropriate handling of binary data than does percent correct (see Barr et al., 2013; Dixon, 2008; Jaeger, 2008); and (3) like ordinary regression, mixed-effects regression permits analysis of continuous as well as categorical…”

“First, we deal with the categorical nature of the dependent variable (using ANOVAs with untransformed proportions would be problematic; see Dixon 2008, Jaeger 2008).”

“-incorrect/timed-out distinction (see Barr et al., 2013; Dixon, 2008; Jaeger, 2008).”

“We analyzed accuracy using a generalized linear mixed model (GLMM) in the lme4 package (Bates et al., 2015) to account for the binomial distribution of trial-level accuracy (Dixon, 2008).”

“…(0 = short, 1 = long) differences in response accuracy (0 = incorrect, 1 = correct) with hierarchical Bayesian binomial logistic regression models, a more appropriate method of analyzing data on the proportion of correct responses than analysis of variance on aggregated proportions (Dixon, 2008).”

“…complexity of verbal descriptions varied according to age, time, feedback condition, and sticker array size, we fitted a mixed-effects logistic regression model to the data using Laplace approximation (Baayen, 2008; Baayen, Davidson, & Bates, 2008; Dixon, 2008; Gelman & Hill, 2007; Jaeger, 2008).”

“In the present study, a mixed-model was considered advantageous as it provides a more flexible method of dealing with missing data [27] as well as accounting for any variability in a participant’s performance across repeated conditions [28].”

“mation of identification results in data analysis (Dixon, 2008).”

“…data averaging and offers the possibility to examine the item effect longitudinally (Baayen et al. 2008); it has better generalization performance than traditional methods such as ANOVA (Barr et al. 2013); and finally, it has greater power to detect effects as compared with ANOVA (Dixon 2008).”

“To avoid the potential pitfalls of conventional methods (e.g., ANOVA and other normal models can lead to spurious results, particularly in the interpretation of interaction effects Dixon, 2008; Jaeger, 2008), data were analyzed using generalized linear mixed effects models (Bolker et al., 2009).”

“This is good practice for binomial data, as the model never reaches mean values below 0 or above 1 (Dixon, 2008).”

“As has been pointed out by Dixon (2008), the assumption of normality (in ANOVA) might often not be met when dealing with proportions, especially when accuracy percentages reach ceiling levels (as is the case here).”

“The mixed effects logistic regression allows using a larger number of data-points to include a more complete and theoretically-motivated random effects structure: specifically, one that specifies a random intercept for subject and a random slope for condition by subject (Barr et al., 2013; Dixon, 2008).”

“Specifically, we used the lme4 package [Bates et al. 2012 ] in R to complete linear mixed effects analyses with random intercepts for each participant (as recommended by Dixon [ 2008 ]), using maximum likelihood estimation (as recommended by Luke [ 2016 ]).”

“…logistic regression model of http://journals.cambridge.org Downloaded: 09 Jul 2016 IP address: 128.197.26.12 the likelihood of accuracy (Dixon, 2008; Jaeger, 2008), starting with random-effect terms for Participant and Item and adding fixed-effect terms for Final (stop, sonorant;…”

“5 Mixed logit models have proven to give more accurate results for proportional data than ANOVA (Dixon, 2008; Jaeger, 2008), which is used in the present study for simplicity.”

“Theorists have advised against the analysis of proportions in this manner, instead recommending the use of statistical models that treat the raw binary scores as arising from a binomial distribution (e.g., Dixon, 2008; Jaeger, 2008).”

“Recently, however, it has been recognized (Dixon, 2008; Jaeger, 2008) that the traditional approach achieves inferior inference relative to those like GLMM, which can handle error data in its native, binomial format.”

“…analyses are not met as the dependent variable, response accuracy, is not continuous, but discrete, and the distribution of the responses is not normal (being either correct or incorrect) and “this can lead to averaging artefacts and distortion in the means, standard errors” (Dixon, 2008, p. 1).”

“However, some individual participants' means were outside the range considered ideal for avoiding scaling artifacts with accuracy data (floor/ceiling effects are less likely in data sets with values that range between 25-75% correct; Dixon, 2008).”

“While task accuracy is easy to interpret, AN OVAs are not well suited to compare this data (43).”

“We estimated generalized linear mixed effect models (GLMMs) with a logit-link function (Dixon, 2008) with the R package lme4 (Bates et al.”

“This approach allowed us to avoid the problems associated with analyzing dichotomous outcomes using linear models (discussed by, e.g., Baayen, 2008; Dixon, 2008; Jaeger, 2008).”

“Because the data of this norming study (and of the subsequent experiments) are categorical, we analysed them with logit mixed effects models (Dixon, 2008; Jaeger, 2008).”

“A logistic linking function was used for the error patterns (cf. Dixon, 2008).”

“, arcsine square root), is liable to producing spurious interactions (Dixon, 2008), which is of particular concern in research on cognitive aging (Salthouse, 2000).”

“It has recently been shown that linear models such as ANOVAs lead to artefacts and distortions in means and standard errors when used with discrete dichotomic dependent variables (Dixon, 2008; Jaeger, 2008).”

“As a result, there is a growing body of literature that argues it is more appropriate to use generalized linear mixed-effects probit/logit models to analyze these types of accuracy data (Dixon, 2008; Jaeger, 2008; Quené and van den Bergh, 2008; Song et al., 2017).”

“Technically, we estimated Generalised Linear Mixed Models (GLMMs) with a logit link function (Dixon, 2008) and random effects (random 238 S.”

“For example, the word matching task employed in both the present study and CG04 results in very low error rates, reducing the power to detect effects on accuracy (for relevant power simulations, see Dixon, 2008).”

“Simulations have suggested that for proportional data, or indeed any data with a binary outcome, multilevel logistic regression may be a superior and less biased technique for analysis than standard analyses of variance (Dixon, 2008; Jaeger, 2008).”

“In this way, variations in accuracy were modeled as differences in log odds, which do not suffer from the limited scale of 0–1 and are therefore relatively robust to the unavoidable floor and ceiling effects observed near the endpoints of accuracy scales (Dixon, 2008).”

“The motivation to do so was to implement a better model of accuracy data and to use a larger number of data-points to include a more complete and theoretically motivated random effects structure: specifically, one that specifies a random intercept for subject and a random slope for condition by subject (Barr, Levy, Sc…”

“item differences by fitting data from individual trials instead of aggregating across all presentations of an item [33], [34].”

“Indeed, a special issue of the Journal of Memory and Language contained several excellent papers illustrating the negative consequences of using linear regression and related techniques to analyze proportions (Barr, 2008; Dixon, 2008; Jaeger, 2008; Mirman, Dixon, & Magnuson, 2008).”

“Following Dixon (2008) and Jaeger (2008), a repeated-measure logistic regression analysis was carried out in order to analyze accuracy data, using the statistical software R (R Development Core Team, 2008).11 ger (2008) demonstrated that, when data are categorical (binomially distributed), mixed…”

“Errors were analyzed with a generalized mixed-effects logistic regression (Dixon, 2008; Jaeger, 2008).”

“Note, however, that a percentage scale should not be treated as an interval scale (Dixon, 2008; cf. Jaeger, 2008), since differences in raw percentages close to floor or ceiling are more meaningful than in the middle range.”

“These problems are alleviated to some extent in binomial approaches via transformation to a logit scale, with accuracy rescaled into an unbounded “response-strength” measure (Dixon, 2008).”

“…data are commonly aggregated into proportions prior to analysis (e.g., using an Analysis of Variance; ANOVA), simulations have consistently demonstrated the superiority of statistical models that treat the raw binary scores as arising from a binomial distribution (e.g., Dixon, 2008; Jaeger, 2008).”

“The logOdds transformation helps to prevent artifacts due to the limited range of proportions and the related co-dependence of mean and variance in raw proportions (Dixon, 2008; Jaeger, 2008).”

“This approach prevents the spurious effects that can sometimes emerge from averaging over accuracy data and using analyses such as ANOVAs (Dixon, 2008).”

“Critically, the use of generalized mixed effect models with trial-wise data reduces ceiling effects in mean accuracy distributions (Dixon, 2008).”

“Accuracy data were analyzed using generalized linear mixed-effects models (Baayen, 2008), fitted with binomial distributions (Dixon, 2008) via a logit transformation (Jaeger, 2008), using package lme4 (Bates, Maechler, & Bolker, 2012) in R (R Core Team, 2012).”

“We analyzed the intrusion error rates, self-corrections rates, and paragraph reading times using mixed effects regressions (linear for reading times; logistic for error rates; Dixon, 2008).”

“4See Jaeger (2008) and Dixon (2008) for reasons to prefer mixed-effects logistic regression to analysis of variance for the analysis of accuracy data.”

“Ecke (2004) included a clever contrast by eliciting TOTs with definitions vs.”

“A mixed-effects method was selected because of its well-established advantages in terms of producing more accurate effect estimates and its ability to handle missing data, nonnormal outcome data, and unbalanced classes (Baayen, Davidson, & Bates, 2008; Dixon, 2008; Jaeger, 2008).”

“…increasingly common because they can accommodate both continuous (Baayen, Davidson, & Bates, 2008; Baayen & Milin, 2010) and binary response data (Dixon, 2008), which regularly violate assumptions of normality and homogeneity of variance in repeated-measures designs (Dixon, 2008; Jaeger, 2008),…”

“This is because a Gaussian has in fi nite support, which means it inevitably places probability mass on values below 0% and above 100% (for an alternative, see Dixon, 2008).”

“Experiment 2: stop identification in English The data from Experiment 2 were analyzed by building a logistic mixed-effects regression model of the log odds of correct identification (Dixon, 2008; Jaeger, 2008) in R (R Development Core Team, 2015).”

“Results and Statistical Analysis: To test our two hypotheses, we take guidance from [24] and perform a binomial logistic regression, where correctness is the outcome variable, and PAE and the baseline function are explanatory variables.”

“linear-mixed effects models using a binomial distribution better model accuracy data [Dixon, 2008].”

“Further, ANOVAs of categorical data may produce spurious results, and are less powerful than mixed effects models (Dixon, 2008).”

“The data were fitted into a generalized linear-mixed (logit) model using the maximum likelihood method (Laplace Approximation) (Baayen 2008, Dixon 2008, Matuschek et al. 2017).”

“Compared with a “classic” analysis of proportions of binary recognition per condition and per participant, GLMMs have several advantages, such as increased statistical power and being less prone to spurious results (Dixon, 2008; Jaeger, 2008).”

“…differences between groups and error types, trial-level data were submitted to generalized linear mixed-effects modeling for binomial distributions (Dixon, 2008) via a logit transformation (Jaeger, 2008) with maximal random structures (Barr, Levy, Scheepers, & Tily, 2013) including crossed random…”

“But note that ANOVA is not recommended for analysis of proportional data because of the restricted range in possible means, and potential for scaling artifact in conditions that approach floor or ceiling (Dixon, 2008; Jaeger, 2008; Winer, 1971).”

“Declerck & Grainger, 2017; Elston-Güttler et al., 2005; Mercier, Pivneva, & Titone, 2016; Paulmann, Elston-Güttler, Gunter, & Kotz, 2006). These studies show that shifting from one language block to the other influences subsequent performance. For instance, Misra et al. (2012) tested Chinese–English bilinguals in a pi…”

“Accuracy data was analyzed similarly but using generalized linear mixed-effects analysis using a binomial link function (Faraway, 2006; Dixon, 2008), an approach tantamount to logistic regression.”