Random effects structure for confirmatory hypothesis testing: Keep it maximal
Open accessPublished version of a preprint
Abstract
Linear mixed-effects models (LMEMs) have become increasingly prominent in psycholinguistics and related areas. However, many researchers do not seem to appreciate how random effects structures affect the generalizability of an analysis. Here, we argue that researchers using LMEMs for confirmatory hypothesis testing should minimally adhere to the standards that have been in place for many decades. Through theoretical arguments and Monte Carlo simulation, we show that LMEMs generalize best when they include the maximal random effects structure justified by the design. The generalization performance of LMEMs including data-driven random effects structures strongly depends upon modeling criteria and sample size, yielding reasonable results on moderately-sized samples when conservative criteria are used, but with little or no power advantage over maximal models. Finally, random-intercepts-only LMEMs used on within-subjects and/or within-items data from populations where subjects and/or items vary in their sensitivity to experimental manipulations always generalize worse than separate F1 and F2 tests, and in many cases, even worse than F1 alone. Maximal LMEMs should be the 'gold standard' for confirmatory hypothesis testing in psycholinguistics and beyond.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When the specified model failed to converge, the random effects structure was simplified following recommended procedures.
“When the specified model failed to converge, we simplified the random effects structure following recommended procedures [7]: first removing correlation parameters among random effects, then if necessary removing random slopes for higher-order interactions, and only as a last resort removing random slopes for main effects of theoretical interest.”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [7]
In “Coping with failures to converge” (page not available), the paper recommends considering removal of random correlations while retaining slopes for predictors of theoretical interest. It does not prescribe the cited sequence of removing higher-order interaction slopes before main-effect slopes, and it advises retaining critical slopes rather than treating their removal as a standard last resort. (LLM, conf=0.86)
Experimental — accuracy not yet measuredNon-balanced design with non-independent observations is accounted for by fitting mixed effects models with random intercepts for strata, challenges, and individuals.
“Essentially, we have a non-balanced design with non-independent observations and we account for it by fitting mixed effects models with random intercepts for strata, challenges, and individuals (Barr, Levy, Scheepers, & Tily, 2013).”
From Relative Feedback Increases Disparities in Effort and Performance in Crowdsourcing Contests (2022) · cites this paper as Barr, Levy, Scheepers, & Tily, 2013
In “Random effects in LMEMs and ANOVA,” the paper explains that random intercepts can account for clustering and non-independence across subjects and items. It does not prescribe intercepts for the citing study’s particular strata, challenges, and individuals. In “Identifying the maximal random effects structure,” it also advises trying random slopes for unbalanced within-unit designs, so intercepts alone are not generally sufficient. (LLM, conf=0.87)
Experimental — accuracy not yet measuredThe statistical model selection started with a maximal random effect structure imposed by the experimental design, and if it did not converge, random slopes and intercepts were iteratively reduced until convergence; if more than one model converged, the best-fit model was selected by AIC.
“To identify the statistical model that would converge for each dependent variable, we started with a model that includes the dependent variable as a response variable with no fixed effects and a maximal random effect structure imposed by the experimental design (Barr et al., 2013). If the model did not reach convergence, we iteratively reduced the random effect structure by excluding random slopes”
From The influence of landmark visualization style on task performance, visual attention, and spatial learning in a real-world navigation task (2024) · cites this paper as Barr et al., 2013
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredWhen non-Gaussian models with multiple random effects fail to converge, researchers often settle for a model that ignores non-independence and yields estimates with inappropriately high confidence and statistical significance.
“which often lets researchers settle for a model that ignores non-independence and yields estimates with inappropriately high confidence and statistical significance (Arnqvist, 2020; Barr et al., 2013; Forstmeier et al., 2017)”
From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Barr et al., 2013
In ‘Coping with failures to converge’ (page not provided), Barr et al. say maximal models often fail to converge, especially for categorical data, and warn against falling back to random-intercepts-only models. Their Results and Discussion say omitting relevant random effects can produce anticonservative tests. The paper does not establish how often researchers actually settle for such models, and it cautions that its results for continuous data may need modification for noncontinuous data. (LLM, conf=0.83)
Experimental — accuracy not yet measuredA simulation that treats all workers as identical would fail to capture variation and might overestimate power.
“A simulation which treated all workers as identical would fail to capture this variation, and hence might overestimate power (Barr et al., 2013).”
From With Little Power Comes Great Responsibility (2020) · cites this paper as Barr et al., 2013
The Method (Generating simulated data; page unavailable) includes variation among subjects in the simulations, and the Method (Performance metrics; page unavailable) warns that anticonservative analyses can have inflated apparent power. This supports the general concern about omitting individual variation, but the paper does not specifically test a simulation that treats workers as identical or show that it overestimates power. (LLM, conf=0.82)
Experimental — accuracy not yet measured
When models failed to converge, the random effects structure was simplified following Barr et al. (2005), documenting all attempted specifications.
“When models failed to converge, we simplified the random effects structure following Barr et al. [7], documenting all attempted specifications.”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [7]
The maximal random-effects structure was maintained as justified by the design.
“Thus, we maintained the maximal random-effects structure justified by the design (Barr et al., 2013).”
From Overconfidence without Understanding: AI Explanations Increase the Illusion of Explanatory Depth (2025) · cites this paper as Barr et al., 2013
Problems with pseudoreplication due to issues with model convergence can lead to anticonservative p values and a high risk of false-positive claims.
“as well as problems with pseudoreplication due to issues with model convergence (Arnqvist, 2020; Barr et al., 2013; Forstmeier et al., 2017).”
From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Barr et al., 2013
The random-effects structure was close to the maximal model, except that interaction terms were not included as random slopes.
“This randomeffects structure was close to the maximal model (Barr, Levy, Scheepers, & Tily, 2013), except that we did not include the interaction terms as random slopes.”
From Effects of Availability, Contingency, and Formulaicity on the Accuracy of English Grammatical Morphemes in Second Language Writing (2022) · cites this paper as Barr, Levy, Scheepers, & Tily, 2013
To keep type I error rate at the nominal 5%, random slopes were included for both proximity and period type within-subject.
“To keep type I error rate at the nominal 5%, we included random slopes (46, 47) for both proximity and period type within-subject.”
From Oxytocin reactivity during intergroup conflict in wild chimpanzees (2017) · cites this paper as [46]
Random slopes were included for the test predictors (event or anticipation) as well as for subgroup size, rank, and proximity within-subject.
“We included random slopes (46, 47) for the test predictors (i.e., event or anticipation) as well as for subgroup size, rank, and proximity within-subject.”
From Oxytocin reactivity during intergroup conflict in wild chimpanzees (2017) · cites this paper as [46]
Following the recommendation of Barr et al., maximal models were used, including participants as a random effect with both random intercepts and random slopes.
“Following the recommendation of Barr et al.87, we used maximal models, including participants as a random effect with both random intercepts and random slopes.”
From Acute social isolation evokes midbrain craving responses similar to hunger (2020) · cites this paper as [87]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Generalized linear mixed models: a practical guide for ecology and evolution
Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J.-S. S. (2009) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2008.10.008 · Cited by 8,910 · see citing works
Mixed-effects modeling with crossed random effects for subjects and items
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.12.005 · Cited by 8,540 · see citing works
Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models
Jaeger, T. F. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.11.007 · Cited by 3,317 · cited 4× · see citing works
“Furthermore, none of the LMEM articles in the JML special issue (Baayen et al., 2008; Barr, 2008; Dixon, 2008; Jaeger, 2008; Mirman, Dixon, & Magnuson, 2008; Quené & van den Bergh, 2008) set a”
InfluentialAn Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · see citing works
Conclusions beyond support: overconfident estimates in mixed models
Schielzeth, H., & Forstmeier, W. (2009) · Behavioral Ecology · doi:10.1093/beheco/arn145 · Cited by 952 · see citing works
Redundancy and reduction: Speakers manage syntactic information density
Jaeger, T. F. (2010) · Cognitive Psychology · doi:10.1016/j.cogpsych.2010.02.002 · Cited by 728 · see citing works
On multi-level modeling of data from repeated measures designs: a tutorial
Quené, H., & van den Bergh, H. (2004) · Speech Communication · doi:10.1016/j.specom.2004.02.004 · Cited by 642 · see citing works
Examples of mixed-effects modeling with crossed random effects and with binomial data
Quené, H., & van den Bergh, H. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2008.02.002 · Cited by 534 · see citing works
Mirman, D., Dixon, J., & Magnuson, J. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.11.006 · Cited by 517 · see citing works
Models of accuracy in repeated-measures designs
Dixon, P. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.11.004 · Cited by 359 · cited 3× · see citing works
“Furthermore, none of the LMEM articles in the JML special issue (Baayen et al., 2008; Barr, 2008; Dixon, 2008; Jaeger, 2008; Mirman, Dixon, & Magnuson, 2008; Quené & van den Bergh, 2008) set a”
On the use of multilevel modeling as an alternative to items analysis in psycholinguistic research
Locker, L., Hoffman, L., & Bovaird, J. A. (2007) · Behavior Research Methods · doi:10.3758/bf03192962 · Cited by 113 · see citing works
Janssen, D. P. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0145-1 · Cited by 45 · see citing works
Flagged first, then by citations
Research works printed but not linked: 16
- Baayen RH Analyzing linguistic data: A practical introduction to statistics using R 2008 Cambridge Cambridge University Press
- Barr DJ Analyzing ‘visual world’ eyetracking data using multilevel logistic regression Journal of Memory and Language 2008 59 457 474
- Clark HH The language-as-fixed-effect fallacy: A critique of language statistics in psychological research Journal of Verbal Learning and Verbal Behavior 1973 12 335 359
- Forster K Dickinson R More on the language-as-fixed-effect fallacy: Monte carlo estimates of error rates for F1,F 2,F′, and min F′ Journal of Verbal Learning and Verbal Behavior 15 1976 135 142
- Gelman A Hill J Data Analysis Using Regression and Multilevel/Hierarchical Models 2007 Cambridge, MA Cambridge University Press
- Hadfield JD MCMC methods for multi-response generalized linear mixed models: The MCMCglmm R package Journal of Statistical Software 2010 33 1 22
- Kliegl R Toward a perceptual-span theory of distributed processing in reading: A reply to Rayner, Pollatsek, Drieghe, Slattery, and Reichle (2007) Journal of Experimental Psychology: General 2007 136 530 537 17696697
- Kuperman V Bertram R Baayen RH Processing trade-offs in the reading of Dutch derived words Journal of Memory and Language 2010 62 83 97
- Pinheiro JC Bates DM Mixed-Effects Models in S and S-PLUS 2000 New York Springer
- Plummer M JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling Proceedings of the 3rd International Workshop on Distributed Statistical Computing 2003
- Raaijmakers JGW Schrijnemakers JMC Gremmen F How to deal with “ the Language-as-Fixed-effect fallacy”: Common misconceptions and alternative solutions Journal of Memory and Language 1999 41 416 426
- Roland D Relative clauses remodeled: The problem with mixed-effect models 2009 Poster presentation at the 2009 CUNY Sentence Processing Conference
- Santa JL Miller JJ Shaw ML Using Quasi F to prevent alpha inflation due to stimulus variation Psychological Bulletin 86 1979 37 46
- Scheffe H The analysis of variance 1959 New York Wiley
- Wickens TD Keppel G On the choice of design and of test statistic in the analysis of experiments with sampled materials Journal of Verbal Learning and Verbal Behavior 1983 22 296 309
- Winer BJ Statistical principles in experimental design 1971 2nd ed. New York McGraw-Hill
Cited by
Sorted by citations
A brief introduction to mixed effects modelling and multi-model inference in ecology
Harrison, X. A., Donaldson, L., Correa-Cano, M. E., Evans, J., Fisher, D. N., Goodwin, C. E. D., Robinson, B. S., Hodgson, D. J., & Inger, R. (2018) · PeerJ · doi:10.7717/peerj.4794 · Cited by 2,094 · see citing works
Fixed and random effects models: making an informed choice
Bell, A., Fairbrother, M., & Jones, K. (2019) · Quality & Quantity · doi:10.1007/s11135-018-0802-x · Cited by 1,085 · see citing works
The effect of word predictability on reading time is logarithmic
Smith, N. J., & Levy, R. (2013) · Cognition · doi:10.1016/j.cognition.2013.02.013 · Cited by 725 · see citing works
Self-citationViolating the normality assumption may be the lesser of two evils
Knief, U., & Forstmeier, W. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01587-5 · Cited by 654 · see citing works
Yarkoni, T. (2022) · Behavioral and Brain Sciences · doi:10.1017/s0140525x20001685 · Cited by 701 · see citing works
Fecal microbiota transplantation plus anti-PD-1 immunotherapy in advanced melanoma: a phase I trial
Routy, B., Lenehan, J. G., Miller, W. H., Jamal, R., Messaoudene, M., Daisley, B. A., Hes, C., Al, K. F., Martinez-Gili, L., Punčochář, M., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02453-x · Cited by 526 · see citing works
Lethal aggression in Pan is better explained by adaptive strategies than human impacts
Wilson, M. L., Boesch, C., Fruth, B., Furuichi, T., Gilby, I. C., Hashimoto, C., Hobaiter, C. L., Hohmann, G., Itoh, N., Koops, K., et al. (2014) · Nature · doi:10.1038/nature13727 · Cited by 474 · see citing works
Experiments with More Than One Random Factor: Designs, Analytic Models, and Statistical Power
Judd, C. M., Westfall, J., & Kenny, D. A. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033702 · Cited by 466 · see citing works
Having a Lot of a Good Thing: Multiple Important Group Memberships as a Source of Self-Esteem
Jetten, J., Branscombe, N. R., Haslam, S. A., Haslam, C., Cruwys, T., Jones, J. M., Cui, L., Dingle, G., Liu, J., Murphy, S., et al. (2015) · PLOS ONE · doi:10.1371/journal.pone.0124609 · Cited by 308 · see citing works
Acute social isolation evokes midbrain craving responses similar to hunger
Tomova, L., Wang, K. L., Thompson, T., Matthews, G. A., Takahashi, A., Tye, K. M., & Saxe, R. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-00742-z · Cited by 283 · see citing works
Toward a principled Bayesian workflow in cognitive science.
Schad, D. J., Betancourt, M., & Vasishth, S. (2021) · Psychological Methods · doi:10.1037/met0000275 · Cited by 252 · see citing works
Encoding of Articulatory Kinematic Trajectories in Human Speech Sensorimotor Cortex
Chartier, J., Anumanchipalli, G. K., Johnson, K., & Chang, E. F. (2018) · Neuron · doi:10.1016/j.neuron.2018.04.031 · Cited by 208 · see citing works
Methods in cognitive pupillometry: Design, preprocessing, and statistical analysis
Mathôt, S., & Vilotijević, A. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01957-7 · Cited by 235 · see citing works
Bornemann, B., & Singer, T. (2017) · Psychophysiology · doi:10.1111/psyp.12790 · Cited by 204 · see citing works
Oxytocin reactivity during intergroup conflict in wild chimpanzees
Samuni, L., Preis, A., Mundry, R., Deschner, T., Crockford, C., & Wittig, R. M. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1616812114 · Cited by 207 · see citing works
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
“Mixed effects models capture such potential item aspects very well (e.g., Barr et al., 2013), and deserve more attention in NLG. Finally, Bayesian models are worth exploring, because they are less sensitive to the aforementioned problems with NHST (e.g., Gelman et al., 2006; Wagenmakers, 2007).”
BackgroundDimsdale-Zucker, H. R., Ritchey, M., Ekstrom, A. D., Yonelinas, A. P., & Ranganath, C. (2018) · Nature Communications · doi:10.1038/s41467-017-02752-1 · Cited by 205 · see citing works
Hesser, H. (2015) · Internet Interventions · doi:10.1016/j.invent.2015.02.003 · Cited by 192 · see citing works
Linde-Domingo, J., Treder, M. S., Kerrén, C., & Wimber, M. (2019) · Nature Communications · doi:10.1038/s41467-018-08080-2 · Cited by 152 · see citing works
Understanding Mixed-Effects Models Through Data Simulation
DeBruine, L. M., & Barr, D. J. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920965119 · Cited by 148 · see citing works
Self-citationNeural correlates of expected risks and returns in risky choice across development
van Duijvenvoorde, A. C. K., Huizenga, H. M., Somerville, L. H., Delgado, M. R., Powers, A., Weeda, W. D., Casey, B. J., Weber, E. U., & Figner, B. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.1924-14.2015 · Cited by 141 · see citing works
Reasoning supports utilitarian resolutions to moral dilemmas across diverse measures.
Patil, I., Zucchelli, M. M., Kool, W., Campbell, S., Fornasier, F., Calò, M., Silani, G., Cikara, M., & Cushman, F. (2021) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000281 · Cited by 133 · see citing works
Bayesian or biased? Analytic thinking and political belief updating
Tappin, B. M., Pennycook, G., & Rand, D. G. (2020) · Cognition · doi:10.1016/j.cognition.2020.104375 · Cited by 127 · see citing works
Reasoning in Reference Games: Individual- vs. Population-Level Probabilistic Modeling
Franke, M., & Degen, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0154854 · Cited by 115 · see citing works
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124 · see citing works
“, all the measurements were not statistically independent) [3].”
BackgroundContextual priming of word meanings is stabilized over sleep
Gaskell, M. G., Cairney, S. A., & Rodd, J. M. (2019) · Cognition · doi:10.1016/j.cognition.2018.09.007 · Cited by 98 · see citing works
Newman, M. G., Jacobson, N. C., Zainal, N. H., Shin, K. E., Szkodny, L. E., & Sliwinski, M. J. (2019) · Clinical Psychological Science · doi:10.1177/2167702619827019 · Cited by 96 · see citing works
Experiential reward learning outweighs instruction prior to adulthood
Decker, J. H., Lourenco, F. S., Doll, B. B., & Hartley, C. A. (2015) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-014-0332-5 · Cited by 96 · see citing works
The social media context interferes with truth discernment
Epstein, Z., Sirlin, N., Arechar, A., Pennycook, G., & Rand, D. (2023) · Science Advances · doi:10.1126/sciadv.abo6169 · Cited by 100 · see citing works
Hayat, H., Marmelshtein, A., Krom, A. J., Sela, Y., Tankus, A., Strauss, I., Fahoum, F., Fried, I., & Nir, Y. (2022) · Nature Neuroscience · doi:10.1038/s41593-022-01107-4 · Cited by 91 · see citing works
Schröder, J., Brückner, K., Fischer, A., Lindenau, M., Köther, U., Vettorazzi, E., & Moritz, S. (2014) · Epilepsia · doi:10.1111/epi.12833 · Cited by 96 · see citing works
Does working memory training have to be adaptive?
von Bastian, C. C., & Eschen, A. (2016) · Psychological Research · doi:10.1007/s00426-015-0655-z · Cited by 83 · see citing works
With Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84 · see citing works
Barthel, M., Sauppe, S., Levinson, S. C., & Meyer, A. S. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01858 · Cited by 83 · see citing works
Quique, Y. M., Evans, W. S., & Dickey, M. W. (2019) · American Journal of Speech-Language Pathology · doi:10.1044/2018_ajslp-17-0155 · Cited by 81 · see citing works
Unifying structural priming effects on syntactic choices and timing of sentence generation
Segaert, K., Wheeldon, L., & Hagoort, P. (2016) · Journal of Memory and Language · doi:10.1016/j.jml.2016.03.011 · Cited by 78 · see citing works
Human Effort and Machine Learnability in Computer Aided Translation
Green, S., Wang, S. I., Chuang, J., Heer, J., Schuster, S., & Manning, C. D. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1130 · Cited by 68 · see citing works
N400 and P600 modulation in presupposition accommodation: The effect of different trigger types
Domaneschi, F., Canal, P., Masia, V., Lombardi Vallauri, E., & Bambini, V. (2018) · Journal of Neurolinguistics · doi:10.1016/j.jneuroling.2017.08.002 · Cited by 69 · see citing works
Stress Impairs Episodic Retrieval by Disrupting Hippocampal and Cortical Mechanisms of Remembering
Gagnon, S. A., Waskom, M. L., Brown, T. I., & Wagner, A. D. (2019) · Cerebral Cortex · doi:10.1093/cercor/bhy162 · Cited by 69 · see citing works
Contextual predictability shapes signal autonomy
Winters, J., Kirby, S., & Smith, K. (2018) · Cognition · doi:10.1016/j.cognition.2018.03.002 · Cited by 54 · see citing works
Martínez-Schuldt, R. D., & Martínez, D. E. (2021) · American Sociological Review · doi:10.1177/0003122420978406 · Cited by 59 · see citing works
Bögels, S., Casillas, M., & Levinson, S. C. (2018) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2017.12.028 · Cited by 55 · see citing works
Endogenous oxytocin, cortisol, and testosterone in response to group singing
Bowling, D. L., Gahr, J., Ancochea, P. G., Hoeschele, M., Canoine, V., Fusani, L., & Fitch, W. T. (2022) · Hormones and Behavior · doi:10.1016/j.yhbeh.2021.105105 · Cited by 51 · see citing works
Overusing the pacifier during infancy sets a footprint on abstract words processing
BARCA, L., MAZZUCA, C., & BORGHI, A. M. (2020) · Journal of Child Language · doi:10.1017/s0305000920000070 · Cited by 48 · see citing works
Breakdown of local information processing may underlie isoflurane anesthesia effects
Wollstadt, P., Sellers, K. K., Rudelt, L., Priesemann, V., Hutt, A., Fröhlich, F., & Wibral, M. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005511 · Cited by 49 · see citing works
Kraus, B., & Kitayama, S. (2019) · Biological Psychology · doi:10.1016/j.biopsycho.2019.107733 · Cited by 46 · see citing works
Beer, Z., Vavra, P., Atucha, E., Rentzing, K., Heinze, H.-J., & Sauvage, M. M. (2018) · PLoS Biology · doi:10.1371/journal.pbio.2006100 · Cited by 44 · see citing works
Articulatory suppression delays processing of abstract words: the role of inner speech
Fini, C., Zannino, G. D., Orsoni, M., Carlesimo, G. A., Benassi, M., & Borghi, A. M. (2022) · Quarterly Journal of Experimental Psychology · doi:10.1177/17470218211053623 · Cited by 41 · see citing works
Speech Planning at Turn Transitions in Dialog Is Associated With Increased Processing Load
Barthel, M., & Sauppe, S. (2019) · Cognitive Science · doi:10.1111/cogs.12768 · Cited by 42 · see citing works
Value attributed to text-based archives generated by artificial intelligence
Darda, K., Carre, M., & Cross, E. (2023) · Royal Society Open Science · doi:10.1098/rsos.220915 · Cited by 37 · see citing works
“However, given recommendations for the ‘ keep it maximal ’ approach to multi-level modelling [44], we further included the maximal number of random slopes and intercepts that the design permitted for our main effects of interest.”
BackgroundMelo, I., Llusia, D., Bastos, R. P., & Signorelli, L. (2021) · Ecological Indicators · doi:10.1016/j.ecolind.2021.108305 · Cited by 40 · see citing works
Zhang, L., Lee, A. J., DeBruine, L. M., & Jones, B. C. (2019) · Evolutionary Psychology · doi:10.1177/1474704919852921 · Cited by 40 · see citing works
Son, J.-Y., Bhandari, A., & FeldmanHall, O. (2019) · Scientific Reports · doi:10.1038/s41598-019-48050-2 · Cited by 37 · see citing works
Wu, T., Weld, D. S., & Heer, J. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3319616 · Cited by 39 · see citing works
Zhou, P., Zhan, L., & Ma, H. (2019) · Journal of Psycholinguistic Research · doi:10.1007/s10936-018-9612-5 · Cited by 38 · see citing works
Mathiasen, K., Andersen, T. E., Lichtenstein, M. B., Ehlers, L. H., Riper, H., Kleiboer, A., & Roessler, K. K. (2022) · Journal of Medical Internet Research · doi:10.2196/36577 · Cited by 34 · see citing works
Teachers’ gestures and students’ learning: sometimes “hands off” is better
Yeo, A., Ledesma, I., Nathan, M. J., Alibali, M. W., & Church, R. B. (2017) · Cognitive Research: Principles and Implications · doi:10.1186/s41235-017-0077-0 · Cited by 34 · see citing works
Groß, J., Blank, H., & Bayen, U. J. (2017) · Clinical Psychological Science · doi:10.1177/2167702617712262 · Cited by 32 · see citing works
An Evaluation of the KiVa Anti-bullying Program in New Zealand
Green, V. A., Woods, L., Wegerhoff, D., Harcourt, S., & Tannahill, S. (2020) · International Journal of Bullying Prevention · doi:10.1007/s42380-019-00034-6 · Cited by 31 · see citing works
Murakami, A., & Ellis, N. C. (2022) · Language Learning · doi:10.1111/lang.12500 · Cited by 26 · see citing works
Binning, K. R., Cook, J. E., Greenaway, V. P., Garcia, J., Apfel, N., Sherman, D. K., & Cohen, G. L. (2021) · Journal of Social Issues · doi:10.1111/josi.12461 · Cited by 25 · see citing works
Kapaj, A., Lanini-Maggi, S., Hilton, C., Cheng, B., & Fabrikant, S. I. (2023) · Cartography and Geographic Information Science · doi:10.1080/15230406.2023.2183525 · Cited by 24 · see citing works
Relationship quality affects social stress buffering in dogs and wolves
Cimarelli, G., Marshall-Pescini, S., Range, F., Berghänel, A., & Virányi, Z. (2021) · Animal Behaviour · doi:10.1016/j.anbehav.2021.06.008 · Cited by 22 · see citing works
Semi-field studies to better understand the impact of eave tubes on mosquito mortality and behaviour
Barreaux, A. M. G., Brou, N., Koffi, A. A., N’Guessan, R., Oumbouke, W. A., Tia, I. Z., & Thomas, M. B. (2018) · Malaria Journal · doi:10.1186/s12936-018-2457-4 · Cited by 19 · see citing works
Enhanced memory for negative social information in borderline personality disorder.
Niedtfeld, I., Renkewitz, F., Mädebach, A., Hillmann, K., Kleindienst, N., Schmahl, C., & Schulze, L. (2020) · Journal of Abnormal Psychology · doi:10.1037/abn0000540 · Cited by 22 · see citing works
Kapaj, A., Hilton, C., Lanini-Maggi, S., & Fabrikant, S. I. (2024) · Spatial Cognition & Computation · doi:10.1080/13875868.2024.2328099 · Cited by 20 · see citing works
“To identify the statistical model that would converge for each dependent variable, we started with a model that includes the dependent variable as a response variable with no fixed effects and a maximal random effect structure imposed by the experimental design (Barr et al., 2013).”
MethodQianchen, L., Gallagher, R. M., & Tsuchiya, N. (2022) · Royal Society Open Science · doi:10.1098/rsos.210394 · Cited by 21 · see citing works
Teng, D., Tan, L., Cao, Q., Xia, Y., Zhang, N., Li, J., & Zhao, D. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02547-z · Cited by 18 · see citing works
Executive control and faithfulness: Only long-term romantic relationships require prefrontal control
Ueda, R., Yanagisawa, K., Ashida, H., & Abe, N. (2018) · Experimental Brain Research · doi:10.1007/s00221-018-5181-y · Cited by 15 · see citing works
Use of morphological and contextual cues in children’s lexical inferencing in L1 and L2
Raudszus, H., Segers, E., & Verhoeven, L. (2021) · Reading and Writing · doi:10.1007/s11145-021-10122-z · Cited by 15 · see citing works
Effects of learner choice over automated, immediate feedback
Kuklick, L. (2025) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.102065 · Cited by 15 · see citing works
Development and Validation of the Single-Item Mindfulness Scale (SIMS)
Meier, B. P., Konrath, S., Fetterman, A. K., Dillard, A. J., James, C., Weinstein, E., & Bushman, B. J. (2023) · Journal of Personality Assessment · doi:10.1080/00223891.2022.2152348 · Cited by 14 · see citing works
McNamara, R. A., Senanayake, R., Willard, A. K., & Henrich, J. (2021) · Evolutionary Human Sciences · doi:10.1017/ehs.2021.1 · Cited by 12 · see citing works
High spatial frequency filtered primes hastens happy faces categorization in autistic adults
Lacroix, A., Nalborczyk, L., Dutheil, F., Kovarski, K., Chokron, S., Garrido, M., Gomot, M., & Mermillod, M. (2021) · Brain and Cognition · doi:10.1016/j.bandc.2021.105811 · Cited by 12 · see citing works
Barreaux, A. M. G., Oumbouke, W. A., Tia, I. Z., Brou, N., Koffi, A. A., N’guessan, R., & Thomas, M. B. (2019) · Malaria Journal · doi:10.1186/s12936-019-2936-2 · Cited by 10 · see citing works
Scimeca, M., Peñaloza, C., & Kiran, S. (2024) · Bilingualism: Language and Cognition · doi:10.1017/s1366728923000391 · Cited by 11 · see citing works
How do listeners time response articulation when answering questions? The role of speech rate.
Corps, R. E., Gambi, C., & Pickering, M. J. (2020) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/xlm0000759 · Cited by 10 · see citing works
Tamminen, K. A., Danyluck, C., Bonk, D., & Chen, R. (2023) · Journal of Sports Sciences · doi:10.1080/02640414.2023.2259205 · Cited by 8 · see citing works
Sustained conscious access to incidental memories in RSVP
Matthews, J., Wu, J., Corneille, V., Hohwy, J., van Boxtel, J., & Tsuchiya, N. (2019) · Attention, Perception, & Psychophysics · doi:10.3758/s13414-018-1600-1 · Cited by 9 · see citing works
Relative Feedback Increases Disparities in Effort and Performance in Crowdsourcing Contests
Tsvetkova, M., Müller, S., Vuculescu, O., Ham, H., & Sergeev, R. A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555649 · Cited by 8 · see citing works
Relevance versus big numbers: Students’ criteria for selecting scholarly references online.
Rouet, J.-F., Skov, O., De Pereyra, G., Ros, C., Le Bigot, L., & Vibert, N. (2018) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000194 · Cited by 6 · see citing works
What is the interference in ‘verbal interference’?
Gerwien, J., von Stutterheim, C., & Rummel, J. (2022) · Acta Psychologica · doi:10.1016/j.actpsy.2022.103774 · Cited by 3 · see citing works
Vahedi, J., Bahic, A., Chaliani, I., Schilbach, L., Ciupka‐Schön, B., Bellebaum, C., Pietrowsky, R., & Peterburs, J. (2025) · Psychophysiology · doi:10.1111/psyp.70142 · Cited by 3 · see citing works
Challenges in Automatic Speech Recognition for Adults with Cognitive Impairment
Cohn, M., Lanzi, A., Ishihara, Y., Chuah, C.-N., Zellou, G., & Weakley, A. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791581 · Cited by 4 · see citing works
“In the cases of singularity or convergence errors, the random effects structure was simplified [9].”
Mathiasen, K., Andersen, T. E., Lichtenstein, M. B., Riper, H., Kleiboer, A., & Roessler, K. K. (2022) · doi:10.2196/preprints.36577 · Cited by 2 · see citing works
All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs?
Kim, S., Janssen, M. A., Wang, J., Min-Venditti, A., Karanjia, N., & Anderies, J. M. (2026) · arXiv · doi:10.48550/arxiv.2604.17247
Predicting emotional valence in autism: a preregistered study in the Bayesian Brain framework
Plank, I. S., Pior, A., Yurova, A., Nowak, J., Shi, Z., & Falter-Wagner, C. M. (2026) · Molecular Autism · doi:10.1186/s13229-026-00730-3 · Cited by 2 · see citing works
Overconfidence without Understanding: AI Explanations Increase the Illusion of Explanatory Depth
Aslanov, I., Felmer, P., & Guerra, E. (2025) · doi:10.31234/osf.io/8psgf_v1
- Authors
- Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J.
- Affiliations
- University of Glasgow (GB)
- Massachusetts Institute of Technology (US)
- Corresponding author
- Barr, Dale J.
- Venue
- Journal of Memory and Language
- Publisher
- Elsevier BV
- Published
- 2013-04
- Volume
- 68
- Issue
- 3
- Pages
- 255-278
- ISSN
- 0749-596X (ISSN-L)
- PMC
- PMC3881361
- OpenAlex
- W2141845152
- Cited by
- 10,328
- References
- 12
- Citations by year
- 2026: 932 · 2025: 1029 · 2024: 937 · 2023: 928 · 2022: 991 · 2021: 1094 · 2020: 952 · 2019: 802 · 2018: 726 · 2017: 597 · 2016: 507 · 2015: 433 · 2014: 290 · 2013: 97 · 2012: 4
- FWCI
- 360.53
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of Memory and Language (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- Funders
- Economic and Social Research Council (ES/G045720/1)
- Eunice Kennedy Shriver National Institute of Child Health and Human Development (R01HD065829)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
- Keywords
- linear mixed models, generalizability, sample size, statistical power, within-subjects design, psycholinguistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
