Conclusions beyond support: overconfident estimates in mixed models
Open accessAbstract
Mixed-effect models are frequently used to control for the nonindependence of data points, for example, when repeated measures from the same individuals are available. The aim of these models is often to estimate fixed effects and to test their significance. This is usually done by including random intercepts, that is, intercepts that are allowed to vary between individuals. The widespread belief is that this controls for all types of pseudoreplication within individuals. Here we show that this is not the case, if the aim is to estimate effects that vary within individuals and individuals differ in their response to these effects. In these cases, random intercept models give overconfident estimates leading to conclusions that are not supported by the data. By allowing individuals to differ in the slopes of their responses, it is possible to account for the nonindependence of data points that pseudoreplicate slope information. Such random slope models give appropriate standard errors and are easily implemented in standard statistical software. Because random slope models are not always used where they are essential, we suspect that many published findings have too narrow confidence intervals and a substantially inflated type I error rate. Besides reducing type I errors, random slope models have the potential to reduce residual variance by accounting for between-individual variation in slopes, which makes it easier to detect treatment effects that are applied between individuals, hence reducing type II errors as well.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Random slopes were included for the test predictors (event or anticipation) as well as for subgroup size, rank, and proximity within-subject.
“We included random slopes (46, 47) for the test predictors (i.e., event or anticipation) as well as for subgroup size, rank, and proximity within-subject.”
From Oxytocin reactivity during intergroup conflict in wild chimpanzees (2017) · cites this paper as [47]
To keep type I error rate at the nominal 5%, random slopes were included for both proximity and period type within-subject.
“To keep type I error rate at the nominal 5%, we included random slopes (46, 47) for both proximity and period type within-subject.”
From Oxytocin reactivity during intergroup conflict in wild chimpanzees (2017) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2075823 · Cited by 18,768 · see citing works
Modern Applied Statistics with S
Venables, W., & Ripley, B. (2010) · Statisctics and computing/Statistics and computing · doi:10.1007/978-0-387-21706-2 · Cited by 16,188 · see citing works
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597 · see citing works
Experimental Design and Data Analysis for Biologists
Quinn, G. P., & Keough, M. J. (2002) · doi:10.1017/cbo9780511806384 · Cited by 10,674 · cited 1× · see citing works
“The problem is most easily illustrated for designs, in which some factor varies between subjects, but other factors (or covariates) vary within subjects (e.g., split-plot and repeatedmeasures designs; Quinn and Keough 2002).”
BackgroundApplied Longitudinal Data Analysis: Modeling Change and Event Occurrence
Singer, J. D., & Willett, J. B. (2003) · doi:10.1093/acprof:oso/9780195152968.001.0001 · Cited by 10,647 · see citing works
Mixed-Effects Models in S and S-PLUS
(2001) · Technometrics · doi:10.1198/tech.2001.s574 · Cited by 9,278 · see citing works
Random-Effects Models for Longitudinal Data
Laird, N. M., & Ware, J. H. (1982) · Biometrics · doi:10.2307/2529876 · Cited by 8,923 · see citing works
Multilevel analysis : an introduction to basic and advanced multilevel modeling
Snijders, T. A. B., & Bosker, R. (1999) · Cited by 6,830 · see citing works
Scaling regression inputs by dividing by two standard deviations
Gelman, A. (2007) · Statistics in Medicine · doi:10.1002/sim.3107 · Cited by 2,536 · see citing works
Extending the Linear Model with R
Faraway, J. J. (2016) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315382722 · Cited by 832 · see citing works
Application of random regression models in animal breeding
Schaeffer, L. R. (2003) · Livestock Production Science · doi:10.1016/s0301-6226(03)00151-9 · Cited by 535 · see citing works
Grindstaff, J. L., Hasselquist, D., Nilsson, J., et al. (2006) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2006.3608 · Cited by 176 · see citing works
Experimental Design and Data Analysis
Carpenter, R. P., Lyon, D. H., & Hasdell, T. A. (2000) · Springer eBooks · doi:10.1007/978-1-4615-4447-0_6 · Cited by 157 · see citing works
Introduction to Mixed Modelling: Beyond Regression and Analysis of Variance
Galwey, N. W. (2006) · Cited by 108 · see citing works
Bolund, E., Schielzeth, H., & Forstmeier, W. (2008) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2008.1251 · Cited by 94 · see citing works
Self-citationSex-specific effects of yolk-androgens on growth of nestling American kestrels
Sockman, K. W., Weiss, J. M., Webster, M. S., Talbott, V., & Schwabl, H. G. (2007) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-007-0486-z · Cited by 91 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328 · see citing works
Nakagawa, S., & Schielzeth, H. (2013) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2012.00261.x · Cited by 10,356 · see citing works
Self-citationSimple means to improve the interpretability of regression coefficients
Schielzeth, H. (2010) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2010.00012.x · Cited by 3,027 · see citing works
Self-citationRepeatability for Gaussian and non-Gaussian data: a practical guide for biologists.
Nakagawa, S., & Schielzeth, H. (2010) · Biological Reviews · doi:10.1111/j.1469-185x.2010.00141.x · Cited by 2,584 · see citing works
A brief introduction to mixed effects modelling and multi-model inference in ecology
Harrison, X. A., Donaldson, L., Correa-Cano, M. E., Evans, J., Fisher, D. N., Goodwin, C. E. D., Robinson, B. S., Hodgson, D. J., & Inger, R. (2018) · PeerJ · doi:10.7717/peerj.4794 · Cited by 2,094 · see citing works
Robustness of linear mixed-effects models to violations of distributional assumptions
Schielzeth, H., Dingemanse, N. J., Nakagawa, S., Westneat, D. F., Allegue, H., Teplitsky, C., Réale, D., Dochtermann, N. A., Garamszegi, L. Z., & Araya‐Ajoy, Y. G. (2020) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13434 · Cited by 1,393 · see citing works
Self-citationLethal aggression in Pan is better explained by adaptive strategies than human impacts
Wilson, M. L., Boesch, C., Fruth, B., Furuichi, T., Gilby, I. C., Hashimoto, C., Hobaiter, C. L., Hohmann, G., Itoh, N., Koops, K., et al. (2014) · Nature · doi:10.1038/nature13727 · Cited by 474 · see citing works
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436 · see citing works
Self-citationNested by design: model fitting and interpretation in a mixed model era
Schielzeth, H., & Nakagawa, S. (2013) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2012.00251.x · Cited by 394 · see citing works
Self-citationNorthrup, J. M., Vander Wal, E., Bonar, M., Fieberg, J., Laforge, M. P., Leclerc, M., Prokopenko, C. M., & Gerber, B. D. (2022) · Ecological Applications · doi:10.1002/eap.2470 · Cited by 227 · see citing works
Oxytocin reactivity during intergroup conflict in wild chimpanzees
Samuni, L., Preis, A., Mundry, R., Deschner, T., Crockford, C., & Wittig, R. M. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1616812114 · Cited by 207 · see citing works
Engert, V., Kok, B. E., Papassotiriou, I., Chrousos, G. P., & Singer, T. (2017) · Science Advances · doi:10.1126/sciadv.1700495 · Cited by 151 · see citing works
Topsoil organic matter build‐up in glacier forelands around the world
Khedim, N., Cécillon, L., Poulenard, J., Barré, P., Baudin, F., Marta, S., Rabatel, A., Dentant, C., Cauvy‐Fraunié, S., Anthelme, F., et al. (2021) · Global Change Biology · doi:10.1111/gcb.15496 · Cited by 106 · see citing works
Engert, V., Koester, A. M., Riepenhausen, A., & Singer, T. (2016) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2016.08.029 · Cited by 103 · see citing works
Contrasting effects of habitat area and connectivity on evenness of pollinator communities
Marini, L., Öckinger, E., Bergman, K., Jauker, B., Krauss, J., Kuussaari, M., Pöyry, J., Smith, H. G., Steffan‐Dewenter, I., & Bommarco, R. (2014) · Ecography · doi:10.1111/j.1600-0587.2013.00369.x · Cited by 51 · see citing works
Relationship quality affects social stress buffering in dogs and wolves
Cimarelli, G., Marshall-Pescini, S., Range, F., Berghänel, A., & Virányi, Z. (2021) · Animal Behaviour · doi:10.1016/j.anbehav.2021.06.008 · Cited by 22 · see citing works
- Authors
- Schielzeth, H., & Forstmeier, W.
- Affiliations
- Max Planck Society (DE)
- Corresponding author
- Schielzeth, H.
- Venue
- Behavioral Ecology
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-11-27
- Volume
- 20
- Issue
- 2
- Pages
- 416-420
- ISSN
- 1045-2249 (ISSN-L)
- 1465-7279
- PMC
- PMC2657178
- OpenAlex
- W2146506147
- Cited by
- 952
- References
- 16
- Citations by year
- 2026: 39 · 2025: 78 · 2024: 69 · 2023: 52 · 2022: 71 · 2021: 85 · 2020: 77 · 2019: 77 · 2018: 86 · 2017: 52 · 2016: 59 · 2015: 65 · 2014: 41 · 2013: 27 · 2012: 29
- FWCI
- 14.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Behavioral Ecology (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- HighWire Press Open Archive (submittedVersion)
- PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion)
- Publikationen an der Universität Bielefeld (Universität Bielefeld) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Funders
- Deutsche Forschungsgemeinschaft
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Life Sciences › Agricultural and Biological Sciences › Ecology, Evolution, Behavior and Systematics › Agriculture, Soil, Plant Science
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Keywords
- mixed models, random slope models, random intercept model, pseudoreplication, type I error rate, confidence intervals, between-person variation, repeated measures, residual variance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
