Violating the normality assumption may be the lesser of two evils
Open accessPublished version of a preprint
The DOI matches its Crossref record; 75 of the 76 references it cites were found in a registry or an index; 34 of the 104 sources its claims cite were checked for support.
Abstract
When data are not normally distributed, researchers are often uncertain whether it is legitimate to use tests that assume Gaussian errors, or whether one has to either model a more specific error structure or use randomization techniques. Here we use Monte Carlo simulations to explore the pros and cons of fitting Gaussian models to non-normal data in terms of risk of type I error, power and utility for parameter estimation. We find that Gaussian models are robust to non-normality over a wide range of conditions, meaning that p values remain fairly reliable except for data with influential outliers judged at strict alpha levels. Gaussian models also performed well in terms of power across all simulated scenarios. Parameter estimates were mostly unbiased and precise except if sample sizes were small or the distribution of the predictor was highly skewed. Transformation of data before analysis is often advisable and visual inspection for outliers and heteroscedasticity is important for assessment. In strong contrast, some non-Gaussian models and randomization techniques bear a range of risks that are often insufficiently known. High rates of false-positive conclusions can arise for instance when overdispersion in count data is not controlled appropriately or when randomization procedures ignore existing non-independencies in the data. Hence, newly developed statistical methods not only bring new opportunities, but they can also pose new threats to reliability. We argue that violating the normality assumption bears risks that are limited and manageable, while several more sophisticated approaches are relatively error prone and particularly difficult to check during peer review. Scientists and reviewers who are not fully aware of the risks might benefit from preferentially trusting Gaussian mixed models in which random effects account for non-independencies in the data.
Overview
CoChat Summary
TL;DR
Monte Carlo simulations show that Gaussian linear (mixed) models are broadly robust to severe non-normality for hypothesis testing, with type I error rates usually near nominal and power largely unaffected. The authors argue that violating normality is often the lesser evil compared with non-Gaussian or randomization approaches that can be more error-prone when overdispersion or non-independence is mishandled.
Key findings
- Gaussian models kept type I error rates close to 0.05: at N = 1000, rates ranged 0.044–0.052 across 100 Y × X distribution combinations; at N = 100, 0.037–0.058. Only at N = 10 did four of 100 combinations show elevated rates of 0.086–0.11, all involving distributions D8 and D9 with extreme leverage observations; some N = 10 cases were too conservative (down to 0.015).
- Power was mostly stable: at N = 1000, power ranged 0.48–0.52 for all but one combination (β = 0.45 when both Y and X were D9); at N = 100, power ranged 0.44–0.60, with 87% of estimates between 0.48 and 0.52; at N = 10, power varied from 0.05 to 0.87.
- Parameter estimates were mostly unbiased and precise, except with small samples or highly skewed predictors: strongest bias occurred at N = 10 with D9 X, shrinking to maximally 1.32-fold at N = 100 and 1.03-fold at N = 1000; precision did not improve from N = 100 to N = 1000.
- In contrast, Poisson models were not robust to distributional violations: inappropriate Poisson fits yielded type I error rates from 0 to as high as 0.55, while Gaussian fits stayed near 0.05. Overdispersion control via Quasipoisson or observation-level random effects fixed some but not all problems; Quasipoisson gave unbiased type I error rates like Gaussian models but is unavailable in lme4.
Methods
- Monte Carlo simulations using ten distributions (D0–D9; five continuous, five discrete) that were severely skewed, platy-/leptokurtic, or zero-inflated. For type I error, Y and X were each sampled from these distributions, yielding 100 combinations per sample size (N = 10, 25, 50, 100, 250, 500, 1000), with 50,000 datasets per combination and no simulated effect; significance was assessed by F-test or likelihood ratio test.
- Power, bias, and precision were evaluated at N = 10, 100, and 1000 using Z-transformed Y and X and an iterative algorithm to introduce predefined effects (r = 0.15, 0.2, 0.25; also effect sizes giving 50% power for normal data: b = 0.59, 0.19, 0.062), with 50,000 simulations. Multivariate and mixed-model simulations used N = 100 and a single random intercept.
- Gaussian, Poisson, and binomial error structures were compared at N = 100 across six Y × X combinations, with 50,000 simulation runs; the R package “TrustGauss” accompanies the paper.
Limitations
- The paper states that it did not cover collinearity between predictors or the distribution of random effects; multivariate and mixed-model simulations were limited to several predictors and a single random intercept.
- The paper states that transformations were avoided to explore major non-normality, and that prediction intervals require the normality assumption; parameter estimates from Gaussian models on non-Gaussian data should be interpreted cautiously outside the typical data range.
- Evident from the text: conclusions about randomization techniques and some non-Gaussian pitfalls are illustrated through examples (Box 1) rather than simulated in the main analyses; the simulations use specific synthetic distributions and parameter settings, so generalization depends on those choices.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
20 Valid · 2 Not valid · 30 Not checked · 67 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“note that such power gains should be quite small; see Table 2 and also Szöcs & Schäfer, 2015; Warton et al., 2016”
§ Violating the normality assumption may be the lesser of two evils > The issue of overdispersion in non-Gaussian models
What the sources say
- Partly supportedcites Szöcs & Schäfer, 2015 matched to “Ecotoxicology is not normal: A comparison of statistical approaches for analysis of count and proportion data in ecotoxicology” (2015) sourceExperimental — accuracy not yet measured
Section 4.2 (Simulations; page not provided) reports that GLMs generally had greater power than linear models applied to transformed data. But Section 3.2.2 (Binomial data; page not provided) reports up to 27% higher power at small sample sizes, with differences vanishing as sample size increased. The paper therefore supports power gains, but not the unqualified claim that they should be quite small. (LLM, conf=0.84)
- Disputedcites Warton et al., 2016 matched to “Three points to consider when choosing a LM or GLM test for count data” (2016) sourceExperimental — accuracy not yet measured
Point 3 says power differences can be substantial in unbalanced designs, with negative binomial tests sometimes having almost twice the power of linear-model tests. The paper does say differences are sometimes small, especially in balanced designs, but that qualification does not support the general claim that gains should be quite small. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Failure to account for overdispersion (as in Fig. 3b, d) will typically result in very high rates of type I errors (Forstmeier et al., 2017; Ives, 2015; Szöcs & Schäfer, 2015; Warton et al., 2016; Warton & Hui, 2011; Young et al., 1999).”
§ Violating the normality assumption may be the lesser of two evils > The issue of overdispersion in non-Gaussian models
What the sources say
- Partly supportedcites Forstmeier et al., 2017 matched to “Detecting and avoiding likely false‐positive findings – a practical guide” (2017) sourceExperimental — accuracy not yet measured
In the section on pseudoreplication due to genetic relatedness (page unavailable), the paper says that failing to account for overdispersion produces anti-conservative P-values. Its Conclusions also link uncorrected overdispersion to incorrect P-values and false-positive findings. These passages support an increased risk of Type I errors, but do not establish that the rates will typically be very high. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites Ives, 2015 matched to “For testing the significance of regression coefficients, go ahead and log‐transform count data” (2015) source
- Partly supportedcites Szöcs & Schäfer, 2015 matched to “Ecotoxicology is not normal: A comparison of statistical approaches for analysis of count and proportion data in ecotoxicology” (2015) sourceExperimental — accuracy not yet measured
Section 4.1 (page unavailable) says that unaccounted-for overdispersion underestimates standard errors and that simulations showed increased Type I error rates. Section 3.2.1 likewise reports inflated rates for some models. The copy supports the direction of the effect, but does not establish that the rates are typically *very high*. (LLM, conf=0.79)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Such inflations of type I error rates in glms already have been reported frequently (Ives, 2015; Szöcs & Schäfer, 2015; Warton et al., 2016; Warton & Hui, 2011; Young et al., 1999)”
§ Violating the normality assumption may be the lesser of two evils > The issue of overdispersion in non-Gaussian models
What the sources say
- Not checked — no full text of the sourcecites Ives, 2015 matched to “For testing the significance of regression coefficients, go ahead and log‐transform count data” (2015) source
- Supportedcites Szöcs & Schäfer, 2015 matched to “Ecotoxicology is not normal: A comparison of statistical approaches for analysis of count and proportion data in ecotoxicology” (2015) source
The Count data results (section 3.2.1) report inflated Type I error rates for the negative-binomial and Poisson GLMs tested. The Case study discussion (section 4.1) also links increased error rates in the Poisson GLM to overdispersion. These findings support citing the paper as an example of reported inflation, though they do not imply that all GLMs have inflated rates. (LLM, conf=0.90)
- Supportedcites Warton et al., 2016 matched to “Three points to consider when choosing a LM or GLM test for count data” (2016) source
In “Point 2 – Poor type I error control can be fixed” (page not supplied), the paper reports that standard GLM tests can have inflated type I error and notes similar findings in other work. Its simulation results also report inflated rates for several GLM tests, supporting the broad statement that such inflation has been reported repeatedly. (LLM, conf=0.90)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“or when randomization procedures ignore existing non-independencies in the data.”
- Also stated in: § Introduction · plain
“inadequate resampling in bootstrapping techniques (e.g., Ihle et al., 2019; Santema et al., 2019)”
§ Violating the normality assumption may be the lesser of two evils > Abstract
What the sources say
- Partly supportedcites Ihle et al., 2019 matched to “Measuring Up to Reality: Null Models and Analysis Simulations to Study Parental Coordination Over Provisioning Offspring” (2019) sourceExperimental — accuracy not yet measured
The Randomization of Observed Inter-feeding Intervals section says the authors designed a randomization to preserve dependencies that affect coordination expected by chance. The Conclusions warn that environmentally induced patterns could explain part or all of the apparent coordination, supporting the risk of a false conclusion. The paper does not establish high false-positive rates specifically caused by randomization procedures that ignore those dependencies; its false-positive-rate analysis concerns statistical models. (LLM, conf=0.82)
- Supportedcites Santema et al., 2019 matched to “Testing the Conditional Cooperation Model: What Can We Learn From Parents Taking Turns When Feeding Offspring?” (2019) source
In “The Problem with Randomizing Inter-Visit Intervals” (page unavailable), the paper explains that randomization removes correlations caused by both parents responding to the same external stimuli. Its simulations show greater apparent turn-taking in the original than in the randomized sequences even though active turn-taking was excluded, supporting the claim that ignoring such dependencies can produce false-positive conclusions. (LLM, conf=0.87)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“High rates of false-positive conclusions can arise for instance when overdispersion in count data is not controlled appropriately”
- Also stated in: § Introduction · plain
“Our examples include failure to account for overdispersion in glmms with Poisson errors (Forstmeier et al., 2017; Harrison, 2014; Ives, 2015)”
§ Violating the normality assumption may be the lesser of two evils > Abstract
What the sources say
- Supportedcites Forstmeier et al., 2017 matched to “Detecting and avoiding likely false‐positive findings – a practical guide” (2017) source
The Conclusions state that failure to control for overdispersion can produce incorrect P-values and false-positive findings. The discussion of overdispersion gives an example involving data modelled as Poisson counts, supporting the claim’s count-data context. (LLM, conf=0.92)
- Supportedcites Harrison, 2014 matched to “Using observation-level random effects to model overdispersion in count data in ecology and evolution” (2014) source
The Introduction says that failing to account for overdispersion in count data can lead researchers to conclude erroneously that variables have meaningful effects, and explicitly identifies a risk of Type I error. The Discussion’s Summary likewise says that ignoring overdispersion can produce erroneous conclusions about significant predictors. These passages support the claim that inadequately controlled overdispersion can produce false-positive conclusions; they do not quantify a rate. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites Ives, 2015 matched to “For testing the significance of regression coefficients, go ahead and log‐transform count data” (2015) source
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
70 more claims · 6 with their cited sources checked
Create a free account to see all 119 claims and to verify the 4 references still open.
- Probably not a substantial claim. 14
- It may state two separate claims as one. 35
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Violating the normality assumption may be the lesser of two evils > Simulations to assess effects on p values, power, and parameter estimates
“We did not simulate any effect, which means that both the regression coefficient b and the intercept a were on average zero”
§ Violating the normality assumption may be the lesser of two evils > Simulations to assess effects on p values, power, and parameter estimates
“| D6 | Gamma with categories# | 10 | 100 | 3 | 0 | 3.45 | 15.09 | DistributionY=“GammaCategorical”, nCategoriesY.cat=3, ShapeY.gamma=1, ScaleY.gamma=10 |”
Citations
Cites
Fitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835 · cited 1×
“We further fitted the above models as mixed-effects models using the lme4 R package (v 1.1-14; Bates et al. 2015).”
MethodInfluentialA heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity
White, H. (1980) · Econometrica · doi:10.2307/1912934 · Cited by 25,861
Mixed effects models and extensions in ecology with R
Zuur, A. F., Ieno, E. N., Walker, N., Saveliev, A. A., & Smith, G. M. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-87458-6 · Cited by 17,963
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Mixed-Effects Models in Sand S-PLUS
Pinheiro, J. C., & Bates, D. M. (2000) · Statistics and Computing · doi:10.1007/978-1-4419-0318-1 · Cited by 11,187
Experimental Design and Data Analysis for Biologists
Quinn, G. P., & Keough, M. J. (2002) · doi:10.1017/cbo9780511806384 · Cited by 10,674
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328
Mixed-Effects Models in S and S-PLUS
(2001) · Technometrics · doi:10.1198/tech.2001.s574 · Cited by 9,278
Generalized linear mixed models: a practical guide for ecology and evolution
Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J.-S. S. (2009) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2008.10.008 · Cited by 8,910
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785 · cited 2×
“We assessed the deviation of observed from expected -log10(P-value) at an expected value of 3 (P = 10−3) and 4 (P = 10−4) and by estimating the scale shift parameter v = σobserved / σexpected (Lin 1989), where σ is the variance in -log10(P-value).”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
A protocol for data exploration to avoid common statistical problems: Data exploration
Zuur, A. F., Ieno, E. N., & Elphick, C. S. (2010) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2009.00001.x · Cited by 8,228
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Casella, G., & Berger, R. (2024) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,711
(1967) · Project Euclid (Cornell University) · Cited by 4,720
Normality Tests for Statistical Analysis: A Guide for Non-Statisticians
Ghasemi, A., & Zahediasl, S. (2012) · International Journal of Endocrinology and Metabolism · doi:10.5812/ijem.3505 · Cited by 4,368 · cited 1×
“Formal tests for normality have been criticized because they have low power at small sample sizes and almost always yield significant deviations from normality at large sample sizes (Ghasemi & Zahediasl 2012).”
Effect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835 · cited 1×
“…coefficient) already work by violating the normality assumption (e.g. fitting a Pearson correlation through the binary data of a 2×2 table; Nakagawa & Cuthill 2007), so why not always report the Gaussian model to begin with, if the primary purpose of the test is to obtain a reliable…”
Zuur, A. F., Ieno, E. N., & Smith, G. M. (2007) · Statistics for Biology and Health · doi:10.1007/978-0-387-45972-1 · Cited by 2,437
The arcsine is asinine: the analysis of proportions in ecology
Warton, D. I., & Hui, F. K. C. (2011) · Ecology · doi:10.1890/10-0340.1 · Cited by 2,280 · cited 3×
“Failure to account for overdispersion will typically result in very high rates of type I errors (Warton & Hui 2011; Ives 2015; Szöcs & Schäfer 2015; Warton et al. 2016; Forstmeier, Wagenmakers & Parker 2017).”
BackgroundMethodQuasi-likelihood functions, generalized linear models, and the Gauss—Newton method
WEDDERBURN, R. W. M. (1974) · Biometrika · doi:10.1093/biomet/61.3.439 · Cited by 2,122
A brief introduction to mixed effects modelling and multi-model inference in ecology
Harrison, X. A., Donaldson, L., Correa-Cano, M. E., Evans, J., Fisher, D. N., Goodwin, C. E. D., Robinson, B. S., Hodgson, D. J., & Inger, R. (2018) · PeerJ · doi:10.7717/peerj.4794 · Cited by 2,094
Glass, G. V., Peckham, P. D., & Sanders, J. R. (1972) · Review of Educational Research · doi:10.3102/00346543042003237 · Cited by 1,826
The Importance of the Normality Assumption in Large Public Health Data Sets
Lumley, T., Diehr, P., Emerson, S., & Chen, L. (2002) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.23.100901.140546 · Cited by 1,803
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
THE COMPARISON OF PERCENTAGES IN MATCHED SAMPLES
COCHRAN, W. G. (1950) · Biometrika · doi:10.1093/biomet/37.3-4.256 · Cited by 1,465
Hayes, A. F., & Cai, L. (2007) · Behavior Research Methods · doi:10.3758/bf03192961 · Cited by 1,432
Robustness of linear mixed-effects models to violations of distributional assumptions
Schielzeth, H., Dingemanse, N. J., Nakagawa, S., Westneat, D. F., Allegue, H., Teplitsky, C., Réale, D., Dochtermann, N. A., Garamszegi, L. Z., & Araya‐Ajoy, Y. G. (2020) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13434 · Cited by 1,393
NON-NORMALITY AND TESTS ON VARIANCES
BOX, G. E. P. (1953) · Biometrika · doi:10.1093/biomet/40.3-4.318 · Cited by 1,302
Do not log‐transform count data
O’Hara, R. B., & Kotze, D. J. (2010) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2010.00021.x · Cited by 1,268
The statistical sleuth : a course in methods of data analysis
Ramsey, F. L., & Schafer, D. W. (2002) · Cited by 1,166
Harrison, X. A. (2014) · PeerJ · doi:10.7717/peerj.616 · Cited by 1,150
Four assumptions of multiple regression that researchers should always test
Osborne, J. W., & Waters, E. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/r222-hv23 · Cited by 1,035
Power and Sample Size Calculations for Studies Involving Linear Regression
Dupont, W. D., & Plummer, W. D. (1998) · Controlled Clinical Trials · doi:10.1016/s0197-2456(98)00037-3 · Cited by 1,024
arm: Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Su, Y.-S. (2007) · CRAN: Contributed Packages · doi:10.32614/cran.package.arm · Cited by 955
Mixed-Effects Models in S and S-PLUS
(2000) · Statistics and Computing · doi:10.1007/b98882 · Cited by 930
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 784
Bishara, A. J., & Hittner, J. B. (2012) · Psychological Methods · doi:10.1037/a0028087 · Cited by 766 · cited 3×
“…regression coefficients (Williams, Grajales & Kurkiewicz 2013) and usually do not impair hypothesis testing (no inflated type I error rate, e.g. Bishara & Hittner 2012; Puth, Neuhauser & Ruxton 2014; Ives 2015; Szöcs & Schäfer 2015; Warton et al. 2016) even at relatively small sample sizes,…”
BackgroundMethodResultNelder, J. A., & Bliss, C. I. (1971) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2343879 · Cited by 758 · cited 1×
“…may occur under some conditions (see Figure 1) it may be most advisable to apply a rank-based inverse normal (RIN) transformation (aka rankit scores Bliss 1967) to the data, which can approximately normalize most distributional shapes and which effectively minimizes type I errors and maximises…”
MethodPermutation, Parametric and Bootstrap Tests of Hypotheses
(2005) · Springer Series in Statistics · doi:10.1007/b138696 · Cited by 707
(1986) · CERN Document Server (European Organization for Nuclear Research) · Cited by 583
Many Labs 3: Evaluating participant pool quality across the academic semester via replication
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.012 · Cited by 468 · cited 1×
“In a growing body of literature, researchers express their concerns about irreproducible results (Open Science Collaboration 2015; Ebersole et al. 2016; Camerer et al. 2018; Silberzahn et al. 2018) and it has been argued that the inappropriate use of statistics is a leading cause of irreproducible…”
BackgroundEffective use of Pearson's product–moment correlation coefficient
Puth, M.-T., Neuhäuser, M., & Ruxton, G. D. (2014) · Animal Behaviour · doi:10.1016/j.anbehav.2014.05.003 · Cited by 446
Modeling zero-inflated count data with glmmTMB
Brooks, M. E., Kristensen, K., van Benthem, K. J., Magnusson, A., Berg, C. W., Nielsen, A., Skaug, H. J., Mächler, M., & Bolker, B. M. (2017) · doi:10.1101/132753 · Cited by 440
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Self-citationHarrison, X. A. (2015) · PeerJ · doi:10.7717/peerj.1114 · Cited by 419
Assumptions of Multiple Regression: Correcting Two Misconceptions
Williams, M. N., Grajales, C. A. G., & Kurkiewicz, D. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/55hn-wk47 · Cited by 359
Freckleton, R. P. (2011) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-010-1045-6 · Cited by 355
For testing the significance of regression coefficients, go ahead and log‐transform count data
Ives, A. R. (2015) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12386 · Cited by 248 · cited 5×
“…that were severely skewed, platy- and leptokurtic or zero-inflated (distributions D0-D9; Figure 1A left column), going beyond previous studies that examined less dramatic violations (Bishara & Hittner 2012; Puth, Neuhauser & Ruxton 2014; Ives 2015; Szöcs & Schäfer 2015; Warton et al. 2016).”
BackgroundMethodResultInfluentialMARDIA, K. V. (1971) · Biometrika · doi:10.1093/biomet/58.1.105 · Cited by 223 · cited 2×
“Searching the literature, we found relatively little pragmatic advice (Box & Watson 1962; Mardia 1971; Lumley et al. 2002; Gelman & Hill 2007; O’Hara 2009; Zuur, Ieno & Elphick 2010) on the question of whether and when it may be safe to violate the assumption of normality of errors, which underlies…”
BackgroundFRANK, S. A. (2009) · Journal of Evolutionary Biology · doi:10.1111/j.1420-9101.2009.01775.x · Cited by 196 · cited 1×
“2016) but that are still of biological relevance (Frank, 2009; Gelman & Hill, 2007; Zuur et al., 2009).”
BackgroundBlock bootstrap for Mann–Kendall trend test of serially dependent data
Önöz, B., & Bayazit, M. (2012) · Hydrological Processes · doi:10.1002/hyp.8438 · Cited by 194
Three points to consider when choosing a LM or GLM test for count data
Warton, D. I., Lyons, M., Stoklosa, J., & Ives, A. R. (2016) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12552 · Cited by 192 · cited 8×
“Such inflations of type I error rates in glms have been reported 169 frequently (Warton & Hui 2011; Ives 2015; Szöcs & Schäfer 2015; Warton et al. 2016) and this 170 problem threatens the reliability of research whenever such models are implemented with insufficient 171 statistical expertise.”
BackgroundMethodResultInfluentialApplication of the Hierarchical Bootstrap to Multi-Level Data in Neuroscience
Saravanan, V., Berman, G. J., & Sober, S. J. (2019) · doi:10.1101/819334 · Cited by 170
Robustness to non-normality of regression tests
BOX, G. E. P., & WATSON, G. S. (1962) · Biometrika · doi:10.1093/biomet/49.1-2.93 · Cited by 149 · cited 2×
“Searching the literature, we found relatively little pragmatic advice (Box & Watson 1962; Mardia 1971; Lumley et al. 2002; Gelman & Hill 2007; O’Hara 2009; Zuur, Ieno & Elphick 2010) on the question of whether and when it may be safe to violate the assumption of normality of errors, which underlies…”
BackgroundMixed Models Offer No Freedom from Degrees of Freedom
Arnqvist, G. (2020) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2019.12.004 · Cited by 135 · cited 2×
“, 2017), which often lets researchers settle for a model that ignores non-independence and yields estimates with inappropriately high confidence and statistical significance (Arnqvist, 2020; Barr et al., 2013; Forstmeier et al., 2017) (B) When observational data do not comply with any distributional assumption, random…”
BackgroundDo not log-transform count dataPreprint
O’Hara, R. B., & Kotze, J. (2010) · Nature Precedings · doi:10.1038/npre.2010.4136.1 · Cited by 127
Ten Simple Rules for Effective Statistical Practice
Kass, R. E., Caffo, B. S., Davidian, M., Meng, X.-L., Yu, B., & Reid, N. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004961 · Cited by 113
Simulating Multivariate Nonnormal Data Using an Iterative Algorithm
Ruscio, J., & Kaczetow, W. (2008) · Multivariate Behavioral Research · doi:10.1080/00273170802285693 · Cited by 101
How to Make Models Add Up — A Primer on GLMMs
O'Hara, R. B. (2009) · Annales Zoologici Fennici · doi:10.5735/086.046.0205 · Cited by 99 · cited 2×
“Recent advances in 41 statistical modelling appear to have solved this pr oblem, because it is now possible to fit generalize d 42 linear mixed models “glmm” with a variety of error distributions (e.g. binomial, Poisson, zero-inflate d 43 Poisson, negative binomial; O'Hara 2009; Harrison et al. 2018) or to use a range…”
BackgroundMethodPhylogenetic Regression for Binary Dependent Variables
Ives, A. R., & Garland, T. (2014) · Modern Phylogenetic Comparative Methods and Their Application in Evolutionary Biology · doi:10.1007/978-3-662-43550-2_9 · Cited by 91 · cited 1×
“Finally, with the inclusion of random effects glmms are much more computationally intensive than lmms and often fail to converge, leading to the recommendation to model all traits as Gaussian (e.g. Ives & Garland 2014).”
BackgroundA Hierarchical Bayesian Approach to Ecological Count Data: A Flexible Tool for Ecologists
Fordyce, J. A., Gompert, Z., Forister, M. L., & Nice, C. C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026785 · Cited by 91
Of rowing boats, ocean liners and tests of the anova homogeneity of variance assumption
McGuinness, K. A. (2002) · Austral Ecology · doi:10.1046/j.1442-9993.2002.01233.x · Cited by 78
Brunner, J., & Austin, P. C. (2009) · Canadian Journal of Statistics · doi:10.1002/cjs.10004 · Cited by 55 · cited 1×
“Brunner and Austin (2009) even regard this as the “primary function of statistical hypothesis testing in the discourse of science”.”
BackgroundSzöcs, E., & Schäfer, R. B. (2015) · Environmental Science and Pollution Research · doi:10.1007/s11356-015-4579-3 · Cited by 52 · cited 5×
“…and usually do not impair hypothesis testing (no inflated type I error rate, e.g. Bishara & Hittner 2012; Puth, Neuhauser & Ruxton 2014; Ives 2015; Szöcs & Schäfer 2015; Warton et al. 2016) even at relatively small sample sizes, and with large sample sizes ≥ 500 the Central Limit Theorem…”
BackgroundMethodResultInfluentialRobustness to nonnormality of regression F-tests
Ali, M. M., & Sharma, S. C. (1996) · Journal of Econometrics · doi:10.1016/0304-4076(94)01700-x · Cited by 45 · cited 1×
“An F-test is usually employed for testing the significance of regression models (Ali & Sharma 1996).”
MethodPlaschke, S., Bulla, M., Cruz-López, M., Gómez del Ángel, S., & Küpper, C. (2019) · Frontiers in Zoology · doi:10.1186/s12983-019-0313-1 · Cited by 31
Ihle, M., Pick, J. L., Winney, I. S., Nakagawa, S., & Burke, T. (2019) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2019.00142 · Cited by 29
Application of the hierarchical bootstrap to multi-level data in neuroscience
Saravanan, V., Berman, G. J., & Sober, S. J. (2020) · arXiv · doi:10.48550/arxiv.2007.07797 · Cited by 26
Analysis of Overdispersed Count Data from Single-Factor Experiments: A Comparative Study
Young, L. J., Campbell, N. L., & Capuano, G. A. (1999) · Journal of Agricultural Biological and Environmental Statistics · doi:10.2307/1400385 · Cited by 24
HACK, H. R. B. (1958) · Biometrika · doi:10.1093/biomet/45.1-2.260 · Cited by 23
Santema, P., Schlicht, E., & Kempenaers, B. (2019) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2019.00094 · Cited by 21
Silberzahn, R., Uhlmann, E. L., Martin, D. P., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810511 · Cited by 21
Knief, U., Schielzeth, H., Backström, N., Hemmrich‐Stanisak, G., Wittig, M., Franke, A., Griffith, S. C., Ellegren, H., Kempenaers, B., & Forstmeier, W. (2017) · Molecular Ecology · doi:10.1111/mec.14009 · Cited by 17
Self-citationBlair, R. C., & Bear Lawson, S. (1982) · Florida Journal of Educational Research · doi:10.62798/tzqq1969 · Cited by 17
The power of outliers (and why researchers should ALWAYS check for them)
Osborne, J. W., & Overbay, A. (2004) · University of Massachusetts Amherst · doi:10.7275/qf69-7k43
Sleuth3: Data Sets from Ramsey and Schafer's "Statistical Sleuth (3rd Ed)"
(2012) · CRAN: Contributed Packages · doi:10.32614/cran.package.sleuth3
Cited by
Mutation bias reflects natural selection in Arabidopsis thaliana
Monroe, J. G., Srikant, T., Carbonell-Bejerano, P., Becker, C., Lensink, M., Exposito-Alonso, M., Klein, M., Hildebrandt, J., Neumann, M., Kliebenstein, D., et al. (2022) · Nature · doi:10.1038/s41586-021-04269-6 · Cited by 396
Discovery and population genomics of structural variation in a songbird genus
Weissensteiner, M. H., Bunikis, I., Catalán, A., Francoijs, K.-J., Knief, U., Heim, W., Peona, V., Pophaly, S. D., Sedlazeck, F. J., Suh, A., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-17195-4 · Cited by 135
Self-citationSample Size and Shapiro-Wilk Test: An Analysis for Soybean Grain Yield
de Souza, R. R., Toebe, M. J., Mello, A. C., & Bittencourt, K. C. (2022) · European Journal of Agronomy · doi:10.1016/j.eja.2022.126666 · Cited by 196
Comprehensive analysis of clustering algorithms: exploring limitations and innovative solutions
Wani, A. A. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2286 · Cited by 198
“The reliance on Gaussian distribution limits it ’ s effectiveness when the data violates this assumption, and the fi xed covariance structure may fail to accurately re fl ect the true spread within each cluster ( Knief & Forstmeier, 2021 ).”
BackgroundSujon, K. M., hassan, R. B., Choi, K., & Samad, M. A. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01313-4 · Cited by 202
“This analysis also helps identify features that may require scaling, transformation, or special handling [57], especially in the context of imbalanced distributions or outliers, which can impact metrics such as F1-Score and MCC that emphasize the balance between true positives and false negatives.”
Metadata
- Authors
- Knief, U., & Forstmeier, W.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- Max Planck Society (DE)
- Corresponding author
- Knief, Ulrich
- Venue
- Behavior Research Methods
- Published
- 2021-05-07
- Volume
- 53
- Issue
- 6
- Pages
- 2576-2590
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- PMC
- PMC8613103
- OpenAlex
- W3158727545
- Cited by
- 654
- References
- 74
- Citations by year
- 2026: 132 · 2025: 165 · 2024: 136 · 2023: 112 · 2022: 68 · 2021: 34
- FWCI
- 65.27
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Behavior Research Methods (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Ludwig-Maximilians-Universität München
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Distribution Estimation and Applications
- Keywords
- normality assumption, non-normal data, robustness to non-normality, type I error, statistical power, parameter estimation bias, outliers, heteroscedasticity, data transformation, permutation tests, peer review
- MeSH terms
- Data Interpretation, Statistical, Humans, Monte Carlo Method, Reproducibility of Results, Normal Distribution, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.