The DOI matches its Crossref record; 75 of the 76 references it cites were found in a registry or an index; 34 of the 104 sources its claims cite were checked for support.
Violating the normality assumption may be the lesser of two evils
Open accessPublished version of a preprint
Abstract
When data are not normally distributed, researchers are often uncertain whether it is legitimate to use tests that assume Gaussian errors, or whether one has to either model a more specific error structure or use randomization techniques. Here we use Monte Carlo simulations to explore the pros and cons of fitting Gaussian models to non-normal data in terms of risk of type I error, power and utility for parameter estimation. We find that Gaussian models are robust to non-normality over a wide range of conditions, meaning that p values remain fairly reliable except for data with influential outliers judged at strict alpha levels. Gaussian models also performed well in terms of power across all simulated scenarios. Parameter estimates were mostly unbiased and precise except if sample sizes were small or the distribution of the predictor was highly skewed. Transformation of data before analysis is often advisable and visual inspection for outliers and heteroscedasticity is important for assessment. In strong contrast, some non-Gaussian models and randomization techniques bear a range of risks that are often insufficiently known. High rates of false-positive conclusions can arise for instance when overdispersion in count data is not controlled appropriately or when randomization procedures ignore existing non-independencies in the data. Hence, newly developed statistical methods not only bring new opportunities, but they can also pose new threats to reliability. We argue that violating the normality assumption bears risks that are limited and manageable, while several more sophisticated approaches are relatively error prone and particularly difficult to check during peer review. Scientists and reviewers who are not fully aware of the risks might benefit from preferentially trusting Gaussian mixed models in which random effects account for non-independencies in the data.
Overview
CoChat Summary
TL;DR
Monte Carlo simulations show that Gaussian linear (mixed) models are broadly robust to severe non-normality for hypothesis testing, with type I error rates usually near nominal and power largely unaffected. The authors argue that violating normality is often the lesser evil compared with non-Gaussian or randomization approaches that can be more error-prone when overdispersion or non-independence is mishandled.
Key findings
- Gaussian models kept type I error rates close to 0.05: at N = 1000, rates ranged 0.044–0.052 across 100 Y × X distribution combinations; at N = 100, 0.037–0.058. Only at N = 10 did four of 100 combinations show elevated rates of 0.086–0.11, all involving distributions D8 and D9 with extreme leverage observations; some N = 10 cases were too conservative (down to 0.015).
- Power was mostly stable: at N = 1000, power ranged 0.48–0.52 for all but one combination (β = 0.45 when both Y and X were D9); at N = 100, power ranged 0.44–0.60, with 87% of estimates between 0.48 and 0.52; at N = 10, power varied from 0.05 to 0.87.
- Parameter estimates were mostly unbiased and precise, except with small samples or highly skewed predictors: strongest bias occurred at N = 10 with D9 X, shrinking to maximally 1.32-fold at N = 100 and 1.03-fold at N = 1000; precision did not improve from N = 100 to N = 1000.
- In contrast, Poisson models were not robust to distributional violations: inappropriate Poisson fits yielded type I error rates from 0 to as high as 0.55, while Gaussian fits stayed near 0.05. Overdispersion control via Quasipoisson or observation-level random effects fixed some but not all problems; Quasipoisson gave unbiased type I error rates like Gaussian models but is unavailable in lme4.
Methods
- Monte Carlo simulations using ten distributions (D0–D9; five continuous, five discrete) that were severely skewed, platy-/leptokurtic, or zero-inflated. For type I error, Y and X were each sampled from these distributions, yielding 100 combinations per sample size (N = 10, 25, 50, 100, 250, 500, 1000), with 50,000 datasets per combination and no simulated effect; significance was assessed by F-test or likelihood ratio test.
- Power, bias, and precision were evaluated at N = 10, 100, and 1000 using Z-transformed Y and X and an iterative algorithm to introduce predefined effects (r = 0.15, 0.2, 0.25; also effect sizes giving 50% power for normal data: b = 0.59, 0.19, 0.062), with 50,000 simulations. Multivariate and mixed-model simulations used N = 100 and a single random intercept.
- Gaussian, Poisson, and binomial error structures were compared at N = 100 across six Y × X combinations, with 50,000 simulation runs; the R package “TrustGauss” accompanies the paper.
Limitations
- The paper states that it did not cover collinearity between predictors or the distribution of random effects; multivariate and mixed-model simulations were limited to several predictors and a single random intercept.
- The paper states that transformations were avoided to explore major non-normality, and that prediction intervals require the normality assumption; parameter estimates from Gaussian models on non-Gaussian data should be interpreted cautiously outside the typical data range.
- Evident from the text: conclusions about randomization techniques and some non-Gaussian pitfalls are illustrated through examples (Box 1) rather than simulated in the main analyses; the simulations use specific synthetic distributions and parameter settings, so generalization depends on those choices.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
20 Valid · 2 Not valid · 30 Not checked · 67 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“note that such power gains should be quite small; see Table 2 and also Szöcs & Schäfer, 2015; Warton et al., 2016”
§ Violating the normality assumption may be the lesser of two evils > The issue of overdispersion in non-Gaussian models
What the sources say
- Partly supportedcites Szöcs & Schäfer, 2015 sourceExperimental — accuracy not yet measured
Section 4.2 (Simulations; page not provided) reports that GLMs generally had greater power than linear models applied to transformed data. But Section 3.2.2 (Binomial data; page not provided) reports up to 27% higher power at small sample sizes, with differences vanishing as sample size increased. The paper therefore supports power gains, but not the unqualified claim that they should be quite small. (LLM, conf=0.84)
- Disputedcites Warton et al., 2016 sourceExperimental — accuracy not yet measured
Point 3 says power differences can be substantial in unbalanced designs, with negative binomial tests sometimes having almost twice the power of linear-model tests. The paper does say differences are sometimes small, especially in balanced designs, but that qualification does not support the general claim that gains should be quite small. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Failure to account for overdispersion (as in Fig. 3b, d) will typically result in very high rates of type I errors (Forstmeier et al., 2017; Ives, 2015; Szöcs & Schäfer, 2015; Warton et al., 2016; Warton & Hui, 2011; Young et al., 1999).”
§ Violating the normality assumption may be the lesser of two evils > The issue of overdispersion in non-Gaussian models
What the sources say
- Partly supportedcites Forstmeier et al., 2017 sourceExperimental — accuracy not yet measured
In the section on pseudoreplication due to genetic relatedness (page unavailable), the paper says that failing to account for overdispersion produces anti-conservative P-values. Its Conclusions also link uncorrected overdispersion to incorrect P-values and false-positive findings. These passages support an increased risk of Type I errors, but do not establish that the rates will typically be very high. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites Ives, 2015 source
- Partly supportedcites Szöcs & Schäfer, 2015 sourceExperimental — accuracy not yet measured
Section 4.1 (page unavailable) says that unaccounted-for overdispersion underestimates standard errors and that simulations showed increased Type I error rates. Section 3.2.1 likewise reports inflated rates for some models. The copy supports the direction of the effect, but does not establish that the rates are typically *very high*. (LLM, conf=0.79)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Such inflations of type I error rates in glms already have been reported frequently (Ives, 2015; Szöcs & Schäfer, 2015; Warton et al., 2016; Warton & Hui, 2011; Young et al., 1999)”
§ Violating the normality assumption may be the lesser of two evils > The issue of overdispersion in non-Gaussian models
What the sources say
- Not checked — no full text of the sourcecites Ives, 2015 source
- Supportedcites Szöcs & Schäfer, 2015 source
The Count data results (section 3.2.1) report inflated Type I error rates for the negative-binomial and Poisson GLMs tested. The Case study discussion (section 4.1) also links increased error rates in the Poisson GLM to overdispersion. These findings support citing the paper as an example of reported inflation, though they do not imply that all GLMs have inflated rates. (LLM, conf=0.90)
- Supportedcites Warton et al., 2016 source
In “Point 2 – Poor type I error control can be fixed” (page not supplied), the paper reports that standard GLM tests can have inflated type I error and notes similar findings in other work. Its simulation results also report inflated rates for several GLM tests, supporting the broad statement that such inflation has been reported repeatedly. (LLM, conf=0.90)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“or when randomization procedures ignore existing non-independencies in the data.”
- Also stated in: § Introduction · plain
“inadequate resampling in bootstrapping techniques (e.g., Ihle et al., 2019; Santema et al., 2019)”
§ Violating the normality assumption may be the lesser of two evils > Abstract
What the sources say
- Partly supportedcites Ihle et al., 2019 sourceExperimental — accuracy not yet measured
The Randomization of Observed Inter-feeding Intervals section says the authors designed a randomization to preserve dependencies that affect coordination expected by chance. The Conclusions warn that environmentally induced patterns could explain part or all of the apparent coordination, supporting the risk of a false conclusion. The paper does not establish high false-positive rates specifically caused by randomization procedures that ignore those dependencies; its false-positive-rate analysis concerns statistical models. (LLM, conf=0.82)
- Supportedcites Santema et al., 2019 source
In “The Problem with Randomizing Inter-Visit Intervals” (page unavailable), the paper explains that randomization removes correlations caused by both parents responding to the same external stimuli. Its simulations show greater apparent turn-taking in the original than in the randomized sequences even though active turn-taking was excluded, supporting the claim that ignoring such dependencies can produce false-positive conclusions. (LLM, conf=0.87)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“High rates of false-positive conclusions can arise for instance when overdispersion in count data is not controlled appropriately”
- Also stated in: § Introduction · plain
“Our examples include failure to account for overdispersion in glmms with Poisson errors (Forstmeier et al., 2017; Harrison, 2014; Ives, 2015)”
§ Violating the normality assumption may be the lesser of two evils > Abstract
What the sources say
- Supportedcites Forstmeier et al., 2017 source
The Conclusions state that failure to control for overdispersion can produce incorrect P-values and false-positive findings. The discussion of overdispersion gives an example involving data modelled as Poisson counts, supporting the claim’s count-data context. (LLM, conf=0.92)
- Supportedcites Harrison, 2014 source
The Introduction says that failing to account for overdispersion in count data can lead researchers to conclude erroneously that variables have meaningful effects, and explicitly identifies a risk of Type I error. The Discussion’s Summary likewise says that ignoring overdispersion can produce erroneous conclusions about significant predictors. These passages support the claim that inadequately controlled overdispersion can produce false-positive conclusions; they do not quantify a rate. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites Ives, 2015 source
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
70 more claims · 6 with their cited sources checked
Create a free account to see all 119 claims and to verify the 3 references still open.
- Probably not a substantial claim. 14
- It may state two separate claims as one. 35
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Violating the normality assumption may be the lesser of two evils > Simulations to assess effects on p values, power, and parameter estimates
“We did not simulate any effect, which means that both the regression coefficient b and the intercept a were on average zero”
§ Violating the normality assumption may be the lesser of two evils > Simulations to assess effects on p values, power, and parameter estimates
“| D6 | Gamma with categories# | 10 | 100 | 3 | 0 | 3.45 | 15.09 | DistributionY=“GammaCategorical”, nCategoriesY.cat=3, ShapeY.gamma=1, ScaleY.gamma=10 |”
Citations
Cites
Fitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835 · cited 1×
“We further fitted the above models as mixed-effects models using the lme4 R package (v 1.1-14; Bates et al. 2015).”
MethodInfluentialA heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity
White, H. (1980) · Econometrica · doi:10.2307/1912934 · Cited by 25,861
Mixed effects models and extensions in ecology with R
Zuur, A. F., Ieno, E. N., Walker, N., Saveliev, A. A., & Smith, G. M. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-87458-6 · Cited by 17,963
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Mixed-Effects Models in Sand S-PLUS
Pinheiro, J. C., & Bates, D. M. (2000) · Statistics and Computing · doi:10.1007/978-1-4419-0318-1 · Cited by 11,187
Experimental Design and Data Analysis for Biologists
Quinn, G. P., & Keough, M. J. (2002) · doi:10.1017/cbo9780511806384 · Cited by 10,674
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328
Mixed-Effects Models in S and S-PLUS
(2001) · Technometrics · doi:10.1198/tech.2001.s574 · Cited by 9,278
Generalized linear mixed models: a practical guide for ecology and evolution
Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J.-S. S. (2009) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2008.10.008 · Cited by 8,910
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785 · cited 2×
“We assessed the deviation of observed from expected -log10(P-value) at an expected value of 3 (P = 10−3) and 4 (P = 10−4) and by estimating the scale shift parameter v = σobserved / σexpected (Lin 1989), where σ is the variance in -log10(P-value).”
Background
Cited by
Mutation bias reflects natural selection in Arabidopsis thaliana
Monroe, J. G., Srikant, T., Carbonell-Bejerano, P., Becker, C., Lensink, M., Exposito-Alonso, M., Klein, M., Hildebrandt, J., Neumann, M., Kliebenstein, D., et al. (2022) · Nature · doi:10.1038/s41586-021-04269-6 · Cited by 396
Discovery and population genomics of structural variation in a songbird genus
Weissensteiner, M. H., Bunikis, I., Catalán, A., Françoijs, K., Knief, U., Heim, W., Peona, V., Pophaly, S. D., Sedlazeck, F. J., Suh, A., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-17195-4 · Cited by 151
Self-citationSample Size and Shapiro-Wilk Test: An Analysis for Soybean Grain Yield
de Souza, R. R., Toebe, M. J., Mello, A. C., & Bittencourt, K. C. (2022) · European Journal of Agronomy · doi:10.1016/j.eja.2022.126666 · Cited by 196
Comprehensive analysis of clustering algorithms: exploring limitations and innovative solutions
Wani, A. A. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2286 · Cited by 197
“The reliance on Gaussian distribution limits it ’ s effectiveness when the data violates this assumption, and the fi xed covariance structure may fail to accurately re fl ect the true spread within each cluster ( Knief & Forstmeier, 2021 ).”
BackgroundSujon, K. M., hassan, R. B., Choi, K., & Samad, M. A. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01313-4 · Cited by 202
“This analysis also helps identify features that may require scaling, transformation, or special handling [57], especially in the context of imbalanced distributions or outliers, which can impact metrics such as F1-Score and MCC that emphasize the balance between true positives and false negatives.”
Metadata
- Authors
- Knief, U., & Forstmeier, W.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- Max Planck Society (DE)
- Corresponding author
- Knief, Ulrich
- Venue
- Behavior Research Methods
- Published
- 2021-05-07
- Volume
- 53
- Issue
- 6
- Pages
- 2576-2590
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- PMC
- PMC8613103
- OpenAlex
- W3158727545
- Cited by
- 654
- References
- 74
- Citations by year
- 2026: 132 · 2025: 165 · 2024: 136 · 2023: 112 · 2022: 68 · 2021: 34
- FWCI
- 65.27
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Behavior Research Methods (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Ludwig-Maximilians-Universität München
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Distribution Estimation and Applications
- Keywords
- normality assumption, non-normal data, robustness to non-normality, type I error, statistical power, parameter estimation bias, outliers, heteroscedasticity, data transformation, permutation tests, peer review
- MeSH terms
- Data Interpretation, Statistical, Humans, Monte Carlo Method, Reproducibility of Results, Normal Distribution, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Violating the normality assumption may be the lesser of two evils. https://app.cochat.ai/papers/10.3758/s13428-021-01587-5
- The paper
- Knief, U., & Forstmeier, W. (2021). Violating the normality assumption may be the lesser of two evils. Behavior Research Methods. https://doi.org/10.3758/s13428-021-01587-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.