Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 75 of the 76 references it cites were found in a registry or an index; 34 of the 104 sources its claims cite were checked for support.

Violating the normality assumption may be the lesser of two evils

Open access
Knief, Ulrich, Forstmeier, Wolfgang · 2021 · Behavior Research Methods

Published version of a preprint

Abstract

When data are not normally distributed, researchers are often uncertain whether it is legitimate to use tests that assume Gaussian errors, or whether one has to either model a more specific error structure or use randomization techniques. Here we use Monte Carlo simulations to explore the pros and cons of fitting Gaussian models to non-normal data in terms of risk of type I error, power and utility for parameter estimation. We find that Gaussian models are robust to non-normality over a wide range of conditions, meaning that p values remain fairly reliable except for data with influential outliers judged at strict alpha levels. Gaussian models also performed well in terms of power across all simulated scenarios. Parameter estimates were mostly unbiased and precise except if sample sizes were small or the distribution of the predictor was highly skewed. Transformation of data before analysis is often advisable and visual inspection for outliers and heteroscedasticity is important for assessment. In strong contrast, some non-Gaussian models and randomization techniques bear a range of risks that are often insufficiently known. High rates of false-positive conclusions can arise for instance when overdispersion in count data is not controlled appropriately or when randomization procedures ignore existing non-independencies in the data. Hence, newly developed statistical methods not only bring new opportunities, but they can also pose new threats to reliability. We argue that violating the normality assumption bears risks that are limited and manageable, while several more sophisticated approaches are relatively error prone and particularly difficult to check during peer review. Scientists and reviewers who are not fully aware of the risks might benefit from preferentially trusting Gaussian mixed models in which random effects account for non-independencies in the data.

Overview

CoChat Summary

TL;DR

Monte Carlo simulations show that Gaussian linear (mixed) models are broadly robust to severe non-normality for hypothesis testing, with type I error rates usually near nominal and power largely unaffected. The authors argue that violating normality is often the lesser evil compared with non-Gaussian or randomization approaches that can be more error-prone when overdispersion or non-independence is mishandled.

Key findings

  • Gaussian models kept type I error rates close to 0.05: at N = 1000, rates ranged 0.044–0.052 across 100 Y × X distribution combinations; at N = 100, 0.037–0.058. Only at N = 10 did four of 100 combinations show elevated rates of 0.086–0.11, all involving distributions D8 and D9 with extreme leverage observations; some N = 10 cases were too conservative (down to 0.015).
  • Power was mostly stable: at N = 1000, power ranged 0.48–0.52 for all but one combination (β = 0.45 when both Y and X were D9); at N = 100, power ranged 0.44–0.60, with 87% of estimates between 0.48 and 0.52; at N = 10, power varied from 0.05 to 0.87.
  • Parameter estimates were mostly unbiased and precise, except with small samples or highly skewed predictors: strongest bias occurred at N = 10 with D9 X, shrinking to maximally 1.32-fold at N = 100 and 1.03-fold at N = 1000; precision did not improve from N = 100 to N = 1000.
  • In contrast, Poisson models were not robust to distributional violations: inappropriate Poisson fits yielded type I error rates from 0 to as high as 0.55, while Gaussian fits stayed near 0.05. Overdispersion control via Quasipoisson or observation-level random effects fixed some but not all problems; Quasipoisson gave unbiased type I error rates like Gaussian models but is unavailable in lme4.

Methods

  • Monte Carlo simulations using ten distributions (D0–D9; five continuous, five discrete) that were severely skewed, platy-/leptokurtic, or zero-inflated. For type I error, Y and X were each sampled from these distributions, yielding 100 combinations per sample size (N = 10, 25, 50, 100, 250, 500, 1000), with 50,000 datasets per combination and no simulated effect; significance was assessed by F-test or likelihood ratio test.
  • Power, bias, and precision were evaluated at N = 10, 100, and 1000 using Z-transformed Y and X and an iterative algorithm to introduce predefined effects (r = 0.15, 0.2, 0.25; also effect sizes giving 50% power for normal data: b = 0.59, 0.19, 0.062), with 50,000 simulations. Multivariate and mixed-model simulations used N = 100 and a single random intercept.
  • Gaussian, Poisson, and binomial error structures were compared at N = 100 across six Y × X combinations, with 50,000 simulation runs; the R package “TrustGauss” accompanies the paper.

Limitations

  • The paper states that it did not cover collinearity between predictors or the distribution of random effects; multivariate and mixed-model simulations were limited to several predictors and a single random intercept.
  • The paper states that transformations were avoided to explore major non-normality, and that prediction intervals require the normality assumption; parameter estimates from Gaussian models on non-Gaussian data should be interpreted cautiously outside the typical data range.
  • Evident from the text: conclusions about randomization techniques and some non-Gaussian pitfalls are illustrated through examples (Box 1) rather than simulated in the main analyses; the simulations use specific synthetic distributions and parameter settings, so generalization depends on those choices.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3758/s13428-021-01587-5

Cites

75 of 76 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 p values from Gaussian linear regression models are in most cases unbiased. a Overview of the ten different distributions that we simulated. Distributions D0 is Gaussian and all remaining distributions are sorted by their tendency to produce strong outliers. Distributions D1, D2, D6, D7, and D8 are discrete. The numbers D0–D9 refer to the plots in b–e where on the Y-axis the distribution of the dependent variable and on the X-axis of the predictor is indicated. b Type I error rate at an α-level of 0.05 for sample sizes of N = 10, 100, and 1000. Red colors represent increased and blue conservative type I error rates. c Scale shift parameter, d bias in p values at an expected p value of 10-3 and e bias in p values at an expected p value of 10-4
Fig. 1 p values from Gaussian linear regression models are in most cases unbiased. a Overview of the ten different distributions that we simulated. Distributions D0 is Gaussian and all remaining distributions are sorted by their tendency to produce strong outliers. Distributions D1, D2, D6, D7, and D8 are discrete. The numbers D0–D9 refer to the plots in b–e where on the Y-axis the distribution of the dependent variable and on the X-axis of the predictor is indicated. b Type I error rate at an α-level of 0.05 for sample sizes of N = 10, 100, and 1000. Red colors represent increased and blue conservative type I error rates. c Scale shift parameter, d bias in p values at an expected p value of 10-3 and e bias in p values at an expected p value of 10-4CC BY · PubMed Central
Fig. 2 Power, bias, and precision of parameter estimates from Gaussian linear regression models are in most cases unaffected by the distributions of the dependent variable Y or the predictor X. a Overview of the different distributions that we simulated, which were the same as in Fig. 1. The numbers D0–D9 refer to the plots in b–e where on the Y-axis the distribution of the dependent variable and on the X-axis of the predictor is indicated. b Power at a regression coefficient b = 0.2 for sample sizes of N = 10, 100, and 1000. Red colors represent increased power. c Power at regression coefficients b = 0.59, 0.19, and 0.06 for sample sizes of N = 10, 100, and 1000, respectively, where the expected power derived from a normally distributed Y and X is 0.5. Red colors represent increased and blue colors decreased power. d Bias and e precision of the regression coefficient estimates at an expected b = 0.2 for sample sizes of N = 10, 100, and 1000
Fig. 2 Power, bias, and precision of parameter estimates from Gaussian linear regression models are in most cases unaffected by the distributions of the dependent variable Y or the predictor X. a Overview of the different distributions that we simulated, which were the same as in Fig. 1. The numbers D0–D9 refer to the plots in b–e where on the Y-axis the distribution of the dependent variable and on the X-axis of the predictor is indicated. b Power at a regression coefficient b = 0.2 for sample sizes of N = 10, 100, and 1000. Red colors represent increased power. c Power at regression coefficients b = 0.59, 0.19, and 0.06 for sample sizes of N = 10, 100, and 1000, respectively, where the expected power derived from a normally distributed Y and X is 0.5. Red colors represent increased and blue colors decreased power. d Bias and e precision of the regression coefficient estimates at an expected b = 0.2 for sample sizes of N = 10, 100, and 1000CC BY · PubMed Central
Fig. 3 Distribution of observed p values (when the null hypothesis is true) as a function of different model specifications (columns) and different distributions of the dependent variable Y (rows a to e). Each panel was summed up across ten different distributions of the predictor X (500,000 simulations per panel with N = 100 data points per simulation). Models were fitted either as glms with a Gaussian error structure that violate the normality assumption (first column), as glms with a Quasipoisson error structure that take overdispersion into account (second column), as glmms with a Poisson error structure and an observation-level random effect (OLRE; Harrison et al., 2018) or as glms with a Poisson error structure that violate the assumption of the Poisson distribution. In each panel, TIER indicates the realized type I error rate (across the ten different predictor distributions), highlighted with a color scheme as in Fig. 1b (blue: below the nominal level of 0.05, red: above the nominal level, grey: closely matching the nominal level). The dependent variable Y was distributed as a distribution D1, b distribution D2, c distribution D6, d distribution D7 or e distribution D8 (see Table 1 and Fig. 1a for details)
Fig. 3 Distribution of observed p values (when the null hypothesis is true) as a function of different model specifications (columns) and different distributions of the dependent variable Y (rows a to e). Each panel was summed up across ten different distributions of the predictor X (500,000 simulations per panel with N = 100 data points per simulation). Models were fitted either as glms with a Gaussian error structure that violate the normality assumption (first column), as glms with a Quasipoisson error structure that take overdispersion into account (second column), as glmms with a Poisson error structure and an observation-level random effect (OLRE; Harrison et al., 2018) or as glms with a Poisson error structure that violate the assumption of the Poisson distribution. In each panel, TIER indicates the realized type I error rate (across the ten different predictor distributions), highlighted with a color scheme as in Fig. 1b (blue: below the nominal level of 0.05, red: above the nominal level, grey: closely matching the nominal level). The dependent variable Y was distributed as a distribution D1, b distribution D2, c distribution D6, d distribution D7 or e distribution D8 (see Table 1 and Fig. 1a for details)CC BY · PubMed Central

Verification

Claims the paper makes

20 Valid · 2 Not valid · 30 Not checked · 67 Own results and interpretations

70 more claims · 6 with their cited sources checked

Create a free account to see all 119 claims and to verify the 3 references still open.

See all claims — free account
  • Probably not a substantial claim. 14
  • It may state two separate claims as one. 35

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 74 references

Cited by

See all 654

Metadata

Authors
Knief, U., & Forstmeier, W.
Affiliations
  • Ludwig-Maximilians-Universität München (DE)
  • Max Planck Society (DE)
Corresponding author
Knief, Ulrich
Venue
Behavior Research Methods
Published
2021-05-07
Volume
53
Issue
6
Pages
2576-2590
ISSN
  • 1554-351X (ISSN-L)
  • 1554-3528
OpenAlex
W3158727545
Cited by
654
References
74
Citations by year
2026: 132 · 2025: 165 · 2024: 136 · 2023: 112 · 2022: 68 · 2021: 34
FWCI
65.27
Type
journal article
Language
en
Open access
Licence
Locations
  • Behavior Research Methods (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Ludwig-Maximilians-Universität München
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Distribution Estimation and Applications
Keywords
normality assumption, non-normal data, robustness to non-normality, type I error, statistical power, parameter estimation bias, outliers, heteroscedasticity, data transformation, permutation tests, peer review
MeSH terms
Data Interpretation, Statistical, Humans, Monte Carlo Method, Reproducibility of Results, Normal Distribution, Sample Size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Violating the normality assumption may be the lesser of two evils. https://app.cochat.ai/papers/10.3758/s13428-021-01587-5
The paper
Knief, U., & Forstmeier, W. (2021). Violating the normality assumption may be the lesser of two evils. Behavior Research Methods. https://doi.org/10.3758/s13428-021-01587-5

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.