The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Cryptic multiple hypotheses testing in linear models: overestimated effect sizes and the winner's curse
Open accessAbstract
Fitting generalised linear models (GLMs) with more than one predictor has become the standard method of analysis in evolutionary and behavioural research. Often, GLMs are used for exploratory data analysis, where one starts with a complex full model including interaction terms and then simplifies by removing non-significant terms. While this approach can be useful, it is problematic if significant effects are interpreted as if they arose from a single a priori hypothesis test. This is because model selection involves cryptic multiple hypothesis testing, a fact that has only rarely been acknowledged or quantified. We show that the probability of finding at least one 'significant' effect is high, even if all null hypotheses are true (e.g. 40% when starting with four predictors and their two-way interactions). This probability is close to theoretical expectations when the sample size (N) is large relative to the number of predictors including interactions (k). In contrast, type I error rates strongly exceed even those expectations when model simplification is applied to models that are over-fitted before simplification (low N/k ratio). The increase in false-positive results arises primarily from an overestimation of effect sizes among significant predictors, leading to upward-biased effect sizes that often cannot be reproduced in follow-up studies ('the winner's curse'). Despite having their own problems, full model tests and P value adjustments can be used as a guide to how frequently type I errors arise by sampling variation alone. We favour the presentation of full models, since they best reflect the range of predictors investigated and ensure a balanced representation also of non-significant results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Model selection and multimodel inference: a practical information-theoretic approach.
Guthery, F. S., Burnham, K. P., & Anderson, D. R. (2003) · The Journal of Wildlife Management · doi:10.2307/3802723 · Cited by 41,495
Multiple regression: testing and interpreting interactions
(1992) · Choice Reviews Online · doi:10.5860/choice.29-3352 · Cited by 36,654
Discovering Statistics Using SPSS
Field, A. P., & Miles, J. N. V. (2000) · Medical Entomology and Zoology · Cited by 28,288
Modern Applied Statistics with S
Venables, W. N., & Ripley, B. D. (2002) · Statistics and Computing · doi:10.1007/978-0-387-21706-2 · Cited by 24,305
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · cited 1×
“The second approach is to control table-wide type I error rates by using sequential Bonferroni correction (Holm 1979; Hochberg 1988; Rice 1989; Wright 1992) or falsediscovery rate (FDR) control (Benjamini and Hochberg 1995; Storey and Tibshirani 2003).”
Analyzing Tables of Statistical Tests
Rice, W. R. (1989) · Evolution · doi:10.2307/2409177 · Cited by 14,571
ANALYZING TABLES OF STATISTICAL TESTS
Rice, W. R. (1989) · Evolution · doi:10.1111/j.1558-5646.1989.tb04220.x · Cited by 14,547
Experimental Design and Data Analysis for Biologists
Quinn, G. P., & Keough, M. J. (2002) · doi:10.1017/cbo9780511806384 · Cited by 10,674 · cited 1×
“In observational studies, model selection procedures are commonly applied in order to find a parsimonious combination of predictors to explain some phenomenon of interest (Miller 1984; Quinn and Keough 2002).”
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Statistical significance for genomewide studies
Storey, J. D., & Tibshirani, R. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1530509100 · Cited by 9,692 · cited 1×
“The second approach is to control table-wide type I error rates by using sequential Bonferroni correction (Holm 1979; Hochberg 1988; Rice 1989; Wright 1992) or falsediscovery rate (FDR) control (Benjamini and Hochberg 1995; Storey and Tibshirani 2003).”
What's wrong with Bonferroni adjustments
Perneger, T. V. (1998) · BMJ · doi:10.1136/bmj.316.7139.1236 · Cited by 5,900
Crawley, M. J. (2007) · Medical Entomology and Zoology · Cited by 5,431
A sharper Bonferroni procedure for multiple tests of significance
HOCHBERG, Y. (1988) · Biometrika · doi:10.1093/biomet/75.4.800 · Cited by 4,829
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 5×
“However, note that the global null hypothesis should not be rejected prematurely with reference to the literature, as the literature necessarily contains a very large number of type I errors as well (Ioannidis 2005).”
InfluentialCrawley, M. J. (2007) · Wiley eBooks · doi:10.1002/9780470515075 · Cited by 3,920
Simple means to improve the interpretability of regression coefficients
Schielzeth, H. (2010) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2010.00012.x · Cited by 3,027 · cited 3×
“Predictor levels were coded −0.5 and 0.5, respectively, to ensure that main effects were interpretable even in the presence of interactions (Aiken and West 1991; Schielzeth 2010).”
InfluentialSelf-citationFORWARD SELECTION OF EXPLANATORY VARIABLES
Blanchet, F. G., Legendre, P., & Borcard, D. (2008) · Ecology · doi:10.1890/07-0986.1 · Cited by 2,116
A farewell to Bonferroni: the problems of low statistical power and publication bias
Nakagawa, S. (2004) · Behavioral Ecology · doi:10.1093/beheco/arh107 · Cited by 2,097
Null Hypothesis Testing: Problems, Prevalence, and an Alternative
Anderson, D. R., Burnham, K. P., & Thompson, W. L. (2000) · The Journal of Wildlife Management · doi:10.2307/3803199 · Cited by 1,930
Krämer, W. (2003) · Computational Statistics & Data Analysis · Cited by 1,858
Why do we still use stepwise modelling in ecology and behaviour?
WHITTINGHAM, M. J., STEPHENS, P. A., BRADBURY, R. B., & FRECKLETON, R. P. (2006) · Journal of Animal Ecology · doi:10.1111/j.1365-2656.2006.01141.x · Cited by 1,520
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
Model uncertainty, data mining and statistical inference
Chatfield, C. (1995) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2983440 · Cited by 1,103 · cited 2×
“…and Anderson (2002) also emphasise that standard errors in linear models are conditional on the model structure and criticise stepwise selection procedures for their failure to incorporate model structure uncertainty into estimates of precision, i.e. the standard errors (see also Chatfield 1995).”
Adjusted P-Values for Simultaneous Inference
Wright, S. P. (1992) · Biometrics · doi:10.2307/2532694 · Cited by 1,026
Implementing false discovery rate control: increasing your power
Verhoeven, K. J. F., Simonsen, K. L., & McIntyre, L. M. (2005) · Oikos · doi:10.1111/j.0030-1299.2005.13727.x · Cited by 974
Conclusions beyond support: overconfident estimates in mixed models
Schielzeth, H., & Forstmeier, W. (2009) · Behavioral Ecology · doi:10.1093/beheco/arn145 · Cited by 952
Self-citationIndividual recognition: it is good to be different
Tibbetts, E. A., & Dale, J. (2007) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2007.09.001 · Cited by 779
Derksen, S., & Keselman, H. J. (1992) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1992.tb00992.x · Cited by 771
Large Upward Bias in Estimation of Locus-Specific Effects from Genomewide Scans
Göring, H. H. H., Terwilliger, J. D., & Blangero, J. (2001) · The American Journal of Human Genetics · doi:10.1086/324471 · Cited by 483 · cited 2×
“This is another aspect of the winner's curse that even if the claimed effects are true, effect size estimates based on selection from a larger set of candidate effects might give misleadingly high effect sizes (Göring et al. 2001; Ioannidis 2008).”
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 463
Stepwise Model Fitting and Statistical Inference: Turning Noise into Signal Pollution
Mundry, R., & Nunn, C. L. (2009) · The American Naturalist · doi:10.1086/593303 · Cited by 440
Overcoming the Winner’s Curse: Estimating Penetrance Parameters from Case-Control Data
Zöllner, S., & Pritchard, J. K. (2007) · The American Journal of Human Genetics · doi:10.1086/512821 · Cited by 382
Validating, augmenting and refining genome-wide association signals
Ioannidis, J. P. A., Thomas, G., & Daly, M. J. (2009) · Nature Reviews Genetics · doi:10.1038/nrg2544 · Cited by 375
Selection of subsets of regression variables
Miller, A. J., & Miller, J. (1984) · Cited by 357
Freckleton, R. P. (2011) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-010-1045-6 · Cited by 355
Model selection bias and Freedman’s paradox
Lukacs, P. M., Burnham, K. P., & Anderson, D. R. (2010) · Annals of the Institute of Statistical Mathematics · doi:10.1007/s10463-009-0234-4 · Cited by 321
A survey of the statistical power of research in behavioral ecology and animal behavior
Jennions, M. D. (2003) · Behavioral Ecology · doi:10.1093/beheco/14.3.438 · Cited by 278 · cited 2×
“This is particularly problematic for typical studies in behavioural ecology, where effect sizes are often small and sample sizes low ( Jennions and Moller 2003 ).”
Inference in ecology and evolution
Stephens, P. A., Buskirk, S. W., & del Rio, C. M. (2007) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2006.12.003 · Cited by 268
Jennions, M. D., & Møller, A. P. (2002) · Proceedings of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rspb.2001.1832 · Cited by 255
Kelly, C. D. (2006) · The Quarterly Review of Biology · doi:10.1086/506236 · Cited by 167
Mundry, R. (2010) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-010-1040-y · Cited by 166
Changing philosophies and tools for statistical inferences in behavioral ecology
Garamszegi, L. Z., Calhim, S., Dochtermann, N. A., Hegyi, G., Hurd, P. L., Jørgensen, C., Kutsukake, N., Lajeunesse, M. J., Pollard, K. A., Schielzeth, H., et al. (2009) · Behavioral Ecology · doi:10.1093/beheco/arp137 · Cited by 135
Self-citationDeveloping multiple hypotheses in behavioral ecology
Dochtermann, N. A., & Jenkins, S. H. (2010) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-010-1039-4 · Cited by 119
Information-theoretic approaches to statistical analysis in behavioural ecology: an introduction
Garamszegi, L. Z. (2010) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-010-1028-7 · Cited by 110
Inference after variable selection in linear regression models
Zhang, P. (1992) · Biometrika · doi:10.1093/biomet/79.4.741 · Cited by 67
Trait duplication by means of sensory bias
Rodríguez, R. L. (2009) · Behavioral Ecology · doi:10.1093/beheco/arp130 · Cited by 16
Cited by
A brief introduction to mixed effects modelling and multi-model inference in ecology
Harrison, X. A., Donaldson, L., Correa-Cano, M. E., Evans, J., Fisher, D. N., Goodwin, C. E. D., Robinson, B. S., Hodgson, D. J., & Inger, R. (2018) · PeerJ · doi:10.7717/peerj.4794 · Cited by 2,094
Quantifying individual variation in behaviour: mixed‐effect modelling approaches
Dingemanse, N. J., & Dochtermann, N. A. (2013) · Journal of Animal Ecology · doi:10.1111/1365-2656.12013 · Cited by 1,081
O'Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G., Moher, D., et al. (2021) · Biological Reviews · doi:10.1111/brv.12721 · Cited by 697
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Self-citationNested by design: model fitting and interpretation in a mixed model era
Schielzeth, H., & Nakagawa, S. (2013) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2012.00251.x · Cited by 394
Self-citation
Metadata
- Authors
- Forstmeier, W., & Schielzeth, H.
- Venue
- Behavioral Ecology and Sociobiology
- Published
- 2010-08-19
- Volume
- 65
- Issue
- 1
- Pages
- 47-55
- ISSN
- 0340-5443 (ISSN-L)
- 1432-0762
- OpenAlex
- W2004872383
- Cited by
- 1,120
- References
- 48
- FWCI
- 13.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.