The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies
Open accessAbstract
Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Self-citationAn improved Bonferroni procedure for multiple tests of significance
SIMES, R. J. (1986) · Biometrika · doi:10.1093/biomet/73.3.751 · Cited by 2,396
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · cited 1×
“…effective remedy is preregistration (e.g., Chambers, 2013; Chambers et al., 2013; de Groot, 1969; Goldacre, 2009; Nosek & Lakens, 2014; Wagenmakers, Wetzels, Borsboom, van der Maas & Kievit, 2012; Wolfe, 2013; for preregistration in medical clinical trials, see, e.g., www.clinicaltrials.gov).”
Self-citationGigerenzer, G. (2004) · The Journal of Socio-Economics · doi:10.1016/j.socec.2004.09.033 · Cited by 890
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · cited 2×
“Remedy 4: Preregistration Another effective remedy is preregistration (e.g., Chambers, 2013; Chambers et al., 2013; de Groot, 1969; Goldacre, 2009; Nosek & Lakens, 2014; Wagenmakers, Wetzels, Borsboom, van der Maas & Kievit, 2012; Wolfe, 2013; for preregistration in medical clinical trials, see,…”
BackgroundMethodGeneralizations of the familywise error rate
Lehmann, E. L., & Romano, J. P. (2005) · The Annals of Statistics · doi:10.1214/009053605000000084 · Cited by 320 · cited 1×
“…setting, all hypotheses implied by the design are considered and tested jointly, rendering this collection of hypotheses a family; in line with the idea that Bthe term ‘family’ refers to the collection of hypotheses ... that is being considered for joint testing^ (Lehmann & Romano, 2005).”
BackgroundSmith, R. A., Levine, T. R., Lachlan, K. A., & Fediuk, T. A. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00821.x · Cited by 77
Cited by
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
afex: Analysis of Factorial Experiments
Singmann, H., Bolker, B., Westfall, J., Aust, F., & Ben-Shachar, M. S. (2012) · CRAN: Contributed Packages · doi:10.32614/cran.package.afex · Cited by 773
Simulation-Based Power Analysis for Factorial Analysis of Variance Designs
Lakens, D., & Caldwell, A. R. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920951503 · Cited by 602
Learning analytics to unveil learning strategies in a flipped classroom
Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017) · The Internet and Higher Education · doi:10.1016/j.iheduc.2017.02.001 · Cited by 376
Rubin, M. (2021) · Synthese · doi:10.1007/s11229-021-03276-4 · Cited by 371
Metadata
- Authors
- Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., Waldorp, L. J., & Wagenmakers, E.-J.
- Venue
- Psychonomic Bulletin & Review
- Published
- 2015-09-15
- Volume
- 23
- Issue
- 2
- Pages
- 640-647
- ISSN
- 1069-9384 (ISSN-L)
- 1531-5320
- OpenAlex
- W1835051027
- Cited by
- 416
- References
- 10
- FWCI
- 24.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies. https://app.cochat.ai/papers/10.3758/s13423-015-0913-5
- The paper
- Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., Waldorp, L. J., & Wagenmakers, E.-J. (2016). Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-015-0913-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
