The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Erroneous analyses of interactions in neuroscience: a problem of significance
Abstract
In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Use of multiple analyses on the same data set and selective reports of statistical methods used, or insufficient correction for multiple comparisons, can cause inflation in effects (29).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [29]
In the citing paper
“priming and awareness are typically tested by comparing two separate tests”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011
In the citing paper
“However, this approach is fundamentally flawed (Nieuwenhuis et al. 2011, Meyen et al. 2022).”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011
In the citing paper
“In this approach, the comparison of two experimental effects is performed by executing two separate t-tests and independently establishing significance for each. When one turns out to be significant and the other is not, one concludes that they are different (Nieuwenhuis et al. 2011).”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011
In the citing paper
“The combination of two Bayesian t-tests (one to determine the presence or absence of awareness and another to determine the presence of a priming effect) essentially suffers from the same problem as executing two frequentist t-tests: the comparison of two experimental effects requires a statistical test on their difference”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011
Citations
Cites
Using confidence intervals in within-subject designs
Loftus, G. R., & Masson, M. E. J. (1994) · Psychonomic Bulletin & Review · doi:10.3758/bf03210951 · Cited by 2,433
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407 · cited 1×
“Although superficially compelling, the latter type of statistical reasoning is erroneous because the difference between significant and not significant need not itself be statistically significan”
BackgroundError bars in experimental biology
Cumming, G., Fidler, F., & Vaux, D. L. (2007) · The Journal of Cell Biology · doi:10.1083/jcb.200611141 · Cited by 940
Statistical procedures and the justification of knowledge in psychological science.
Rosnow, R. L., & Rosenthal, R. (1989) · American Psychologist · doi:10.1037/0003-066x.44.10.1276 · Cited by 863
Statistical procedures and the justification of knowledge in psychological science.
Rosnow, R. L., & Rosenthal, R. (1992) · American Psychological Association eBooks · doi:10.1037/10109-027 · Cited by 653
Guidelines for reporting an fMRI study
Poldrack, R. A., Fletcher, P. C., Henson, R. N., Worsley, K. J., Brett, M., & Nichols, T. E. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.11.048 · Cited by 550
What can Functional Neuroimaging Tell the Experimental Psychologist?
Henson, R. (2005) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/02724980443000502 · Cited by 327
The Interpretation of Levels of Significance by Psychological Researchers
Rosenthal, R., & Gaito, J. M. (1963) · The Journal of Psychology · doi:10.1080/00223980.1963.9916596 · Cited by 183
Flagged first, then by citations
Cited by
Sorted by citations
cocor: A Comprehensive Solution for the Statistical Comparison of Correlations
Diedenhofen, B., & Musch, J. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121945 · Cited by 1,919
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
McLaren, D. G., Ries, M. L., Xu, G., & Johnson, S. C. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.03.068 · Cited by 1,445
Scanning the horizon: towards transparent and reproducible neuroimaging research
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.167 · Cited by 1,410
Analysis of drug combinations: current methodological landscape
Foucquier, J., & Guedj, M. (2015) · Pharmacology Research & Perspectives · doi:10.1002/prp2.149 · Cited by 1,115
Metadata
- Authors
- Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J.
- Affiliations
- Leiden University (NL)
- University of Amsterdam (NL)
- Corresponding author
- Nieuwenhuis, Sander
- Venue
- Nature Neuroscience
- Published
- 2011-08-26
- Volume
- 14
- Issue
- 9
- Pages
- 1105-1107
- ISSN
- 1097-6256 (ISSN-L)
- 1546-1726
- OpenAlex
- W2165810797
- Semantic Scholar
- 6ea9a180f01c0c6a4b559511cf362bbfcfd73b87
- Cited by
- 864
- References
- 8
- Citations by year
- 2026: 51 · 2025: 32 · 2024: 37 · 2023: 60 · 2022: 52 · 2021: 83 · 2020: 61 · 2019: 63 · 2018: 61 · 2017: 57 · 2016: 57 · 2015: 47 · 2014: 69 · 2013: 57 · 2012: 65
- FWCI
- 68.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature Neuroscience (publishedVersion)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- interaction effects, effect size comparison, significance testing, statistical errors, two-sample t-test, neuroscience literature
- MeSH terms
- Animals, Humans, Neurons, Neurosciences, Statistics as Topic, Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
