The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Erroneous analyses of interactions in neuroscience: a problem of significance
Abstract
In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Use of multiple analyses on the same data set and selective reports of statistical methods used, or insufficient correction for multiple comparisons, can cause inflation in effects (29).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [29]
In the citing paper
“priming and awareness are typically tested by comparing two separate tests”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011
In the citing paper
“However, this approach is fundamentally flawed (Nieuwenhuis et al. 2011, Meyen et al. 2022).”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011
In the citing paper
“In this approach, the comparison of two experimental effects is performed by executing two separate t-tests and independently establishing significance for each. When one turns out to be significant and the other is not, one concludes that they are different (Nieuwenhuis et al. 2011).”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011
In the citing paper
“The combination of two Bayesian t-tests (one to determine the presence or absence of awareness and another to determine the presence of a priming effect) essentially suffers from the same problem as executing two frequentist t-tests: the comparison of two experimental effects requires a statistical test on their difference”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011
Citations
Cites
Using confidence intervals in within-subject designs
Loftus, G. R., & Masson, M. E. J. (1994) · Psychonomic Bulletin & Review · doi:10.3758/bf03210951 · Cited by 2,433
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407 · cited 1×
“Although superficially compelling, the latter type of statistical reasoning is erroneous because the difference between significant and not significant need not itself be statistically significan”
BackgroundError bars in experimental biology
Cumming, G., Fidler, F., & Vaux, D. L. (2007) · The Journal of Cell Biology · doi:10.1083/jcb.200611141 · Cited by 940
Statistical procedures and the justification of knowledge in psychological science.
Rosnow, R. L., & Rosenthal, R. (1989) · American Psychologist · doi:10.1037/0003-066x.44.10.1276 · Cited by 863
Statistical procedures and the justification of knowledge in psychological science.
Rosnow, R. L., & Rosenthal, R. (1992) · American Psychological Association eBooks · doi:10.1037/10109-027 · Cited by 653
Guidelines for reporting an fMRI study
Poldrack, R. A., Fletcher, P. C., Henson, R. N., Worsley, K. J., Brett, M., & Nichols, T. E. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.11.048 · Cited by 550
What can Functional Neuroimaging Tell the Experimental Psychologist?
Henson, R. (2005) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/02724980443000502 · Cited by 327
The Interpretation of Levels of Significance by Psychological Researchers
Rosenthal, R., & Gaito, J. M. (1963) · The Journal of Psychology · doi:10.1080/00223980.1963.9916596 · Cited by 183
Cited by
Sorted by citations
Items 701–800 of 868
Use of significance test logic by scientists in a novel reasoning taskPreprint
Morey, R. D., & Hoekstra, R. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/sxj3r · Cited by 1
Statistical reminders involving indirect comparisons and non-significant results: a case studyPreprint
Puri, R. (2021) · doi:10.31234/osf.io/6dz5b · Cited by 1
The science of science: Clinical Science launches a new translational meta-research collection
Weissgerber, T. L. (2021) · Clinical Science · doi:10.1042/cs20210777 · Cited by 1
Harris, C. A. (2022) · Utrecht University Repository (Utrecht University) · doi:10.33540/873 · Cited by 1
Drug targets slip-sliding away
(2011) · Nature Medicine · doi:10.1038/nm.2530 · Cited by 1
Trying to make it work: Compositional effects in the processing of compound "nonwords"Preprint
Guenther, F., & Marelli, M. (2019) · doi:10.31234/osf.io/72szg · Cited by 1
The Implicit Relational Assessment Procedure’s trial-types are not independentPreprint
Hussey, I. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4gsxq · Cited by 1
Hirschhorn, R., & Schönberg, T. (2023) · Elsevier eBooks · doi:10.1016/b978-0-12-820480-1.00014-0 · Cited by 1
Baker, A. L., & Gulledge, A. T. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.05.04.652144 · Cited by 1
Neural correlates of consciousness in an auditory no-report fMRI studyPreprint
Dellert, T., Balster, H., Schloßmacher, I., Bruchmann, M., Moeck, R., & Straube, T. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.05.16.654468 · Cited by 2
Translational neuroscience requires better design and analysis of preclinical studiesPreprint
Lazic, S. E. (2012) · Nature Precedings · doi:10.1038/npre.2012.6987.1 · Cited by 1
de Zubicaray, G. I. (2026) · Language Cognition and Neuroscience · doi:10.1080/23273798.2026.2683465 · Cited by 1
Does statistical significance demonstrate a change in prognostic utility after TAVI?
Sarıoğlu, G. (2026) · Journal of Echocardiography · doi:10.1007/s12574-026-00751-0 · Cited by 1
Evidence for unconscious priming using a Bayesian single-subject approach
Sánchez-Fuenzalida, N., van Gaal, S., van den Hurk, Z., Stein, T., & Fahrenfort, J. J. (2026) · Neuroscience of Consciousness · doi:10.1093/nc/niag052
Allen, B. N. (2015) · Scholarship@Western (Western University)
The EEG activity of girls with attention-deficit/hyperactivity disorder
Dupuy, F. E. (2014) · Research Online (University of Wollongong)
The temporal dynamics of speeded decision making
Dutilh, G. (2012) · UvA-DARE (University of Amsterdam)
The quality of perception without attention
Vandenbroucke, A. R. E. (2013) · UvA-DARE (University of Amsterdam)
Teixeira, I. P., Smirmaul, B. P. C., LUCHINI, P. E. H., Gobbi, R. B., de Moura, L. P., & Fernandes, R. A. (2015) · Revista Brasileira de Educação Física e Esporte · doi:10.1590/1807-55092015000100139
5. What do we do now? Implications for the clinical practice
DePaolis, M. D., Cho, P., Cunningham, D., Morgan, P. B., QUINN, T. G., Saks, A. P., & Shovlin, J. P. (2013) · Contact Lens and Anterior Eye · doi:10.1016/s1367-0484(13)60007-7
The Failings of Modern Medicine
Bauer, H. H. (2014) · EXPLORE · doi:10.1016/j.explore.2014.08.009
van Maanen, H. (2014) · Huisarts en Wetenschap · doi:10.1007/s12445-014-0122-x
Probing visual consciousness with transcranial magnetic stimulation
Allen, C. P. G. (2012) · ORCA Online Research @Cardiff (Cardiff University)
Rezlescu, C., & Susilo, T. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00289
ARE NEUROREDUCTIONIST EXPLANATIONS OF COGNITION POSSIBLE
Uttal, W. R. (2014)
Mayes, A. (2013) · Swinburne figshare (Swinburne University of Technology) · doi:10.25916/sut.26257748
Vieth, H. E., McMahon, K. L., & de Zubicaray, G. I. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01984
NEUROSCIENTISTS STUMBLE ON STATS
Pulver, S. R. (2011) · Journal of Experimental Biology · doi:10.1242/jeb.063826
Phonological working memory and finiteness marking in typical development
Ostrovskaya, I. (2015)
Hubert, L. (2012) · Chapman & Hall/CRC eBooks · doi:10.1201/b13103-20
Medial Prefrontal Cortex Signals Prediction Errors Across Domains of Pain and Cognitive Control
Jahn, A. (2015) · IUScholarWorks (Indiana University)
Macroscopic Neural Theories of Cognition
Uttal, W. R. (2014) · Defense Technical Information Center (DTIC) · doi:10.21236/ada597175
The preproglucagon derived peptides and energy homeostasis
Parker, J. (2013) · doi:10.25560/10956
Campbell, K. L. (2014) · TSpace (University of Toronto)
Topics to Consider When Analyzing Data
Smalheiser, N. R. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-811306-6.00008-7
Fight Dispersion Through Education: The Results of the First Cycle of the NoOut Project
Batini, F., Bartolucci, M., & Carlo, E. D. (2017) · Mind Brain and Education · doi:10.1111/mbe.12157
Tyurenkov, I. N., Волотова, Е. В., Куркин, Д. В., & Bakulin, D. A. (2019) · Reviews on Clinical Pharmacology and Drug Therapy · doi:10.17816/rcf17157-64
Why do we foster and grant wrong innovative scientific methods? The Neuroscientific Challenge
Vallverdú, J. (2018) · Journal of Psychology and Psychiatry · doi:10.15761/jpp.1000114
Decipher the complexity of cis-regulatory regions by a modified Cas9
Kirchner, S., Reuter, S., Westphal, A., & Mrowka, R. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235530
Kashyap, V. L. (2020) · Springer eBooks · doi:10.1007/978-981-15-6337-9_6
On the effects of offloading memory on memory performance
Kelly, M. O. (2020) · UWSpace (University of Waterloo)
Bacons wetenschapsethiek: onze hoop op een humanere academie
CORNELIS, G. C. (2014) · Ethiek en Maatschappij · doi:10.21825/em.94602
Liu, T. T., Chen, Z., Oxner, M., Wang, W., Ku, Y., & Hayward, W. G. (2019) · doi:10.31234/osf.io/t89aw
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.001
Construct and Data-Evaluation Validity
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.004
Single-Case Experimental Research Designs
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.009
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.007
Cautions, Negative Effects, and Replication
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.016
Selecting Measures for Research
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.011
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.021
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.013
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.022
Ideas that Begin the Research Process
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.005
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.020
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.025
Presenting and Analyzing the Data
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.015
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.024
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.018
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.002
Assessment: Types of Measures and Their Use
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.012
Experimental Research Using Group Designs
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.006
Ethical Issues and Guidelines for Research
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.017
Communication of Research Findings
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.019
Good, P. I., & Hardin, J. W. (2012) · Wiley eBooks · doi:10.1002/9781118360125.biblio
Motulsky, H. (2019) · De Boeck Supérieur eBooks · doi:10.3917/dbu.motul.2019.01.0533
Ethical Considerations in Data Collection
Hubert, L. (2012) · Chapman & Hall/CRC eBooks · doi:10.1201/b13103-23
Dissociation Between Speech and Emotion Effects in Short-Term Memory: A Data Reanalysis.
Wiens, S. (2021) · Meta-Psychology · doi:10.15626/mp.2019.1635
Statistical Use and Misuse in Scientific Publications
Hoffman, J. I. E. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-817084-7.00002-4
(2011) · CHANCE · doi:10.1007/s00144-011-0041-6
Proactively maintaining publication standards
(2012) · Biochemical Pharmacology · doi:10.1016/j.bcp.2012.05.016
Sala, C. (2018) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.732860113.793545803
Pinsault, N., Dolgopoloff, M., & Guillaud, A. (2017) · Manual Therapy Posturology & Rehabilitation Journal · doi:10.17784/mtprehabjournal.2017.15.497
(2016) · Journal of Experimental Psychology General · doi:10.1037/xge0000166.supp
(2014) · Presses universitaires de Grenoble eBooks · doi:10.3917/pug.guill.2014.01.0175
Schmid, B. (2020) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.736726087.793578052
Legg, E. W., Farrar, B. G., Lazić, A., Thiele, M., Kampis, D., Mani, N., Sesar, K., Klapwijk, E. T., Schlingloff, L., Reindl, E., et al. (2021) · doi:10.31234/osf.io/mxac9
(2020) · Journal of Experimental Psychology General · doi:10.1037/xge0000733.supp
Drug Discovery and Personalized Medicine
Sontheimer, H. W. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-821228-8.00013-5
Decision letter for "That's a Lot to Process! Pitfalls of Popular Path Models"
(2022) · Publons reviews and discussion · doi:10.1177/25152459221095827/v2/decision1
Decision letter for "That's a Lot to Process! Pitfalls of Popular Path Models"
(2022) · Publons reviews and discussion · doi:10.1177/25152459221095827/v3/decision1
Author response for "That's a Lot to Process! Pitfalls of Popular Path Models"
Rohrer, J. M., Hünermund, P., Arslan, R. C., & Elson, M. (2022) · Publons reviews and discussion · doi:10.1177/25152459221095827/v2/response1
Author response for "That's a Lot to Process! Pitfalls of Popular Path Models"
Rohrer, J. M., Hünermund, P., Arslan, R. C., & Elson, M. (2022) · Publons reviews and discussion · doi:10.1177/25152459221095827/v3/response1
Decision letter for "That's a Lot to Process! Pitfalls of Popular Path Models"
(2021) · Publons reviews and discussion · doi:10.1177/25152459221095827/v1/decision1
Review for "That's a Lot to Process! Pitfalls of Popular Path Models"
(2021) · Publons reviews and discussion · doi:10.1177/25152459221095827/v1/review3
Review for "That's a Lot to Process! Pitfalls of Popular Path Models"
(2021) · Publons reviews and discussion · doi:10.1177/25152459221095827/v1/review2
Review for "That's a Lot to Process! Pitfalls of Popular Path Models"
(2021) · Publons reviews and discussion · doi:10.1177/25152459221095827/v1/review1
Sığırlı, D., & Taşar, P. (2023) · Turkiye Klinikleri Journal of Biostatistics · doi:10.5336/biostatic.2022-94692
Kołodziej, S., Łoboda, J., Święcka, A., Sirko, W., & Białek, M. (2018) · Polish Psychological Bulletin · doi:10.24425/119486
Grid Cell Firing Patterns Maintain their Hexagonal Firing Patterns on a Circular TrackPreprint
Yim, M. Y., Walton, S., & Hedrick, K. R. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.09.14.557783
Stein, T., van Gaal, S., & Fahrenfort, J. J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573995
Stein, T., van Gaal, S., & Fahrenfort, J. J. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/2zpy8
Pièges et mésusages en analyse de données
Supplisson, O., & Sofonea, M. T. (2023) · Anesthésie & Réanimation · doi:10.1016/j.anrea.2023.08.002
Comment on Jakob, Dechamps & Maier
Bancel, P. A. (2024) · Journal of Anomalous Experience and Cognition · doi:10.31156/jaex.25420
Grid cell populations maintain their hexagonal firing patterns on a circular trackPreprint
Yim, M. Y., Walton, S., & Hedrick, K. R. (2024) · Research Square · doi:10.21203/rs.3.rs-4468831/v1
Lerch, J. P., & Johansen‐Berg, H. (2024) · Elsevier eBooks · doi:10.1016/b978-0-12-820480-1.00170-4
Sampling, study design, and statistics. A didactic discussion
Welsby, P. D., & Weatherall, M. (2024) · Postgraduate Medical Journal · doi:10.1093/postmj/qgae114
Metadata
- Authors
- Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J.
- Affiliations
- Leiden University (NL)
- University of Amsterdam (NL)
- Corresponding author
- Nieuwenhuis, Sander
- Venue
- Nature Neuroscience
- Published
- 2011-08-26
- Volume
- 14
- Issue
- 9
- Pages
- 1105-1107
- ISSN
- 1097-6256 (ISSN-L)
- 1546-1726
- OpenAlex
- W2165810797
- Semantic Scholar
- 6ea9a180f01c0c6a4b559511cf362bbfcfd73b87
- Cited by
- 864
- References
- 8
- Citations by year
- 2026: 51 · 2025: 32 · 2024: 37 · 2023: 60 · 2022: 52 · 2021: 83 · 2020: 61 · 2019: 63 · 2018: 61 · 2017: 57 · 2016: 57 · 2015: 47 · 2014: 69 · 2013: 57 · 2012: 65
- FWCI
- 68.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature Neuroscience (publishedVersion)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- interaction effects, effect size comparison, significance testing, statistical errors, two-sample t-test, neuroscience literature
- MeSH terms
- Animals, Humans, Neurons, Neurosciences, Statistics as Topic, Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
