Erroneous analyses of interactions in neuroscience: A problem of significance

Nieuwenhuis, Sander; Forstmann, Birte U; Wagenmakers, Eric-Jan

doi:10.1038/nn.2886

Cited by 868Meta-analysis and systematic reviewsPsychology Research and BibliometricsStatistics Education and Methodologies

Erroneous analyses of interactions in neuroscience: A problem of significance

Nieuwenhuis, Sander, Forstmann, Birte U, Wagenmakers, Eric-Jan·2011·Nature Neuroscience

DOI 10.1038/nn.2886

Open this paper in CoChat — free account

Abstract

In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling.

Integrity signals

Registry

Crossref · 10.1038/nn.2886

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Use of multiple analyses on the same data set and selective reports of statistical methods used, or insufficient correction for multiple comparisons, can cause inflation in effects.
“Use of multiple analyses on the same data set and selective reports of statistical methods used, or insufficient correction for multiple comparisons, can cause inflation in effects (29).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [29]
Priming and awareness are typically tested by comparing two separate tests.
“priming and awareness are typically tested by comparing two separate tests”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011
The double t-test approach is fundamentally flawed.
“However, this approach is fundamentally flawed (Nieuwenhuis et al. 2011, Meyen et al. 2022).”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011
In the double t-test approach, the comparison of two experimental effects is performed by executing two separate t-tests and independently establishing significance for each; when one is significant and the other is not, one concludes they are different.
“In this approach, the comparison of two experimental effects is performed by executing two separate t-tests and independently establishing significance for each. When one turns out to be significant and the other is not, one concludes that they are different (Nieuwenhuis et al. 2011).”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011
The double t-test approach is problematic because comparing two experimental effects requires a statistical test on their difference.
“The combination of two Bayesian t-tests (one to determine the presence or absence of awareness and another to determine the presence of a priming effect) essentially suffers from the same problem as executing two frequentist t-tests: the comparison of two experimental effects requires a statistical test on their difference”
From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Nieuwenhuis et al. 2011

Using confidence intervals in within-subject designs
Loftus, G. R., & Masson, M. E. J. (1994) · Psychonomic Bulletin & Review · doi:10.3758/bf03210951 · Cited by 2,448 · see citing works
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,316 · cited 1× · see citing works
“Although superficially compelling, the latter type of statistical reasoning is erroneous because the difference between significant and not significant need not itself be statistically significan”
Background
Statistical Procedures and the Justification of Knowledge in Psychological Science
Rosnow, R., & Rosenthal, R. (1989) · American Psychologist · doi:10.1037/0003-066x.44.10.1276 · Cited by 951 · see citing works
Error bars in experimental biology
Cumming, G., Fidler, F., & Vaux, D. L. (2007) · The Journal of Cell Biology · doi:10.1083/jcb.200611141 · Cited by 940 · see citing works
Statistical procedures and the justification of knowledge in psychological science.
Rosnow, R. L., & Rosenthal, R. (1992) · American Psychological Association eBooks · doi:10.1037/10109-027 · Cited by 653 · see citing works
Guidelines for reporting an fMRI study
Poldrack, R. A., Fletcher, P. C., Henson, R. N., Worsley, K. J., Brett, M., & Nichols, T. E. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.11.048 · Cited by 550 · see citing works
What can Functional Neuroimaging Tell the Experimental Psychologist?
Henson, R. (2005) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/02724980443000502 · Cited by 327 · see citing works
The Interpretation of Levels of Significance by Psychological Researchers
Rosenthal, R., & Gaito, J. M. (1963) · The Journal of Psychology · doi:10.1080/00223980.1963.9916596 · Cited by 183 · see citing works

Flagged first, then by citations

Sorted by citations

cocor: A Comprehensive Solution for the Statistical Comparison of Correlations
Diedenhofen, B., & Musch, J. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121945 · Cited by 1,919 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
A generalized form of context-dependent psychophysiological interactions (gPPI): A comparison to standard approaches
McLaren, D. G., Ries, M. L., Xu, G., & Johnson, S. C. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.03.068 · Cited by 1,445 · see citing works
Scanning the horizon: towards transparent and reproducible neuroimaging research
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.167 · Cited by 1,410 · see citing works
Analysis of drug combinations: current methodological landscape
Foucquier, J., & Guedj, M. (2015) · Pharmacology Research & Perspectives · doi:10.1002/prp2.149 · Cited by 1,115 · see citing works
Broad domain generality in focal regions of frontal and parietal cortex
Fedorenko, E., Duncan, J., & Kanwisher, N. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1315235110 · Cited by 952 · see citing works
BoxPlotR: a web tool for generation of box plots
Spitzer, M., Wildenhain, J., Rappsilber, J., & Tyers, M. (2014) · Nature Methods · doi:10.1038/nmeth.2811 · Cited by 809 · see citing works
Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754 · see citing works
Statistical aspects of equivalent dose and error calculation and display in OSL dating: An overview and some recommendations
Galbraith, R. F., & Roberts, R. G. (2012) · Quaternary Geochronology · doi:10.1016/j.quageo.2012.04.020 · Cited by 730 · see citing works
Using Bayes factor hypothesis testing in neuroscience to establish evidence of absence
Keysers, C., Gazzola, V., & Wagenmakers, E.-J. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-0660-4 · Cited by 654 · see citing works
Self-citation
A solution to dependency: using multilevel analysis to accommodate nested data
Aarts, E., Verhage, M., Veenvliet, J. V., Dolan, C. V., & van der Sluis, S. (2014) · Nature Neuroscience · doi:10.1038/nn.3648 · Cited by 631 · see citing works
How robust is the language architecture? The case of mood
Berkum, J. J. A. V., Goede, D. D., van Alphen, P. M., Mulder, E. R., & Kerstholt, J. H. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00505 · Cited by 562 · see citing works
Diet quality and depression risk: A systematic review and dose-response meta-analysis of prospective studies
Molendijk, M., Molero, P., Ortuño Sánchez-Pedreño, F., Van der Does, W., & Angel Martínez-González, M. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.09.022 · Cited by 550 · see citing works
Autism-associated neuroligin-3 mutations commonly impair striatal circuits to boost repetitive behaviors
Rothwell, P. E., Fuccillo, M. V., Maxeiner, S., Hayton, S. J., Gokce, O., Lim, B. K., Fowler, S. C., Malenka, R. C., & Südhof, T. C. (2014) · Cell · doi:10.1016/j.cell.2014.04.045 · Cited by 477 · see citing works
Computation of measures of effect size for neuroscience data sets
Hentschke, H., & Stüttgen, M. C. (2011) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07902.x · Cited by 405 · see citing works
Neuroimaging of the periaqueductal gray: State of the field
Linnman, C., Moulton, E. A., Barmettler, G., Becerra, L. R., & Borsook, D. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.11.095 · Cited by 402 · see citing works
Structural brain plasticity in adult learning and development
Lövdén, M., Wenger, E., Mårtensson, J., Lindenberger, U., & Bäckman, L. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.02.014 · Cited by 400 · see citing works
Consensus on the reporting and experimental design of clinical and cognitive-behavioural neurofeedback studies (CRED-nf checklist)
Ros, T., Enriquez-Geppert, S., Zotev, V., Young, K. D., Wood, G., Whitfield-Gabrieli, S., Wan, F., Vuilleumier, P., Vialatte, F., Van De Ville, D., et al. (2020) · Brain · doi:10.1093/brain/awaa009 · Cited by 399 · see citing works
Sleep deprivation and stressors: evidence for elevated negative affect in response to mild stressors when sleep deprived
Minkel, J. D., Banks, S., Htaik, O., Moreta, M. C., Jones, C. W., McGlinchey, E. L., Simpson, N. S., & Dinges, D. F. (2012) · Emotion · doi:10.1037/a0026871 · Cited by 379 · see citing works
Scientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar‐Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 366 · see citing works
The anova to mixed model transition
Boisgontier, M. P., & Cheval, B. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.05.034 · Cited by 357 · see citing works
CHD8 haploinsufficiency results in autistic-like phenotypes in mice
Katayama, Y., Nishiyama, M., Shoji, H., Ohkawa, Y., Kawamura, A., Sato, T., Suyama, M., Takumi, T., Miyakawa, T., & Nakayama, K. I. (2016) · Nature · doi:10.1038/nature19357 · Cited by 353 · see citing works
Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies
Walum, H., Waldman, I. D., & Young, L. J. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.06.016 · Cited by 345 · see citing works
“Use of multiple analyses on the same data set and selective reports of statistical methods used, or insufficient correction for multiple comparisons, can cause inflation in effects (29).”
Method
Reflecting on the mirror neuron system in autism: A systematic review of current theories
Hamilton, A. F. D. C. (2012) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2012.09.008 · Cited by 343 · see citing works
Quantifying Sources of Variability in Infancy Research Using the Infant-Directed-Speech Preference
Frank, M. C., Alcock, K., Arias‐Trejo, N., Aschersleben, G., Baldwin, D. A., Barbu, S., Bergelson, E., Bergmann, C., Black, A. K., Blything, R. P., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919900809 · Cited by 343 · see citing works
Evaluation of synergism in drug combinations and reference models for future orientations in oncology
Duarte, D., & Vale, N. (2022) · Current Research in Pharmacology and Drug Discovery · doi:10.1016/j.crphar.2022.100110 · Cited by 340 · see citing works
Sex differences in hippocampal cognition and neurogenesis
Yagi, S., & Galea, L. A. M. (2018) · Neuropsychopharmacology · doi:10.1038/s41386-018-0208-4 · Cited by 340 · see citing works
The language network as a natural kind within the broader landscape of the human brain
Fedorenko, E. G., Ivanova, A. A., & Regev, T. I. (2024) · Nature reviews. Neuroscience · doi:10.1038/s41583-024-00802-4 · Cited by 327 · see citing works
An open investigation of the reproducibility of cancer biology research
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A. (2014) · eLife · doi:10.7554/elife.04333 · Cited by 312 · see citing works
Big data, bigger dilemmas: A critical review
Ekbia, H. R., Mattioli, M., Kouper, I., Arave, G., Ghazinejad, A., Bowman, T. D., Suri, V. R., Tsou, A., Weingart, S., & Sugimoto, C. R. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23294 · Cited by 305 · see citing works
Neurofeedback with fMRI: A critical systematic review
Thibault, R. T., MacPherson, A., Lifshitz, M., Roth, R. R., & Raz, A. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.12.071 · Cited by 300 · see citing works
Bayesian tests to quantify the result of a replication attempt.
Verhagen, J., & Wagenmakers, E. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0036731 · Cited by 287 · see citing works
Self-citation
Explicit memory creation during sleep demonstrates a causal role of place cells in navigation
de Lavilléon, G., Lacroix, M. M., Rondi-Reig, L., & Benchenane, K. (2015) · Nature Neuroscience · doi:10.1038/nn.3970 · Cited by 279 · see citing works
Qualitative Research Methods
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.010 · Cited by 279 · see citing works
Meaningful Mediation Analysis: Plausible Causal Inference and Informative Communication
Pieters, R. G. M. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx081 · Cited by 270 · see citing works
Statistical methods for linguistic research: Foundational Ideas—Part II
Nicenboim, B., & Vasishth, S. (2016) · Language and Linguistics Compass · doi:10.1111/lnc3.12207 · Cited by 267 · see citing works
Quantitative set analysis for gene expression: a method to quantify gene set differential expression including gene-gene correlations
Yaari, G., Bolen, C. R., Thakar, J., & Kleinstein, S. H. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt660 · Cited by 262 · see citing works
Recommendations for sex/gender neuroimaging research: key principles and implications for research design, analysis, and interpretation
Rippon, G., Jordan-Young, R. M., Kaiser, A., & Fine, C. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00650 · Cited by 253 · see citing works
Comparisons against baseline within randomised groups are often used and can be highly misleading
Bland, M., & Altman, D. G. (2011) · Trials · doi:10.1186/1745-6215-12-264 · Cited by 251 · see citing works
Improving basic and translational science by accounting for litter-to-litter variation in animal models
Lazic, S. E., & Essioux, L. (2013) · BMC Neuroscience · doi:10.1186/1471-2202-14-37 · Cited by 247 · see citing works
Reporting and misreporting of sex differences in the biological sciences
Garcia-Sifuentes, Y., & Maney, D. L. (2021) · eLife · doi:10.7554/elife.70817 · Cited by 241 · see citing works
Research Design in Clinical Psychology
Kazdin, A. E. (2021) · doi:10.1017/9781108993647 · Cited by 240 · see citing works
Ten common statistical mistakes to watch out for when writing or reviewing a manuscript
Makin, T. R., & Orban de Xivry, J.-J. (2019) · eLife · doi:10.7554/elife.48175 · Cited by 238 · see citing works
Nav1.1-Overexpressing Interneuron Transplants Restore Brain Rhythms and Cognition in a Mouse Model of Alzheimer’s Disease
Martínez‐Losa, M., Tracy, T. E., Ma, K., Verret, L., Clemente-Perez, A., Khan, A. S., Cobos, I., Ho, K., Gan, L., Mucke, L., et al. (2018) · Neuron · doi:10.1016/j.neuron.2018.02.029 · Cited by 233 · see citing works
Enhancing the Delivery of Resveratrol in Humans: If Low Bioavailability is the Problem, What is the Solution?
Smoliga, J. M., & Blanchard, O. L. (2014) · Molecules · doi:10.3390/molecules191117154 · Cited by 231 · see citing works
The Long Way From α-Error Control to Validity Proper
Fiedler, K., Kutzner, F., & Krueger, J. I. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462587 · Cited by 229 · see citing works
The Significance of Meaning: Why Do Over 90% of Behavioral Neuroscience Results Fail to Translate to Humans, and What Can We Do to Fix It?
Garner, J. P. (2014) · ILAR Journal · doi:10.1093/ilar/ilu047 · Cited by 225 · see citing works
Improving standards in brain-behavior correlation analyses
Rousselet, G. A., & Pernet, C. R. (2012) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2012.00119 · Cited by 222 · see citing works
fMRI reveals language-specific predictive coding during naturalistic sentence comprehension
Shain, C., Blank, I. A., van Schijndel, M., Schuler, W., & Fedorenko, E. (2020) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2019.107307 · Cited by 215 · see citing works
Use and misuse of corrections for multiple testing
García-Pérez, M. Á. (2023) · Methods in Psychology · doi:10.1016/j.metip.2023.100120 · Cited by 212 · see citing works
Teaching an adult brain new tricks: A critical review of evidence for training-dependent structural plasticity in humans
Thomas, C., & Baker, C. I. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2012.03.069 · Cited by 204 · see citing works
Sleeping on the motor engram: The multifaceted nature of sleep-related motor memory consolidation
King, B. R., Hoedlmoser, K., Hirschauer, F., Dolfen, N., & Albouy, G. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.04.026 · Cited by 202 · see citing works
Subcortical brain volume, regional cortical thickness, and cortical surface area across disorders: findings from the ENIGMA ADHD, ASD, and OCD working groups
Boedhoe, P. S. W., van Rooij, D., Hoogman, M., Twisk, J. W. R., Schmaal, L., Abe, Y., Alonso, P., Ameis, S. H., Anikin, A., Anticevic, A., et al. (2020) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2020.19030331 · Cited by 201 · see citing works
The statistical significance filter leads to overoptimistic expectations of replicability
Vasishth, S., Mertzen, D., Jäger, L. A., & Gelman, A. (2018) · Journal of Memory and Language · doi:10.1016/j.jml.2018.07.004 · Cited by 194 · see citing works
Quantifying selection in high-throughput Immunoglobulin sequencing data sets
Yaari, G., Uduman, M., & Kleinstein, S. H. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks457 · Cited by 189 · see citing works
Antiphasic 40 Hz Oscillatory Current Stimulation Affects Bistable Motion Perception
Strüber, D., Rach, S., Trautmann-Lengsfeld, S. A., Engel, A. K., & Herrmann, C. S. (2013) · Brain Topography · doi:10.1007/s10548-013-0294-x · Cited by 187 · see citing works
Perils and pitfalls of reporting sex differences
Maney, D. L. (2016) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2015.0119 · Cited by 186 · see citing works
Acquisition of Paleolithic toolmaking abilities involves structural remodeling to inferior frontoparietal regions
Hecht, E. E., Gutman, D. A., Khreisheh, N., Taylor, S. V., Kilner, J. M., Faisal, A. A., Bradley, B. A., Chaminade, T., & Stout, D. (2014) · Brain Structure and Function · doi:10.1007/s00429-014-0789-6 · Cited by 185 · see citing works
Improving basic and translational science by accounting for litter-to-litter variation in animal modelsPreprint
Lazic, S. E., & Essioux, L. (2016) · Cited by 182 · see citing works
From ERPs to MVPA Using the Amsterdam Decoding and Modeling Toolbox (ADAM)
Fahrenfort, J. J., van Driel, J., van Gaal, S., & Olivers, C. N. L. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00368 · Cited by 177 · see citing works
A history of the evolution of guidelines for reporting medical research: the long road to the EQUATOR Network
Altman, D. G., & Simera, I. (2016) · Journal of the Royal Society of Medicine · doi:10.1177/0141076815625599 · Cited by 174 · see citing works
Lack of selectivity for syntax relative to word meanings throughout the language network
Fedorenko, E., Blank, I. A., Siegelman, M., & Mineroff, Z. (2020) · Cognition · doi:10.1016/j.cognition.2020.104348 · Cited by 170 · see citing works
The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known
Calin-Jageman, R. J., & Cumming, G. (2019) · The American Statistician · doi:10.1080/00031305.2018.1518266 · Cited by 170 · see citing works
Application of the Hierarchical Bootstrap to Multi-Level Data in Neuroscience
Saravanan, V., Berman, G. J., & Sober, S. J. (2019) · doi:10.1101/819334 · Cited by 170 · see citing works
Understanding synergy
Geary, N. (2012) · American Journal of Physiology-Endocrinology and Metabolism · doi:10.1152/ajpendo.00308.2012 · Cited by 168 · see citing works
Analyzing multiple outcomes in clinical research using multivariate multilevel models.
Baldwin, S. A., Imel, Z. E., Braithwaite, S. R., & Atkins, D. C. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0035628 · Cited by 167 · see citing works
Neural Population Evidence of Functional Heterogeneity along the CA3 Transverse Axis: Pattern Completion versus Pattern Separation
Lee, H., Wang, C., Deshmukh, S. S., & Knierim, J. J. (2015) · Neuron · doi:10.1016/j.neuron.2015.07.012 · Cited by 164 · see citing works
Syntactic processing is distributed across the language system
Blank, I., Balewski, Z. Z., Mahowald, K., & Fedorenko, E. G. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.11.069 · Cited by 164 · see citing works
Statistical Significance and the Dichotomization of Evidence
McShane, B. B., & Gal, D. (2017) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1289846 · Cited by 164 · see citing works
The Potential of Functional Near-Infrared Spectroscopy-Based Neurofeedback—A Systematic Review and Recommendations for Best Practice
Kohl, S. H., Mehler, D. M. A., Lührs, M., Thibault, R. T., Konrad, K., & Sorger, B. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00594 · Cited by 160 · see citing works
The critical period hypothesis in second language acquisition: A statistical critique and a reanalysis
Vanhove, J. (2013) · PLoS ONE · doi:10.1371/journal.pone.0069172 · Cited by 158 · see citing works
Promoting Replicability in Developmental Research Through Meta-analyses: Insights From Language Acquisition Research
Bergmann, C., Tsuji, S., Piccinini, P. E., Lewis, M. L., Braginsky, M., Frank, M. C., & Cristià, A. (2018) · Child Development · doi:10.1111/cdev.13079 · Cited by 155 · see citing works
A Bayesian model-averaged meta-analysis of the power pose effect with informed and default priors: the case of felt power
Gronau, Q. F., van Erp, S., Heck, D. W., Cesario, J., Jonas, K. J., & Wagenmakers, E. (2017) · Comprehensive Results in Social Psychology · doi:10.1080/23743603.2017.1326760 · Cited by 153 · see citing works
Self-citation
The Design and Statistical Analysis of Animal Experiments
Bate, S. T., & Clark, R. A. (2014) · doi:10.1017/cbo9781139344319 · Cited by 150 · see citing works
Voodoo and circularity errors
Vul, E., & Pashler, H. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.027 · Cited by 147 · see citing works
A meta-analytical evaluation of the dual-hormone hypothesis: Does cortisol moderate the relationship between testosterone and status, dominance, risk taking, aggression, and psychopathy?
Dekkers, T. J., van Rentergem, J. A. A., Meijer, B., Popma, A., Wagemaker, E., & Huizenga, H. M. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.12.004 · Cited by 147 · see citing works
Estimation for Better Inference in Neuroscience
Calin-Jageman, R. J., & Cumming, G. (2019) · eNeuro · doi:10.1523/eneuro.0205-19.2019 · Cited by 145 · see citing works
Sex Differences in Opioid and Psychostimulant Craving and Relapse: A Critical Review
Nicolas, C., Zlebnik, N. E., Farokhnia, M., Leggio, L., Ikemoto, S., & Shaham, Y. (2022) · Pharmacological Reviews · doi:10.1124/pharmrev.121.000367 · Cited by 144 · see citing works
Interference patterns in subject-verb agreement and reflexives revisited: A large-sample study
Jäger, L. A., Mertzen, D., Van Dyke, J. A., & Vasishth, S. (2020) · Journal of Memory and Language · doi:10.1016/j.jml.2019.104063 · Cited by 142 · see citing works
Comm Research—Views from Europe| Five Challenges for the Future of Media-Effects Research
Valkenburg, P. M., & Peter, J. (2013) · International journal of communication · Cited by 139 · see citing works
STRengthening Analytical Thinking for Observational Studies: the STRATOS initiative
Sauerbrei, W., Abrahamowicz, M., Altman, D. G., le Cessie, S., Carpenter, J. R., & initiative, O. B. O. T. S. (2014) · Statistics in Medicine · doi:10.1002/sim.6265 · Cited by 135 · see citing works
Symptom improvement in children with autism spectrum disorder following bumetanide administration is associated with decreased GABA/glutamate ratios
Zhang, L., Huang, C., Dai, Y., Luo, Q., Ji, Y., Wang, K., Deng, S., Yu, J., Xu, M., Du, X., et al. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-0692-2 · Cited by 134 · see citing works
Avoiding common pitfalls in machine learning omic data science
Teschendorff, A. E. (2019) · Nature Materials · doi:10.1038/s41563-018-0241-z · Cited by 132 · see citing works
Task preparation processes related to reward prediction precede those related to task-difficulty expectation
Schevernels, H., Krebs, R. M., Santens, P., Woldorff, M. G., & Boehler, C. N. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.09.039 · Cited by 132 · see citing works
High‐throughput sequencing of TCR repertoires in multiple sclerosis reveals intrathecal enrichment of EBV‐reactive CD8+ T cells
Lossius, A., Johansen, J. N., Vartdal, F., Robins, H. S., Benth, J. Š., Holmøy, T., & Olweus, J. (2014) · European Journal of Immunology · doi:10.1002/eji.201444662 · Cited by 130 · see citing works
Multisensory integration across the senses in young and old adults
Mahoney, J. R., Li, P. C. C., Oh‐Park, M., Verghese, J., & Holtzer, R. (2011) · Brain Research · doi:10.1016/j.brainres.2011.09.017 · Cited by 129 · see citing works
Noradrenergic-dependent functions are associated with age-related locus coeruleus signal intensity differences
Liu, K. Y., Kievit, R., Tsvetanov, K. A., Betts, M. J., Düzel, E., Rowe, J. B., Cam-CAN, TYLER, L. K., Brayne, C., Bullmore, E. T., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-15410-w · Cited by 125 · see citing works
Do ‘early’ brain responses reveal word form prediction during language comprehension? A critical review
Nieuwland, M. S. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.11.019 · Cited by 122 · see citing works
Network-level reorganisation of functional connectivity following arm amputation
Makin, T. R., Filippini, N., Duff, E. P., Slater, D. H., Tracey, I., & Johansen‐Berg, H. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.02.067 · Cited by 121 · see citing works
Gender differences in cognitive Theory of Mind revealed by transcranial direct current stimulation on medial prefrontal cortex
Adenzato, M., Brambilla, M., Manenti, R., Lucia, L. D., Trojano, L., Garofalo, S., Enrici, I., & Cotelli, M. (2017) · Scientific Reports · doi:10.1038/srep41219 · Cited by 121 · see citing works
Moderate levels of activation lead to forgetting in the think/no-think paradigm
Detre, G. J., Natarajan, A., Gershman, S. J., & Norman, K. A. (2013) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2013.02.017 · Cited by 120 · see citing works
Retronasal Odor Perception Requires Taste Cortex, but Orthonasal Does Not
Blankenship, M. L., Grigorova, M., Katz, D. B., & Maier, J. X. (2018) · Current Biology · doi:10.1016/j.cub.2018.11.011 · Cited by 120 · see citing works
The Basal Pharmacology of Palmitoylethanolamide
Rankin, L., & Fowler, C. J. (2020) · International Journal of Molecular Sciences · doi:10.3390/ijms21217942 · Cited by 119 · see citing works
What event-related potentials (ERPs) bring to social neuroscience?
Ibáñez, A. M., Melloni, M., Huepe, D., Helgiu, E., Rivera-Rei, Á. A., Canales‐Johnson, A., Baker, P., & Moya, Á. (2012) · Social Neuroscience · doi:10.1080/17470919.2012.691078 · Cited by 119 · see citing works
The Experimental Design Assistant
Percie du Sert, N., Bamsey, I., Bate, S. T., Berdoy, M., Clark, R. A., Cuthill, I., Fry, D., Karp, N. A., Macleod, M., Moon, L., et al. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2003779 · Cited by 118 · see citing works
Which of these things is not like the others?
Kaufman, J. S., & MacLehose, R. F. (2013) · Cancer · doi:10.1002/cncr.28359 · Cited by 115 · see citing works
White matter pathology in Parkinson's disease: The effect of imaging protocol differences and relevance to executive function
Rae, C. L., Correia, M., Altena, E., Hughes, L. E., Barker, R. A., & Rowe, J. B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.06.012 · Cited by 112 · see citing works
Validation of the driver behaviour questionnaire using behavioural data from an instrumented vehicle and high-fidelity driving simulator
Helman, S., & Reed, N. (2014) · Accident Analysis & Prevention · doi:10.1016/j.aap.2014.12.008 · Cited by 112 · see citing works
Considerations for Studying Sex as a Biological Variable in Spinal Cord Injury
Stewart, A. N., MacLean, S. M., Stromberg, A. J., Whelan, J. P., Bailey, W. M., Gensel, J. C., & Wilson, M. E. (2020) · Frontiers in Neurology · doi:10.3389/fneur.2020.00802 · Cited by 110 · see citing works
An introduction to the bootstrap: a versatile method to make inferences by using data-driven simulations.
Rousselet, G. A., Pernet, C. R., & Wilcox, R. R. (2023) · Meta-Psychology · doi:10.15626/mp.2019.2058 · Cited by 110 · see citing works

Authors: Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J.
Affiliations: Leiden University (NL)
University of Amsterdam (NL)
Corresponding author: Nieuwenhuis, Sander
ORCID: Nieuwenhuis, Sander
Forstmann, Birte U
Wagenmakers, Eric-Jan
Venue: Nature Neuroscience
Publisher: Springer Science and Business Media LLC
Published: 2011-08-26
Volume: 14
Issue: 9
Pages: 1105-1107
ISSN: 1097-6256 (ISSN-L)
1546-1726
DOI: 10.1038/nn.2886
OpenAlex: W2165810797
Semantic Scholar: 6ea9a180f01c0c6a4b559511cf362bbfcfd73b87
Cited by: 864
References: 8
Citations by year: 2026: 51 · 2025: 32 · 2024: 37 · 2023: 60 · 2022: 52 · 2021: 83 · 2020: 61 · 2019: 63 · 2018: 61 · 2017: 57 · 2016: 57 · 2015: 47 · 2014: 69 · 2013: 57 · 2012: 65
FWCI: 68.19
Type: journal article
Language: en
Open access: closed
Licence: www.springer.com/tdm (tdm)
Locations: Nature Neuroscience (publishedVersion)
PubMed (publishedVersion)
UvA-DARE (University of Amsterdam) (publishedVersion)
UvA-DARE (University of Amsterdam) (submittedVersion)
Data Archiving and Networked Services (DANS) (submittedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
Keywords: interaction effects, effect size comparison, significance testing, statistical errors, two-sample t-test, neuroscience literature
MeSH terms: Animals, Humans, Neurons, Neurosciences, Statistics as Topic, Reproducibility of Results, Biomedical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Erroneous analyses of interactions in neuroscience: A problem of significance

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks