The DOI matches its Crossref record; 31 of the 32 references it cites were found in a registry or an index.
A peculiar prevalence of p values just below .05
Abstract
In null hypothesis significance testing (NHST), p values are judged relative to an arbitrary threshold for significance (.05). The present work examined whether that standard influences the distribution of p values reported in the psychology literature. We examined a large subset of papers from three highly regarded journals. Distributions of p were found to be similar across the different journals. Moreover, p values were much more common immediately below .05 than would be expected based on the number of p values occurring in other ranges. This prevalence of p values just below the arbitrary criterion for significance was observed in all three journals. We discuss potential sources of this pattern, including publication bias and researcher degrees of freedom.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many published studies only barely achieve nominal statistical significance 15 .”
From Power failure: why small sample size undermines the reliability of neuroscience (2013) · cites this paper as [15]
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Sackett, D. L. (1979) · Journal of Chronic Diseases · doi:10.1016/0021-9681(79)90012-2 · Cited by 2,075
The review of educational research.
(1931) · Journal of Educational Psychology · doi:10.1037/h0066448 · Cited by 1,575
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
Schmidt, F. L. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.115 · Cited by 1,044
Beyond significance testing: Reforming data analysis methods in behavioral research.
Kline, R. B. (2004) · doi:10.1037/10693-000 · Cited by 1,034
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 1×
“Authors may assume that they must obtain statistical significance in their studies in order to publish (Sterling, 1959).”
BackgroundThe case against statistical significance testing
Carver, R. (1978) · Harvard Educational Review · doi:10.17763/haer.48.3.t490261645281841 · Cited by 931
Calibration of ρ Values for Testing Precise Null Hypotheses
Sellke, T., Bayarri, M. J., & Berger, J. O. (2001) · The American Statistician · doi:10.1198/000313001300339950 · Cited by 856
The fallacy of the null-hypothesis significance test.
Rozeboom, W. W. (1960) · Psychological Bulletin · doi:10.1037/h0042040 · Cited by 740
Cumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451
An Alternative to Null-Hypothesis Significance Tests
Killeen, P. R. (2005) · Psychological Science · doi:10.1111/j.0956-7976.2005.01538.x · Cited by 443
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424
Becoming a behavioral science researcher: a guide to producing research that matters
Kline, R. B. (2009) · Choice Reviews Online · doi:10.5860/choice.46-3545 · Cited by 240
Why P Values Are Not a Useful Measure of Evidence in Statistical Significance Testing
Hubbard, R., & Lindsay, R. M. (2008) · Theory & Psychology · doi:10.1177/0959354307086923 · Cited by 226
The Interpretation of Levels of Significance by Psychological Researchers
Rosenthal, R., & Gaito, J. M. (1963) · The Journal of Psychology · doi:10.1080/00223980.1963.9916596 · Cited by 183
Trafimow, D. (2003) · Psychological Review · doi:10.1037/0033-295x.110.3.526 · Cited by 158
Statistical Reform in Psychology: Is Anything Changing?
Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A., Lo, J., McMenamin, N., & Wilson, S. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01881.x · Cited by 122
When are ‘positive’ clinical trials in oncology truly positive?
Ocana, A., & Tannock, I. F. (2011) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djq463 · Cited by 110
Post Hoc Pair-Wise Comparisons for the Chi-Square Test of Homogeneity of Proportions
Cox, M. K., & Key, C. H. (1993) · Educational and Psychological Measurement · doi:10.1177/0013164493053004008 · Cited by 64
Cicchetti, D. V., Koenig, K., Klin, A. J., Volkmar, F. R., Paul, R., & Sparrow, S. S. (2010) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-010-1035-6 · Cited by 49
Poitevineau, J., & Lecoutre, B. (2001) · Psychonomic Bulletin & Review · doi:10.3758/bf03196227 · Cited by 41
Hypothesis Testing in Relation to Statistical Methodology
Clark, C. A. (1963) · Review of Educational Research · doi:10.2307/1169648 · Cited by 35
p rep misestimates the probability of replication
Iverson, G., Lee, M. D., & Wagenmakers, E. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.424 · Cited by 27
Chapter I: Hypothesis Testing in Relation to Statistical Methodology
Clark, C. A. (1963) · Review of Educational Research · doi:10.3102/00346543033005455 · Cited by 15
How often is prep close to the true replication probability?
Trafimow, D., MacDonald, J. A., Rice, S. C., & Clason, D. L. (2010) · Psychological Methods · doi:10.1037/a0018533 · Cited by 6
Cited by
Sorted by citations
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Metadata
- Authors
- Masicampo, E. J., & Lalande, D. R.
- Affiliations
- Wake Forest University (US)
- Université du Québec à Chicoutimi (CA)
- Corresponding author
- Masicampo, E.J.
- Lalande, Daniel R.
- Venue
- Quarterly Journal of Experimental Psychology
- Publisher
- SAGE Publications
- Published
- 2012-11
- Volume
- 65
- Issue
- 11
- Pages
- 2271-2279
- ISSN
- 0033-555X (ISSN-L)
- 1747-0218
- 1747-0226
- OpenAlex
- W2112039868
- Cited by
- 266
- References
- 32
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 17 · 2023: 18 · 2022: 23 · 2021: 24 · 2020: 24 · 2019: 28 · 2018: 26 · 2017: 23 · 2016: 15 · 2015: 21 · 2014: 22 · 2013: 11 · 2012: 4
- FWCI
- 21.52
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Quarterly Journal of Experimental Psychology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- p-values, null hypothesis significance testing, significance threshold, p-value distribution, publication bias, researcher degrees of freedom, statistical reporting, psychology journals
- MeSH terms
- Humans, Probability, Psychology, Statistics as Topic, Prevalence, Databases, Bibliographic, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
