The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
P-curve: A key to the file-drawer.
Abstract
Because scientists tend to report only studies (publication bias) or analyses (p-hacking) that "work," readers must ask, "Are these effects true, or do they merely reflect selective reporting?" We introduce p-curve as a way to answer this question. P-curve is the distribution of statistically significant p values for a set of studies (ps < .05). Because only true effects are expected to generate right-skewed p-curves-containing more low (.01s) than high (.04s) significant p values--only right-skewed p--curves are diagnostic of evidential value. By telling us whether we can rule out selective reporting as the sole explanation for a set of findings, p-curve offers a solution to the age-old inferential problems caused by file-drawers of failed studies and analyses.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Incorrect p-values could affect the outcome of tests that analyze the distribution of p-values, such as the p-curve (Simonsohn, Nelson, & Simmons, 2014) and p-uniform (Van Assen, Van Aert, & Wicherts, 2014).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Simonsohn, Nelson, & Simmons, 2014
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [33]
In the citing paper
“After excluding studies which assessed maltreatment as part of broader measures of victimization (k=3) (25, 27, 29) or ACEs (k=5) (30, 34, 39, 44, 49), or focused on institutional neglect (k=3) (15, 45, 46), the meta-analytic effect size was also similar to the original estimate (Cohen’s d=0.34, 95% CI=0.26-0.42).”
From Childhood Maltreatment and Mental Health Problems: A Systematic Review and Meta-Analysis of Quasi-Experimental Studies (2023) · cites this paper as [25]
In the citing paper
“Trim‐and‐fill procedures do suffer from some well‐known limitations that could result in the underestimation or overestimation of publication bias (Rothstein, 2008; Simonsohn, Nelson, & Simmons, 2014).”
From Hot spots policing of small geographic areas effects on crime (2019) · cites this paper as Simonsohn, Nelson, & Simmons, 2014
In the citing paper
“Applying selection modeling approaches like p-uniform or 3PSM to the simple effects may therefore substantially overestimate the true effect size.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Simonsohn et al., 2014
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
(1963) · Medical Entomology and Zoology · Cited by 15,085
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Self-citationTesting Statistical Hypotheses
(2021) · Wiley Series in Probability and Statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,106
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“1 Trying multiple analyses to obtain statistical significance has received many names, including: bias (Ioannidis, 2005), significance chasing (Ioannidis & Trikalinos, 2007), data snooping (White, 2003), fiddling (Bross, 1971), publication bias in situ (Phillips, 2004) and specification searching (Gerber & Malhotra, 2…”
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
A Fail-Safe N for Effect Size in Meta-Analysis
Orwin, R. G. (1983) · Journal of Educational Statistics · doi:10.2307/1164923 · Cited by 2,064 · cited 1×
“effect size statistically (Rosenthal, 1979) or practically (Orwin, 1983) non-significant.”
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1×
“In general, one can distinguish between these two alternative accounts of a null finding by considering a different null hypothesis: not that the effect is zero, but that it is very small instead (see e.g., Cohen, 1988, pp. 16-17; Greenwald, 1975; Hodges & Lehmann, 1954; Serlin & Lapsley, 1985).”
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 2×
“Scientists have been grappling with the problem of publication bias for at least 50 years (Sterling, 1959).”
Erroneous analyses of interactions in neuroscience: A problem of significance
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2886 · Cited by 864
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2012) · doi:10.2139/ssrn.2160588 · Cited by 680
Self-citationTime-Series Minimum-Wage Studies: A Meta-analysis
Card, D., & Krueger, A. B. (2007) · American Economic Review · Cited by 657
Becoming famous overnight: Limits on the ability to avoid unconscious influences of the past.
Jacoby, L. L., Kelley, C., Brown, J., & Jasechko, J. (1989) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.56.3.326 · Cited by 602
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
Cumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451
Misleading funnel plot for detection of bias in meta-analysis
Tang, J.-L., & Liu, J. L. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00204-8 · Cited by 420
Wohl, M. J. A., & Branscombe, N. R. (2005) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.88.2.288 · Cited by 418
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356
Feeling close: Emotional intensity reduces perceived psychological distance.
Van Boven, L., Kane, J., McGraw, A. P., & Dale, J. (2010) · Journal of Personality and Social Psychology · doi:10.1037/a0019262 · Cited by 352
Gerber, A., & Malhotra, N. (2008) · Quarterly Journal of Political Science · doi:10.1561/100.00008024 · Cited by 284
The Behavior of the P-Value When the Alternative Hypothesis is True
Hung, H. M. J., O'Neill, R. T., Bauer, P., & Kohne, K. (1997) · Biometrics · doi:10.2307/2533093 · Cited by 281
Clarkson, J. J., Hirt, E. R., Jia, L., & Alexander, M. B. (2010) · Journal of Personality and Social Psychology · doi:10.1037/a0017539 · Cited by 280
A peculiar prevalence of p values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266
Gerber, A. S., & Malhotra, N. (2008) · Sociological Methods & Research · doi:10.1177/0049124108318973 · Cited by 246
Rationality in psychological research: The good-enough principle.
Serlin, R. C., & Lapsley, D. K. (1985) · American Psychologist · doi:10.1037//0003-066x.40.1.73 · Cited by 217
Testing the Approximate Validity of Statistical Hypotheses
Hodges, J. L., & Lehmann, E. L. (2011) · Springer eBooks · doi:10.1007/978-1-4614-1412-4_15 · Cited by 117
Compounding Probabilities from Independent Significance Tests
Wallis, W. A. (1942) · Econometrica · doi:10.2307/1905466 · Cited by 113
Phillips, C. V. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-20 · Cited by 94
Biological Clock in the Unicorn
Cole, L. C. (1957) · Science · doi:10.1126/science.125.3253.874 · Cited by 93
False fame prevented: Avoiding fluency effects without judgmental correction.
Topolinski, S., & Strack, F. (2010) · Journal of Personality and Social Psychology · doi:10.1037/a0019260 · Cited by 68
Gadbury, G. L., & Allison, D. B. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046363 · Cited by 61
Simonsohn, U. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463399 · Cited by 37
Self-citationRidley, J., Kolm, N., FRECKELTON, R. P., & Gage, M. J. G. (2007) · Journal of Evolutionary Biology · doi:10.1111/j.1420-9101.2006.01291.x · Cited by 36
It really just does not follow, comments on Francis (2013)
Simonsohn, U. (2013) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2013.03.006 · Cited by 24
Self-citation
Cited by
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationScherer, R., Siddiq, F., & Tondeur, J. (2019) · Computers & Education · doi:10.1016/j.compedu.2018.09.009 · Cited by 1,923
“Third, we analyzed the p -curves derived from the aggregated correlations (Simonsohn, Nelson, & Simmons, 2014).”
MethodScientific method: Statistical errors
Nuzzo, R. (2014) · Nature · doi:10.1038/506150a · Cited by 1,536
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388
“…Abstracts are more likely to contain p -values that relate to primary hypotheses, which are expected to be more strongly p-hacked than p -values from less interesting, ancillary tests [41], lower power and reporting bias may impede detection of p-hacking using p -values obtained from Abstracts.”
BackgroundMethodResultInfluentialIncreasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,372
Metadata
- Authors
- Simonsohn, U., Nelson, L. D., & Simmons, J. P.
- Affiliations
- University of Pennsylvania (US)
- Venue
- Journal of Experimental Psychology: General
- Published
- 2014-04
- Volume
- 143
- Issue
- 2
- Pages
- 534-547
- ISSN
- 0096-3445 (ISSN-L)
- 1939-2222
- OpenAlex
- W3124690569
- Cited by
- 1,500
- References
- 49
- Citations by year
- 2026: 75 · 2025: 96 · 2024: 110 · 2023: 121 · 2022: 146 · 2021: 157 · 2020: 158 · 2019: 154 · 2018: 143 · 2017: 113 · 2016: 103 · 2015: 80 · 2014: 35
- FWCI
- 82.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Experimental Psychology General (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-curve, publication bias, selective reporting, p-hacking, file drawer problem, statistical significance, evidentiary value
- MeSH terms
- Data Interpretation, Statistical, Humans, Psychology, Experimental, Statistics as Topic, Reproducibility of Results, Models, Statistical, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.