P-curve: A key to the file-drawer.
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Because scientists tend to report only studies (publication bias) or analyses (p-hacking) that "work," readers must ask, "Are these effects true, or do they merely reflect selective reporting?" We introduce p-curve as a way to answer this question. P-curve is the distribution of statistically significant p values for a set of studies (ps < .05). Because only true effects are expected to generate right-skewed p-curves-containing more low (.01s) than high (.04s) significant p values--only right-skewed p--curves are diagnostic of evidential value. By telling us whether we can rule out selective reporting as the sole explanation for a set of findings, p-curve offers a solution to the age-old inferential problems caused by file-drawers of failed studies and analyses.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Incorrect p-values could affect the outcome of tests that analyze the distribution of p-values, such as the p-curve (Simonsohn, Nelson, & Simmons, 2014) and p-uniform (Van Assen, Van Aert, & Wicherts, 2014).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Simonsohn, Nelson, & Simmons, 2014
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [33]
In the citing paper
“After excluding studies which assessed maltreatment as part of broader measures of victimization (k=3) (25, 27, 29) or ACEs (k=5) (30, 34, 39, 44, 49), or focused on institutional neglect (k=3) (15, 45, 46), the meta-analytic effect size was also similar to the original estimate (Cohen’s d=0.34, 95% CI=0.26-0.42).”
From Childhood Maltreatment and Mental Health Problems: A Systematic Review and Meta-Analysis of Quasi-Experimental Studies (2023) · cites this paper as [25]
In the citing paper
“Trim‐and‐fill procedures do suffer from some well‐known limitations that could result in the underestimation or overestimation of publication bias (Rothstein, 2008; Simonsohn, Nelson, & Simmons, 2014).”
From Hot spots policing of small geographic areas effects on crime (2019) · cites this paper as Simonsohn, Nelson, & Simmons, 2014
In the citing paper
“Applying selection modeling approaches like p-uniform or 3PSM to the simple effects may therefore substantially overestimate the true effect size.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Simonsohn et al., 2014
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
(1963) · Medical Entomology and Zoology · Cited by 15,085
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Self-citationTesting Statistical Hypotheses
(2021) · Wiley Series in Probability and Statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,106
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“1 Trying multiple analyses to obtain statistical significance has received many names, including: bias (Ioannidis, 2005), significance chasing (Ioannidis & Trikalinos, 2007), data snooping (White, 2003), fiddling (Bross, 1971), publication bias in situ (Phillips, 2004) and specification searching (Gerber & Malhotra, 2…”
Cited by
Items 301–400 of 1,005
Sakaluk, J. K. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.013 · Cited by 79
Do people learn about politics on social media? A meta-analysis of 76 studies
Amsalem, E., & Zoizner, A. (2022) · Journal of Communication · doi:10.1093/joc/jqac034 · Cited by 106
The roles, challenges, and merits of the p value
Chén, O. Y., Bodelet, J., Saraiva, R. G., Phan, H., Di, J., Nagels, G., Schwantje, T., Cao, H., Gou, J., Reinen, J., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100878 · Cited by 98
Dataset decay and the problem of sequential analyses on open datasets
Thompson, W. H., Wright, J., Bissett, P. G., & Poldrack, R. A. (2020) · eLife · doi:10.7554/elife.53498 · Cited by 88
Rahman, T., & Lewis, S. E. (2019) · Journal of Research in Science Teaching · doi:10.1002/tea.21610 · Cited by 79
David, D., Costescu, C. A., Cardoș, R. A.-I., & Mogoaşe, C. (2020) · Child & Youth Care Forum · doi:10.1007/s10566-020-09566-1 · Cited by 78
Emotion-related impulsivity and risky decision-making: A systematic review and meta-regression
Elliott, M., Johnson, S. L., Pearlstein, J. G., Lopez, D. E. M., & Keren, H. (2022) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102232 · Cited by 91
“Third, in addition to the pre-registered checks, we conducted a P-Curve analysis (http://p-curve.com) to examine the distribution of significant p-values and test for evidence of a true aggregate effect (Simonsohn, Nelson, and Simmons, 2014; Simonsohn, Simmons, and Nelson, 2015).”
MethodThe Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates
Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015) · Review of General Psychology · doi:10.1037/gpr0000034 · Cited by 69
de Winter, J. C., & Dodou, D. (2015) · PeerJ · doi:10.7717/peerj.733 · Cited by 78
How (and Whether) to Teach Undergraduates About the Replication Crisis in Psychological Science
Chopik, W. J., Bremner, R. H., Defever, A. M., & Keller, V. N. (2018) · Teaching of Psychology · doi:10.1177/0098628318762900 · Cited by 78
The quest for an optimal alpha
Miller, J. O., & Ulrich, R. (2019) · PLoS ONE · doi:10.1371/journal.pone.0208631 · Cited by 89
Pre-Analysis Plans: An Early Stocktaking
Ofosu, G. K., & Posner, D. (2021) · Perspectives on Politics · doi:10.1017/s1537592721000931 · Cited by 100
Pabst, A., Proksch, S., Médé, B., Comstock, D. C., Ross, J. M., & Balasubramaniam, R. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104587 · Cited by 68
Replicating and fixing failed replications: The case of need for cognition and argument quality
Luttrell, A., Petty, R. E., & Xu, M. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.09.006 · Cited by 72
Pavlacic, J. M., Buchanan, E. M., Maxwell, N. P., Hopke, T. G., & Schulenberg, S. E. (2019) · Review of General Psychology · doi:10.1177/1089268019831645 · Cited by 87
van Kuijk, I., Verkoeijen, P., Dijkstra, K., & Zwaan, R. A. (2018) · Collabra: Psychology · doi:10.1525/collabra.117 · Cited by 67
Schoemann, M., O’Hora, D. P., Dale, R. A. C., & Scherbaum, S. (2020) · Psychonomic Bulletin & Review · doi:10.3758/s13423-020-01851-3 · Cited by 76
Electrifying the motor engram: effects of tDCS on motor learning and control
de Xivry, J. O., & Shadmehr, R. (2014) · Experimental Brain Research · doi:10.1007/s00221-014-4087-6 · Cited by 75
“(aka the file-drawer problem; Simonsohn et al. 2014) These problems impact all fields of neuroscience , but solutions have been recently proposed (Lakens and Evers 2014; Simonsohn et al. 2014).”
BackgroundSocial Dilemma Cooperation (Unlike Dictator Game Giving) Is Intuitive for Men As Well As WomenPreprint
Rand, D. G. (2017) · SSRN Electronic Journal · doi:10.1016/j.jesp.2017.06.013 · Cited by 71
Vadillo, M. A., Hardwicke, T. E., & Shanks, D. R. (2016) · Journal of Experimental Psychology General · doi:10.1037/xge0000157 · Cited by 71
“…effect size estimates in Table 2 (which are inflated by the inclusion of at least one clear outlier; see Figure 1). p-Curve Analysis Recently, Simonsohn, Nelson, and Simmons (2014) have developed p-curve, a new tool to assess the evidential value of a series of statistical contrasts that is…”
BackgroundMethodResultInfluentialReward-driven distraction: A meta-analysis.
Rusz, D., Pelley, M. E. L., Kompier, M. A. J., Mait, L., & Bijleveld, E. (2020) · Psychological Bulletin · doi:10.1037/bul0000296 · Cited by 69
Apšvalka, D., Ramsey, R., & Cross, E. S. (2018) · Neural Plasticity · doi:10.1155/2018/1237962 · Cited by 72
“Given concerns over publication bias in general [51] and in the domain of tDCS in particular [52], it is important to report null results in order to provide a less biased estimate of the likely effect sizes that tDCS may have on behaviour.”
Verschuère, B., Köbis, N., Bereby‐Meyer, Y., Rand, D. G., & Shalvi, S. (2018) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2018.04.005 · Cited by 89
Challenges in Building Usable Knowledge in Education
Hedges, L. V. (2017) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2017.1375583 · Cited by 71
Smith, M. M., Sherry, S., Vidovic, V., Hewitt, P. L., & Flett, G. L. (2020) · Journal of Research in Personality · doi:10.1016/j.jrp.2020.103954 · Cited by 81
Is Pornography Consumption a Risk Factor for Condomless Sex?
Tokunaga, R. S., Wright, P. J., & Vangeel, L. (2020) · Human Communication Research · doi:10.1093/hcr/hqaa005 · Cited by 70
Cai, Z., Gui, Y., Mao, P., Wang, Z., Hao, X., Fan, X., & Tai, R. H. (2023) · Educational Research Review · doi:10.1016/j.edurev.2023.100521 · Cited by 98
Lai, L., Liu, Y., McCracken, L. M., Li, Y., & Ren, Z. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104308 · Cited by 60
Elliott, G., Kudrin, N., & Wüthrich, K. (2022) · Econometrica · doi:10.3982/ecta18583 · Cited by 113
Religion and Prejudice Toward Immigrants and Refugees: A Meta-Analytic Review
Deslandes, C., & Anderson, J. (2019) · International Journal for the Psychology of Religion · doi:10.1080/10508619.2019.1570814 · Cited by 74
“Thus, we can infer that the effects observed in this literature are not falsely inflated due to p-hacking practices (for more, see Simonsohn et al., 2014).”
ResultTransparency in Practice in Qualitative Research
Kapiszewski, D., & Karcher, S. (2020) · PS Political Science & Politics · doi:10.1017/s1049096520000955 · Cited by 101
Toward a radically embodied neuroscience of attachment and relationships
Beckes, L., IJzerman, H. R., & Tops, M. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00266 · Cited by 79
The Emergence of Statistical Objectivity: Changing Ideas of Epistemic Vice and Virtue in Science
Freese, J., & Peterson, D. (2018) · Sociological Theory · doi:10.1177/0735275118794987 · Cited by 69
“Particularly influential in psychology has been the p-curve method (Simonsohn, Nelson, and Simmons 2014), which is based on one way that the results of a set of experiments detecting a true effect would differ from a set of experiments in which statistically significant results were the illusory…”
MethodCôté, S., Stellar, J. E., Willer, R., Forbes, R. C., Martín, S., & Bianchi, E. C. (2020) · Personality and Social Psychology Bulletin · doi:10.1177/0146167220916633 · Cited by 65
Héroux, M. E., Taylor, J. L., & Gandevia, S. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0144151 · Cited by 66
“Approximately half the respondents to our survey indicated they could not reproduce the original TMS effects: is it any wonder we also found evidence of a large filedrawer problem? Many TMS studies are not written up or accepted for publication and end up in a real or metaphorical file-drawer, never to be seen again […”
BackgroundMonden, R., de Vos, S., Morey, R. D., Wagenmakers, E., de Jonge, P., & Roest, A. M. (2016) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1507 · Cited by 62
“However, the NHST framework, which is based on frequentist statistics, is associated with a number of problems (Goodman, 1999a; Ioannidis, 2005a;Wagenmakers et al., 2011; Grendár, 2012; Simonsohn et al., 2014).”
MethodRouy, M., Saliou, P., Nalborczyk, L., Pereira, M., Roux, P., & Faivre, N. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.03.017 · Cited by 71
Gehlbach, H., & Robinson, C. D. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1898961 · Cited by 67
“As Simonsohn et al. (2014) explained, a left-skewed p-curve (i.e., finding p-values of .03 and .04 s disproportionately more than p-values of .01 and .001) suggest the presence of selective reporting and, therefore, illusory results.”
BackgroundEvidence on Research Transparency in Economics
Miguel, E. A. (2021) · The Journal of Economic Perspectives · doi:10.1257/jep.35.3.193 · Cited by 85
Microbe-Mediated Thermotolerance in Plants and Pertinent Mechanisms- A Meta-Analysis and Review
Dastogeer, K. M. G., Zahan, M. I., Rhaman, M. S., Sarker, M. S. A., & Chakraborty, A. (2022) · Frontiers in Microbiology · doi:10.3389/fmicb.2022.833566 · Cited by 64
Success of post-fire plant recovery strategies varies with shifting fire seasonality
Tangney, R., Paroissien, R., Breton, T. D. L., Thomsen, A. M., Doyle, C., Ondik, M. M., Miller, R. G., Miller, B. P., & Ooi, M. K. J. (2022) · Communications Earth & Environment · doi:10.1038/s43247-022-00453-2 · Cited by 78
Abramovitch (2019) · doi:10.1016/j.cpr.2018.09.001 · Cited by 59
Maternal perinatal depression and child executive function: A systematic review and meta-analysis
Power, J., van IJzendoorn, M. H., Lewis, A. J., Chen, W., & Galbally, M. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.05.003 · Cited by 69
Human estrus: implications for relationship science
Gangestad, S. W., & Haselton, M. G. (2015) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2014.12.007 · Cited by 64
“05 [43].”
A Bayesian approach to mitigation of publication bias
Guan, M., & Vandekerckhove, J. (2015) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0868-6 · Cited by 68
Trendel, O., & Werle, C. O. C. (2015) · Appetite · doi:10.1016/j.appet.2015.10.005 · Cited by 70
Sherman, L. E., Steinberg, L. D., & Chein, J. M. (2017) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2017.05.007 · Cited by 60
Power to Detect What? Considerations for Planning and Evaluating Sample SizePreprint
Giner‐Sorolla, R., Montoya, A. K., Aberson, C. L., Carpenter, T., Lewis, N. A., Bostyn, D. H., Conrique, B. G., Ng, B., Reifman, A. S., Schoemann, A. M., et al. (2023) · doi:10.31234/osf.io/rv3kw · Cited by 62
Transparency, Replication, and Cumulative Learning: What Experiments Alone Cannot Achieve
Dunning, T. (2016) · Annual Review of Political Science · doi:10.1146/annurev-polisci-042114-015939 · Cited by 62
“For discussion and a review of applications, see Tanner (2014); for reasons why using STCM is a bad idea, see Collier et al. (2015).”
BackgroundThe Effects of Disgust on Moral Judgments: Testing Moderators
Johnson, D. J., Wortman, J., Cheung, F., Hein, M., Lucas, R. E., Donnellan, M. B., Ebersole, C. R., & Narr, R. K. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616654211 · Cited by 62
Larson, L. M., Kubes, J., Ramirez-Luzuriaga, M. J., Khishen, S., Shankar, A. H., & Prado, E. L. (2019) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.14105 · Cited by 62
Nitschke, F., McKimmie, B. M., & Vanman, E. J. (2019) · Psychological Bulletin · doi:10.1037/bul0000206 · Cited by 84
Hosseini‐Kamkar, N., Lowe, C. J., & Morton, J. B. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.04.006 · Cited by 70
Meta-Analytic Methods to Detect Publication Bias in Behavior Science Research
Dowdy, A., Hantula, D. A., Travers, J. C., & Tincani, M. (2021) · Perspectives on Behavior Science · doi:10.1007/s40614-021-00303-0 · Cited by 67
Trust, Transparency, and Replication in Political Science
Laitin, D. D., & Reich, R. (2017) · PS Political Science & Politics · doi:10.1017/s1049096516002365 · Cited by 54
“On the ubiquity of p-hacking in psychology, see Simonsohn, Nelson, and Simmons (2014).”
BackgroundGateway illusion or cultural cognition confusion?
van der Linden, S., Leiserowitz, A., & Maibach, E. W. (2017) · Journal of Science Communication · doi:10.22323/2.16050204 · Cited by 61
Bessa, C., Hastie, P. A., Ramos, A., & Mesquita, I. (2021) · Journal of Sports Science and Medicine · doi:10.52082/jssm.2021.110 · Cited by 78
“Following the recommendations of Simonsohn et al. (2013) eligibility criteria were defined before the electronic search.”
MethodEffectiveness of cognitive stimulation for dementia: A systematic review and meta-analysis.
Cafferata, R. M. T., Hicks, B., & von Bastian, C. C. (2021) · Psychological Bulletin · doi:10.1037/bul0000325 · Cited by 72
“We further pre-registered to conduct a p-curve analysis (Simonsohn et al., 2014); however, 29 out of the 44 studies had to be excluded from p-curve analysis for various reasons (e.”
MethodComment: What p -hacking really looks like: A comment on Masicampo and LaLande (2012)
Lakens, D. (2014) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2014.982664 · Cited by 56
From Big Data to Knowledge in the Social Sciences
Hesse, B. W., Moser, R. P., & Riley, W. T. (2015) · The Annals of the American Academy of Political and Social Science · doi:10.1177/0002716215570007 · Cited by 68
“…pressure on investigators to strain the assumptions of a priori hypothesis testing to explore ways of reaching a .05 level of significance for at least some findings in their dataset (Ioannidis et al. 2014), a custom referred to by some as “p hacking” (Simonsohn, Nelson, and Simmons 2014).”
BackgroundUeno, T., Fastrich, G. M., & Murayama, K. (2016) · Journal of Experimental Psychology: General · doi:10.1037/xge0000159 · Cited by 56
Lu, S. C., Kong, D. T., Ferrin, D. L., & Dirks, K. T. (2017) · Journal of Trust Research · doi:10.1080/21515581.2017.1285241 · Cited by 73
Davis-Kean, P. E., & Ellis, A. (2019) · Infant Behavior and Development · doi:10.1016/j.infbeh.2019.101339 · Cited by 61
Wang, Y. A., Sparks, J., Gonzales, J. E., Hess, Y. D., & Ledgerwood, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.04.011 · Cited by 62
“One easy way to reconcile these contrasting narratives is simply to advocate including a covariate in the analysis of experimental data if and only if the covariate is specified a priori—for instance, in a preanalysis plan (see Ledgerwood et al., 2017; Simonsohn et al., 2014).”
BackgroundClayson, P. E., Carbine, K. A., & Larson, M. J. (2020) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2020.01.005 · Cited by 61
Biddlestone, M., Green, R., Douglas, K. M., Azevedo, F., Sutton, R. M., & Cichocka, A. K. (2022) · doi:10.31234/osf.io/rxjqc · Cited by 55
West, G., Melby‐Lervåg, M., & Hulme, C. (2021) · Developmental Psychology · doi:10.1037/dev0001172 · Cited by 69
Reproducibility in the Social Sciences
Moody, J. W., Keister, L. A., & Ramos, M. C. (2022) · Annual Review of Sociology · doi:10.1146/annurev-soc-090221-035954 · Cited by 67
Adversarial Collaboration: The Next Science Reform
Clark, C. J., & Tetlock, P. E. (2023) · Springer eBooks · doi:10.1007/978-3-031-29148-7_32 · Cited by 72
Validation of Italian rebus puzzles and compound remote associate problems
Salvi, C., Costantini, G., Bricolo, E., Perugini, M., & Beeman, M. J. (2015) · Behavior Research Methods · doi:10.3758/s13428-015-0597-9 · Cited by 63
Cristea, I. A., & Ioannidis, J. P. A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0197440 · Cited by 61
P-hacking in clinical trials and how incentives shape the distribution of results across phases
Adda, J., Decker, C., & Ottaviani, M. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1919906117 · Cited by 67
“Our analysis builds on and develops the methods proposed in the literature that investigates “p-hacking,” publication bias, and the “file-drawer problem” (22, 23) for academic journal publications in a number of fields, ranging from life sciences (24) to psychology (25, 26), political science (27, 28), and economics (…”
Beyond Statistical Ritual: Theory in Psychological Science
Proulx, T., & Morey, R. D. (2021) · Perspectives on Psychological Science · doi:10.1177/17456916211017098 · Cited by 70
la Fuente, I. M., Pastor, A., Herrador, Z., Benito, A., & Berzosa, P. J. (2021) · Malaria Journal · doi:10.1186/s12936-021-03812-0 · Cited by 62
“Additionally, the asymmetry of the funnel plot was tested using the Egger test, and possible p-hacking was tested by performing a P-curve analysis [33].”
MethodSiegel, M., Eder, J. S. N., Wicherts, J. M., & Pietschnig, J. (2022) · Journal of Applied Psychology · doi:10.1037/apl0000991 · Cited by 71
Sarıbay, S. A., & Yılmaz, O. (2017) · Personality and Individual Differences · doi:10.1016/j.paid.2017.03.056 · Cited by 59
Does Neonatal Imitation Exist? Insights From a Meta-Analysis of 336 Effect Sizes
Davis, J., Redshaw, J., Suddendorf, T., Nielsen, M., Kennedy‐Costantini, S., Oostenbroek, J., & Slaughter, V. P. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620959834 · Cited by 66
Tymofiyeva, O., & Gaschler, R. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.497245 · Cited by 75
Four-component instructional design (4C/ID) model: a meta-analysis on use and effect
Costa, J. M., Miranda, G. L., & Melo, M. (2021) · Learning Environments Research · doi:10.1007/s10984-021-09373-y · Cited by 68
Open science, the replication crisis, and environmental public health
Hicks, D. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1962713 · Cited by 63
“Another example is the p-curve, a graphical method developed to detect p-hacking by examining a distribution of p-values (Simonsohn, Nelson, and Simmons 2014).”
MethodKrawczyk, M. W. (2015) · PLoS ONE · doi:10.1371/journal.pone.0127872 · Cited by 58
“Indeed, it has been shown [27,28] that generally speaking a downward sloping distribution is to be expected when at least some alternative hypotheses are true.”
BackgroundTielbeek, J. J., Linnér, R. K., Beers, K., Posthuma, D., Popma, A., & Polderman, T. J. C. (2016) · American Journal of Medical Genetics Part B Neuropsychiatric Genetics · doi:10.1002/ajmg.b.32442 · Cited by 61
Bem, D., Tressoldi, P. E., Rabeyron, T., & Duggan, M. (2015) · F1000Research · doi:10.12688/f1000research.7177.1 · Cited by 72
Opportunistic biases: Their origins, effects, and an integrated solution.
DeCoster, J., Sparks, E., Sparks, J. C., Sparks, G. G., & Sparks, C. W. (2015) · American Psychologist · doi:10.1037/a0039191 · Cited by 58
“The Center for Open Science has recently developed the Open Science Framework (OSF) to facilitate this process via open collaboration (http://openscienceframework .org, Spies & Nosek, 2014).”
BackgroundEmotion regulation and mental health across cultures: a systematic review and meta-analysis
Chen, S., Cai, Q., Omari, D. I., Sanghvi, D. E., Lyu, S., & Bonanno, G. A. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02168-8 · Cited by 71
“Therefore, we also performed P -curve 350 and PET–PEESE 351 procedures to further examine and correct for potential publication bias.”
Weintraub, M. J., Hall, D. L., Carbonella, J. Y., de Mamani, A. W., & Hooley, J. M. (2016) · Family Process · doi:10.1111/famp.12208 · Cited by 61
“As Simonsohn et al. (2014) note, researchers and journals tend to publish results that are statistically significant, so are these effects true and valid or do they reflect publication bias or statistical p-hacking?”
InfluentialHow to avoid mismodelling in GLM-based fMRI data analysis: cross-validated Bayesian model selection
Soch, J., Haynes, J., & Allefeld, C. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.07.047 · Cited by 50
“This strategy, a form of so-called p-hacking (Simonsohn et al., 2014), may render the false positive rate much higher than indicated by the nominal significance level (Simmons et al.”
Holgado, D., Zandonai, T., Ciria, L. F., Zabala, M., Hopker, J., & Sanabria, D. (2019) · PLoS ONE · doi:10.1371/journal.pone.0210873 · Cited by 62
“, the tendency to only publish positive outcomes) might be biasing the literature to positive findings [37].”
BackgroundQuestionable Research Practices and Open Science in Quantitative Criminology
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2021) · Journal of Quantitative Criminology · doi:10.1007/s10940-021-09525-6 · Cited by 82
“2014, 2018), such as trim-and-fill, which are known to be ineffective (Simonsohn et al. 2014; van Assen et al. 2015).”
BackgroundEstimating the evidential value of significant results in psychological science
Aczél, B., Pálfi, B., & Szászi, B. (2017) · PLoS ONE · doi:10.1371/journal.pone.0182651 · Cited by 56
Eitan, O., Viganola, D., Inbar, Y., Dreber, A., Johannesson, M., Pfeiffer, T., Thau, S., & Uhlmann, E. L. (2018) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2018.06.004 · Cited by 60
Three Decades of Research on Induced Hypocrisy: A Meta-Analysis
Priolo, D., Pelt, A., Saint-Bauzel, R., Rubens, L., Voisin, D., & Fointiat, V. (2019) · Personality and Social Psychology Bulletin · doi:10.1177/0146167219841621 · Cited by 68
“Then, we used APP 4.052 (http://www.p-curve.com/app4/) to run p-curve analyses (Simonsohn et al., 2014; Simonsohn, Simmons, & Nelson, 2015).”
InfluentialTransparency, Reproducibility, and the Credibility of Economics ResearchPreprint
Christensen, G., & Miguel, E. A. (2017) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/9a3rw · Cited by 59
Wyatt, T. D. (2020) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2019.0262 · Cited by 56
The effectiveness of pre-performance routines in sports: a meta-analysis
Rupprecht, A. G. O., Tran, U. S., & Gröpel, P. (2021) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2021.1944271 · Cited by 72
“In addition, the contour-enhanced funnel plots displayed several studies that were just barely signi fi cant, which might be an indication of p -hacking (Simonsohn et al., 2014).”
BackgroundFeter, N., Ligeza, T. S., Bashir, N., Shanmugam, R., Herrera, B. M., Aldabbagh, T., Usman, A.-F., Yonezawa, A., McCarthy, S., Herrera, D., et al. (2024) · British Journal of Sports Medicine · doi:10.1136/bjsports-2024-108444 · Cited by 56
Americans Still Overestimate Social Class Mobility: A Pre-Registered Self-Replication
Kraus, M. W. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01709 · Cited by 58
“…us to practically assess whether a number of statistical techniques for detecting publication bias (i.e., r-index, test of insufficient variance, p-curve; Schimmack, 2014a,b; Simonsohn et al., 2014) can be used as predictors of success or failure of an individual high-powered replication attempt.”
BackgroundMethodInfluentialThe Consequences of Strategic News Coverage for Democracy: A Meta-Analysis
Zoizner, A. (2018) · Communication Research · doi:10.1177/0093650218808691 · Cited by 59
Manipulating Belief in Free Will and Its Downstream Consequences: A Meta-Analysis
Genschow, O., Cracco, E., Schneider, J., Protzko, J., Wisniewski, D., Braß, M., & Schooler, J. W. (2022) · Personality and Social Psychology Review · doi:10.1177/10888683221087527 · Cited by 72
“The p-values used in the p-curve analysis were coded according to the guidelines laid out in Simonsohn et al. (2014).”
BackgroundMethodInfluentialSeven Pervasive Statistical Flaws in Cognitive Training Interventions
Moreau, D., Kirk, I. J., & Waldie, K. E. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00153 · Cited by 60
Metadata
- Authors
- Simonsohn, U., Nelson, L. D., & Simmons, J. P.
- Affiliations
- University of Pennsylvania (US)
- Venue
- Journal of Experimental Psychology: General
- Published
- 2014-04
- Volume
- 143
- Issue
- 2
- Pages
- 534-547
- ISSN
- 0096-3445 (ISSN-L)
- 1939-2222
- OpenAlex
- W3124690569
- Cited by
- 1,500
- References
- 49
- Citations by year
- 2026: 75 · 2025: 96 · 2024: 110 · 2023: 121 · 2022: 146 · 2021: 157 · 2020: 158 · 2019: 154 · 2018: 143 · 2017: 113 · 2016: 103 · 2015: 80 · 2014: 35
- FWCI
- 82.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Experimental Psychology General (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-curve, publication bias, selective reporting, p-hacking, file drawer problem, statistical significance, evidentiary value
- MeSH terms
- Data Interpretation, Statistical, Humans, Psychology, Experimental, Statistics as Topic, Reproducibility of Results, Models, Statistical, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.