P-curve: A key to the file-drawer.
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Because scientists tend to report only studies (publication bias) or analyses (p-hacking) that "work," readers must ask, "Are these effects true, or do they merely reflect selective reporting?" We introduce p-curve as a way to answer this question. P-curve is the distribution of statistically significant p values for a set of studies (ps < .05). Because only true effects are expected to generate right-skewed p-curves-containing more low (.01s) than high (.04s) significant p values--only right-skewed p--curves are diagnostic of evidential value. By telling us whether we can rule out selective reporting as the sole explanation for a set of findings, p-curve offers a solution to the age-old inferential problems caused by file-drawers of failed studies and analyses.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Incorrect p-values could affect the outcome of tests that analyze the distribution of p-values, such as the p-curve (Simonsohn, Nelson, & Simmons, 2014) and p-uniform (Van Assen, Van Aert, & Wicherts, 2014).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Simonsohn, Nelson, & Simmons, 2014
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [33]
In the citing paper
“After excluding studies which assessed maltreatment as part of broader measures of victimization (k=3) (25, 27, 29) or ACEs (k=5) (30, 34, 39, 44, 49), or focused on institutional neglect (k=3) (15, 45, 46), the meta-analytic effect size was also similar to the original estimate (Cohen’s d=0.34, 95% CI=0.26-0.42).”
From Childhood Maltreatment and Mental Health Problems: A Systematic Review and Meta-Analysis of Quasi-Experimental Studies (2023) · cites this paper as [25]
In the citing paper
“Trim‐and‐fill procedures do suffer from some well‐known limitations that could result in the underestimation or overestimation of publication bias (Rothstein, 2008; Simonsohn, Nelson, & Simmons, 2014).”
From Hot spots policing of small geographic areas effects on crime (2019) · cites this paper as Simonsohn, Nelson, & Simmons, 2014
In the citing paper
“Applying selection modeling approaches like p-uniform or 3PSM to the simple effects may therefore substantially overestimate the true effect size.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Simonsohn et al., 2014
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
(1963) · Medical Entomology and Zoology · Cited by 15,085
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Self-citationTesting Statistical Hypotheses
(2021) · Wiley Series in Probability and Statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,106
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“1 Trying multiple analyses to obtain statistical significance has received many names, including: bias (Ioannidis, 2005), significance chasing (Ioannidis & Trikalinos, 2007), data snooping (White, 2003), fiddling (Bross, 1971), publication bias in situ (Phillips, 2004) and specification searching (Gerber & Malhotra, 2…”
Cited by
Items 501–600 of 1,005
Experimental design studies in hospitality and tourism research: constructive recommendations
Kim, J., Kim, S., Jhang, J., Kwon, Y., & Baah, N. G. (2023) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-06-2022-0720 · Cited by 39
Raugh, I. M., Berglund, A. M., & Strauss, G. P. (2024) · Affective Science · doi:10.1007/s42761-024-00281-x · Cited by 49
“…2000), limit meta-analysis (Rücker, Carpenter et al., 2011; Rücker, Schwarzer et al., 2011), Rosenthal’s fail-safe N to reduce a significant effect to be nonsignificant, Orwin’s fail-safe N to reduce the observed effect size by 50% (Rosenberg, 2005), and p curve analysis (Simonsohn et al., 2014).”
A Review of Latino/Latinx Participants in Mindfulness-Based Intervention Research
Cotter, E. W., & Jones, N. (2019) · Mindfulness · doi:10.1007/s12671-019-01266-9 · Cited by 41
Sivaramakrishnan, H., Gucciardi, D. F., McDonald, M., Quested, E., Thøgersen‐Ntoumani, C., Cheval, B., & Ntoumanis, N. (2021) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2021.2004611 · Cited by 43
“A p-curve analysis was conducted to assess the evidentiary value of the summarised literature (Simonsohn et al., 2014).”
MethodWill Happiness-Trainings Make Us Happier? A Research Synthesis Using an Online Findings-Archive
Bergsma, A., Buijt, I., & Veenhoven, R. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01953 · Cited by 41
“A more formal check for this “filedrawer” problem and “p-hacking” as the “sole explanation of the evidential value of a set of significant findings” is a p-curve analysis (Simonsohn et al., 2014).”
BackgroundEconomic distress and voting: evidence from the subprime mortgage crisis
Hall, A. B., Yoder, J., & Karandikar, N. (2021) · Political Science Research and Methods · doi:10.1017/psrm.2021.3 · Cited by 36
Emotion Recognition and Traumatic Brain Injury: a Systematic Review and Meta-Analysis
Murphy, J. M., Bennett, J. M., Garcia, X. D. L. P., & Willis, M. L. (2021) · Neuropsychology Review · doi:10.1007/s11065-021-09510-7 · Cited by 38
Big Little Lies: A Compendium and Simulation of p-Hacking StrategiesPreprint
Stefan, A. M., & Schönbrodt, F. D. (2022) · doi:10.31234/osf.io/xy2dk · Cited by 35
Donolato, E., Toffalini, E., Rogde, K., Nordahl‐Hansen, A., Lervåg, A., Norbury, C., & Melby‐Lervåg, M. (2023) · Campbell Systematic Reviews · doi:10.1002/cl2.1368 · Cited by 72
“We employed the p ‐ curve method on the whole set of effect sizes (Simonsohn et al., 2014), as stated in the protocol.”
The Heuristic Value of p in Inductive Statistical Inference
Krueger, J. I., & Heck, P. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00908 · Cited by 34
Rødvik, A. K., Torkildsen, J. V. K., Wie, O. B., Storaker, M. A., & Silvola, J. T. (2018) · Journal of Speech Language and Hearing Research · doi:10.1044/2018_jslhr-h-16-0463 · Cited by 36
The natural selection of good science
Stewart, A. J., & Plotkin, J. B. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01111-x · Cited by 35
“There is ample evidence to suggest that in some fields a large portion of reported results may be false (Kerr, 1998; Ioannidis, 2005; Simmons et al., 2011; John et al., 2012; Simonsohn et al., 2014; Rahal et al., 2015; Begley and Ioannidis, 2015).”
BackgroundLooming Cognitive Style and Its Associations with Anxiety and Depression: A Meta-analysis
Yeo, G. C., Hong, R. Y. S., & Riskind, J. H. (2020) · Cognitive Therapy and Research · doi:10.1007/s10608-020-10089-1 · Cited by 36
Implications of the open science era for educational psychology research syntheses
Patall, E. A. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1897009 · Cited by 38
“These practices, also described as selective reporting, p-hacking, data dredging, data fishing, or significance chasing (e.g., Nuzzo, 2014; Simmons et al., 2011; Simonsohn et al., 2014), include practices such as examining many comparisons without adjustment, excluding data points or including covariates based on post…”
BackgroundHagger, M. S. (2022) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2021.1966824 · Cited by 44
“The p-curve (Simonsohn et al., 2014) is the curve of the distribution of values”
BackgroundNew Guidelines for Null Hypothesis Significance Testing in Hypothetico-Deductive IS Research
Mertens, W., & Recker, J. (2023) · Technology, work and globalization · doi:10.1007/978-3-031-38719-7_13 · Cited by 36
Kappenman, E. S., & Keil, A. (2016) · Psychophysiology · doi:10.1111/psyp.12787 · Cited by 36
Transgender and cisgender children's essentialist beliefs about sex and gender identity
Gülgöz, S., Alonso, D. J., Olson, K. R., & Gelman, S. A. (2021) · Developmental Science · doi:10.1111/desc.13115 · Cited by 32
Exogenous nitric oxide promotes salinity tolerance in plants: A meta-analysis
Tahjib‐Ul‐Arif, M., Wei, X., Jahan, I., Hasanuzzaman, M., Sabuj, Z. H., Zulfiqar, F., Chen, J., Iqbal, R., Dastogeer, K. M. G., Sohag, A. A. M., et al. (2022) · Frontiers in Plant Science · doi:10.3389/fpls.2022.957735 · Cited by 36
“We visually examined asymmetry in funnel plots, employed “trim-and-fill” analysis, and ran Begg and Mazumdar rank correlation tests based on Kendall’s tau, the Egger regression test, and p-curve analysis (Begg and Mazumdar, 1994; Egger et al., 1997; Simonsohn et al., 2014).”
MethodMcKay, B., Yantha, Z. D., Hussien, J., Carter, M. J., & Ste‐Marie, D. M. (2022) · Meta-Psychology · doi:10.15626/mp.2021.2803 · Cited by 52
Adam, A. M., Giller, K. E., Rusinamhodzi, L., Rasche, F., Koomson, E., Marohn, C., & Cadisch, G. (2024) · Field Crops Research · doi:10.1016/j.fcr.2024.109663 · Cited by 45
Bumblebee size polymorphism and worker response to queen pheromone
Holman, L. (2014) · PeerJ · doi:10.7717/peerj.604 · Cited by 32
“…reasons, and the false positive rate is probably substantially higher than the 5% implied by the familiar p<0.05 threshold due to the combined influence of many forms of experimental and statistical bias (Ioannidis 2005; Simonsohn et al. 2014), making replication essential to scientific progress.”
BackgroundInfluentialWebster, G. D., & Duffy, R. D. (2016) · Intelligence · doi:10.1016/j.intell.2016.01.001 · Cited by 37
Weingarten, E., Chen, Q., McAdams, M., Yi, J., Hepler, J., & Albarracín, D. (2016) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2016.04.015 · Cited by 38
“p-curve on all studies conducted using p-values based on the researchers’ focal hypotheses [27], and (b) p-curves based on studies with the largest error degrees of freedom.”
MethodThe development of infants’ responses to mispronunciations: A meta-analysis.
Holzen, K. V., & Bergmann, C. (2020) · Developmental Psychology · doi:10.1037/dev0001141 · Cited by 35
When Does Monetary Policy Sway House Prices? A Meta-Analysis
Ehrenbergerová, D., Bajzík, J., & Havránek, T. (2022) · IMF Economic Review · doi:10.1057/s41308-022-00185-5 · Cited by 49
Intuitions About the Reference of Proper Names: a Meta-Analysis
van Dongen, N. N. N., Colombo, M., Romero, F., & Sprenger, J. (2020) · Review of Philosophy and Psychology · doi:10.1007/s13164-020-00503-8 · Cited by 39
Ulrich, R., Erdfelder, E., Deutsch, R., Strauss, B. M., Brüggemann, A., Hannover, B., Tuschen‐Caffier, B., Kirschbaum, C., Blickle, G., Møller, J., et al. (2016) · Psychologische Rundschau · doi:10.1026/0033-3042/a000296 · Cited by 37
Writing Empirical Articles: Transparency, Reproducibility, Clarity, and Memorability
Gernsbacher, M. A. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918754485 · Cited by 47
Valkenburg, C., Slot, D. E., & van der Weijden, G. A. (2019) · International Journal of Dental Hygiene · doi:10.1111/idh.12423 · Cited by 38
The action-sentence compatibility effect (ACE): Meta-analysis of a benchmark finding for embodiment
Winter, A., Dudschig, C., Miller, J. O., Ulrich, R., & Kaup, B. (2022) · Acta Psychologica · doi:10.1016/j.actpsy.2022.103712 · Cited by 35
The Contact Hypothesis RevisitedPreprint
Paluck, E. L., & Green, S. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2973474 · Cited by 31
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22989 · Cited by 48
Ibanez, A., Maito, M., Botero-Rodríguez, F., Fittipaldi, S., Coronel, C., Migeot, J., Lacroix, A., Lawlor, B., Duran-Aniotz, C., Baez, S., et al. (2024) · Nature Aging · doi:10.1038/s43587-024-00648-6 · Cited by 30
Kim, J. H., Kim, J. Y., Lee, S., Lee, S., Stubbs, B., Koyanagi, A., Dragioti, E., Jacob, L., Carvalho, A. F., Raduà, J., et al. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104761 · Cited by 39
Salivary cortisol reactivity to psychological stressors in infancy: A meta-analysis
Puhakka, I., & Peltola, M. J. (2020) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2020.104603 · Cited by 37
P‐Hacking , P‐Curves , and the PSM –Performance Relationship: Is There Evidential Value?
Vogel, D., & Homberg, F. (2020) · Public Administration Review · doi:10.1111/puar.13273 · Cited by 37
Fatigue severity and avoidance among individuals with chronic disease: A meta-analysis
Adamowicz, J. L., Velez-Bermudez, M. E., & Thomas, E. B. K. (2022) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2022.110951 · Cited by 34
Siegel, M., Legler, M., Neziraj, F., Goldberg, A. E., & Zemp, M. (2022) · Children · doi:10.3390/children9091364 · Cited by 37
Yeager, D. S., Krosnick, J. A., Visser, P. S., Holbrook, A. L., & Tahk, A. M. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000171 · Cited by 33
The Weapons Identification Task: Recommendations for adequately powered research
Rivers, A. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0177857 · Cited by 34
Turner, M., & Hodis, F. A. (2023) · Educational Psychology Review · doi:10.1007/s10648-023-09838-x · Cited by 44
Open science: Considerations and issues for TESOL research
Al‐Hoorie, A. H., Cinaglia, C., Hiver, P. V., Huensch, A., Isbell, D. R., Leung, C., & Sudina, E. (2024) · TESOL Quarterly · doi:10.1002/tesq.3304 · Cited by 33
Barriers to advancing the science and practice of marketing
Baron, A. S., Zaltman, G., & Olson, J. C. (2017) · Journal of Marketing Management · doi:10.1080/0267257x.2017.1323839 · Cited by 32
Stone, D. N. (2017) · Accounting Horizons · doi:10.2308/acch-51949 · Cited by 29
Centralized scientific communities are less likely to generate replicable results
Danchev, V., Rzhetsky, A., & Evans, J. A. (2019) · eLife · doi:10.7554/elife.43094 · Cited by 35
The impact of speculation on commodity prices: A Meta-Granger analysis
Wimmer, T., Geyer‐Klingeberg, J., Hütter, M., Schmid, F., & Rathgeber, A. (2020) · Journal of commodity markets · doi:10.1016/j.jcomm.2020.100148 · Cited by 36
McKay, B., & Ste‐Marie, D. M. (2020) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.2020.1795056 · Cited by 32
“A meta-analysis of the self-controlled learning literature with bias-correction and estimation techniques, such as the Vevea-Hedges weight function model (Hedges & Vevea, 1996) and p-curve (Simonsohn et al., 2014), would be a fruitful next step.”
MethodA systematic review comparing two popular methods to assess a Type D personality effect
Lodder, P., Kupper, N. H. M., Antens, M., & Wicherts, J. M. (2021) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2021.04.002 · Cited by 33
“Under the truth of the null hypothesis the pvalue distribution is expected to be uniform, with an equal chance on observing any p-value [77,78].”
BackgroundDavydenko, M., Kolbuszewska, M. T., & Peetz, J. (2021) · PLoS ONE · doi:10.1371/journal.pone.0253938 · Cited by 52
Focused Ultrasound Stimulation as a Neuromodulatory Tool for Parkinson’s Disease: A Scoping Review
Lee, K. S., Clennell, B., Steward, T. G. J., Gialeli, A., Llana, O. C., & Whitcomb, D. J. (2022) · Brain Sciences · doi:10.3390/brainsci12020289 · Cited by 30
Pietschnig, J., Gerdesmann, D., Zeiler, M. D., & Voracek, M. (2022) · Royal Society Open Science · doi:10.1098/rsos.211621 · Cited by 62
Cárdenas-Egúsquiza, A. L., & Berntsen, D. (2022) · Consciousness and Cognition · doi:10.1016/j.concog.2022.103333 · Cited by 34
“S1 and S2; Simonsohn et al., 2014a; 2014b), which showed similar results as the bar chart.”
Zur Methodologie von Replikationsstudien
Erdfelder, E., & Ulrich, R. (2018) · Psychologische Rundschau · doi:10.1026/0033-3042/a000387 · Cited by 33
Teaching Good Research Practices: Protocol of a Research Master Course
Sarafoglou, A., Hoogeveen, S., Matzke, D., & Wagenmakers, E. (2019) · Psychology Learning & Teaching · doi:10.1177/1475725719858807 · Cited by 38
Facilitating open-science with realistic fMRI simulation: validation and application
Ellis, C. T., Baldassano, C. A., Schapiro, A. C., Cai, M. B., & Cohen, J. D. (2020) · PeerJ · doi:10.7717/peerj.8564 · Cited by 37
“Synergistic with their use in data analysis, simulations can also facilitate open science and reproducibility ( Simonsohn, Nelson & Simmons, 2014 ).”
MethodPre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 34
Integrative methods for post-selection inference under convex constraints
Panigrahi, S., Taylor, J., & Weinstein, A. (2021) · The Annals of Statistics · doi:10.1214/21-aos2057 · Cited by 57
Sheng, H., Ding, Z., Lin, X., & Zhu, Y. (2022) · Global Strategy Journal · doi:10.1002/gsj.1452 · Cited by 28
From Mentally Doing to Actually Doing: A Meta-Analysis of Induced Positive Consumption Simulations
Ceylan, G., Diehl, K., & Wood, W. (2023) · Journal of Marketing · doi:10.1177/00222429231181071 · Cited by 35
“Third, we created a p -curve from all published studies in our fi nal data set to test whether the underlying studies had evidential value, meaning that effects were not due to selective reporting (Simonsohn, Nelson, and Simmons 2014).”
MethodLet’s Do It Again: A Call for Replications in Psi Chi Journal of Psychological Research
Edlund, J. E. (2016) · Psi Chi Journal of Psychological Research · doi:10.24839/2164-8204.jn21.1.59 · Cited by 28
Who Should Do Replication Labor?
Romero, F. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918803619 · Cited by 29
Montes, K. S., & Pearson, M. R. (2021) · Psychology of Addictive Behaviors · doi:10.1037/adb0000721 · Cited by 36
Sotola, L. K., & Credé, M. (2022) · European Journal of Social Psychology · doi:10.1002/ejsp.2858 · Cited by 28
Krefeld-Schwalb, A., & Scheibehenne, B. (2022) · Marketing Letters · doi:10.1007/s11002-022-09662-3 · Cited by 31
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · Behavior Research Methods · doi:10.3758/s13428-017-0967-6 · Cited by 36
Expertise, Policy-making and Democracy
Holst, C., Molander, A., & Christensen, J. (2022) · Routledge eBooks · doi:10.4324/9781003106555 · Cited by 68
Wright, S. R., Graham, C. D., Houghton, R., Ghiglieri, C. G., & Berry, E. (2023) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2022.12.003 · Cited by 34
Examining publication bias—a simulation-based evaluation of statistical tests on publication bias
Schneck, A. (2017) · PeerJ · doi:10.7717/peerj.4115 · Cited by 32
“…studies).9 161 p-uniform (PU) 162 The tests discussed so far focus on the empirical effect sizes, whereas the p-curve method, 163 proposed by Simonsohn et al. (2014b), and the similar PU, a method proposed by van Assen et al. 164 (2015), focus entirely on the distribution of significant…”
BackgroundMethodInfluentialEnduring Association Between Parenting and Cortisol: A Meta-analysis
Hackman, D. A., O’Brien, J. R., & Zalewski, M. (2018) · Child Development · doi:10.1111/cdev.13077 · Cited by 32
Chun, C. A., Ciceron, L., & Kwapil, T. R. (2018) · Journal of Abnormal Psychology · doi:10.1037/abn0000383 · Cited by 29
McKay, B., & Ste‐Marie, D. M. (2020) · Human Movement Science · doi:10.1016/j.humov.2020.102612 · Cited by 33
Ramos, A., Coutinho, P., Leitão, J. C., Cortinhas, A., Davids, K., & Mesquita, I. (2020) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2020.101754 · Cited by 42
“The publication bias consists of considering only studies that provide statistically significant results (i.e., p 0.05) (Simonsohn, Nelson, & Simmons, 2014).”
BackgroundInfluentialThe effects of temperature on prosocial and antisocial behaviour: A review and meta‐analysis
Lynott, D., Corker, K. S., Connell, L., & O’Brien, K. (2023) · British Journal of Social Psychology · doi:10.1111/bjso.12626 · Cited by 29
What limitations are reported in short articles in social and personality psychology?
Clarke, B., Schiavone, S. R., & Vazire, S. (2023) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000458 · Cited by 38
p-Hacking and False Discovery in A/B TestingPreprint
Berman, R., Pekelis, L., Scott, A., & den Bulte, C. V. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3204791 · Cited by 36
“Unlike previous analyses using a p-curve (Simonsohn et al. 2014) or a funnel plot of effect sizes vs. standard errors at the time of termination, our data allows us to directly infer the relationship between achieved p-values and stopping behavior, including discontinuities within very narrow…”
MethodLin, L., Lin, X., Zhang, X., & Ginns, P. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09933-7 · Cited by 55
“We used three approaches to examining the publication bias, i.e., the funnel plots (Sterne et al., 2005), Egger’s regression test (Egger et al., 1997), and the p-curve analysis (Simonsohn et al., 2014, 2015).”
The Research Reproducibility Crisis and Economics of Science
Maniadis, Z., & Tufano, F. (2017) · The Economic Journal · doi:10.1111/ecoj.12526 · Cited by 32
Tovmasyan, A., Monk, R. L., Sawicka, I., & Heim, D. (2022) · The American Journal of Drug and Alcohol Abuse · doi:10.1080/00952990.2022.2082300 · Cited by 32
“To perform a p-curve analysis and assess potential p-hacking (98), single-level models were used (computed using metagen function from meta package; (99)).”
MethodHarper, C. A., & Bartels, R. M. (2016) · Journal of Sexual Aggression · doi:10.1080/13552600.2016.1250963 · Cited by 26
Meta‐Regression Models and Observational Research
Bruns, S. B. (2017) · Oxford Bulletin of Economics and Statistics · doi:10.1111/obes.12172 · Cited by 31
Kinesiophobia and physical activity: A systematic review and meta-analysisPreprint
Goubran, M., Farajzadeh, A., Lahart, I. M., Bilodeau, M., & Boisgontier, M. P. (2023) · medRxiv · doi:10.1101/2023.08.17.23294240 · Cited by 26
“01 16 Elfving (2007) 64 (39) 47 (19-64) Chronic low back pain n.”
Video games and attitude change: A meta-analysis
Kolek, L., Ropovik, I., Šisler, V., van Oostendorp, H., & Brom, C. (2023) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2023.102225 · Cited by 45
Clarke, B., Alley, L. J., Ghai, S., Flake, J. K., Rohrer, J. M., Simmons, J. P., Schiavone, S. R., & Vazire, S. (2024) · Social and Personality Psychology Compass · doi:10.1111/spc3.12979 · Cited by 35
Self-citationReanalyzing Head et al. (2015): investigating the robustness of widespread p -hacking
Hartgerink, C. H. J. (2017) · PeerJ · doi:10.7717/peerj.3068 · Cited by 28
“05 (Simonsohn et al., 2014), 67 sufficient evidence for p-hacking is present if the last bin has a significantly higher frequency than the 68 penultimate bin in a binomial test.”
BackgroundBruns, S. B., & Kalthaus, M. (2019) · Research Policy · doi:10.1016/j.respol.2019.103877 · Cited by 31
Preventing persistent postsurgical pain: A systematic review and component network meta‐analysis
Allen, C., Walker, A. M., Premji, Z., Beauchemin‐Turcotte, M., Wong, J., Soh, S., Hawboldt, G. S., Shinkaruk, K., & Archer, D. P. (2022) · European Journal of Pain · doi:10.1002/ejp.1915 · Cited by 31
“, 2008) and with p- curve analysis (Simonsohn et al., 2014).”
BackgroundMethodInfluentialEffects of victim presence and coercion in restorative justice: An experimental paradigm.
Saulnier, A., & Sivasubramaniam, D. (2015) · Law and Human Behavior · doi:10.1037/lhb0000130 · Cited by 38
Pair-Bonded Relationships and Romantic Alternatives
Durante, K. M., Eastwick, P. W., Finkel, E. J., Gangestad, S. W., & Simpson, J. A. (2015) · Advances in experimental social psychology · doi:10.1016/bs.aesp.2015.09.001 · Cited by 29
“(2014b) present independent evidence for the existence of real ovulatory shifts: Within the set of published effects that are statistically significant, the distribution of p-values (i.e., the “p-curve”; Simonsohn et al., 2014) is consistent with the existence of true effects.”
Wilson, J. P., Young, S. G., Rule, N. O., & Hugenberg, K. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.07.007 · Cited by 31
Statistical reporting inconsistencies in experimental philosophy
Colombo, M., Duev, G., Nuijten, M. B., & Sprenger, J. (2018) · PLoS ONE · doi:10.1371/journal.pone.0194360 · Cited by 29
“Finally, Fig 5 presents a p -curve analysis that uses the distribution of significant p -values to quantify the evidential value of a set of results [23–24].”
MethodExamining Psychological Science Through Systematic Meta-Method Analysis: A Call for Research
Elson, M. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919863296 · Cited by 26
“…such as the test of insufficient variance (Schimmack, 2014), the p-uniform method (van Assen, van Aert, & Wicherts, 2015), the p-curve method (Simonsohn et al., 2014a, 2014b), trim-and-fill-based and funnel-plot-based methods (Duval & Tweedie, 2000), the precision-effect test and the use of…”
MethodHead, Heart, or Hands: How Do Employees Respond to a Radical Global Language Change over Time?
Reiche, B. S., & Neeley, T. B. (2019) · Organization Science · doi:10.1287/orsc.2019.1289 · Cited by 28
Hirst, T. C., Klasen, M. G., Rhodes, J. K. J., Macleod, M. R., & Andrews, P. (2020) · Journal of Neurotrauma · doi:10.1089/neu.2019.6923 · Cited by 25
“We had planned to use p-curve analysis, which appears to be more promising in this regard, 58,59 but un-fortunately the author skilled in this technique left the study.”
Sorting the File Drawer: A Typology for Describing Unpublished Studies
Lishner, D. A. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620979831 · Cited by 25
“Examples of techniques used to detect the potential for publication bias include funnel plots, cumulative meta-analysis, and p-curve analysis; examples of techniques that may be used to investigate the nature of suspected publication bias include subgroup comparison and model-selection analysis (e.g., Borenstein et al…”
MethodBerry, D. R., Wall, C. S. J., Tubbs, J. D., Zeidan, F. Z., & Brown, K. W. (2021) · Social Psychological and Personality Science · doi:10.1177/19485506211053095 · Cited by 31
“True effects are expected to generate an abundance of low p values (ps \ .01; Simonsohn et al., 2014), and thus, we encourage scientists to reproduce these findings in well-powered, preregistered studies.”
Mikneviciute, G., Ballhausen, N., Rimmele, U., & Kliegel, M. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.12.009 · Cited by 32
Macchia, L., & Whillans, A. V. (2021) · Social Psychology · doi:10.1027/1864-9335/a000466 · Cited by 33
A meta‐analysis on the effects of just‐below versus round prices
Troll, E. S., Frankenbach, J., Friese, M., & Loschelder, D. D. (2023) · Journal of Consumer Psychology · doi:10.1002/jcpy.1353 · Cited by 28
Kivimäki, M., Batty, G. D., Kawachi, I. I., Virtanen, M., Singh‐Manoux, A., & Brunner, E. J. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwu164 · Cited by 31
“…process, from what questions are asked by the investigator, to which method is chosen to gather data, to which analyses are conducted (e.g., “ P-hacking, ” wherein the method of statistical analysis and the degrees of freedom are manipulated until they yield statistically signi fi cant results) (1).”
BackgroundMethod
Metadata
- Authors
- Simonsohn, U., Nelson, L. D., & Simmons, J. P.
- Affiliations
- University of Pennsylvania (US)
- Venue
- Journal of Experimental Psychology: General
- Published
- 2014-04
- Volume
- 143
- Issue
- 2
- Pages
- 534-547
- ISSN
- 0096-3445 (ISSN-L)
- 1939-2222
- OpenAlex
- W3124690569
- Cited by
- 1,500
- References
- 49
- Citations by year
- 2026: 75 · 2025: 96 · 2024: 110 · 2023: 121 · 2022: 146 · 2021: 157 · 2020: 158 · 2019: 154 · 2018: 143 · 2017: 113 · 2016: 103 · 2015: 80 · 2014: 35
- FWCI
- 82.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Experimental Psychology General (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-curve, publication bias, selective reporting, p-hacking, file drawer problem, statistical significance, evidentiary value
- MeSH terms
- Data Interpretation, Statistical, Humans, Psychology, Experimental, Statistics as Topic, Reproducibility of Results, Models, Statistical, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.