P-curve: A key to the file-drawer.
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Because scientists tend to report only studies (publication bias) or analyses (p-hacking) that "work," readers must ask, "Are these effects true, or do they merely reflect selective reporting?" We introduce p-curve as a way to answer this question. P-curve is the distribution of statistically significant p values for a set of studies (ps < .05). Because only true effects are expected to generate right-skewed p-curves-containing more low (.01s) than high (.04s) significant p values--only right-skewed p--curves are diagnostic of evidential value. By telling us whether we can rule out selective reporting as the sole explanation for a set of findings, p-curve offers a solution to the age-old inferential problems caused by file-drawers of failed studies and analyses.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Incorrect p-values could affect the outcome of tests that analyze the distribution of p-values, such as the p-curve (Simonsohn, Nelson, & Simmons, 2014) and p-uniform (Van Assen, Van Aert, & Wicherts, 2014).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Simonsohn, Nelson, & Simmons, 2014
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [33]
In the citing paper
“After excluding studies which assessed maltreatment as part of broader measures of victimization (k=3) (25, 27, 29) or ACEs (k=5) (30, 34, 39, 44, 49), or focused on institutional neglect (k=3) (15, 45, 46), the meta-analytic effect size was also similar to the original estimate (Cohen’s d=0.34, 95% CI=0.26-0.42).”
From Childhood Maltreatment and Mental Health Problems: A Systematic Review and Meta-Analysis of Quasi-Experimental Studies (2023) · cites this paper as [25]
In the citing paper
“Trim‐and‐fill procedures do suffer from some well‐known limitations that could result in the underestimation or overestimation of publication bias (Rothstein, 2008; Simonsohn, Nelson, & Simmons, 2014).”
From Hot spots policing of small geographic areas effects on crime (2019) · cites this paper as Simonsohn, Nelson, & Simmons, 2014
In the citing paper
“Applying selection modeling approaches like p-uniform or 3PSM to the simple effects may therefore substantially overestimate the true effect size.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Simonsohn et al., 2014
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
(1963) · Medical Entomology and Zoology · Cited by 15,085
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Self-citationTesting Statistical Hypotheses
(2021) · Wiley Series in Probability and Statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,106
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“1 Trying multiple analyses to obtain statistical significance has received many names, including: bias (Ioannidis, 2005), significance chasing (Ioannidis & Trikalinos, 2007), data snooping (White, 2003), fiddling (Bross, 1971), publication bias in situ (Phillips, 2004) and specification searching (Gerber & Malhotra, 2…”
Cited by
Items 101–200 of 1,005
p-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210
“Simonsohn et al. [1] who coined the distribution of statistically significant p -values “ p - curve ” have argued that true effects generate right-skewed p -curves whereas null effects should result in uniformly distributed p -curves.”
BackgroundMethodInfluentialTowards reproducible computational drug discovery
Schaduangrat, N., Lampa, S., Simeon, S., Gleeson, M. P., Spjuth, O., & Nantasenamat, C. (2020) · Journal of Cheminformatics · doi:10.1186/s13321-020-0408-x · Cited by 208
Gangestad, S. W., Haselton, M. G., Welling, L. L. M., Gildersleeve, K., Pillsworth, E. G., Burriss, R. P., Larson, C. M., & Puts, D. A. (2016) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2015.09.001 · Cited by 207
“Stirnemann et al. (2013) used ultrasound fetal biometry to estimate the day of conception (number of days following beginning of last menses to day of conception) on a sample of nearly 6000 women.”
MethodThe relation between students’ socioeconomic status and ICT literacy: Findings from a meta-analysis
Scherer, R., & Siddiq, F. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.04.011 · Cited by 202
The cognitive benefits of learning computer programming: A meta-analysis of transfer effects.
Scherer, R., Siddiq, F., & Sánchez Viveros, B. (2019) · Journal of Educational Psychology · doi:10.1037/edu0000314 · Cited by 205
“.0001; continuous test for a right-skewed curve; Simonsohn et al., 2014) and that the p -curve was right-skewed (see Figure 2b).”
BackgroundMethodCombs, H. L., Folley, B. S., Berry, D. T. R., Segerstrom, S. C., Han, D. Y., Anderson-Mooney, A. J., Walls, B. D., & van Horne, C. (2015) · Neuropsychology Review · doi:10.1007/s11065-015-9302-0 · Cited by 205
Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 202
Transparency in Ecology and Evolution: Real Problems, Real Solutions
Parker, T. H., Forstmeier, W., Koricheva, J., Fidler, F., Hadfield, J. D., Chee, Y. E., Kelly, C. D., Gurevitch, J., & Nakagawa, S. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2016.07.002 · Cited by 201
“This practice has become known as P-hacking [4,5].”
InfluentialStar Wars: The Empirics Strike Back
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016) · American Economic Journal: Applied Economics · doi:10.1257/app.20150044 · Cited by 430
Childhood maltreatment and intimate partner violence victimization: A meta-analysis
Li, S., Zhao, F., & Yu, G. (2019) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2018.11.012 · Cited by 202
No evidence for a bilingual executive function advantage in the ABCD study
Dick, A. S., Garcia, N. L., Pruden, S. M., Thompson, W. K., Hawes, S. W., Sutherland, M. T., Riedel, M. C., Laird, A. R., & Gonzalez, R. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0609-3 · Cited by 201
Leder, J., Horlitz, T., Puschmann, P., Wittstock, V., & Schütz, A. (2019) · Safety Science · doi:10.1016/j.ssci.2018.07.021 · Cited by 199
Nudging: Progress to date and future directions
Beshears, J., & Kosowsky, H. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.09.001 · Cited by 198
“We also perform a p-curve analysis (Simonsohn et al., 2014) for each of the categories that we consider.”
MethodAnimal-Assisted Intervention for trauma: a systematic literature review
O'Haire, M. E., Guérin, N. A., & Kirkham, A. C. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01121 · Cited by 202
Chua, L.-K., Jimenez-Diaz, J., Lewthwaite, R., Kim, T., & Wulf, G. (2021) · Psychological Bulletin · doi:10.1037/bul0000335 · Cited by 188
“…bias, we also reported (see Supplementary Material 2) the results of a trim-and-fill analysis (Duval & Tweedie, 2000) as well as the p-curve method (Simonsohn et al., 2014) that corrected any ES overestimation on the assumption that publication bias from inclusion of small studies with large ES…”
Areas of Interest as a Signal Detection Problem in Behavioral Eye‐Tracking Research
Orquin, J. L., Ashby, N. J. S., & Clarke, A. D. F. (2016) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1867 · Cited by 191
“Finally, roughly a third of responders analyzed more than one AOI size, which could be an indication of data fishing depending on what the researcher chose to report (Simonsohn et al., 2014; Simmons et al., 2011).”
Backgroundvan Elk, M., Matzke, D., Gronau, Q. F., Guan, M., Vandekerckhove, J., & Wagenmakers, E.-J. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01365 · Cited by 192
“However, as argued in detail elsewhere (Simonsohn et al., 2014), this method rests on the unlikely assumption that selective reporting is related to researchers being hesitant to report small effect sizes, whereas in practice results are left unreported because the p-value does not reach the…”
McGuire, J., Kaiser, C., & Bach-Mortensen, A. M. (2022) · Nature Human Behaviour · doi:10.1038/s41562-021-01252-z · Cited by 185
Afonso, J., Ramirez-Campillo, R., Clemente, F. M., Büttner, F. C., & Andrade, R. (2024) · Sports Medicine · doi:10.1007/s40279-023-01927-9 · Cited by 187
Narcissism and social media use: A meta-analytic review.
McCain, J. L., & Campbell, W. K. (2018) · Psychology of Popular Media Culture · doi:10.1037/ppm0000137 · Cited by 247
RCTs to Scale: Comprehensive Evidence from Two Nudge Units
DellaVigna, S., & Linos, E. (2020) · National Bureau of Economic Research · doi:10.3386/w27594 · Cited by 189
“Thaler and Sunstein (2008) define nudges as “choice architecture that alters people’s behavior in a predictable way without forbidding any options or significantly changing their economic incentives.”
BackgroundThe Acute Effect of High-Intensity Exercise on Executive Function: A Meta-Analysis
Moreau, D., & Chou, E. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850568 · Cited by 185
Clayson, P. E., Carbine, K. A., Baldwin, S. A., & Larson, M. J. (2019) · Psychophysiology · doi:10.1111/psyp.13437 · Cited by 184
Six Questions for the Resource Model of Control (and Some Answers)
Inzlicht, M., & Berkman, E. (2015) · Social and Personality Psychology Compass · doi:10.1111/spc3.12200 · Cited by 181
Estimating the Reproducibility of Experimental Philosophy
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J., Berniūnas, R., Boudesseul, J., Colombo, M., et al. (2021) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0400-9 · Cited by 179
“…Average replication effect size Content-based 90.3% 0.41 (0.17) 0.39 (0.21) Context-based 25.0% 0.22 (0.08) 0.01 (0.06) Demographic effect 25.0% 0.29 (0.10) 0.12 (0.10) (Simonsohn et al. 2014), whereas the p-curve for our target set of x-phi findings revealed a substantial rightward skew (see Fig.”
BackgroundInfluentialEffective Teacher Professional Development: New Theory and a Meta-Analytic Test
Sims, S., Fletcher-Wood, H., O’Mara-Eves, A., Cottingham, S., Stansfield, C., Goodrich, J., Van Herwegen, J., & Anders, J. (2025) · Review of Educational Research · doi:10.3102/00346543231217480 · Cited by 168
“Our p -curve analysis does not indicate motivated p hacking in our sample (Simonsohn et al., 2014a, 2014b).”
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 174
Dubois, J., Galdi, P., Han, Y., Paul, L. K., & Adolphs, R. (2018) · Personality Neuroscience · doi:10.1017/pen.2018.8 · Cited by 172
“…is available in principle (Carp, 2012), and to the all-too-common practice of keeping such explorations “ behind the scenes ” and only reporting the “ best ” strategy, leading to an inflation of positive findings reported in the literature (Neuroskeptic, 2012; Simonsohn, Nelson, & Simmons, 2014).”
Finkel, E. J. (2014) · Advances in Experimental Social Psychology · doi:10.1016/b978-0-12-800052-6.00001-9 · Cited by 169
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
Scherer, R., Siddiq, F., & Sánchez Viveros, B. (2020) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106349 · Cited by 164
“Third, we examined the p -curve underlying the all intervention effects on computer programming in the data set (Simonsohn, Nelson, & Simmons, 2014).”
MethodA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
Decision making for others involving risk: A review and meta-analysis
Polman, E., & Wu, K. (2020) · Journal of Economic Psychology · doi:10.1016/j.joep.2019.06.007 · Cited by 164
Hilgard, J., Engelhardt, C. R., & Rouder, J. N. (2017) · Psychological Bulletin · doi:10.1037/bul0000074 · Cited by 162
Power Posing: P-Curving the Evidence
Simmons, J. P., & Simonsohn, U. (2017) · Psychological Science · doi:10.1177/0956797616658563 · Cited by 159
“P-Curve Analysis In Simonsohn et al. (2014a), we introduced p-curve analysis, a statistical tool that tests whether a set of findings contains evidential value.”
BackgroundMethodInfluentialSelf-citationWhat Have We (Not) Learnt from Millions of Scientific Papers with P Values?
Ioannidis, J. P. A. (2019) · The American Statistician · doi:10.1080/00031305.2018.1447512 · Cited by 159
“However, P-curves are sensitive to even tiny bias, corresponding to distortions of the effect sizes by 0.01 standard deviations in a setting of observational studies with confounding.”
BackgroundInfluentialThreats of a replication crisis in empirical computer science
Cockburn, A., Dragicevic, P., Besançon, L., & Gutwin, C. A. (2020) · Communications of the ACM · doi:10.1145/3360311 · Cited by 208
“40 Studies of true effects should produce a right-skewed p -curve, with many lower statistically significant p - values (for example, .01s) than high values (for example, .04s); but a set of p -hacked studies are likely to show a left-skewed p -curve, indicative of selecting variables that tipped…”
BackgroundDörrenberg, S., Rakoczy, H., & Liszkowski, U. (2018) · Cognitive Development · doi:10.1016/j.cogdev.2018.01.001 · Cited by 186
Publication Bias in Meta-Analysis: Confidence Intervals for Rosenthal’s Fail-Safe Number
Fragkos, K. C., Tsagris, M., & Frangos, C. C. (2014) · International Scholarly Research Notices · doi:10.1155/2014/825383 · Cited by 159
Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are Feasible
Coffman, L. C., & Niederle, M. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.81 · Cited by 156
Replicability and Reproducibility in Comparative Psychology
Stevens, J. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00862 · Cited by 190
“This “phacking” (Simmons et al., 2011; Simonsohn et al., 2014) generates an over-reporting of results with p-values just under 0.05 (Masicampo and Lalande, 2012) and is a particular problem for psychology (Head et al., 2015).”
BackgroundInfluentialQazi, A., Hasan, N., Abayomi‐Alli, O., Hardaker, G., Scherer, R., Sarker, Y., Paul, S. K., & Maitama, J. Z. (2021) · Education and Information Technologies · doi:10.1007/s10639-021-10775-x · Cited by 250
“Finally, we inspected the p -curve underlying the sample of effect sizes and their respective p -values to rule out a possible selection bias (Simonsohn et al., 2014).”
MethodOn the Reproducibility of Psychological Science
Johnson, V. E., Payne, R. D., Wang, T., Asher, A., & Mandal, S. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1240079 · Cited by 164
“…acute in the social sciences, where the effects of publication bias and other sources of nonreproducibility are now widely recognized, and a number of researchers have gone as far as to propose new methods for detecting irregularities in reported test results (Francis 2013; Simonsohn et al. 2014).”
MethodSarstedt, M., Adler, S. J., Rau, L., & Schmitt, B. (2024) · Psychology & Marketing · doi:10.1002/mar.21982 · Cited by 149
“…confirmatory study into an exploratory fishing expedition — a practice that has been criticized as p ‐ hacking (Guo & Ma, 2022; Sarstedt & Adler, 2023; Simonsohn et al., 2014) and which is commonly viewed as a major contributor to low replication rates in various fields (Ioannidis, 2005; Miller &…”
BackgroundDekkers, T. J., van Rentergem, J. A. A., Meijer, B., Popma, A., Wagemaker, E., & Huizenga, H. M. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.12.004 · Cited by 160
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617698146 · Cited by 155
Self-citationMedina, J., & Cason, S. (2017) · Cortex · doi:10.1016/j.cortex.2017.06.021 · Cited by 168
Lakens, D., & Etz, A. J. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693058 · Cited by 169
Philosophy of science and the replicability crisis
Romero, F. (2019) · Philosophy Compass · doi:10.1111/phc3.12633 · Cited by 184
Bartos̆, F., Maier, M., Wagenmakers, E., Doucouliagos, H., & Stanley, T. D. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1594 · Cited by 180
Robust Bayesian meta-analysis: Addressing publication bias with model-averaging.
Maier, M., Bartos̆, F., & Wagenmakers, E. (2022) · Psychological Methods · doi:10.1037/met0000405 · Cited by 174
Technology acceptance research: Meta-analysis
Marikyan, D., Papagiannidis, S., & Stewart, G. (2026) · Journal of Information Science · doi:10.1177/01655515231191177 · Cited by 140
“Publication bias may happen not because of unreported data on non-significant relationships, but manipulations to achieve significant results [114,115].”
Scientific apophenia in strategic management research: Significance tests & mistaken inference
Goldfarb, B., & King, A. A. (2016) · Strategic Management Journal · doi:10.1002/smj.2459 · Cited by 138
Liu, D., Kirschner, P. A., & Karpinski, A. C. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.039 · Cited by 139
The Teasing Effect: An Underappreciated Benefit of Creating and Resolving an Uncertainty
Ruan, B., Hsee, C. K., & Lu, Z. Y. (2018) · Journal of Marketing Research · doi:10.1509/jmr.15.0346 · Cited by 138
“We also submitted key statistical results into a p-curve analysis (Simonsohn, Nelson, and Simmons 2014).”
MethodKorrel, H., Mueller, K. L., Silk, T., Anderson, V., & Sciberras, E. (2017) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12688 · Cited by 138
Potential Reporting Bias in Neuroimaging Studies of Sex Differences
David, S. P., Naudet, F., Laude, J., Radua, J., Fusar-Poli, P., Chu, I., Stefanick, M. L., & Ioannidis, J. P. A. (2018) · Scientific Reports · doi:10.1038/s41598-018-23976-1 · Cited by 138
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
Narcissism and Social Networking Behavior: A Meta‐Analysis
Gnambs, T., & Appel, M. (2018) · Journal of Personality · doi:10.1111/jopy.12305 · Cited by 133
Lane, A., Luminet, O., Nave, G., & Mikolajczak, M. (2016) · Journal of Neuroendocrinology · doi:10.1111/jne.12384 · Cited by 129
A meta-analysis of social network site use and social support
Liu, D., Wright, K. B., & Hu, B. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.08.024 · Cited by 127
Carney, D. R., Cuddy, A. J. C., & Yap, A. J. (2015) · Psychological Science · doi:10.1177/0956797614566855 · Cited by 144
The Production of Knowledge: Enhancing Progress in Social Science
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 241
Trash-talking: Competitive incivility motivates rivalry, performance, and unethical behavior
Yip, J. A., Schweitzer, M. E., & Nurmohamed, S. (2017) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2017.06.002 · Cited by 164
Impacts and Determinants of Health Levels in Low-Income Countries
Dupas, P., & Miguel, E. A. (2016) · Handbook of economic field experiments · doi:10.1016/bs.hefe.2016.09.003 · Cited by 202
Do ‘watching eyes’ influence antisocial behavior? A systematic review & meta-analysis
Dear, K., Dutton, K., & Fox, E. (2019) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2019.01.006 · Cited by 166
College football, elections, and false-positive results in observational research
Fowler, A., & Montagnes, B. P. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1502615112 · Cited by 129
“Recently, researchers have pointed out that many published, experimental results in medicine (10) and psychology (11, 12) are false, and the same concerns apply broadly to experimental and nonexperimental findings in all academic disciplines.”
BackgroundReal, but Limited: A Meta-Analytic Assessment of Framing Effects in the Political Domain
Amsalem, E., & Zoizner, A. (2020) · British Journal of Political Science · doi:10.1017/s0007123420000253 · Cited by 174
Mutaf, B., Sasanguie, D., Smedt, B. D., & Reynvoet, B. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.02074 · Cited by 155
Re-examining the Experiential Advantage in Consumption: A Meta-Analysis and Review
Weingarten, E., & Goodman, J. K. (2020) · Journal of Consumer Research · doi:10.1093/jcr/ucaa047 · Cited by 145
“Evidential value conveys that selective reporting cannot entirely explain the results (Simonsohn et al. 2014a), which would occur for right-skewed p -curves.”
BackgroundMethodInfluentialA meta-analysis of fear-avoidance and pain intensity: the paradox of chronic pain
Kroska, E. B. (2016) · Scandinavian Journal of Pain · doi:10.1016/j.sjpain.2016.06.011 · Cited by 128
The frequency of excess success for articles in Psychological Science
Francis, G. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0601-x · Cited by 135
Effectiveness of digital educational game and game design in STEM learning: a meta-analytic review
Gui, Y., Cai, Z., Yang, Y., Kong, L., Fan, X., & Tai, R. H. (2023) · International Journal of STEM Education · doi:10.1186/s40594-023-00424-9 · Cited by 284
“…tampering with data in changing a non-significant result into a significant result ( p -hacking) could lead to the presence of publication bias (Simonsohn et al., 2014). p -curve, as a new method for assessing p -hacking, was used to assess the publication bias due to p-hacking, with the p…”
MethodBerry, D. R., Hoerr, J. P., Cesko, S., Alayoubi, A., Carpio, K., Zirzow, H., Walters, W., Scram, G., Rodriguez, K., & Beaver, V. (2020) · Personality and Social Psychology Bulletin · doi:10.1177/0146167219900418 · Cited by 123
Dastogeer, K. M. G., Zahan, M. I., Tahjib‐Ul‐Arif, M., Akter, M. A., & Okazaki, S. (2020) · Frontiers in Plant Science · doi:10.3389/fpls.2020.588550 · Cited by 144
“We visually inspected asymmetry in funnel plots, used “trim-and-fill” analysis and performed Begg andMazumdar rank correlation tests based on Kendall’s tau, the Egger regression test and p-curve analysis (Begg andMazumdar, 1994; Egger et al., 1997; Simonsohn et al., 2014) to examine publication biases in the datasets.”
MethodThe flipped classroom in second language learning: A meta-analysis
Vitta, J. P., & Al‐Hoorie, A. H. (2020) · Language Teaching Research · doi:10.1177/1362168820981403 · Cited by 179
Patterns of MHC ‐dependent mate selection in humans and nonhuman primates: a meta‐analysis
Winternitz, J. C., Abbate, J. L., Huchard, É., Havlíček, J., & Garamszegi, L. Z. (2016) · Molecular Ecology · doi:10.1111/mec.13920 · Cited by 141
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122
How to run an experimental auction: a review of recent advances
Canavari, M., Drichoutis, A. C., Lusk, J. L., & Nayga, R. M. (2019) · European Review of Agricultural Economics · doi:10.1093/erae/jbz038 · Cited by 140
“The problem of false positives is further exacerbated by the fact that researchers may not only file-drawer entire studies but also file-drawer subsets of analyses that produce non-significant results (Simonsohn et al., 2014).”
BackgroundMethodCurrent Research and Statistical Practices in Sport Science and a Need for Change
Bernards, J., Sato, K., Haff, G., & Bazyler, C. (2017) · Sports · doi:10.3390/sports5040087 · Cited by 120
“Examples include manipulation of statistics to produce statistically significant results (p-hacking) and hypothesis after results are known (HARKing) [7,8].”
BackgroundLearning From Testimony on Quantitative Research in Management
King, A. A., Goldfarb, B. D., & Simcoe, T. S. (2019) · Academy of Management Review · doi:10.2139/ssrn.3816190 · Cited by 135
Big little lies: a compendium and simulation of p -hacking strategies
Stefan, A. M., & Schönbrodt, F. D. (2023) · Royal Society Open Science · doi:10.1098/rsos.220346 · Cited by 206
“The term ‘p-hacking’ appeared shortly after the onset of the replication crisis [9,11], and the practice has since been discussed as one of the driving factors of false-positive results in the social sciences and beyond [12–14].”
BackgroundMethodInfluentialThe hedonic character of nostalgia: an integrative data analysis
Leunissen, J., Wildschut, T., Sedikides, C., & Routledge, C. (2021) · Emotion Review · doi:10.1177/1754073920950455 · Cited by 119
“As a formal test for publication bias, we conducted a p-curve analysis (Simonsohn et al., 2014) of the nostalgia effects on “happy,” which supported the evidential value of the included experiments.”
MethodAddressing the inconsistencies in fear of crime research: A meta-analytic review
Collins, R. E. (2016) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2016.06.004 · Cited by 154
Is personality associated with dementia risk? A meta-analytic investigation
Aschwanden, D., Strickhouser, J. E., Luchetti, M., Stéphan, Y., Sutin, A. R., & Terracciano, A. (2021) · Ageing Research Reviews · doi:10.1016/j.arr.2021.101269 · Cited by 137
A kinked meta‐regression model for publication bias correction
Bom, P. R. D., & Rachinger, H. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1352 · Cited by 133
van IJzendoorn, M. H., Schuengel, C., Wang, Q., & Bakermans-Kranenburg, M. J. (2023) · Development and Psychopathology · doi:10.1017/s0954579421001462 · Cited by 116
“Evidential value for the meta-analyses was established with the p -curve approach (Simonsohn et al., 2014), and no evidence for p -hacking could be detected for any of the outcomes.”
InfluentialGrowth mindset is not associated with scholastic aptitude in a large sample of university applicants
Bahník, Š., & Vranka, M. A. (2017) · Personality and Individual Differences · doi:10.1016/j.paid.2017.05.046 · Cited by 165
Pair-bonding, fatherhood, and the role of testosterone: a meta-analytic review
Grebe, N. M., Sarafin, R. E., Strenth, C. R., & Zilioli, S. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.01.010 · Cited by 116
Nakagawa, S., & Parker, T. H. (2015) · BMC Biology · doi:10.1186/s12915-015-0196-3 · Cited by 110
“Simonsohn and colleagues [22] recently termed”
Past-focused environmental comparisons promote proenvironmental outcomes for conservatives
Baldwin, M. W., & Lammers, J. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1610834113 · Cited by 132
The effect of horizontal eye movements on free recall: A preregistered adversarial collaboration.
Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H. A., van der Molen, M. W., & Wagenmakers, E. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000038 · Cited by 118
“The steps involved in the selection of p values that met these selection criteria are documented in the recommended p -curve disclosure table (cf. Simonsohn et al., 2014) available in the supplemental materials at http://osf.io/pXT3M/.”
Brain volume and intelligence: The moderating role of intelligence measurement quality
Gignac, G. E., & Bates, T. C. (2017) · Intelligence · doi:10.1016/j.intell.2017.06.004 · Cited by 155
“Simonsohn, Nelson, and Simmons (2014a) introduced the p-curve analysis as a method capable of formally evaluating the likelihood that published literature relevant to a particular hypothesis may be the result of p-hacking.”
MethodHeterogeneity of Striatal Dopamine Function in Schizophrenia: Meta-analysis of Variance
Brugger, S. P., Angelescu, I., Abi‐Dargham, A., Mizrahi, R., Shahrezaei, V., & Howes, O. D. (2019) · Biological Psychiatry · doi:10.1016/j.biopsych.2019.07.008 · Cited by 120
Prevention of eating disorders at universities: A systematic review and meta‐analysis
Harrer, M., Adam, S. H., Messner, E., Baumeister, H., Cuijpers, P., Bruffaerts, R., Auerbach, R. P., Kessler, R. C., Jacobi, C., Taylor, C. B., et al. (2020) · International Journal of Eating Disorders · doi:10.1002/eat.23224 · Cited by 114
Everything Matters: The ReproNim Perspective on Reproducible Neuroimaging
Kennedy, D. N., Abraham, S. A., Bates, J. F., Crowley, A., Ghosh, S., Gillespie, T., Goncalves, M., Grethe, J. S., Halchenko, Y. O., Hanke, M., et al. (2019) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2019.00001 · Cited by 113
“At a high level, a premium is put on novel, high-profile publications (in contrast to replications and negative findings) and a specific p-value (typically 0.05) as a proxy for truth has been adopted (Simonsohn et al., 2014; Wasserstein and Lazar, 2016).”
BackgroundInfluentialDissanayaka, T., Zoghi, M., Farrell, M. J., Egan, G. F., & Jaberzadeh, S. (2017) · European Journal of Neuroscience · doi:10.1111/ejn.13640 · Cited by 127
“lysis is due to P-hacking or true effect (Simonsohn et al., 2014).”
BackgroundChester, D. S., & Lasko, E. N. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620950684 · Cited by 158
“Little evidence of p-hacking (Simonsohn et al., 2014) or publication bias was observed, suggesting that researchers are neither simply selectively reporting their pilot validity data to artificially evince an underlying effect nor merely submitting unsuccessful pilot validity studies to the “file drawer” and cherry-pi…”
Dekkers, T. J., Hornstra, R., der Oord, S. V., Luman, M., Hoekstra, P. J., Groenman, A. P., & van den Hoofdakker, B. J. (2021) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2021.06.015 · Cited by 150
Motyl, M. S., Demos, A. P., Carsel, T. S., Hanson, B. E., Melton, Z. J., Mueller, A. B., Prims, J., Sun, J., Washburn, A. N., Wong, K., et al. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000084 · Cited by 123
“…have generally only been applied to small sets of suspicious studies (e.g., Bem, 2011) and have done well in generating some statistic that suggests the findings were false positives and contain little or no evidentiary value (e.g., Simonsohn et al., 2014; for a review, see Schimmack, 2014b).”
Background
Metadata
- Authors
- Simonsohn, U., Nelson, L. D., & Simmons, J. P.
- Affiliations
- University of Pennsylvania (US)
- Venue
- Journal of Experimental Psychology: General
- Published
- 2014-04
- Volume
- 143
- Issue
- 2
- Pages
- 534-547
- ISSN
- 0096-3445 (ISSN-L)
- 1939-2222
- OpenAlex
- W3124690569
- Cited by
- 1,500
- References
- 49
- Citations by year
- 2026: 75 · 2025: 96 · 2024: 110 · 2023: 121 · 2022: 146 · 2021: 157 · 2020: 158 · 2019: 154 · 2018: 143 · 2017: 113 · 2016: 103 · 2015: 80 · 2014: 35
- FWCI
- 82.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Experimental Psychology General (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-curve, publication bias, selective reporting, p-hacking, file drawer problem, statistical significance, evidentiary value
- MeSH terms
- Data Interpretation, Statistical, Humans, Psychology, Experimental, Statistics as Topic, Reproducibility of Results, Models, Statistical, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.