The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
P-curve: A key to the file-drawer.
Abstract
Because scientists tend to report only studies (publication bias) or analyses (p-hacking) that "work," readers must ask, "Are these effects true, or do they merely reflect selective reporting?" We introduce p-curve as a way to answer this question. P-curve is the distribution of statistically significant p values for a set of studies (ps < .05). Because only true effects are expected to generate right-skewed p-curves-containing more low (.01s) than high (.04s) significant p values--only right-skewed p--curves are diagnostic of evidential value. By telling us whether we can rule out selective reporting as the sole explanation for a set of findings, p-curve offers a solution to the age-old inferential problems caused by file-drawers of failed studies and analyses.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Incorrect p-values could affect the outcome of tests that analyze the distribution of p-values, such as the p-curve (Simonsohn, Nelson, & Simmons, 2014) and p-uniform (Van Assen, Van Aert, & Wicherts, 2014).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Simonsohn, Nelson, & Simmons, 2014
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [33]
In the citing paper
“After excluding studies which assessed maltreatment as part of broader measures of victimization (k=3) (25, 27, 29) or ACEs (k=5) (30, 34, 39, 44, 49), or focused on institutional neglect (k=3) (15, 45, 46), the meta-analytic effect size was also similar to the original estimate (Cohen’s d=0.34, 95% CI=0.26-0.42).”
From Childhood Maltreatment and Mental Health Problems: A Systematic Review and Meta-Analysis of Quasi-Experimental Studies (2023) · cites this paper as [25]
In the citing paper
“Trim‐and‐fill procedures do suffer from some well‐known limitations that could result in the underestimation or overestimation of publication bias (Rothstein, 2008; Simonsohn, Nelson, & Simmons, 2014).”
From Hot spots policing of small geographic areas effects on crime (2019) · cites this paper as Simonsohn, Nelson, & Simmons, 2014
In the citing paper
“Applying selection modeling approaches like p-uniform or 3PSM to the simple effects may therefore substantially overestimate the true effect size.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Simonsohn et al., 2014
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
(1963) · Medical Entomology and Zoology · Cited by 15,085
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Self-citationTesting Statistical Hypotheses
(2021) · Wiley Series in Probability and Statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,106
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“1 Trying multiple analyses to obtain statistical significance has received many names, including: bias (Ioannidis, 2005), significance chasing (Ioannidis & Trikalinos, 2007), data snooping (White, 2003), fiddling (Bross, 1971), publication bias in situ (Phillips, 2004) and specification searching (Gerber & Malhotra, 2…”
Cited by
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationScherer, R., Siddiq, F., & Tondeur, J. (2019) · Computers & Education · doi:10.1016/j.compedu.2018.09.009 · Cited by 1,923
“Third, we analyzed the p -curves derived from the aggregated correlations (Simonsohn, Nelson, & Simmons, 2014).”
MethodScientific method: Statistical errors
Nuzzo, R. (2014) · Nature · doi:10.1038/506150a · Cited by 1,536
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388
“…Abstracts are more likely to contain p -values that relate to primary hypotheses, which are expected to be more strongly p-hacked than p -values from less interesting, ancillary tests [41], lower power and reporting bias may impede detection of p-hacking using p -values obtained from Abstracts.”
BackgroundMethodResultInfluentialIncreasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,372
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293
Sisk, V. F., Burgoyne, A. P., Sun, J., Butler, J. L., & Macnamara, B. N. (2018) · Psychological Science · doi:10.1177/0956797617739704 · Cited by 1,159
Mini Meta‐Analysis of Your Own Studies: Some Arguments on Why and a Primer on How
Goh, J. X., Hall, J. A., & Rosenthal, R. (2016) · Social and Personality Psychology Compass · doi:10.1111/spc3.12267 · Cited by 994
Muris, P., Merckelbach, H., Otgaar, H., & Meijer, E. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691616666070 · Cited by 946
The contact hypothesis re-evaluated
PALUCK, E. L., GREEN, S. A., & GREEN, D. P. (2019) · Behavioural Public Policy · doi:10.1017/bpp.2018.25 · Cited by 856
“The same may be said for our analysis of p-hacking (Simonsohn et al., 2014; Head et al., 2015), which is symptomatic of research discretion that favors significant relationships.”
BackgroundRevised standards for statistical evidence
Johnson, V. E. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1313476110 · Cited by 808
Small Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742
Self-citationMany Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 784
p -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717
“01) than high significant p values (.04 < p < .05; Cumming, 2008; Hung, O’Neill, Bauer, & Kohne, 1997; Simonsohn et al., 2014; Wallis, 1942).”
InfluentialSelf-citationPrejudice Reduction: Progress and Challenges
Paluck, E. L., Porat, R., Clark, C. S., & Green, D. P. (2021) · Annual Review of Psychology · doi:10.1146/annurev-psych-071620-030619 · Cited by 686
“F or p er so na l u se o nl y. by those reviewing the literature; it also creates a “p-hacking” problem (Simonsohn et al. 2014) whereby researchers make statistical choices that militate in favor of producing larger, statistically significant estimates.”
BackgroundA meta-analysis of procedures to change implicit measures
Forscher, P. S., Lai, C. K., Axt, J. R., Ebersole, C. R., Herman, M., Devine, P. G., & Nosek, B. A. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000160 · Cited by 700
Coping flexibility and psychological adjustment to stressful life changes: A meta-analytic review.
Cheng, C., Lau, H.-P. B., & Chan, M.-P. S. (2014) · Psychological Bulletin · doi:10.1037/a0037913 · Cited by 664
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
“…to identify and accommodate potential publication and related reporting biases (Stanley, 2005, 2008; Ioannidis and Trikalinos, 2007; Stanley et al., 2010; Stanley and Doucouliagos, 2014) and others have proposed methods to detect and evaluate the extent of p-value hacking (Simonsohn et al., 2014).”
MethodBuilding a Science of Individual Differences from fMRI
Dubois, J., & Adolphs, R. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.03.014 · Cited by 628
“Culprits for the lack of replicability are small sample sizes [138], analytical flexibility [179] which fosters p-hacking [180] without appropriate control for the overall risk of false positives [181], and the bias to publish only statistically significant results (also known as the file drawer problem [182]).”
BackgroundNelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
Self-citationOpen science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
“For instance, the open software p-curve analysis was developed in response to the skew in the literature in favour of positive results and facilitates estimating publication bias within research areas ([62]; although, see [63]).”
ResultSequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (2017) · Psychological Methods · doi:10.1037/met0000061 · Cited by 567
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510
“Checking articles with statcheck can also prevent such inconsistencies from distorting meta-analyses or analyses of p -value distributions (Simonsohn et al., 2014; Van Assen et al., 2014).”
Heisig, J. P., & Schaeffer, M. (2019) · European Sociological Review · doi:10.1093/esr/jcy053 · Cited by 552
“Consistent with ‘phacking’ (Simonsohn et al., 2014), the distribution of absolute t-ratios for models estimated without a random slope exhibits a marked peak just above the critical value of 1.”
BackgroundPromoting Transparency in Social Science Research
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 540
Self-citationGeneralizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515
Creativity and academic achievement: A meta-analysis
Gajda, A., Karwowski, M., & Beghetto, R. A. (2017) · Journal of Educational Psychology · doi:10.1037/edu0000133 · Cited by 515
RCTs to Scale: Comprehensive Evidence From Two Nudge Units
DellaVigna, S., & Linos, E. (2022) · Econometrica · doi:10.3982/ecta18709 · Cited by 480
Religious Priming: A Meta-Analysis With a Focus on Prosociality
Shariff, A. F., Willard, A. K., Andersen, T., & Norenzayan, A. (2016) · Personality and Social Psychology Review · doi:10.1177/1088868314568811 · Cited by 488
“Although the p-curve architects provide a set of selection criteria for which p values should be included (Simonsohn et al., 2014), they recognize that, as with any meta-analysis, ambiguous situations will require subjective judgments.”
InfluentialHarrer, M., Adam, S. H., Baumeister, H., Cuijpers, P., Karyotaki, E., Auerbach, R. P., Kessler, R. C., Bruffaerts, R., Berking, M., & Ebert, D. D. (2019) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1759 · Cited by 445
“We therefore additionally used p curve, a novel method to detect selective reporting bias in meta‐analysis (Simonsohn et al., 2014a), which assumes that bias is caused because only statistically significant results (p 0.05) are considered for publication.”
MethodInfluentialDetecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
Presidential Address: The Scientific Outlook in Financial Economics
HARVEY, C. R. (2017) · The Journal of Finance · doi:10.1111/jofi.12530 · Cited by 461
A meta-analysis of biological impacts of artificial light at night
Sanders, D., Frago, E., Kehoe, R., Patterson, C., & Gaston, K. J. (2020) · Nature Ecology & Evolution · doi:10.1038/s41559-020-01322-x · Cited by 458
Methods for testing publication bias in ecological and evolutionary meta‐analyses
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez‐Tójar, A., Yang, Y., & O'Dea, R. E. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13724 · Cited by 399
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
Capaldi, C. A., Passmore, H., Nisbet, E. K. L., Zelenski, J. M., & Dopko, R. L. (2015) · International Journal of Wellbeing · doi:10.5502/ijw.v5i4.1 · Cited by 442
Kleven, H. J. (2016) · Annual Review of Economics · doi:10.1146/annurev-economics-080315-015234 · Cited by 416
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410
How Bayes factors change scientific practice
Dienes, Z. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.10.003 · Cited by 401
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Rand, D. G. (2016) · Psychological Science · doi:10.1177/0956797616654455 · Cited by 395
“Furthermore, p -curve analysis (Simonsohn, Nelson, & Simmons, 2014), using p -curve app 4.03 (accessed at www.p-curve.com/app4/pcurve4.php), indicated the presence of strong evidential value, p .0001 for both full and half p -curves, and did not find indications of inadequate evidential value, p…”
InfluentialEvolution of Reporting P Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370
Weingarten, E., Chen, Q., McAdams, M., Yi, J., Hepler, J., & Albarracín, D. (2016) · Psychological Bulletin · doi:10.1037/bul0000030 · Cited by 380
“…two sets of p-curve analyses (based on continuous tests) of the studies in this meta-analysis: (a) a p-curve on all studies conducted using p-values based on the researchers’ focal hypotheses (Simonsohn et al., 2014), and (b) p-curves based on studies with the largest error degrees of freedom.”
InfluentialStatistical Inference as Severe Testing: How to Get Beyond the Statistics Wars
Mayo, D. G. (2018) · doi:10.1017/9781107286184 · Cited by 376
Large Cross-National Differences in Gene × Socioeconomic Status Interaction on Intelligence
Tucker-Drob, E. M., & Bates, T. C. (2016) · Psychological Science · doi:10.1177/0956797615612727 · Cited by 359
“(2012), and Tucker-Drob et al. (2013). We also consulted a table of previous studies included in Hanscombe et al.”
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000104 · Cited by 343
“.05 (see Figure 3A in Simonsohn et al., 2014).”
InfluentialSelf-citationUsing large language models in psychology
Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00241-5 · Cited by 369
“For example, LLMs could help to discover red flags for P -hacking 75 in a contextually attuned way, rather than using keyword searches (such as ‘covariate’ or ‘transformation’).”
BackgroundHot spots policing and crime reduction: an update of an ongoing systematic review and meta-analysis
Braga, A. A., Turchan, B. S., Papachristos, A. V., & Hureau, D. M. (2019) · Journal of Experimental Criminology · doi:10.1007/s11292-019-09372-3 · Cited by 352
“Trim-and-fill procedures do suffer from some well-known limitations that could result in the underestimation or overestimation of publication bias (Rothstein 2008; Simonsohn et al. 2014).”
BackgroundJahrami, H. A., Alhaj, O. A., Humood, A. M., Alenezi, A. F., Fekih-Romdhane, F., AlRasheed, M. M., Saif, Z. Q., Bragazzi, N. L., Pandi-Perumal, S. R., BaHammam, A. S., et al. (2022) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2022.101591 · Cited by 335
“The p-curve approach, which focuses on p-values as the major driver of publication bias, was used to determine whether there is a real impact behind our meta-analysis data and to estimate its size [32,33].”
Heterogeneity and Homogeneity of Regional Brain Structure in Schizophrenia: A Meta-analysis
Brugger, S. P., & Howes, O. D. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.2663 · Cited by 332
A meta-analysis of syntactic priming in language production
Mahowald, K., James, A., Futrell, R., & Gibson, E. (2016) · Journal of Memory and Language · doi:10.1016/j.jml.2016.03.009 · Cited by 322
Freese, J., & Peterson, D. (2017) · Annual Review of Sociology · doi:10.1146/annurev-soc-060116-053450 · Cited by 310
A Meta-Analytic Review of Social Identification and Health in Organizational Contexts
Steffens, N. K., Haslam, S. A., Schuh, S. C., Jetten, J., & van Dick, R. (2017) · Personality and Social Psychology Review · doi:10.1177/1088868316656701 · Cited by 311
Baldwin, J. R., Wang, B., Karwatowska, L., Schoeler, T., Tsaligopoulou, A., Munafò, M. R., & Pingault, J.-B. (2023) · American Journal of Psychiatry · doi:10.1176/appi.ajp.20220174 · Cited by 313
“We also inspected funnel plots and conducted a p-curve analysis (24) based on an average effect size per study to test whether statistically significant findings were likely to reflect selective reporting and/or p-hacking.”
MethodConducting perception research over the internet: A tutorial review
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015) · PeerJ · doi:10.7717/peerj.1058 · Cited by 301
Unpacking teachers’ intentions to integrate technology: A meta-analysis
Scherer, R., & Teo, T. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.03.001 · Cited by 302
Is there a gender gap? A meta-analysis of the gender differences in students' ICT literacy
Siddiq, F., & Scherer, R. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.03.007 · Cited by 297
“Fourth, we examined the p - curve that resulted from the statistics underlying the gender differences (Simonsohn, Nelson, & Simmons, 2014).”
MethodDetecting Selection Bias in Meta-Analyses with Multiple Outcomes: A Simulation Study
Fernández-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Van den Noortgate, W. (2021) · The Journal of Experimental Education · doi:10.1080/00220973.2019.1582470 · Cited by 285
Lakens, D., & Evers, E. R. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528520 · Cited by 266
When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298
Comparing meta-analyses and preregistered multiple-laboratory replication projects
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0787-z · Cited by 271
“For examples of direct and indirect evidence on publication bias and selective reporting see, for instance, Simmons, Nelson and Simonsohn14, Brodeur17, John, Loewensten and Prelec20 and Franco, Malhotra and Simonovits21,22.”
BackgroundMarginally Significant Effects as Evidence for Hypotheses: Changing Attitudes Over Four Decades
Pritschet, L., Powell, D., & Horne, Z. (2016) · Psychological Science · doi:10.1177/0956797616645672 · Cited by 268
Retracted: Digital Communication Media Use and Psychological Well-Being: A Meta-Analysis
Liu, D., Baumeister, R. F., Yang, C.-C., & Hu, B. (2019) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmz013 · Cited by 260
RetractedMeta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
“An exhaustive pre-registration can provide evidence against the findings being a result of selective reporting or p-hacking [178] and thus strengthen the credibility of a study.”
BackgroundBraga, A. A., Weisburd, D., & Turchan, B. (2018) · Criminology & Public Policy · doi:10.1111/1745-9133.12353 · Cited by 261
“Trim-and-fill procedures do suffer from some well-known limitations that could result in the underestimation or overestimation of publication bias (Rothstein, 2008; Simonsohn, Nelson, and Simmons, 2014).”
Can Policies Stall the Fertility Fall? A Systematic Review of the (Quasi‐) Experimental Literature
Bergsvik, J., Fauske, A., & Hart, R. K. (2021) · Population and Development Review · doi:10.1111/padr.12431 · Cited by 259
Self-Enhancement and Psychological Adjustment: A Meta-Analytic Review
Dufner, M., Gebauer, J. E., Sedikides, C., & Denissen, J. J. A. (2019) · Personality and Social Psychology Review · doi:10.1177/1088868318756467 · Cited by 258
“Finally, a p-curve analysis (Simonsohn et al., 2014) indicated that in all cases the p curve was right skewed.”
InfluentialGotfredsen, K., & Holm, B. (2000) · PubMed · Cited by 258
Underpowered samples, false negatives, and unconscious learning
Vadillo, M. A., Konstantinidis, E., & Shanks, D. R. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0892-6 · Cited by 255
“On 26 November 2013 we searched in the Web of Science for all the papers citing the original report from Chun and Jiang (1998).”
MethodIncreasing the Power of Your Study by Increasing the Effect Size
Meyvis, T., & Van Osselaer, S. M. J. (2018) · Journal of Consumer Research · doi:10.1093/jcr/ucx110 · Cited by 250
Mathur, M. B., Epel, E., Kind, S., Desai, M., Parks, C. G., Sandler, D. P., & Khazeni, N. (2016) · Brain, Behavior, and Immunity · doi:10.1016/j.bbi.2016.02.002 · Cited by 252
“The ‘‘p-curve” (Simonsohn et al., 2014), a more precise test of publication bias that does not rely on sample standard error, was not feasible in this case due to the small number of positive findings.”
BackgroundMind full of ideas: A meta-analysis of the mindfulness–creativity link
Lebuda, I., Zabelina, D. L., & Karwowski, M. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2015.09.040 · Cited by 250
Winer, E. S., & Salem, T. (2016) · Psychological Bulletin · doi:10.1037/bul0000022 · Cited by 248
Sample Size, Statistical Power, and False Conclusions in Infant Looking‐Time Research
Oakes, L. M. (2017) · Infancy · doi:10.1111/infa.12186 · Cited by 248
A Short (Personal) Future History of Revolution 2.0
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615609918 · Cited by 213
Cherry Picking with Synthetic Controls
Ferman, B., Pinto, C., & Possebom, V. (2020) · Journal of Policy Analysis and Management · doi:10.1002/pam.22206 · Cited by 243
“However, it is important to note that testing all the possible SC specifications separately would not provide a valid hypothesis test since there would not be a defined decision rule (see White (2000)). One alternative is to consider a test statistic for the permutation test that combines the test statistics for all i…”
BackgroundMethodInfluentialCarter, E. C., & McCullough, M. E. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00823 · Cited by 381
“This method is thoroughly discussed in a working paper by Simonsohn et al. (2014b).”
MethodHow Much Can We Generalize From Impact Evaluations?
Vivalt, E. (2020) · Journal of the European Economic Association · doi:10.1093/jeea/jvaa019 · Cited by 237
Meaningful associations in the adolescent brain cognitive development study
Dick, A. S., Lopez, D. A., Watts, A. L., Heeringa, S., Reuter, C., Bartsch, H., Fan, C. C., Kennedy, D. N., Palmer, C., Marshall, A., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118262 · Cited by 241
Estimating the Difference Between Published and Unpublished Effect Sizes: A Meta-Review
Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016) · Review of Educational Research · doi:10.3102/0034654315582067 · Cited by 234
A meta-analysis of virtual reality training programs for social skill development
Howard, M. C., & Gutworth, M. B. (2020) · Computers & Education · doi:10.1016/j.compedu.2019.103707 · Cited by 234
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307
On the reproducibility of meta-analyses: six practical recommendations
Lakens, D., Hilgard, J., & Staaks, J. (2016) · BMC Psychology · doi:10.1186/s40359-016-0126-3 · Cited by 230
“, [30]), a reference to a specific study should preferably be accompanied by a direct quote from the original article describing the test the effect size is derived from (containing information that underlies the coding of study characteristics upon which studies are organized into subgroups, see [55]).”
BackgroundMethodInfluentialBig Five and HEXACO Personality Traits, Proenvironmental Attitudes, and Behaviors: A Meta-Analysis
Soutter, A. R. B., Bates, T. C., & Mõttus, R. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620903019 · Cited by 224
Limitations of PET-PEESE and Other Meta-Analysis Methods
Stanley, T. D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693062 · Cited by 229
Coles, N. A., Larsen, J. T., & Lench, H. C. (2019) · Psychological Bulletin · doi:10.1037/bul0000194 · Cited by 228
Hot spots policing of small geographic areas effects on crime
Braga, A. A., Turchan, B., Papachristos, A. V., & Hureau, D. M. (2019) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1046 · Cited by 260
Power to Detect What? Considerations for Planning and Evaluating Sample Size
Giner-Sorolla, R., Montoya, A. K., Reifman, A., Carpenter, T., Lewis, N. A., Aberson, C. L., Bostyn, D. H., Conrique, B. G., Ng, B. W., Schoemann, A. M., et al. (2024) · Personality and Social Psychology Review · doi:10.1177/10888683241228328 · Cited by 221
“…bias can be assumed in most topics of psychology, some authors have proposed adjusting down-ward any effect sizes taken from published literature or otherwise treating them with skepticism (S. F. Anderson et al., 2017; McShane et al., 2016; Perugini et al., 2014; Simonsohn et al., 2014).”
BackgroundMethodThe Big Five personality traits, Big Two metatraits and social media: A meta-analysis
Liu, D., & Campbell, W. K. (2017) · Journal of Research in Personality · doi:10.1016/j.jrp.2017.08.004 · Cited by 222
Narcissism and Self-Insight: A Review and Meta-Analysis of Narcissists’ Self-Enhancement Tendencies
Grijalva, E., & Zhang, L. (2016) · Personality and Social Psychology Bulletin · doi:10.1177/0146167215611636 · Cited by 221
p-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210
“Simonsohn et al. [1] who coined the distribution of statistically significant p -values “ p - curve ” have argued that true effects generate right-skewed p -curves whereas null effects should result in uniformly distributed p -curves.”
BackgroundMethodInfluentialSchwens, C., Zapkau, F. B., Bierwerth, M., Isidor, R., Knight, G., & Kabst, R. (2018) · Entrepreneurship Theory and Practice · doi:10.1177/1042258718795346 · Cited by 220
Weidman, A. C., Steckler, C. M., & Tracy, J. L. (2017) · Emotion · doi:10.1037/emo0000226 · Cited by 217
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616650874 · Cited by 219
“Moreover, use of the quite popular trim-and-fill method is discouraged because it often provides inaccurate results (Moreno, Sutton, Abrams, et al., 2009; Simonsohn et al., 2014a; Stanley & Doucouliagos, 2014; van Assen et al., 2015).”
BackgroundMethodResultInfluentialP. Simmons, J., D. Nelson, L., & Simonsohn, U. (2021) · Journal of Consumer Psychology · doi:10.1002/jcpy.1208 · Cited by 213
Self-citationEffects of lavender on anxiety: A systematic review and meta-analysis
Donelli, D., Antonelli, M., Bellinazzi, C., Gensini, G. F., & Firenzuoli, F. (2019) · Phytomedicine · doi:10.1016/j.phymed.2019.153099 · Cited by 210
Family correlates of false belief understanding in early childhood: A meta-analysis
Devine, R. T., & Hughes, C. (2018) · Child Development · doi:10.1111/cdev.12682 · Cited by 265
Metadata
- Authors
- Simonsohn, U., Nelson, L. D., & Simmons, J. P.
- Affiliations
- University of Pennsylvania (US)
- Venue
- Journal of Experimental Psychology: General
- Published
- 2014-04
- Volume
- 143
- Issue
- 2
- Pages
- 534-547
- ISSN
- 0096-3445 (ISSN-L)
- 1939-2222
- OpenAlex
- W3124690569
- Cited by
- 1,500
- References
- 49
- Citations by year
- 2026: 75 · 2025: 96 · 2024: 110 · 2023: 121 · 2022: 146 · 2021: 157 · 2020: 158 · 2019: 154 · 2018: 143 · 2017: 113 · 2016: 103 · 2015: 80 · 2014: 35
- FWCI
- 82.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Experimental Psychology General (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-curve, publication bias, selective reporting, p-hacking, file drawer problem, statistical significance, evidentiary value
- MeSH terms
- Data Interpretation, Statistical, Humans, Psychology, Experimental, Statistics as Topic, Reproducibility of Results, Models, Statistical, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.