P-hacking and publication bias interact to distort meta-analytic effect size estimates

Friese, Malte; Frankenbach, Julius

doi:10.1037/met0000246

Cited by 122Meta-analysis and systematic reviewsPsychology Research and BibliometricsAcademic integrity and plagiarism

P-hacking and publication bias interact to distort meta-analytic effect size estimates

Friese, Malte, Frankenbach, Julius·2020·Psychological Methods

DOI 10.1037/met0000246

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Science depends on trustworthy evidence. Thus, a biased scientific record is of questionable value because it impedes scientific progress, and the public receives advice on the basis of unreliable evidence that has the potential to have far-reaching detrimental consequences. Meta-analysis is a technique that can be used to summarize research evidence. However, meta-analytic effect size estimates may themselves be biased, threatening the validity and usefulness of meta-analyses to promote scientific progress. Here, we offer a large-scale simulation study to elucidate how p -hacking and publication bias distort meta-analytic effect size estimates under a broad array of circumstances that reflect the reality that exists across a variety of research areas. The results revealed that, first, very high levels of publication bias can severely distort the cumulative evidence. Second, p -hacking and publication bias interact: At relatively high and low levels of publication bias, p -hacking does comparatively little harm, but at medium levels of publication bias, p -hacking can considerably contribute to bias, especially when the true effects are very small or are approaching zero. Third, p -hacking can severely increase the rate of false positives. A key implication is that, in addition to preventing p -hacking, policies in research institutions, funding agencies, and scientific journals need to make the prevention of publication bias a top priority to ensure a trustworthy base of evi…

Integrity signals

Registry

Crossref · 10.1037/met0000246

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Researchers often engage in analytical practices that capitalize on chance to obtain positive findings, such as p-hacking and the garden of forking paths.
“In addition, researchers often engage in analytical practices that capitalize on chance to obtain positive findings, such as p hacking-repeatedly running statistical analyses and only reporting those that are significant-and”
From Acceptance and Commitment Therapy Processes and Mediation: Challenges and How to Address Them (2023) · cites this paper as Friese & Frankenbach, 2020

R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Introduction to Meta‐Analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225 · see citing works
Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · see citing works
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Signals from chloroplasts converge to regulate nuclear gene expression.
Koussevitzky, S., Nott, A., Mockler, T. C., Hong, F., Sachetto‐Martins, G., Surpin, M. A., Lim, J. C.-S., Mittler, R., & Chory, J. (2007) · PubMed · doi:10.1126/science.1140516 · Cited by 5,076 · see citing works
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397 · see citing works
Effect size guidelines for individual differences researchers
Gignac, G. E., & Szodorai, E. T. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.06.069 · Cited by 2,860 · see citing works
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · see citing works
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989 · see citing works
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709 · see citing works
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558 · see citing works
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · see citing works
Untitled work
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · see citing works
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · see citing works
Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · see citing works
Stereotype Threat
Spencer, S. J., Logel, C., & Davies, P. G. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-073115-103235 · Cited by 920 · see citing works
Meta-regression approximations to reduce publication selection bias: T. D. STANLEY AND H. DOUCOULIAGOS
Stanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916 · see citing works
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · see citing works
Correlational effect size benchmarks
Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015) · Journal of Applied Psychology · doi:10.1037/a0038047 · Cited by 768 · see citing works
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · see citing works
<i>p</i> -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717 · see citing works
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
The Persistence of Underpowered Studies in Psychological Research: Causes, Consequences, and Remedies.
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 655 · see citing works
A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science’s Aversion to the Null
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · see citing works
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588 · see citing works
Psychology's Renaissance
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · see citing works
Publication Decisions and Their Possible Effects on Inferences Drawn from Tests of Significance--Or Vice Versa
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.2307/2282137 · Cited by 524 · see citing works
Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses.
Ferguson, C. J., & Brannick, M. Τ. (2011) · Psychological Methods · doi:10.1037/a0024445 · Cited by 493 · see citing works
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410 · see citing works
The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical Power
Fraley, R. C., & Vazire, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109019 · Cited by 409 · see citing works
Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size
Kühberger, A., Fritz, A., & Scherndl, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105825 · Cited by 399 · see citing works
Selection Models and the File Drawer Problem
Iyengar, S., & Greenhouse, J. B. (1988) · Statistical Science · doi:10.1214/ss/1177013012 · Cited by 399 · see citing works
Meta-assessment of bias in science
Fanelli, D., Costas, R., & Ioannidis, J. P. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618569114 · Cited by 364 · see citing works
Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking: A reply to Ulrich and Miller (2015)
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000104 · Cited by 343 · see citing works
Does Self-Control Training Improve Self-Control? A Meta-Analysis
Friese, M., Frankenbach, J., Job, V., & Loschelder, D. D. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617697076 · Cited by 335 · see citing works
Self-citation
Early extreme contradictory estimates may appear in published research: The Proteus phenomenon in molecular genetics research and randomized trials
Ioannidis, J. P. A., & Trikalinos, T. A. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.019 · Cited by 326 · see citing works
Questionable Research Practices Revisited
Fiedler, K., & Schwarz, N. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550615612150 · Cited by 279 · see citing works
Science or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science
Giner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274 · see citing works
A peculiar prevalence of <i>p</i> values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266 · see citing works
Meta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265 · see citing works
A meta-analysis of the facial feedback literature: Effects of facial feedback on emotional experience are small and variable.
Coles, N. A., Larsen, J. T., & Lench, H. C. (2019) · Psychological Bulletin · doi:10.1037/bul0000194 · Cited by 231 · see citing works
Multilevel Meta-analysis: a comparison with traditional meta-analytical procedures
Van Den Noortgate, W., & Onghena, P. (2003) · Educational and Psychological Measurement · doi:10.1177/0013164403251027 · Cited by 218 · see citing works
Can Stereotype Threat Explain the Gender Gap in Mathematics Performance and Achievement?
Stoet, G., & Geary, D. C. (2012) · Review of General Psychology · doi:10.1037/a0026617 · Cited by 211 · see citing works
p-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210 · see citing works
Research in Social Psychology Changed Between 2011 and 2016: Larger Sample Sizes, More Self-Report Measures, and More Online Studies
Sassenberg, K., & Ditrich, L. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919838781 · Cited by 176 · see citing works
Registered Reports: Realigning incentives in scientific publishing
Chambers, C., Dienes, Z. P., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015) · Cortex · doi:10.1016/j.cortex.2015.03.022 · Cited by 171 · see citing works
Publication Decisions Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa
Sterling, T. D., Rosenbaum, W. L., & Jj, W. (1995) · The American Statistician · doi:10.2307/2684823 · Cited by 168 · see citing works
Sample size in psychological research over the past 30 years
Marszalek, J. M., Barber, C., Kohlhart, J., & Cooper, B. H. (2011) · Perceptual and Motor Skills · doi:10.2466/03.11.pms.112.2.331-348 · Cited by 167 · see citing works
Systematic review of publication bias in studies on publication bias
Dubben, H.-H., & Beck‐Bornholdt, H. (2005) · BMJ · doi:10.1136/bmj.38478.497164.f7 · Cited by 153 · see citing works
Correcting for Publication Bias in a Meta-Analysis with the P-uniform* MethodPreprint
van Aert, R. C. M., & van Assen, M. A. L. M. (2018) · doi:10.31222/osf.io/zqjr9 · Cited by 108 · see citing works
Synthesizing Evidence: Shifting the Focus From Individual Studies to the Body of Evidence
Murad, M. H., & Montori, V. M. (2013) · JAMA · doi:10.1001/jama.2013.5616 · Cited by 104 · see citing works
Bite-size science and its undesired side effects
Bertamini, M., & Munafò, M. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611429353 · Cited by 62 · see citing works
Comment: What p -hacking really looks like: A comment on Masicampo and LaLande (2012)
Lakens, D. (2014) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2014.982664 · Cited by 56 · see citing works
How can preregistration contribute to research in our field?
Jonas, K. J., & Cesario, J. (2015) · Comprehensive Results in Social Psychology · doi:10.1080/23743603.2015.1070611 · Cited by 36 · see citing works
Pre-Registration in Social Psychology—a Discussion and Suggested TemplatePreprint
Veer, A. V. '., & Giner‐Sorolla, R. (2016) · doi:10.31234/osf.io/4frms · Cited by 31 · see citing works
Reanalyzing Head et al. (2015): investigating the robustness of widespread p -hacking
Hartgerink, C. H. J. (2017) · PeerJ · doi:10.7717/peerj.3068 · Cited by 25 · see citing works
Meta-analysis of social-personality psychological research
Johnson, B. T., & Eagly, A. H. (2014) · OpenCommons - UConn (University of Connecticut) · doi:10.13140/2.1.4045.5685 · Cited by 21 · see citing works
How to detect publication bias in psychological research? A comparative evaluation of six statistical methodsPreprint
Renkewitz, F., & Keiner, M. (2018) · doi:10.31234/osf.io/w94ep · Cited by 14 · see citing works

Flagged first, then by citations

Sorted by citations

Empirically derived guidelines for effect size interpretation in social psychology
Lovakov, A. V., & Рафиковна, А. Е. (2021) · European Journal of Social Psychology · doi:10.1002/ejsp.2752 · Cited by 692 · see citing works
Visual attentional orienting by eye gaze: A meta-analytic review of the gaze-cueing effect.
McKay, K. T., Grainger, S. A., Coundouris, S. P., Skorich, D. P., Phillips, L. H., & Henry, J. D. (2021) · Psychological Bulletin · doi:10.1037/bul0000353 · Cited by 443 · see citing works
Do growth mindset interventions impact students’ academic achievement? A systematic review and meta-analysis with recommendations for best practices.
Macnamara, B. N., & Burgoyne, A. P. (2022) · Psychological Bulletin · doi:10.1037/bul0000352 · Cited by 358 · see citing works
Internet addiction in young adults: A meta-analysis and systematic review
Lozano-Blasco, R., Quílez-Robres, A., & Soto, A. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107201 · Cited by 278 · see citing works
Power to Detect What? Considerations for Planning and Evaluating Sample Size
Giner‐Sorolla, R., Montoya, A. K., Reifman, A. S., Carpenter, T., Lewis, N. A., Aberson, C. L., Bostyn, D. H., Conrique, B. G., Ng, B., Schoemann, A. M., et al. (2024) · Personality and Social Psychology Review · doi:10.1177/10888683241228328 · Cited by 292 · see citing works
The Past, Present, and Future of Ego Depletion
Inzlicht, M., & Friese, M. (2019) · Social Psychology · doi:10.1027/1864-9335/a000398 · Cited by 180 · see citing works
Self-citation
The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews
Steel, P. D. G., Beugelsdijk, S., & Aguinis, H. (2021) · Journal of International Business Studies · doi:10.1057/s41267-020-00385-z · Cited by 5 · see citing works
Strong Effort Manipulations Reduce Response Caution: A Preregistered Reinvention of the Ego-Depletion Paradigm
Lin, H., Saunders, B., Friese, M., Evans, N. J., & Inzlicht, M. (2020) · Psychological Science · doi:10.1177/0956797620904990 · Cited by 111 · see citing works
Self-citation
Single-case intervention research design standards: Additional proposed upgrades and future directions
Kratochwill, T. R., Horner, R. H., Levin, J. R., Machalicek, W., Ferron, J. M., & Johnson, A. H. (2023) · Journal of School Psychology · doi:10.1016/j.jsp.2022.12.002 · Cited by 92 · see citing works
Effects of Different Therapeutic Exercise Modalities on Migraine or Tension-Type Headache: A Systematic Review and Meta-Analysis with a Replicability Analysis
Varangot‐Reille, C., Suso‐Martí, L., Romero-Palau, M., Suárez-Pastor, P., & Cuenca‐Martínez, F. (2021) · Journal of Pain · doi:10.1016/j.jpain.2021.12.003 · Cited by 89 · see citing works
Focusing on cybersickness: pervasiveness, latent trajectories, susceptibility, and effects on the virtual reality experience
Garrido, L. E., Frías-Hiciano, M., Moreno-Jiménez, M., Cruz, G. N., García-Batista, Z. E., Guerra-Peña, K., & Medrano, L. A. (2022) · Virtual Reality · doi:10.1007/s10055-022-00636-4 · Cited by 87 · see citing works
Acceptance and Commitment Therapy Processes and Mediation: Challenges and How to Address Them
Arch, J. J., Fishbein, J. N., Finkelstein, L. B., & Luoma, J. B. (2023) · Behavior Therapy · doi:10.1016/j.beth.2022.07.005 · Cited by 72 · see citing works
Sex drive: Theoretical conceptualization and meta-analytic review of gender differences.
Frankenbach, J., Weber, M., Loschelder, D. D., Kilger, H., & Friese, M. (2022) · Psychological Bulletin · doi:10.1037/bul0000366 · Cited by 68 · see citing works
Self-citation
Screen addicts: A meta-analysis of internet addiction in adolescence
Lozano-Blasco, R., Martínez, M. P. L., & Pascual, M. D. P. A. C. (2022) · Children and Youth Services Review · doi:10.1016/j.childyouth.2022.106373 · Cited by 67 · see citing works
Types of Intelligence and Academic Performance: A Systematic Review and Meta-Analysis
Lozano-Blasco, R., Quílez-Robres, A., Supervía, P. U., Salavera, C., & López, R. C. (2022) · Journal of Intelligence · doi:10.3390/jintelligence10040123 · Cited by 84 · see citing works
Message Design Choices Don't Make Much Difference to Persuasiveness and Can't Be Counted On—Not Even When Moderating Conditions Are Specified
O’Keefe, D. J., & Hoeken, H. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.664160 · Cited by 59 · see citing works
The relation of mindfulness and prosocial behavior: What do we (not) know?
Schindler, S., & Friese, M. (2021) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2021.09.010 · Cited by 62 · see citing works
Self-citation
Power to Detect What? Considerations for Planning and Evaluating Sample SizePreprint
Giner‐Sorolla, R., Montoya, A. K., Aberson, C. L., Carpenter, T., Lewis, N. A., Bostyn, D. H., Conrique, B. G., Ng, B., Reifman, A. S., Schoemann, A. M., et al. (2023) · doi:10.31234/osf.io/rv3kw · Cited by 62 · see citing works
A meta-analysis of Libet-style experiments
Braun, M. N., Wessler, J., & Friese, M. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.06.018 · Cited by 59 · see citing works
Self-citation
Socio‐Economic Status and Creativity: A Meta‐Analysis
Acar, S., Tadık, H., Uysal, R., Myers, D., & Inetas, B. (2022) · The Journal of Creative Behavior · doi:10.1002/jocb.568 · Cited by 56 · see citing works
Neglect of publication bias compromises meta-analyses of educational research
Ropovik, I., Adamkovic, M., & Greger, D. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252415 · Cited by 52 · see citing works
How to enable healthier and more sustainable food practices in collective meal contexts: A scoping review.
Graça, J., Campos, L., Guedes, D., Roque, L., Brazão, V., Trüninger, M., & Godinho, C. I. A. (2023) · Appetite · doi:10.1016/j.appet.2023.106597 · Cited by 40 · see citing works
The validity of situation awareness for performance: a meta-analysis
Bakdash, J. Z., Marusich, L. R., Cox, K. R., Geuss, M. N., Zaroukian, E. G., & Morris, K. (2021) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2021.1921310 · Cited by 28 · see citing works
Meta-analyzing the multiverse: A peek under the hood of selective reporting.
Olsson-Collentine, A., van Aert, R. C. M., Bakker, M., & Wicherts, J. M. (2023) · Psychological Methods · doi:10.1037/met0000559 · Cited by 34 · see citing works
On the predicted replicability of two decades of experimental research on system justification: A Z‐curve analysis
Sotola, L. K., & Credé, M. (2022) · European Journal of Social Psychology · doi:10.1002/ejsp.2858 · Cited by 28 · see citing works
Predictive Models of Life Satisfaction in Older People: A Machine Learning Approach
Shen, X., Yin, F., & Jiao, C. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20032445 · Cited by 27 · see citing works
Empirically Derived Guidelines for Effect Size Interpretation in Social PsychologyPreprint
Lovakov, A. V., & Рафиковна, А. Е. (2017) · doi:10.31234/osf.io/2epc4 · Cited by 26 · see citing works
The impact of big data on research methods in information science
Zhang, J., Wolfram, D., & Ma, F. (2023) · Data and Information Management · doi:10.1016/j.dim.2023.100038 · Cited by 24 · see citing works
A meta‐analysis on the effects of just‐below versus round prices
Troll, E. S., Frankenbach, J., Friese, M., & Loschelder, D. D. (2023) · Journal of Consumer Psychology · doi:10.1002/jcpy.1353 · Cited by 25 · see citing works
Self-citation
Reliable measurement in sport psychology: The case of performance outcome measures
Schweizer, G., Furley, P., Rost, N., & Barth, K. (2020) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2020.101663 · Cited by 24 · see citing works
The Family Context in Cybervictimization: A Systematic Review and Meta-Analysis
Lozano-Blasco, R., Barreiro-Collazo, A., Romero‐Gonzalez, B., & Soto, A. (2023) · Trauma Violence & Abuse · doi:10.1177/15248380231207894 · Cited by 27 · see citing works
Does social psychology persist over half a century? A direct replication of Cialdini et al.’s (1975) classic door-in-the-face technique.
Genschow, O., Westfal, M., Crusius, J., Bartosch, L., Feikes, K. I., Pallasch, N., & Wozniak, M. (2020) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000261 · Cited by 21 · see citing works
Alpha Oscillations and Temporal Binding Windows in Perception—A Critical Review and Best Practice Guidelines
Schoffelen, J., Pesci, U. G., & Noppeney, U. (2024) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_02118 · Cited by 21 · see citing works
Digital Competence in University Lecturers: A Meta-Analysis of Teaching Challenges
Orús, M. L., Lozano-Blasco, R., & Romeral, L. A. (2023) · Education Sciences · doi:10.3390/educsci13050508 · Cited by 29 · see citing works
Sensory recruitment in visual short-term memory: A systematic review and meta-analysis of sensory visual cortex interference using transcranial magnetic stimulation
Phylactou, P., Traikapi, A., Papadatou-Pastou, M., & Konstantinou, N. (2022) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02107-y · Cited by 15 · see citing works
Modelling publication bias and p‐hacking
Moss, J., & de Bin, R. (2019) · Biometrics · doi:10.1111/biom.13560 · Cited by 20 · see citing works
The Unknown Prevalence of Postrandomization Bias in 15 Physical Therapy Journals: A Methods Review
Riley, S. P., Swanson, B. T., Shaffer, S. M., Sawyer, S. F., & Cleland, J. A. (2021) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2021.10491 · Cited by 16 · see citing works
Systematic Analysis of Publication Bias in Neurosurgery Meta-Analyses
Phua, Q. S., Lu, L., Harding, M., Poonnoose, S. I., Jukes, A., & To, M. (2022) · Neurosurgery · doi:10.1227/neu.0000000000001788 · Cited by 16 · see citing works
Campbell’s Law Explains the Replication Crisis: Pre-Registration Badges Are History Repeating
Klonsky, E. D. (2024) · Assessment · doi:10.1177/10731911241253430 · Cited by 21 · see citing works
Why Change is Hard
McLean, K. C. (2024) · doi:10.1093/oso/9780197764640.001.0001 · Cited by 14 · see citing works
Beyond statistical significance: Five principles for the new era of data analysis and reporting
Wedel, M., & Gal, D. (2023) · Journal of Consumer Psychology · doi:10.1002/jcpy.1379 · Cited by 12 · see citing works
Do Owners Know How Impulsive Their Dogs Are?
Stevens, J. R., Mathias, M., Herridge, M., Hughes-Duvall, K., Wolff, L., & Yohe, M. (2022) · Animal Behavior and Cognition · doi:10.26451/abc.09.03.02.2022 · Cited by 12 · see citing works
A revised and expanded taxonomy for understanding heterogeneity in research and reporting practices.
Manapat, P. D., Anderson, S. F., & Edwards, M. C. (2022) · Psychological Methods · doi:10.1037/met0000488 · Cited by 11 · see citing works
The Inner Workings of Registered Reports
Dienes, Z. P. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009000796.015 · Cited by 9 · see citing works
The power and peril of first offers in negotiations: a conceptual, meta-analytic, and experimental synthesis
Petrowsky, H. M., Boecker, L., Escher, Y. A., Frech, M.-L., Friese, M., Galinsky, A. D., Gunia, B., Lee, A. J., Schaerer, M., Schweinsberg, M., et al. (2025) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2025.104448 · Cited by 10 · see citing works
Self-citation
Estimating the replicability of technology education research
Buckley, J., Hyland, T., & Seery, N. (2022) · International Journal of Technology and Design Education · doi:10.1007/s10798-022-09787-6 · Cited by 9 · see citing works
Quantifying, and correcting for, the impact of questionable research practices on false discovery rates in psychological science
Kravitz, D. J., & Mitroff, S. R. (2023) · Journal for Reproducibility in Neuroscience · doi:10.36850/jrn.2023.e44 · Cited by 11 · see citing works
Are most published research findings false? Trends in statistical power, publication selection bias, and the false discovery rate in psychology (1975–2017)
Schneck, A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292717 · Cited by 10 · see citing works
Current Status and Future Opportunities in Modeling Clinical Characteristics of Multiple Sclerosis
Liu, J., Kelly, E., & Bielekova, B. (2022) · Frontiers in Neurology · doi:10.3389/fneur.2022.884089 · Cited by 10 · see citing works
Launching Registered Report Replications in Computer Science Education Research
Brown, N. C. C., Marinus, E., & Cheuoua, A. H. (2022) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3501385.3543971 · Cited by 10 · see citing works
A Systematic Review on the Evolution of Power Analysis Practices in Psychological Research
Vankelecom, L., Schacht, O., Laroy, N., Loeys, T., & Moerkerke, B. (2025) · Psychologica Belgica · doi:10.5334/pb.1318 · Cited by 12 · see citing works
Publication bias is bad for science if not necessarily scientists
Heesen, R., & Bright, L. K. (2025) · Royal Society Open Science · doi:10.1098/rsos.240688 · Cited by 9 · see citing works
Statistical significance misuse in public health research: an investigation of the current situation and possible solutions
Rovetta, A. (2024) · Journal of Health Policy & Outcomes Research · doi:10.7365/jhpor.2024.1.7 · Cited by 9 · see citing works
Perspective taking does not moderate the price precision effect, but indirectly affects counteroffers to asking prices
Leib, M., Kee, K., Loschelder, D. D., & Roskes, M. (2022) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2022.104323 · Cited by 8 · see citing works
The Myriad Forms of p-Hacking
Reis, D., & Friese, M. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_5 · Cited by 10 · see citing works
Self-citation
How to survive the doctorate? A meta-analysis of success in PhD Candidates
Lozano-Blasco, R., Romero‐Gonzalez, B., & Soto, A. (2024) · Educación XX1 · doi:10.5944/educxx1.36566 · Cited by 8 · see citing works
Wissenschaftskommunikation wissenschaftlich arbeitender Psycholog_innen im deutschsprachigen Raum
Genschow, O., Sauerland, M., Crusius, J., & Friese, M. (2022) · Psychologische Rundschau · doi:10.1026/0033-3042/a000606 · Cited by 8 · see citing works
Self-citation
Noise in the process: Assessing the evidential value of mediation analyses in marketing journalsPreprint
Montoya, A. K., Charlton, A., Price, J., Hilgard, J., & Krefeld-Schwalb, A. (2021) · doi:10.31234/osf.io/ck2r5 · Cited by 8 · see citing works
Navigating Unmeasured Confounding in Nonexperimental Psychological Research: A Practical Guide to Computing and Interpreting E-Value
Bi, K., Merrin, G. J., Li, T., Sun, X., Chai, Y., Lu, Z., & Chen, S. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251326571 · Cited by 7 · see citing works
How cognitive issue bracketing affects interdependent decision-making in negotiations
Warsitzka, M., Zhang, H., Loschelder, D. D., Majer, J. M., & Trötschel, R. (2021) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2021.104268 · Cited by 8 · see citing works
Does who I am and what I feel determine what I see (or say)? A meta-analytic systematic review exploring the influence of real and perceived bodily state on spatial perception of the external environment
MacIntyre, E., BRAITHWAITE, F., Mouatt, B., Wilson, D., & Stanton, T. R. (2022) · PeerJ · doi:10.7717/peerj.13383 · Cited by 7 · see citing works
Publish or perish in the era of artificial intelligence: which way for the Kenyan research community?
Ajwang, S. O., & Ikoha, A. P. (2024) · Library Hi Tech News · doi:10.1108/lhtn-04-2024-0065 · Cited by 6 · see citing works
Cobblers, Let’s Stick to Our Lasts! A Song of Sorrow (and of Hope) About the State of Personnel and Human Resource Management Science
Götz, M., & O’Boyle, E. H. (2023) · Emerald Group Publishing Limited eBooks · doi:10.1108/s0742-730120230000041004 · Cited by 6 · see citing works
The small impact of p-hacking marginally significant results on the meta-analytic estimation of effect size
Botella, J., Suero, M., Durán, J. I., & Blazquez, D. (2021) · Anales de Psicología · doi:10.6018/analesps.433051 · Cited by 4 · see citing works
The Word on the Street: Science Is Not Advocacy, but Publishing Research Is
Rice, D. B. (2022) · Group & Organization Management · doi:10.1177/10596011221097807 · Cited by 5 · see citing works
Breaking bad? Playing the fool and constructing the ‘bad researcher’ in entrepreneurship
Harrison, R. (2024) · Journal of Business Venturing Insights · doi:10.1016/j.jbvi.2024.e00484 · Cited by 4 · see citing works
Appropriately estimating the standardized average treatment effect with missing data: A simulation and primer
Anderson, S. F. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-02043-8 · Cited by 4 · see citing works
Estimating the replicability of statistically significant moderation effects in personality research using z-curve analysis
Sotola, L. K., & Credé, M. (2023) · Journal of Research in Personality · doi:10.1016/j.jrp.2023.104435 · Cited by 4 · see citing works
Prosocial behaviour enhances evaluation of physical beauty
Kononov, N., & Ein‐Gar, D. (2024) · British Journal of Social Psychology · doi:10.1111/bjso.12800 · Cited by 4 · see citing works
Open Science: Recommendations for Research on School Bullying
Noret, N. F., Hunter, S. C., Pimenta, S. M., Taylor, R., & Johnson, R. (2022) · International Journal of Bullying Prevention · doi:10.1007/s42380-022-00130-0 · Cited by 5 · see citing works
Increasing Value and Reducing Waste of Research on Neurofeedback Effects in Post-traumatic Stress Disorder: A State-of-the-Art-Review
Marcu, G. M., Dumbravă, A., Bacila, C. I., Szekely‐Copîndean, R. D., & Zăgrean, A. (2023) · Applied Psychophysiology and Biofeedback · doi:10.1007/s10484-023-09610-5 · Cited by 4 · see citing works
The replication crisis as mere indicator of two fundamental misalignments: Methodological confirmation bias in hypothesis testing and anthropological oversimplification in theory-building
Schnepf, J., & Groeben, N. (2024) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2024.101110 · Cited by 3 · see citing works
Servitization and firm performance: a p-curve analysis
Latan, H., Jabbour, A. B. L. D. S., Jabbour, C. J. C., Ali, M., & Filho, M. G. (2024) · Journal of Business and Industrial Marketing · doi:10.1108/jbim-07-2023-0392 · Cited by 3 · see citing works
Questionable Research Practices: p -Hacking
Taylor, J. M. (2025) · Journal of Nursing Education · doi:10.3928/01484834-20250210-01 · Cited by 3 · see citing works
Anchoring and Sleep Inertia
Frech, M.-L., Häusser, J. A., Siems, M.-C., & Loschelder, D. D. (2022) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1027/1618-3169/a000552 · Cited by 2 · see citing works
Les pratiques de recherche ouvertes en psychologie
Lantian, A. (2020) · Psychologie Française · doi:10.1016/j.psfr.2020.09.001 · Cited by 3 · see citing works
The lack of robust evidence for cleansing effects
Ropovik, I., Sparacio, A., & IJzerman, H. (2021) · Behavioral and Brain Sciences · doi:10.1017/s0140525x20000448 · Cited by 3 · see citing works
Opening Doors to Open Science and Scholarship for School Psychology Research, Training, and PracticePreprint
Roberson, A. J., Farmer, R. L., Shaw, S. R., Upton, S. R., & Zaheer, I. (2020) · doi:10.31234/osf.io/su53p · Cited by 3 · see citing works
Focusing on Cybersickness: Pervasiveness, Latent Trajectories, Susceptibility, and Effects on the Virtual Reality ExperiencePreprint
Garrido, L. E., Frías-Hiciano, M., Moreno-Jiménez, M., Cruz, G. N., García-Batista, Z. E., Guerra-Peña, K., & Medrano, L. A. (2022) · doi:10.31234/osf.io/nt5mz · Cited by 5 · see citing works
Letters to our Future Selves? High-Powered Replication Attempts Question Effects on Future Orientation, Delinquent Decisions, and Risky Investments
Quinten, L. K. J. L., Murmann, A. C., Genau, H. A., Warkentin, R., & Banse, R. (2020) · Social Cognition · doi:10.1521/soco.2020.38.6.521 · Cited by 2 · see citing works
The Power and Peril of Precise vs. Round Health Message Interventions to Increase Stair Use
Krull, S., Boecker, L., & Loschelder, D. D. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.624198 · Cited by 2 · see citing works
Perception of Social and Educational Quality of Life of Minors Diagnosed with Rare Diseases: A Systematic Review and Meta-Analysis
Coca, J. R., Gómez-Redondo, S., Soto-Sánchez, A., Lozano-Blasco, R., & Romero-Gonzalez, B. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20020933 · Cited by 2 · see citing works
The Influence of Alpha Frequency on Temporal Binding across the Senses: Response to the Special Focus
Noppeney, U., Pesci, U. G., & Schoffelen, J. (2024) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_02112 · Cited by 2 · see citing works
Evidential Value of Intervention Studies in Two School Psychology Journals: A p -Curve Analysis of Research from 2011 to 2021
Ni, H., Price, P. C., & Jones, C. J. (2024) · School Psychology Review · doi:10.1080/2372966x.2024.2365618 · Cited by 2 · see citing works
Functional Diversity and Access to the World of Work in Young People: A Meta-Analysis and Systematic Review
Lozano-Blasco, R., Cardona, I. P., & Mira-Aladrén, M. (2024) · Child & Youth Services · doi:10.1080/0145935x.2024.2389085 · Cited by 2 · see citing works
How to Produce, Identify, and Motivate Robust Psychological Science: A Roadmap and a Response to Vize et al.
Klonsky, E. D. (2024) · Assessment · doi:10.1177/10731911241299723 · Cited by 2 · see citing works
Cumulative Evidence Synthesis and Consideration of “Research Waste” Using Bayesian Methods
Corcoran, H., & Steele, J. (2024) · Communications in Kinesiology · doi:10.51224/cik.2024.74 · Cited by 3 · see citing works
P-Curve analysis of autonomous and controlling motivation priming effects supports their evidential value
Murphy, S. L., & Steel, R. P. (2021) · Motivation and Emotion · doi:10.1007/s11031-021-09919-w · Cited by 2 · see citing works
Geschichte der BWL – Die Konsumentenverhaltensforschung in Marketing und Betriebswirtschaftslehre
Gröppel‐Klein, A. (2022) · Springer eBooks · doi:10.1007/978-3-658-35155-7_8 · Cited by 2 · see citing works
Preprint - Meta-Analyzing the Multiverse: A Peek Under the Hood of Selective ReportingPreprint
Olsson-Collentine, A., van Aert, R. C. M., Bakker, M., & Wicherts, J. M. (2021) · doi:10.31234/osf.io/43yae · Cited by 2 · see citing works
Focusing on Cybersickness: Pervasiveness, Latent Trajectories, Susceptibility, and Effects on the Virtual Reality ExperiencePreprint
Garrido, L. E., Frías-Hiciano, M., Moreno-Jiménez, M., Cruz, G. N., García-Batista, Z. E., Guerra-Peña, K., & Medrano, L. A. (2021) · doi:10.31234/osf.io/7qvcf · Cited by 2 · see citing works
A Meta-Meta-Analysis of Behavior Change Interventions: Two Tales of Behavior ChangePreprint
Kaiser, T., Kloidt, J., Mata, J., & Hertwig, R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5251012 · Cited by 2 · see citing works
Evaluation of Transparency and Openness Guidelines in Physical Therapist Journals
Plante, J., Langerwerf, L., Klopper, M., Rhon, D. I., & Young, J. L. (2023) · Physical Therapy · doi:10.1093/ptj/pzad133 · Cited by 1 · see citing works
Theoretical Foundations
McLean, K. C. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197764640.003.0003 · Cited by 1 · see citing works
Implicit evidence for price-ending effects on quality and image perceptions
Seitz, L., Petrowsky, H. M., Neumann, M., & Loschelder, D. D. (2026) · Journal of Revenue and Pricing Management · doi:10.1057/s41272-026-00578-5 · Cited by 1 · see citing works
A Framework to Avoid Significance Fallacy
Rovetta, A. (2023) · Cureus · doi:10.7759/cureus.40242 · Cited by 1 · see citing works
Breaking the mold: examining the effectiveness of techniques to reduce motivated reasoning
Parvizi, J., & Hmielowski, J. D. (2023) · Atlantic Journal of Communication · doi:10.1080/15456870.2023.2224482 · Cited by 1 · see citing works
Tekrarlanabilirlik Krizi ve Geçerlilik Krizi Kıskacındaki Psikoloji ve Sosyal Bilimlerde Krizden Çıkış İçin Öne Çıkan İki Trend: “Yeni İstatistik” ve “Bayesyen İstatistik”
Kara, E. (2023) · Anadolu Journal Of Educational Sciences International · doi:10.18039/ajesi.1240655 · Cited by 1 · see citing works
Considering Registered Reports at C&RL
Riegelman, A. (2021) · College & Research Libraries · doi:10.5860/crl.82.1.2 · Cited by 1 · see citing works
How do psychology researchers interpret the results of multiple replication studies?
van den Akker, O. R., Wicherts, J. M., Alvarez, L. D., Bakker, M., & van Assen, M. A. L. M. (2023) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02235-5 · Cited by 1 · see citing works

The Impact of Publication Bias and Single and Combined p-Hacking Practices on Effect Size and Heterogeneity Estimates in Meta-AnalysisPreprint
Maassen, E., van Assen, M. A. L. M., Nuijten, M. B., & Wicherts, J. M. (2025) · doi:10.31234/osf.io/2uynm_v1
Estimating effect size and heterogeneity in meta-analysis in the presence of publication bias and questionable research practicesPreprint
Nuijten, M. B., Wicherts, J. M., van Assen, M. A. L. M., & Maassen, E. (2021) · doi:10.17605/osf.io/tfu3m
The small impact of p-hacking marginally significant results on the meta-analytic estimation of effect size
Botella, J., Suero, M., Durán, J. I., & Blazquez, D. (2021) · Anales de Psicología · doi:10.6018/analesps.433051 · Cited by 4 · see citing works
P ‐hacking in meta‐analyses: A formalization and new meta‐analytic methods
Mathur, M. B. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1701 · Cited by 24 · see citing works
Does p-Hacking Mitigate or Exacerbate the Effects of Publication Bias?Preprint
Cai, Y., Pernoud, A., & Xu, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.05372

Authors: Friese, M., & Frankenbach, J.
ORCID: Friese, Malte
Frankenbach, Julius
Venue: Psychological Methods
Publisher: American Psychological Association (APA)
Published: 2020-08
Volume: 25
Issue: 4
Pages: 456-471
ISSN: 1082-989X (ISSN-L)
1939-1463
DOI: 10.1037/met0000246
OpenAlex: W2992095815
Cited by: 122
References: 61
Citations by year: 2026: 5 · 2025: 8 · 2024: 27 · 2023: 24 · 2022: 28 · 2021: 22 · 2020: 6 · 2019: 1 · 2017: 1
FWCI: 9.56
Type: journal article
Language: en
Open access: closed
Locations: Psychological Methods (publishedVersion)
PubMed (publishedVersion)
OSF Preprints (OSF Preprints) (submittedVersion)
TUbilio (Technical University of Darmstadt) (submittedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Social Sciences › Safety Research › Academic integrity and plagiarism
Keywords: p-hacking, publication bias, research integrity, statistical significance testing
MeSH terms: Computer Simulation, Humans, Psychology, Research, Research Design, Meta-Analysis as Topic, Publication Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

P-hacking and publication bias interact to distort meta-analytic effect size estimates

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks