Publication bias as a threat to the validity of meta-analytic results
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Publication bias presents a strong challenge to any review of evaluation studies (Rothstein, 2008).”
From Problem-oriented policing for reducing crime and disorder: An updated systematic review and meta-analysis (2020) · cites this paper as Rothstein, 2008
In the citing paper
“Trim‐and‐fill procedures do suffer from some well‐known limitations that could result in the underestimation or overestimation of publication bias (Rothstein, 2008; Simonsohn, Nelson, & Simmons, 2014).”
From Hot spots policing of small geographic areas effects on crime (2019) · cites this paper as Rothstein, 2008
In the citing paper
“Following the advice of Rothstein (2008), we apply two of the conventional methods that address publication bias.”
From A Meta-analysis of Attachment to Parents and Delinquency (2012) · cites this paper as Rothstein 2008
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“( Rosenthal 1979 ). He was concerned that metaanalytic results could be wrong, due to non-significant studies that remained in researchers’ file drawers.”
ResultPerceived organizational support: a review of the literature.
Rhoades, L., & Eisenberger, R. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.4.698 · Cited by 7,592 · cited 1×
“Mullen et al. (2001) proposed Rosenthal’s guideline as a formal rule, and the authors of several recent psychology meta-analyses (Del Vecchio and O’Leary 2004; Rhoades and Eisenberger 2002 ) have used this formula to assess the results of their file drawer analyses.”
MethodJournal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955
Systematic Reviews in Health Care : Meta-Analysis in Context
Egger, M., Smith, G. D., & Altman, D. G. (2000) · Cited by 2,412
Research works printed but not linked: 2
- Ioannidis, J. (2005). Differentiating biases from genuine heterogeneity. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.). Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.
- Wall Street Journal, (2004, November 1). E-Mails Suggest Merck Knew Vioxx’s Dangers at Early Stage.
Cited by
Emotion-regulation strategies across psychopathology: A meta-analytic review
Aldao, A., Nolen-Hoeksema, S., & Schweizer, S. (2010) · Clinical Psychology Review · doi:10.1016/j.cpr.2009.11.004 · Cited by 6,686
Dewald, J. F., Meijer, A. M., Oort, F. J., Kerkhof, G. A., & Bögels, S. M. (2010) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2009.10.004 · Cited by 1,526
Gender differences in emotion expression in children: A meta-analytic review.
Chaplin, T. M., & Aldao, A. (2013) · Psychological Bulletin · doi:10.1037/a0030737 · Cited by 1,247
Emotion regulation strategies in depressive and anxiety symptoms in youth: A meta-analytic review
Schäfer, J. Ö., Naumann, E., Holmes, E. A., Tuschen-Caffier, B., & Samson, A. C. (2017) · Journal of Youth and Adolescence · doi:10.1007/s10964-016-0585-0 · Cited by 941
Relation between Emotion Regulation and Mental Health: A Meta-Analysis Review
Hu, T., Zhang, D., Wang, J., Mistry, R., Ran, G., & Wang, X. (2014) · Psychological Reports · doi:10.2466/03.20.pr0.114k22w4 · Cited by 633
“The meta-analysis was not likely to have been aff ected by publication bias ( Rothstein, 2008 ).”
BackgroundThe Effects of Hot Spots Policing on Crime: An Updated Systematic Review and Meta-Analysis
Braga, A. A., Papachristos, A. V., & Hureau, D. M. (2014) · Justice Quarterly · doi:10.1080/07418825.2012.673632 · Cited by 694
“Publication Bias Publication bias presents a strong challenge to any review of evaluation studies (Rothstein, 2008).”
BackgroundDealing with the positive publication bias: Why you should really publish your negative results
Mlinarić, A., Horvat, M., & Šupak Smolčić, V. (2017) · Biochemia Medica · doi:10.11613/bm.2017.030201 · Cited by 514
“However, it is clear that a positive bias is introduced when studies with negative results remain unreported, thereby jeopardizing the validity of meta-analysis (25,26).”
BackgroundGill, C., Weisburd, D., Telep, C. W., Vitter, Z., & Bennett, T. (2014) · Journal of Experimental Criminology · doi:10.1007/s11292-014-9210-y · Cited by 486
“Study authors and journal editors may be more inclined to submit or accept reports of statistically significant findings, perceiving them to be more ‘interesting,’ while studies showing no effect may never be published, or written up only in harder to find gray literature such as technical reports (Rothstein 2008; Rot…”
BackgroundA Meta-analysis of Attachment to Parents and Delinquency
Hoeve, M., Stams, G. J. J. M., van der Put, C. E., Dubas, J. S., van der Laan, P. H., & Gerris, J. R. M. (2012) · Journal of Abnormal Child Psychology · doi:10.1007/s10802-011-9608-1 · Cited by 466
“Several methods exist to address potential effects of publication bias, but each has its own shortcoming (Rothstein 2008).”
Beelmann, A., & Heinemann, K. S. (2014) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2013.11.002 · Cited by 412
“A potential threat to the interpretation and validity of any metaanalytic result is the problem of publication bias (Beelmann & Lipsey, in press; Rothstein, 2008).”
BackgroundDoes stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410
Emotion regulation and posttraumatic stress symptoms: a meta-analysis
Seligowski, A. V., Lee, D. J., Bardeen, J. R., & Orcutt, H. K. (2015) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2014.980753 · Cited by 408
“A common problem in meta-analyses is the file drawer phenomenon, where only studies with significant results are reported in the literature (Rosenthal, 1979; Rothstein, 2007; Rothstein, Sutton, & Borenstein, 2006; Sutton, Duval, Tweedie, Abrams, & Jones, 2000).”
BackgroundBraga, A. A., & Weisburd, D. L. (2012) · Journal of Research in Crime and Delinquency · doi:10.1177/0022427811419368 · Cited by 334
“Publication Bias Publication bias presents a strong challenge to any review of evaluation studies (Rothstein 2008).”
BackgroundHot spots policing and crime reduction: an update of an ongoing systematic review and meta-analysis
Braga, A. A., Turchan, B. S., Papachristos, A. V., & Hureau, D. M. (2019) · Journal of Experimental Criminology · doi:10.1007/s11292-019-09372-3 · Cited by 352
“Publication bias, generally defined as the concern that the collection of studies easily available to a reviewer represents those studies most likely to have statistically significant results, presents a strong challenge to any review of evaluation studies (Rothstein 2008).”
BackgroundThe relationship between math anxiety and math performance: a meta-analytic investigation
Zhang, J., Zhao, N., & Kong, Q. P. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01613 · Cited by 347
“According to instruction of Rosenthal, when value of failsafe-N was larger than 5K + 10 (K represented the number of individual studies), we could safely refuse possible publication bias (Rothstein, 2008).”
BackgroundJeong, H., Hmelo-Silver, C. E., & Jo, K. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.100284 · Cited by 339
Does Working Memory Training Transfer? A Meta-Analysis Including Training Conditions as Moderators
Schwaighofer, M., Fischer, F., & Bühner, M. (2015) · Educational Psychologist · doi:10.1080/00461520.2015.1036274 · Cited by 332
“When the yintercept differs significantly from zero, bias is indicated (Egger et al., 1997; Rothstein, 2008).”
BackgroundMethodAmpatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., & Chatzigeorgiou, A. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.10.006 · Cited by 305
A Meta‐Analytic Review of Child‐Centered Play Therapy Approaches
Lin, Y., & Bratton, S. C. (2015) · Journal of Counseling & Development · doi:10.1002/j.1556-6676.2015.00180.x · Cited by 248
Weisburd, D., Telep, C. W., Hinkle, J. C., & Eck, J. E. (2010) · Criminology & Public Policy · doi:10.1111/j.1745-9133.2010.00617.x · Cited by 285
Can Policing Disorder Reduce Crime? A Systematic Review and Meta-analysis
Braga, A. A., Welsh, B. C., & Schnell, C. (2015) · Journal of Research in Crime and Delinquency · doi:10.1177/0022427815576576 · Cited by 254
“Publication bias presents a strong challenge to any review of evaluation studies (Rothstein 2008).”
BackgroundBraga, A. A., Weisburd, D., & Turchan, B. (2018) · Criminology & Public Policy · doi:10.1111/1745-9133.12353 · Cited by 261
Hot spots policing of small geographic areas effects on crime
Braga, A. A., Turchan, B., Papachristos, A. V., & Hureau, D. M. (2019) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1046 · Cited by 260
Associations between microaggression and adjustment outcomes: A meta-analytic and narrative review.
Lui, P. P., & Quezada, L. (2019) · Psychological Bulletin · doi:10.1037/bul0000172 · Cited by 217
Emotion Regulation and Substance Use: A Meta-Analysis
Weiss, N. H., Kiefer, R., Goncharenko, S., Raudales, A. M., Forkus, S. R., Schick, M. R., & Contractor, A. A. (2022) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2021.109131 · Cited by 235
Lang, C., Kalak, N., Brand, S., Holsboer-Trachsler, E., Pühse, U., & Gerber, M. (2016) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2015.07.004 · Cited by 201
The Effects of “Pulling Levers” Focused Deterrence Strategies on Crime
Braga, A. A., & Weisburd, D. L. (2012) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.4073/csr.2012.6 · Cited by 204
Hinkle, J. C., Weisburd, D., Telep, C. W., & Petersen, K. (2020) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1089 · Cited by 194
Boemo, T., Nieto, I., Vazquez, C., & Sanchez-Lopez, A. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104747 · Cited by 191
“Whenever studies are asymmetrically represented around the pooled effect size, it suggests the presence of publication bias (Rothstein, 2008).”
BackgroundHot spots policing effects on crime
Braga, A., Papachristos, A., & Hureau, D. (2012) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.4073/csr.2012.8 · Cited by 189
Levine, T. R., Asada, K. J., & Carpenter, C. (2009) · Communication Monographs · doi:10.1080/03637750903074685 · Cited by 131
“Examination of a funnel plot has also been criticized as highly subjective and therefore difficult to assess consistently (Rothstein, 2008).”
BackgroundNguyen, T. D., Kramer, J. W., & Evans, B. J. (2019) · Review of Educational Research · doi:10.3102/0034654319877156 · Cited by 131
“Furthermore, there may be publication bias in which only statistically significant results are published, and the peer review moderator allows us to explore the existence of this bias (Borenstein, Hedges, Higgins, & Rothstein, 2009; Rothstein, 2008).”
BackgroundEarly maladaptive schemas and depression in adulthood: A systematic review and meta‐analysis
Bishop, A., Younan, R., Low, J., & Pilkington, P. D. (2022) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2630 · Cited by 135
Do cultural differences affect users’ e‐learning adoption? A meta‐analysis
Zhao, Y., Wang, N., Li, Y., Zhou, R., & Li, S. (2021) · British Journal of Educational Technology · doi:10.1111/bjet.13002 · Cited by 132
Correlates of teachers’ classroom management self-efficacy: A systematic review and meta-analysis
Duan, S., Bissaker, K. A., & Xu, Z. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09881-2 · Cited by 168
“In general, articles with positive or statistically significant results are more likely to be published, which can lead to publication bias (Rothstein, 2008).”
Song, Y., Nam, S., Park, S., Shin, I., & Ku, B. J. (2017) · The Science of Diabetes Self-Management and Care · doi:10.1177/0145721717712457 · Cited by 147
Emotional Awareness in Depressive and Anxiety Symptoms in Youth: A Meta-Analytic Review
Sendzik, L., Ö. Schäfer, J., C. Samson, A., Naumann, E., & Tuschen-Caffier, B. (2017) · Journal of Youth and Adolescence · doi:10.1007/s10964-017-0629-0 · Cited by 111
“Publication bias is a potential threat to the validity of metaanalytic results (Rothstein 2008).”
BackgroundMethodResultInfluentialEffects of mobile devices on K–12 students' achievement: a meta‐analysis
Tingir, S., Cavlazoglu, B., Caliskan, O., Koklu, O., & Intepe‐Tingir, S. (2017) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12184 · Cited by 103
Goldman, S. E., & Burke, M. M. (2017) · Exceptionality · doi:10.1080/09362835.2016.1196444 · Cited by 97
“In combination, these methods and their results were planned to assess the possibility of publication bias (Rothstein, 2008).”
MethodClauss, K. E., Gorday, J. Y., & Bardeen, J. R. (2022) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102142 · Cited by 137
What Works? A Systematic Review of Corporate Crime Deterrence
Schell-Busey, N. M., Simpson, S. S., Rorie, M. L., & Alper, M. E. (2016) · Criminology & Public Policy · doi:10.1111/1745-9133.12195 · Cited by 148
Halasa, T., Østerås, O., Hogeveen, H., van Werven, T., & Nielen, M. (2009) · Journal of Dairy Science · doi:10.3168/jds.2008-1740 · Cited by 119
van der Pol, T. M., Hoeve, M., Noom, M. J., Stams, G. J. J. M., Doreleijers, T. A. H., van Domburgh, L., & Vermeiren, R. R. J. M. (2017) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12685 · Cited by 108
“Several methods exist to address potential effects of publication bias, but each has its own shortcomings (Rothstein, 2008).”
MethodPrasetya, F., Fortuna, A., Samala, A. D., Rawas, S., Mystakidis, S., Syahril, S., Waskito, W., Primawati, Wulansari, R. E., & Kassymova, G. K. (2024) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2024.100926 · Cited by 131
Public project success? Measuring the nuances of success through ex post evaluation
Volden, G. H., & Welde, M. (2022) · International Journal of Project Management · doi:10.1016/j.ijproman.2022.06.006 · Cited by 115
Bushman, B. J., Rothstein, H. R., & Anderson, C. A. (2010) · Psychological Bulletin · doi:10.1037/a0018718 · Cited by 104
“One of us (Rothstein) is an expert on publication bias in meta-analysis (McDaniel, Rothstein, & Whetzel, 2006; Rothstein, 2004, 2008a, 2008b; Rothstein & Hopewell, 2009; Rothstein, Sutton, & Borenstein, 2005).”
InfluentialSelf-citationThe cohesion–performance relationship in sport: a 10-year retrospective meta-analysis
Filho, E., Dobersek, U., Gershgoren, L., Becker, B. J., & Tenenbaum, G. (2014) · Sport Sciences for Health · doi:10.1007/s11332-014-0188-7 · Cited by 124
“Not surprisingly, researchers adopting various methodological designs may still reach similar conclusions when studying a particular phenomenon [23, 27].”
BackgroundThe Effects of Problem-Oriented Policing on Crime and Disorder
Weisburd, D., Telep, C. W., Hinkle, J. C., & Eck, J. E. (2008) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.4073/csr.2008.14 · Cited by 100
Beelmann, A., & Lösel, F. A. (2020) · Journal of Developmental and Life-Course Criminology · doi:10.1007/s40865-020-00142-8 · Cited by 103
Focused deterrence strategies effects on crime: A systematic review
Braga, A. A., Weisburd, D., & Turchan, B. (2019) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1051 · Cited by 92
Zhaı, X., Shi, L., & Nehm, R. H. (2020) · Journal of Science Education and Technology · doi:10.1007/s10956-020-09875-z · Cited by 99
“According to Rothstein (2008), a failsafe-N equal to or greater than five times the number of studies (i.e., κ ) in the original meta-analysis, plus ten studies (i.e., (5 κ + 10) κ ), would indicate that the meta-analytic results were robust to the threat of publication bias.”
BackgroundBurglar Target Selection: A Cross-national Comparison
Townsley, M., Birks, D., Bernasco, W., Ruiter, S., Johnson, S. D., White, G., & Baum, S. (2014) · Journal of Research in Crime and Delinquency · doi:10.1177/0022427814541447 · Cited by 91
“Thus, publication bias may limit the extent to which the results of replications are disseminated, and in turn, the degree to which the generalizability of findings can be established (Rothstein 2008).”
After-School Programs for Delinquency Prevention: A Systematic Review and Meta-Analysis
Taheri, S. A., & Welsh, B. C. (2016) · Youth Violence and Juvenile Justice · doi:10.1177/1541204014567542 · Cited by 72
“This is important because it concerns publication bias, which can present a strong challenge to the findings of meta-analyses (Rothstein, 2008).”
BackgroundZheng, H., Han, F., Huang, Y., Wu, Y., & Wu, X. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-13299-2 · Cited by 94
“The Fail-Safe N values are presented in Table 2, which all exceeded the commonly accepted threshold of 5 k + 10, suggesting that inclusion of unpublished studies with null results would not alter the meta-analysis findings substantially (Rothstein, 2008).”
Understanding the combined impacts of weeds and climate change on crops
Vilà, M., Beaury, E. M., Blumenthal, D. M., Bradley, B. A., Early, R., Laginhas, B. B., Trillo, A., Dukes, J. S., Sorte, C. J. B., & Ibáñez, I. (2021) · Environmental Research Letters · doi:10.1088/1748-9326/abe14b · Cited by 89
de Vries, I., & Goggin, K. (2018) · Trauma Violence & Abuse · doi:10.1177/1524838018801332 · Cited by 86
“Because all studies were published, there is a risk of publication bias (Rothstein, 2008).”
BackgroundBaziliansky, S., & Cohen, M. W. (2020) · Stress and Health · doi:10.1002/smi.2972 · Cited by 81
Meta‐analysis of social science research: A practitioner's guide
Iršová, Z., Doucouliagos, H., Havránek, T., & Stanley, T. D. (2023) · Journal of Economic Surveys · doi:10.1111/joes.12595 · Cited by 101
The Effects of Using Mobile Devices on Student Achievement in Language Learning: A Meta-Analysis
Cho, K., Lee, S., Joo, M.-H., & Becker, B. J. (2018) · Education Sciences · doi:10.3390/educsci8030105 · Cited by 72
“Based on both Q B and Q W statistics in our moderator analysis, we employed the random-effects model and the analysis-of-variance or ANOVA-like mixed-effects model.”
BackgroundMethodHoeve, M., Stams, G. J. J. M., van der Zouwen, M., Vergeer, M. M., Jurrius, K. H. M., & Asscher, J. J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0104909 · Cited by 83
“Several methods exist to address potential effects of publication bias, but each has its own shortcoming [20].”
Holmgren, J., & Merkel, A. (2017) · Research in Transportation Economics · doi:10.1016/j.retrec.2017.05.001 · Cited by 74
Beelmann, A., Arnold, L. S., & Hercher, J. (2022) · Aggression and Violent Behavior · doi:10.1016/j.avb.2022.101798 · Cited by 77
Effects of Systemic Therapy on Mental Health of Children and Adolescents: A Meta-Analysis
Riedinger, V., Pinquart, M., & Teubert, D. (2015) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2015.1063427 · Cited by 66
“Finally, we checked whether our results may have been affected by a “publication bias,” as non-significant results may be less likely to be published (Rothstein, 2008).”
ResultRamamoorthi, R., Gahreman, D., Skinner, T., & Moss, S. (2019) · PLoS ONE · doi:10.1371/journal.pone.0221067 · Cited by 55
Roest, J. J., de Poll, M. J. W.-V., van der Helm, P. G., Stams, G. J. J. M., & Hoeve, M. (2022) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2022.2093210 · Cited by 66
SPATIAL DISPLACEMENT AND DIFFUSION OF BENEFITS AMONG GEOGRAPHICALLY FOCUSED POLICING INITIATIVES
Bowers, K., Johnson, S., Guerette, R. T., Summers, L., & Poynton, S. (2011) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.4073/csr.2011.3 · Cited by 60
A Gentle Introduction to Bayesian Network Meta-Analysis Using an Automated R Package
Liu, Y., Béliveau, A., Wei, Y., Chen, M. Y., Record-Lemon, R., Kuo, P., Pritchard, E., Tang, X., & Chen, G. (2022) · Multivariate Behavioral Research · doi:10.1080/00273171.2022.2115965 · Cited by 64
“It is also commonly referred to as the “ file drawer ” problem, which arises when not all relevant findings are published and are therefore missing in a meta-analysis (Hedges, 1992; Rothstein, 2008).”
BackgroundWhat are the effects of different elements of media on radicalization outcomes? A systematic review
Wolfowicz, M., Hasisi, B., & Weisburd, D. L. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1244 · Cited by 70
Hoogsteder, L. M., Stams, G. J. J. M., Figge, M. A., Changoe, K., van Horn, J. E., Hendriks, J., & Wissink, I. B. (2014) · Journal of Forensic Psychiatry and Psychology · doi:10.1080/14789949.2014.971851 · Cited by 56
“Additionally, we applied one of the conventional methods to examine the possibility of publication bias, the fail-safe number (Rothstein, 2008).”
McDaniel, J., Slaboch, K. D., & Yoder, P. J. (2017) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2017.11.010 · Cited by 57
“A uthor M anuscript A uthor M anuscript A uthor M anuscript A uthor M anuscript the population of completed studies on that topic” (Rothstein, 2008, p. 61).”
BackgroundResearch fraud: a long-term problem exacerbated by the clamour for research grants
Harvey, L. (2020) · Quality in Higher Education · doi:10.1080/13538322.2020.1820126 · Cited by 53
Disorder policing to reduce crime: An updated systematic review and meta‐analysis
Braga, A. A., Schnell, C., & Welsh, B. C. (2024) · Criminology & Public Policy · doi:10.1111/1745-9133.12667 · Cited by 57
Claridge, A. M. (2014) · Archives of Women's Mental Health · doi:10.1007/s00737-013-0391-6 · Cited by 44
“Funnel plots were created using inverse standard errors and average effect sizes, and Egger tests (Egger et al. 1997) were conducted to assess for potential publication bias (Rothstein 2008).”
MethodBeelmann, A. (2014) · Zeitschrift für Erziehungswissenschaft · doi:10.1007/s11618-014-0509-2 · Cited by 46
“Rothstein (2008) und Kollegen (Rothstein et al. 2005) haben sich daher eingehend mit Publikationsverzerrungen auseinander gesetzt.”
BackgroundInterpreting t-Statistics Under Publication Bias: Rough Rules of Thumb
Winship, C., & Zhuo, X. (2018) · Journal of Quantitative Criminology · doi:10.1007/s10940-018-9387-8 · Cited by 46
Zhang, J., Liu, Y., & Hu, T. (2019) · Early Child Development and Care · doi:10.1080/03004430.2019.1679127 · Cited by 42
Chin, M., Robson, D. A., Woodbridge, H., & Hawes, D. J. (2025) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-024-00512-4 · Cited by 42
“Publication bias can be due to a number of reasons, for example, reporting biases and editorial decisions favoring larger statistically significant studies (Egger et al., 1997; Rothstein, 2008).”
Sample size, effect size, and statistical power: a replication study of Weisburd’s paradox
Nelson, M. S., Wooditch, A., & Dario, L. M. (2014) · Journal of Experimental Criminology · doi:10.1007/s11292-014-9212-9 · Cited by 44
“This results in a publication bias, where negative or null findings are often unreported (Chan and Altman 2005; Chan et al. 2004; Dickersin 2005; Rothstein 2008), such as when a researcher abstains from submitting null findings for publication or an editor showing 144 M.”
BackgroundInfluentialDeslandes, C., Kaufmann, L. M., & Anderson, J. (2023) · International Journal of Intercultural Relations · doi:10.1016/j.ijintrel.2023.101928 · Cited by 38
“While the risk of publication bias must be noted for these analyses, we endeavoured to include all relevant literature both published and unpublished within each analysis, and the methods used to examine publication bias are not without limitations themselves (Rothstein, 2008).”
McDaniel, J., Brady, N. C., & Warren, S. F. (2021) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-021-05331-y · Cited by 38
Motivations behind problematic short video use: A three-level meta-analysis
Chen, Y., Zhang, W., Zhong, N., & Zhao, M. (2024) · Telematics and Informatics · doi:10.1016/j.tele.2024.102196 · Cited by 35
Roest, J. J., de Poll, M. J. W.-V., van der Helm, P. G., Stams, G. J. J. M., & Hoeve, M. (2022) · Research on Child and Adolescent Psychopathology · doi:10.1007/s10802-022-00986-2 · Cited by 38
“The tendency of selected publication by journals to accept papers that report significant associations – referred to as publication bias – can influence the overall estimates of effect sizes in a meta-analysis and therefore its conclusions (Rosenthal, 1979, 1991; Rothstein, 2008).”
BackgroundHow accurately do teachers’ judge students? Re-analysis of Hoge and Coladarci (1989) meta-analysis
Kaufmann, E. (2020) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2020.101902 · Cited by 22
Roman, C. G. (2021) · Journal of Aggression Conflict and Peace Research · doi:10.1108/jacpr-02-2021-0576 · Cited by 33
Wong‐Anuchit, C., Chantamit-O-Pas, C., Schneider, J. K., & Mills, A. C. (2018) · Journal of the American Psychiatric Nurses Association · doi:10.1177/1078390318761790 · Cited by 33
“We examined publication bias using the funnel plot, Egger’s regression intercept, Orwin’s fail-safe N, and Begg Mazumdar rank correlation test (Borenstein et al., 2009; Rothstein, 2008).”
MethodResultInfluentialKaufmann, E., Reips, U.-D., & Maag Merki, K. (2016) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000251 · Cited by 31
“This is an important point, as there has been controversy and critical discussion on the attempts and methods of estimating publication bias (see Rothstein, 2008).”
MethodWeisburd, D. L., Telep, C. W., Hinkle, J. C., & Eck, J. E. (2015) · SSRN Electronic Journal · Cited by 31
Type M Error Might Explain Weisburd’s Paradox
Gelman, A., Skarðhamar, T., & Aaltonen, M. (2017) · Journal of Quantitative Criminology · doi:10.1007/s10940-017-9374-5 · Cited by 32
“One meta-analysis of drug-court studies has concluded that there are indications of publication bias (Rothstein 2008).”
BackgroundDisorder policing to reduce crime: A systematic review
Braga, A. A., Welsh, B. C., & Schnell, C. (2019) · Campbell Systematic Reviews · doi:10.1002/cl2.1050 · Cited by 39
“Publication bias presents a serious challenge to any review of evaluation studies (Rothstein, 2008).”
A Critical Meta-Analysis of Lens Model Studies in Human Judgment and Decision-Making
Kaufmann, E., Reips, U., & Wittmann, W. W. (2013) · PLoS ONE · doi:10.1371/journal.pone.0083528 · Cited by 29
“Indeed, Rothstein [73] asserted that disentangling the effects of publication bias from other sources of heterogeneity can be difficult.”
BackgroundThe risk factors for avian influenza on poultry farms: A meta-analysis
Wang, Y., Li, P., Wu, Y., Sun, X., Yu, K., Yu, C., & Qin, A. (2014) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2014.06.008 · Cited by 28
“…and Lee, 2000); meanwhile, using the fail-safe N method, the number of studies needed to reverse the effect can be calculated quantitatively, but this method cannot calculate the number of studies that are needed to nullify the effect, and it has low power (Li and Wang, 2008; Rothstein, 2008).”
MethodBeelmann, A., Arnold, L. S., & Schulz, S. (2020) · International Journal of Psychology · doi:10.1002/ijop.12725 · Cited by 31
Geng, X., & Su, Y.-S. (2024) · International Journal of Science and Mathematics Education · doi:10.1007/s10763-024-10484-0 · Cited by 31
Liu, Y., Wang, W., & Xu, E. (2025) · SAGE Open · doi:10.1177/21582440251336707 · Cited by 34
“Research studies demonstrating statistically significant findings are more prone to being published and referenced (Rothstein, 2008 ).”
Maneuvrier, A., Nguyen, N.-D.-T., & Renaud, P. (2023) · Frontiers in Virtual Reality · doi:10.3389/frvir.2023.1307925 · Cited by 30
Ending publication bias: A values-based approach to surface null and negative results
Curry, S., Mercado-Lara, E., Arechavala‐Gomeza, V., Begley, C. G., Bernard, C., Bernard, R., Bertuzzi, S., Bhalla, N., Bowers, D., Brod, S., et al. (2025) · PLoS Biology · doi:10.1371/journal.pbio.3003368 · Cited by 26
“This bias persists despite decades of warnings against its consequences [3–8].”
Benfer, N., Spitzer, E. G., & Bardeen, J. R. (2021) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2021.102360 · Cited by 31
The effectiveness of EQUIP on sociomoral development and recidivism reduction: A meta-analytic study
van Stam, M. A., van der Schuur, W. A., Tserkezis, S., van Vugt, E., Asscher, J. J., Gibbs, J. C., & Stams, G. J. J. M. (2014) · Children and Youth Services Review · doi:10.1016/j.childyouth.2014.01.002 · Cited by 49
And We Wonder Why Criminology Is Sometimes Considered Irrelevant in Real‐World Policy Conversations
Braga, A. A., & Apel, R. J. (2016) · Criminology & Public Policy · doi:10.1111/1745-9133.12235 · Cited by 27
A retrospective analysis of dissemination biases in the brief alcohol intervention literature.
Tanner-Smith, E. E., & Polanin, J. R. (2015) · Psychology of Addictive Behaviors · doi:10.1037/adb0000014 · Cited by 26
“As other social scientists have suggested, preventing publication bias in the first place may be the best approach to deal with it (Rothstein, 2008).”
Background
Metadata
- Authors
- Rothstein, H. R.
- Affiliations
- Baruch College (US)
- Corresponding author
- Rothstein, Hannah R.
- ORCID
- Rothstein, Hannah R.
- Venue
- Journal of Experimental Criminology
- Published
- 2007-12-19
- Volume
- 4
- Issue
- 1
- Pages
- 61-81
- ISSN
- 1572-8315 (ISSN-L)
- 1573-3750
- OpenAlex
- W2010026440
- Cited by
- 216
- References
- 51
- Citations by year
- 2026: 11 · 2025: 23 · 2024: 23 · 2023: 12 · 2022: 20 · 2021: 10 · 2020: 10 · 2019: 13 · 2018: 20 · 2017: 11 · 2016: 9 · 2015: 8 · 2014: 16 · 2013: 4 · 2012: 9
- FWCI
- 4.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Journal of Experimental Criminology (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- publication bias, trim and fill, cumulative meta-analysis, effect size estimation, drug courts, recidivism, criminology research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.