Publication bias as a threat to the validity of meta-analytic results
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Publication bias presents a strong challenge to any review of evaluation studies (Rothstein, 2008).”
From Problem-oriented policing for reducing crime and disorder: An updated systematic review and meta-analysis (2020) · cites this paper as Rothstein, 2008
In the citing paper
“Trim‐and‐fill procedures do suffer from some well‐known limitations that could result in the underestimation or overestimation of publication bias (Rothstein, 2008; Simonsohn, Nelson, & Simmons, 2014).”
From Hot spots policing of small geographic areas effects on crime (2019) · cites this paper as Rothstein, 2008
In the citing paper
“Following the advice of Rothstein (2008), we apply two of the conventional methods that address publication bias.”
From A Meta-analysis of Attachment to Parents and Delinquency (2012) · cites this paper as Rothstein 2008
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“( Rosenthal 1979 ). He was concerned that metaanalytic results could be wrong, due to non-significant studies that remained in researchers’ file drawers.”
ResultPerceived organizational support: a review of the literature.
Rhoades, L., & Eisenberger, R. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.4.698 · Cited by 7,592 · cited 1×
“Mullen et al. (2001) proposed Rosenthal’s guideline as a formal rule, and the authors of several recent psychology meta-analyses (Del Vecchio and O’Leary 2004; Rhoades and Eisenberger 2002 ) have used this formula to assess the results of their file drawer analyses.”
MethodJournal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955
Systematic Reviews in Health Care : Meta-Analysis in Context
Egger, M., Smith, G. D., & Altman, D. G. (2000) · Cited by 2,401
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282
Investigating and dealing with publication and other biases in meta-analysis
Sterne, J. A. C., Egger, M., & Smith, G. D. (2001) · BMJ · doi:10.1136/bmj.323.7304.101 · Cited by 2,225
A Fail-Safe N for Effect Size in Meta-Analysis
Orwin, R. G. (1983) · Journal of Educational Statistics · doi:10.2307/1164923 · Cited by 2,064
The Handbook of Research Synthesis.
Light, R. J., Cooper, H., & Hedges, L. V. (1994) · Journal of the American Statistical Association · doi:10.2307/2291021 · Cited by 1,868
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 1×
“In addition to the evidence that whole studies are selectively missing are demonstrations by Chan and colleagues ( Chan et al. 2004; Chan and Altman 2005) that reporting of outcomes within studies is often incomplete and biased, due to selective withholding of statistically nonsignificant results.”
BackgroundSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
Cumulative Meta-Analysis of Therapeutic Trials for Myocardial Infarction
Lau, J., Antman, E. M., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992) · New England Journal of Medicine · doi:10.1056/nejm199207233270406 · Cited by 1,140
Landenberger, N. A., & Lipsey, M. W. (2005) · Journal of Experimental Criminology · doi:10.1007/s11292-005-3541-7 · Cited by 850
Regression Methods to Detect Publication and Other Bias in Meta‐Analysis
Sterne, J. A. C., & Egger, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch6 · Cited by 826
Publication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814 · cited 1×
“errors) of these effects ( Begg and Berlin 1988; Begg and Mazumdar 1994; Begg 1994).”
BackgroundAdjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809 · cited 1×
“Simulations by Terrin et al. (2003) have shown that when trim and fill is applied to heterogeneous data sets, it can adjust for publication bias when none actually exists.”
BackgroundThe effectiveness of treatment for sexual offenders: a comprehensive meta-analysis
L�sel, F., & Schmucker, M. (2005) · Journal of Experimental Criminology · doi:10.1007/s11292-004-6466-7 · Cited by 806 · cited 1×
“Illescas et al. 2001; Losel and Schmucker 2005 ) or they have conducted a file-drawer analysis and calculated a failsafe N (cf.”
BackgroundChan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613
Duval, S. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch8 · Cited by 496
A meta-analytic review of the effects of high stress on eyewitness memory
Deffenbacher, K. A., Bornstein, B. H., Penrod, S. D., & McGorty, E. K. (2004) · Law and Human Behavior · doi:10.1007/s10979-004-0565-x · Cited by 489 · cited 1×
“Deffenbacher et al. 2004; Dreznick 2003; Pratt et al. 2002).”
BackgroundArnqvist, G. (1998) · The Quarterly Review of Biology · doi:10.1086/420179 · Cited by 475
A systematic review of drug court effects on recidivism
Wilson, D. B., Mitchell, O., & MacKenzie, D. L. (2007) · Journal of Experimental Criminology · doi:10.1007/s11292-006-9019-4 · Cited by 456
Cooper, H. M. (1979) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.37.1.131 · Cited by 412
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359 · cited 1×
“The third type asks how the results would change after they had been adjusted for the possible effects of publication bias under some explicit model of publication selection., This type includes Duval and Tweedie’s trim-and-fill method (Duval 2005; Duval and Tweedie 2000a, b), Hedges and Vevea’s general selection mode…”
MethodSterne, J. A. C., Becker, B. J., & Egger, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch5 · Cited by 341
Cures That Harm: Unanticipated Outcomes of Crime Prevention Programs
McCord, J. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716202250781 · Cited by 338 · cited 1×
“McCord (2003) has done an excellent job highlighting the fact that interventions in criminal justice intended to help may sometimes harm; to the extent that publication bias exists in systematic reviews in this area, this problem may be more widespread than previously thought.”
BackgroundZhang, J., Ding, E. L., & Song, Y. (2006) · JAMA · doi:10.1001/jama.296.13.jrv60015 · Cited by 332 · cited 1×
“Merck maintained that it recalled Vioxx as soon as the data indicated a high prevalence of adverse effects, but it became embroiled in controversy as others claimed that Merck had concealed adverse event data for years (Wall Street Journal, November 1, 2004), a view supported by a meta-analysis of all available random…”
ResultEffectiveness of anger treatments for specific anger problems: A meta-analytic review
Del Vecchio, T., & O'Leary, K. D. (2004) · Clinical Psychology Review · doi:10.1016/j.cpr.2003.09.006 · Cited by 302
Sterne, J. A. C., Bradburn, M. J., & Egger, M. (2001) · Systematic Reviews in Health Care · doi:10.1002/9780470693926.ch18 · Cited by 297
Publication Bias: Recognizing the Problem, Understanding Its Origins and Scope, and Preventing Harm
Dickersin, K. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch2 · Cited by 291
Those confounded moderators in meta-analysis: good, bad, and ugly
Lipsey, M. W. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716202250791 · Cited by 283 · cited 1×
“This is likely to be the case in most criminology metaanalyses (cf. Lipsey 2003; Weisburd et al. 1993).”
BackgroundA sensitivity analysis for publication bias in systematic reviews
Copas, J. B., & Shi, J. Q. (2001) · Statistical Methods in Medical Research · doi:10.1177/096228020101000402 · Cited by 259 · cited 1×
“The third type asks how the results would change after they had been adjusted for the possible effects of publication bias under some explicit model of publication selection., This type includes Duval and Tweedie’s trim-and-fill method (Duval 2005; Duval and Tweedie 2000a, b), Hedges and Vevea’s general selection mode…”
MethodHot spots policing and crime prevention: A systematic review of randomized controlled trials
Braga, A. A. (2005) · Journal of Experimental Criminology · doi:10.1007/s11292-005-8133-z · Cited by 252 · cited 2×
“For example, although Braga (2005) , in his meta-analysis of hotspots policing, found a relationship between sample size and effect size that he suggested was a function of implementation problems in two of the included studies [those from the Repeat Call Address Policing (RECAP) experiment], he did not empirically te…”
BackgroundKrleža-Jerić, K., Chan, A.-W., Dickersin, K., Sim, I., Grimshaw, J., & Gluud, C. (2005) · BMJ · doi:10.1136/bmj.330.7497.956 · Cited by 236
Cumulative Meta-Analysis: A Consideration of Indicators of Sufficiency and Stability
Mullen, B., Muellerleile, P., & Bryant, B. (2001) · Personality and Social Psychology Bulletin · doi:10.1177/01461672012711006 · Cited by 234 · cited 1×
“Mullen et al. (2001) proposed Rosenthal’s guideline as a formal rule, and the authors of several recent psychology meta-analyses (Del Vecchio and O’Leary 2004; Rhoades and Eisenberger 2002) have used this formula to assess the results of their file drawer analyses.”
MethodHedges, L. V., & Vevea, J. L. (1996) · Journal of Educational and Behavioral Statistics · doi:10.2307/1165338 · Cited by 221 · cited 1×
“The third type asks how the results would change after they had been adjusted for the possible effects of publication bias under some explicit model of publication selection., This type includes Duval and Tweedie’s trim-and-fill method (Duval 2005; Duval and Tweedie 2000a, b), Hedges and Vevea’s general selection mode…”
MethodBorenstein, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch11 · Cited by 178 · cited 3×
“If, however, the point estimate does shift when the smaller studies are added, “there is at least a prima facie case for bias” ( Borenstein 2005 ).”
BackgroundMethodInfluentialPratt, T. C., Cullen, F. T., Blevins, K. R., Daigle, L., & Unnever, J. D. (2002) · International Journal of Police Science & Management · doi:10.1350/ijps.4.4.344.10873 · Cited by 173 · cited 1×
“Deffenbacher et al. 2004; Dreznick 2003; Pratt et al. 2002 ).”
BackgroundWhat Works?: Selectivity Models and Meta-Analysis
Copas, J. (1999) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/1467-985x.00123 · Cited by 168 · cited 1×
“The third type asks how the results would change after they had been adjusted for the possible effects of publication bias under some explicit model of publication selection., This type includes Duval and Tweedie’s trim-and-fill method (Duval 2005; Duval and Tweedie 2000a, b), Hedges and Vevea’s general selection mode…”
MethodDesign Sensitivity in Criminal Justice Experiments
Weisburd, D., Petrosino, A., & Mason, G. (1993) · Crime and Justice · doi:10.1086/449216 · Cited by 124 · cited 1×
“This is likely to be the case in most criminology metaanalyses (cf. Lipsey 2003; Weisburd et al. 1993 ).”
BackgroundThe Internet Journal of Epidemiology
(2014) · The Internet Journal of Epidemiology · doi:10.5580/ije · Cited by 107
Genovés, V. J. G., Illescas, S. R., & Sánchez‐Meca, J. (2001) · Psychology in Spain · Cited by 52
Heterosocial competence of rapists and child molesters: A meta‐analysis
Dreznick, M. T. (2003) · The Journal of Sex Research · doi:10.1080/00224490309552178 · Cited by 51 · cited 1×
“Deffenbacher et al. 2004; Dreznick 2003; Pratt et al. 2002).”
BackgroundA Simple Method For Detecting And Adjusting Meta-Analyses For Publication Bias
Richy, F., & Reginster, J.-Y. (2006) · The Internet Journal of Epidemiology · doi:10.5580/1980 · Cited by 4
A simple method for detecting and adjusting meta-analyses for publication Bias
Richy, F., & Reginster, J.-Y. (2006) · ORBi (University of Liège) · Cited by 2
Research works printed but not linked: 2
- Ioannidis, J. (2005). Differentiating biases from genuine heterogeneity. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.). Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.
- Wall Street Journal, (2004, November 1). E-Mails Suggest Merck Knew Vioxx’s Dangers at Early Stage.
Cited by
Emotion-regulation strategies across psychopathology: A meta-analytic review
Aldao, A., Nolen-Hoeksema, S., & Schweizer, S. (2010) · Clinical Psychology Review · doi:10.1016/j.cpr.2009.11.004 · Cited by 6,686
Dewald, J. F., Meijer, A. M., Oort, F. J., Kerkhof, G. A., & Bögels, S. M. (2010) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2009.10.004 · Cited by 1,526
Gender differences in emotion expression in children: A meta-analytic review.
Chaplin, T. M., & Aldao, A. (2013) · Psychological Bulletin · doi:10.1037/a0030737 · Cited by 1,247
Emotion regulation strategies in depressive and anxiety symptoms in youth: A meta-analytic review
Schäfer, J. Ö., Naumann, E., Holmes, E. A., Tuschen-Caffier, B., & Samson, A. C. (2017) · Journal of Youth and Adolescence · doi:10.1007/s10964-016-0585-0 · Cited by 941
Relation between Emotion Regulation and Mental Health: A Meta-Analysis Review
Hu, T., Zhang, D., Wang, J., Mistry, R., Ran, G., & Wang, X. (2014) · Psychological Reports · doi:10.2466/03.20.pr0.114k22w4 · Cited by 633
“The meta-analysis was not likely to have been aff ected by publication bias ( Rothstein, 2008 ).”
Background
Metadata
- Authors
- Rothstein, H. R.
- Affiliations
- Baruch College (US)
- Corresponding author
- Rothstein, Hannah R.
- ORCID
- Rothstein, Hannah R.
- Venue
- Journal of Experimental Criminology
- Published
- 2007-12-19
- Volume
- 4
- Issue
- 1
- Pages
- 61-81
- ISSN
- 1572-8315 (ISSN-L)
- 1573-3750
- OpenAlex
- W2010026440
- Cited by
- 216
- References
- 51
- Citations by year
- 2026: 11 · 2025: 23 · 2024: 23 · 2023: 12 · 2022: 20 · 2021: 10 · 2020: 10 · 2019: 13 · 2018: 20 · 2017: 11 · 2016: 9 · 2015: 8 · 2014: 16 · 2013: 4 · 2012: 9
- FWCI
- 4.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Journal of Experimental Criminology (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- publication bias, trim and fill, cumulative meta-analysis, effect size estimation, drug courts, recidivism, criminology research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.