Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 334

The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.

Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis

Open access
van Aert, Robbie C. M., Wicherts, Jelte M., van Assen, Marcel A. L. M.·2019·PLOS ONE
DOI 10.1371/journal.pone.0215052
Open this paper in CoChat — free account

Abstract

Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-known. We examined evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and 499 systematic reviews from the Cochrane Database of Systematic Reviews (CDSR; representing meta-analyses from medicine). Publication bias was assessed on all homogeneous subsets (3.8% of all subsets of meta-analyses published in Psychological Bulletin) of primary studies included in meta-analyses, because publication bias methods do not have good statistical properties if the true effect size is heterogeneous. Publication bias tests did not reveal evidence for bias in the homogeneous subsets. Overestimation was minimal but statistically significant, providing evidence of publication bias that appeared to be similar in both fields. However, a Monte-Carlo simulation study revealed that the creation of homogeneous subsets resulted in challenging conditions for publication bias methods since the number of effect sizes in a subset was rather small (median number of effect sizes equaled 6). Our findings are in line with, in its most extreme case, publication bias ranging from no bias until only 5% statistically nonsignificant effect sizes being published. These and other findings, in combination with the small percentages of statistically significant primary effect sizes (28.9% and 18.9% for subsets published in Psychological Bulletin and CDSR), led to the conclusion that evidence for publication bias in the studied homogeneous subsets is weak, but suggestive of mild publication bias in both psychology and medicine.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0215052

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 51

  1. DM Lane, 1978, Estimating effect size: Bias resulting from the significance criterion in editorial decisions., British Journal of Mathematical & Statistical Psychology, 31, 107, 10.1111/j.2044-8317.1978.tb00578.x · DOI
  2. MB Nuijten, 2015, The replication paradox: Combining studies can decrease accuracy of effect size estimates, Review of General Psychology, 19, 172, 10.1037/gpr0000034 · DOI
  3. TD Sterling, 1995, Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa, The American Statistician, 49, 108, 10.1080/00031305.1995.10476125 · DOI · Paper page
  4. A Coursol, 1986, Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Professional Psychology, Research and Practice, 17, 136
  5. A Franco, 2016, Underreporting in psychology experiments: Evidence from a study registry, Social Psychological and Personality Science, 7, 8, 10.1177/1948550615598377 · DOI
  6. CHJ Hartgerink, 2015, Distributions of p-values smaller than .05 in psychology: What is going on?, PeerJ., 4, e1935, 10.7717/peerj.1935 · DOI
  7. CHJ Hartgerink, 2017, Investigating the robustness of widespread p-hacking, PeerJ, 5
  8. A Brodeur, 2016, Star Wars: The empirics strike back., American Economic Journal: Applied Economics, 8, 1
  9. M. Krawczyk, 2015, The search for significance: A few peculiarities in the distribution of p values in experimental psychology literature, PLOS ONE, 10, e0127872, 10.1371/journal.pone.0127872 · DOI
  10. K Auspurg, 2014, Ausmaß und Risikofaktoren des Publication Bias in der deutschen Soziologie, KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie, 66, 549, 10.1007/s11577-014-0284-3 · DOI
  11. S Hopewell, 2005, Grey literature and systematic reviews, Publication bias in meta-analysis: Prevention, assessment and adjustments, 99
  12. GC Banks, 2012, Publication bias: The antagonist of meta-analytic reviews and effective policymaking, Educational Evaluation and Policy Analysis, 34, 259, 10.3102/0162373712446144 · DOI
  13. ZG Aytug, 2012, Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses, Organizational Research Methods, 15, 103, 10.1177/1094428111403495 · DOI
  14. GC Banks, 2012, Publication bias: A call for improved meta-analytic practice in the organizational sciences, International Journal of Selection and Assessment, 20, 182, 10.1111/j.1468-2389.2012.00591.x · DOI
  15. S Kepes, 2012, Publication bias in the organizational sciences, Organizational Research Methods, 15, 624, 10.1177/1094428112452760 · DOI · Paper page
  16. KM Coburn, 2015, Publication bias as a function of study characteristics, Psychological methods, 20, 310, 10.1037/met0000046 · DOI
  17. CJ Ferguson, 2012, Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses, Psychological Methods, 17, 120, 10.1037/a0024445 · DOI · Paper page
  18. G Jürgens, 2004, Effects of low sodium diet versus high sodium diet on blood pressure, renin, aldosterone, catecholamines, cholesterols, and triglyceride, Cochrane Database of Systematic Reviews, 10.1002/14651858.CD004022.pub2 · DOI
  19. M Egger, 1997, Bias in meta-analysis detected by a simple, graphical test, British Medical Journal, 315, 629, 10.1136/bmj.315.7109.629 · DOI · Paper page
  20. N Terrin, 2005, In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias, Journal of Clinical Epidemiology, 58, 894, 10.1016/j.jclinepi.2005.01.006 · DOI · Paper page
  21. JP Ioannidis, 2007, An exploratory test for an excess of significant findings, Clinical Trials, 4, 245, 10.1177/1740774507079441 · DOI · Paper page
  22. Francis G. Replication, 2013, statistical consistency, and publication bias, Journal of Mathematical Psychology, 57, 153, 10.1016/j.jmp.2013.02.003 · DOI
  23. WR Reed, 2015, A Monte Carlo analysis of alternative meta-analysis estimators in the presence of publication bias, Economics: The Open-Access, Open-Assessment E-Journal
  24. N Alinaghi, 2018, Meta‐analysis and publication bias: How well does the FAT‐PET‐PEESE procedure work?, Research Synthesis Methods
  25. TD Stanley, 2010, Could it be better to discard 90% of the data? A statistical paradox, The American Statistician, 64, 70, 10.1198/tast.2009.08205 · DOI · Paper page
  26. JP Ioannidis, 2006, Extreme between-study homogeneity in meta-analyses could offer useful insights, Journal of Clinical Epidemiology, 59, 1023, 10.1016/j.jclinepi.2006.02.013 · DOI
  27. JPT Higgins, 2008, Commentary: Heterogeneity in meta-analysis should be expected and appropriately quantified, International Journal of Epidemiology, 37, 1158, 10.1093/ije/dyn204 · DOI · Paper page
  28. KM Rhodes, 2015, Predictive distributions were developed for the extent of heterogeneity in meta-analyses of continuous outcome data, Journal of Clinical Epidemiology, 68, 52, 10.1016/j.jclinepi.2014.08.012 · DOI · Paper page
  29. HR Rothstein, 2012, Publication bias in psychological science: Comment on Ferguson and Brannick (2012)., Psychological Methods, 17, 129, 10.1037/a0027128 · DOI
  30. S Iyengar, 1988, Selection models and the file drawer problem, Statistical Science, 3, 109, 10.1214/ss/1177013012 · DOI
  31. KBG Dear, 1992, An approach for assessing publication bias prior to performing a meta-analysis, Statistical Science, 7, 237, 10.1214/ss/1177011363 · DOI
  32. JL Vevea, 1995, A general linear model for estimating effect size in the presence of publication bias., Psychometrika, 60, 419, 10.1007/BF02294384 · DOI · Paper page
  33. JB Copas, 1999, What works?: Selectivity models and meta-analysis, Journal of the Royal Statistical Society Series A, 162, 95, 10.1111/1467-985X.00123 · DOI
  34. M Guan, 2015, A Bayesian approach to mitigation of publication bias, Psychonomic Bulletin and Review
  35. S Iyengar, 1988, Selection models and the file drawer problem, Rejoinder. Statistical Science, 3, 133, 10.1214/ss/1177013019 · DOI
  36. W. Viechtbauer, 2007, Approximate confidence intervals for standardized effect sizes in the two-independent and two-dependent samples design, Journal of Educational and Behavioral Statistics, 32, 39, 10.3102/1076998606298034 · DOI
  37. JPT Higgins, 2003, Measuring inconsistency in meta-analyses, British Medical Journal, 327, 557, 10.1136/bmj.327.7414.557 · DOI · Paper page
  38. G Rücker, 2008, Undue reliance on I2 in assessing heterogeneity may mislead., BMC Medical Research Methodology, 8, 79, 10.1186/1471-2288-8-79 · DOI · Paper page
  39. JP Ioannidis, 2007, Uncertainty in heterogeneity estimates in meta-analyses, BMJ, 335, 914, 10.1136/bmj.39343.408449.80 · DOI · Paper page
  40. J. Loevinger, 1948, The technic of homogeneous tests compared with some aspects of scale analysis and factor analysis, Psychological Bulletin, 45, 507, 10.1037/h0055827 · DOI
  41. RC Paule, 1982, Consensus values and weighting factors, Journal of Research of the National Bureau of Standards, 87, 377, 10.6028/jres.087.022 · DOI · Paper page
  42. D. Jackson, 2006, The implications of publication bias for meta-analysis' other parameter, Statistics in Medicine, 25, 2911, 10.1002/sim.2293 · DOI
  43. HEM Augusteijn, The effect of publication bias on the Q-test and assessment of heterogeneity, Psychological Methods
  44. R. Koenker, 2005, Quantile regression, 10.1017/CBO9780511754098 · DOI · Paper page
  45. 2018, R: A language and environment for statistical computing, R: A language and environment for statistical computing
  46. RCM van Aert, 2018, Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication, Behavior Research Methods, 50, 1515, 10.3758/s13428-017-0967-6 · DOI
  47. RCM van Aert, Correcting for publication bias in a meta-analysis with the p-uniform* method, 10.31222/osf.io/zqjr9 · DOI
  48. D Fanelli, 2013, US studies may overestimate effect sizes in softer research, Proc Natl Acad Sci USA, 110, 15031, 10.1073/pnas.1302997110 · DOI
  49. RJ Hardy, 1998, Detecting and describing heterogeneity in meta-analysis, Statistics in Medicine, 17, 841, 10.1002/(SICI)1097-0258(19980430)17:8<841::AID-SIM781>3.0.CO;2-D · DOI · Paper page
  50. W. Viechtbauer, 2007, Hypothesis tests for population heterogeneity in meta-analysis, British Journal of Mathematical and Statistical Psychology, 60, 29, 10.1348/000711005X64042 · DOI
  51. M Citkowicz, 2017, A parsimonious weight function for modeling publication bias, Psychological Methods, 22, 28, 10.1037/met0000119 · DOI

Cited by

See all 334

Metadata

Authors
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M.
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2019-04-12
Volume
14
Issue
4
Pages
e0215052
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0215052
OpenAlex
W2788406885
Cited by
334
References
56
FWCI
27.21
Type
journal article
Language
en
Open access
gold
Licence
creativecommons.org/licenses/by/4.0/ (vor)
Funders
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (406-13-050)
  • Berkeley Initiative for Transparency in the Social Sciences and the Laura and John Arnold Foundation
  • The European Research Council (726361 (IMPROVE))

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.