The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research
Open accessAbstract
BACKGROUND: A meta-analysis as part of a systematic review aims to provide a thorough, comprehensive and unbiased statistical summary of data from the literature. However, relevant study results could be missing from a meta-analysis because of selective publication and inadequate dissemination. If missing outcome data differ systematically from published ones, a meta-analysis will be biased with an inaccurate assessment of the intervention effect. As part of the EU-funded OPEN project (www.open-project.eu) we conducted a systematic review that assessed whether the inclusion of data that were not published at all and/or published only in the grey literature influences pooled effect estimates in meta-analyses and leads to different interpretation. METHODS AND FINDINGS: Systematic review of published literature (methodological research projects). Four bibliographic databases were searched up to February 2016 without restriction of publication year or language. Methodological research projects were considered eligible for inclusion if they reviewed a cohort of meta-analyses which (i) compared pooled effect estimates of meta-analyses of health care interventions according to publication status of data or (ii) examined whether the inclusion of unpublished or grey literature data impacts the result of a meta-analysis. Seven methodological research projects including 187 meta-analyses comparing pooled treatment effect estimates according to different publication status were identified. Two research projects showed that published data showed larger pooled treatment effects in favour of the intervention than unpublished or grey literature data (Ratio of ORs 1.15, 95% CI 1.04-1.28 and 1.34, 95% CI 1.09-1.66). In the remaining research projects pooled effect estimates and/or overall findings were not significantly changed by the inclusion of unpublished and/or grey literature data. The precision of the pooled estimate was increased with narrower 95% confidence interval. CONCLUSIONS: Although we may anticipate that systematic reviews and meta-analyses not including unpublished or grey literature study results are likely to overestimate the treatment effects, current empirical research shows that this is only the case in a minority of reviews. Therefore, currently, a meta-analyst should particularly consider time, effort and costs when adding such data to their analysis. Future research is needed to identify which reviews may benefit most from including unpublished or grey data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As suggested by Schmucker et al. (2017), although studies excluding grey literature might be likely to overestimate the treatment effects, current empirical research shows that this is the case only in a minority of reviews; further, publication bias might particularly affect specific research fields where there is need of publishing positive results more rapidly. In addition, grey literature is”
From The relationship between mind wandering and reading comprehension: A meta-analysis (2023) · cites this paper as Schmucker et al. (2017)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709
Self-citationSelective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435
Industry sponsorship and research outcome
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub3 · Cited by 1,697
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181 · cited 1×
“## Egger et al 2003 also separated effect estimates for selected medical fields.”
BackgroundMcAuley, L., Pham, B., Tugwell, P., & Moher, D. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02786-0 · Cited by 812
Industry sponsorship and research outcome
Lundh, A., Sismondo, S., Lexchin, J., Busuioc, O. A., & Bero, L. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub2 · Cited by 605
Grey literature in meta-analyses of randomized trials of health care interventions
Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000010.pub3 · Cited by 598
Publication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427
Golder, S., Loke, Y. K., Wright, K., & Norman, G. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002127 · Cited by 277
Schmucker, C., Schell, L. K., Portalupi, S., Oeller, P., Cabrera, L., Bassler, D., Schwarzer, G., Scherer, R. W., Antes, G., von Elm, E., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114023 · Cited by 269
Self-citationShould Unpublished Data Be Included in Meta-analyses?: Current Convictions and Controversies
Cook, D. J. (1993) · JAMA · doi:10.1001/jama.1993.03500210049030 · Cited by 259
Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses
Hart, B., Lundh, A., & Bero, L. (2012) · BMJ · doi:10.1136/bmj.d7202 · Cited by 255
Driessen, E., Hollon, S. D., Bockting, C. L. H., Cuijpers, P., & Turner, E. H. (2015) · PLOS ONE · doi:10.1371/journal.pone.0137864 · Cited by 208
Potential Reporting Bias in fMRI Studies of the Brain
David, S. P., Ware, J. J., Chu, I. M., Loftus, P. D., Fusar-Poli, P., Radua, J., Munafò, M. R., & Ioannidis, J. P. A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0070104 · Cited by 130
Mueller, K. F., Meerpohl, J. J., Briel, M., Antes, G., von Elm, E., Lang, B., Motschall, E., Schwarzer, G., & Bassler, D. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.015 · Cited by 48 · cited 1×
“[2] [34–36] However, the empirical research work of Mueller et al 2016 concluded that it remains difficult to advise which method should be used as they are all limited and only few methods have been validated in empirical evaluations using unpublished studies obtained from regulators (e.g., FDA…”
MethodSelf-citation
Cited by
Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., & Garside, R. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0545-3 · Cited by 561
Digital interventions for the treatment of depression: a meta-analytic review
Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., Pulkki-Råback, L., Baumeister, H., & Sander, L. B. (2021) · Psychological Bulletin · doi:10.1037/bul0000334 · Cited by 390
Sibhatu, K. T., & Qaim, M. (2018) · Food Policy · doi:10.1016/j.foodpol.2018.04.013 · Cited by 366
The effects of green exercise on physical and mental wellbeing: a systematic review
Lahart, I., Darcy, P., Gidlow, C., & Calogiuri, G. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16081352 · Cited by 353
Assessing risk of bias due to missing results in a synthesis
Page, M. J., Higgins, J. P., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch13 · Cited by 402
Metadata
- Authors
- Schmucker, C. M., Blümle, A., Schell, L. K., Schwarzer, G., Oeller, P., Cabrera, L., von Elm, E., Briel, M., Meerpohl, J. J., & on behalf of the OPEN consortium
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-04-25
- Volume
- 12
- Issue
- 4
- Pages
- e0176210
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2609678219
- Cited by
- 227
- References
- 16
- FWCI
- 4.61
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.