The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research
Open accessAbstract
BACKGROUND: A meta-analysis as part of a systematic review aims to provide a thorough, comprehensive and unbiased statistical summary of data from the literature. However, relevant study results could be missing from a meta-analysis because of selective publication and inadequate dissemination. If missing outcome data differ systematically from published ones, a meta-analysis will be biased with an inaccurate assessment of the intervention effect. As part of the EU-funded OPEN project (www.open-project.eu) we conducted a systematic review that assessed whether the inclusion of data that were not published at all and/or published only in the grey literature influences pooled effect estimates in meta-analyses and leads to different interpretation. METHODS AND FINDINGS: Systematic review of published literature (methodological research projects). Four bibliographic databases were searched up to February 2016 without restriction of publication year or language. Methodological research projects were considered eligible for inclusion if they reviewed a cohort of meta-analyses which (i) compared pooled effect estimates of meta-analyses of health care interventions according to publication status of data or (ii) examined whether the inclusion of unpublished or grey literature data impacts the result of a meta-analysis. Seven methodological research projects including 187 meta-analyses comparing pooled treatment effect estimates according to different publication status were identified. Two research projects showed that published data showed larger pooled treatment effects in favour of the intervention than unpublished or grey literature data (Ratio of ORs 1.15, 95% CI 1.04-1.28 and 1.34, 95% CI 1.09-1.66). In the remaining research projects pooled effect estimates and/or overall findings were not significantly changed by the inclusion of unpublished and/or grey literature data. The precision of the pooled estimate was increased with narrower 95% confidence interval. CONCLUSIONS: Although we may anticipate that systematic reviews and meta-analyses not including unpublished or grey literature study results are likely to overestimate the treatment effects, current empirical research shows that this is only the case in a minority of reviews. Therefore, currently, a meta-analyst should particularly consider time, effort and costs when adding such data to their analysis. Future research is needed to identify which reviews may benefit most from including unpublished or grey data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As suggested by Schmucker et al. (2017), although studies excluding grey literature might be likely to overestimate the treatment effects, current empirical research shows that this is the case only in a minority of reviews; further, publication bias might particularly affect specific research fields where there is need of publishing positive results more rapidly. In addition, grey literature is”
From The relationship between mind wandering and reading comprehension: A meta-analysis (2023) · cites this paper as Schmucker et al. (2017)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709
Self-citationSelective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435
Industry sponsorship and research outcome
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub3 · Cited by 1,697
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181 · cited 1×
“## Egger et al 2003 also separated effect estimates for selected medical fields.”
BackgroundMcAuley, L., Pham, B., Tugwell, P., & Moher, D. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02786-0 · Cited by 812
Industry sponsorship and research outcome
Lundh, A., Sismondo, S., Lexchin, J., Busuioc, O. A., & Bero, L. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub2 · Cited by 605
Grey literature in meta-analyses of randomized trials of health care interventions
Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000010.pub3 · Cited by 598
Publication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427
Golder, S., Loke, Y. K., Wright, K., & Norman, G. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002127 · Cited by 277
Cited by
Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., & Garside, R. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0545-3 · Cited by 561
Digital interventions for the treatment of depression: a meta-analytic review
Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., Pulkki-Råback, L., Baumeister, H., & Sander, L. B. (2021) · Psychological Bulletin · doi:10.1037/bul0000334 · Cited by 390
Sibhatu, K. T., & Qaim, M. (2018) · Food Policy · doi:10.1016/j.foodpol.2018.04.013 · Cited by 366
The effects of green exercise on physical and mental wellbeing: a systematic review
Lahart, I., Darcy, P., Gidlow, C., & Calogiuri, G. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16081352 · Cited by 353
Assessing risk of bias due to missing results in a synthesis
Page, M. J., Higgins, J. P., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch13 · Cited by 402
How should systematic reviewers handle conference abstracts? A view from the trenches
Scherer, R. W., & Saldanha, I. J. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1188-0 · Cited by 336
Attention-deficit/hyperactivity disorder and very preterm/very low birth weight: a meta-analysis
Franz, A. P., Bolat, G. U., Bolat, H., Matijasevich, A., Santos, I. S., Silveira, R. C., Procianoy, R. S., Rohde, L. A., & Moreira-Maia, C. R. (2018) · Pediatrics · doi:10.1542/peds.2017-1645 · Cited by 317
Full publication of results initially presented in abstracts
Scherer, R. W., Meerpohl, J. J., Pfeifer, N., Schmucker, C., Schwarzer, G., & von Elm, E. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000005.pub4 · Cited by 307
Self-citationIs there a gender gap? A meta-analysis of the gender differences in students' ICT literacy
Siddiq, F., & Scherer, R. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.03.007 · Cited by 297
“Second, we compared the effect sizes obtained from published studies and grey literature (Schmucker et al., 2017).”
ResultPage, M. J., Sterne, J. A. C., Higgins, J. P. T., & Egger, M. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1468 · Cited by 294
The cognitive benefits of learning computer programming: A meta-analysis of transfer effects.
Scherer, R., Siddiq, F., & Sánchez Viveros, B. (2019) · Journal of Educational Psychology · doi:10.1037/edu0000314 · Cited by 205
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
The use of commercial computerised cognitive games in older adults: a meta-analysis
Bonnechère, B., Langley, C., & Sahakian, B. J. (2020) · Scientific Reports · doi:10.1038/s41598-020-72281-3 · Cited by 126
The relationship between mind wandering and reading comprehension: A meta-analysis
Bonifacci, P., Viroli, C., Vassura, C., Colombini, E., & Desideri, L. (2023) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02141-w · Cited by 83
Mental health and psychological wellbeing in the early stages of doctoral study: a systematic review
Jackman, P. C., Jacobs, L., Hawkins, R. M., & Sisson, K. (2022) · European Journal of Higher Education · doi:10.1080/21568235.2021.1939752 · Cited by 76
Reeves, R., Curran, D., Gleeson, A., & Hanna, D. (2022) · Behavior Modification · doi:10.1177/0145445521991102 · Cited by 69
“However, recent studies have indicated that including unpublished literature has a minimal effect on effect sizes in metaanalysis (Hartling et al., 2017; Schmucker et al., 2017).”
BackgroundDechartres, A., Atal, I., Riveros, C., Meerpohl, J., & Ravaud, P. (2018) · Annals of Internal Medicine · doi:10.7326/m18-1517 · Cited by 59
Self-citationBashir, R., Bourgeois, F. T., & Dunn, A. G. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0518-3 · Cited by 47
Herbert, M. S., Dochat, C., Wooldridge, J. S., Materna, K., Blanco, B. H., Tynan, M., Lee, M. W., Gasperi, M., Camodeca, A., Harris, D., et al. (2022) · Behaviour Research and Therapy · doi:10.1016/j.brat.2021.103995 · Cited by 44
Karnjanapiboonwong, A., Anothaisintawee, T., Chaikledkaew, U., Dejthevaporn, C., Attia, J., & Thakkinstian, A. (2020) · BMC Cardiovascular Disorders · doi:10.1186/s12872-020-01736-2 · Cited by 27
A Meta-analysis of the Effect of Multimedia Technology on Creative Performance
Liu, M., Pang, W., Guo, J., & Zhang, Y. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-10981-1 · Cited by 25
Rehman, Y., Ferguson, H., Bozek, A., Blair, J., Allison, A., & Johnston, R. (2020) · Journal of Osteopathic Medicine · doi:10.7556/jaoa.2020.128 · Cited by 21
de Oliveira, T. M. D., Felício, D. C., Filho, J. E., Fonseca, D. S., Durigan, J. L. Q., & Malaguti, C. (2022) · Journal of Bodywork and Movement Therapies · doi:10.1016/j.jbmt.2022.03.010 · Cited by 19
Metadata
- Authors
- Schmucker, C. M., Blümle, A., Schell, L. K., Schwarzer, G., Oeller, P., Cabrera, L., von Elm, E., Briel, M., Meerpohl, J. J., & on behalf of the OPEN consortium
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-04-25
- Volume
- 12
- Issue
- 4
- Pages
- e0176210
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2609678219
- Cited by
- 227
- References
- 16
- FWCI
- 4.61
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.