CoChat

Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates: empirical study

Open access

The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.

Abstract

Objective: To explore whether systematic reviewers selectively include trial effect estimates in meta-analyses when multiple are available, and what impact this may have on meta-analytic effects. Design: Cross-sectional study. Data sources: We randomly selected systematic reviews of interventions from 2 clinical specialties published between January 2010 and 2012. The first presented meta-analysis of a continuous outcome in each review was selected (index meta-analysis), and all trial effect estimates that were eligible for inclusion in the meta-analysis (eg, from multiple scales or time points) were extracted from trial reports. Analysis: We calculated a statistic (the Potential Bias Index (PBI)) to quantify and test for evidence of selective inclusion. The PBI ranges from 0 to 1; values above or below 0.5 are suggestive of selective inclusion of effect estimates more or less favourable to the intervention, respectively. The impact of any potential selective inclusion was investigated by comparing the index meta-analytic standardised mean difference (SMD) to the median of a randomly constructed distribution of meta-analytic SMDs (representing the meta-analytic SMD expected when there is no selective inclusion). Results: 31 reviews (250 trials) were included. The estimated PBI was 0.57 (95% CI 0.50 to 0.63), suggesting that trial effect estimates that were more favourable to the intervention were included in meta-analyses slightly more often than expected under a process consistent with random selection; however, the 95% CI included the null hypothesis of no selective inclusion. Any potential selective inclusion did not have an important impact on the meta-analytic effects. Conclusion: There was no clear evidence that selective inclusion of trial effect estimates occurred in this sample of meta-analyses. Further research on selective inclusion in other clinical specialties is needed. To enable readers to assess the risk of selective inclusion bias, we recommend that systematic reviewers report the methods used to select effect estimates to include in meta-analyses.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjopen-2016-011863

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Flow diagram of identification, screening and inclusion of systematic reviews. RCT, randomised controlled trial.
Figure 1 Flow diagram of identification, screening and inclusion of systematic reviews. RCT, randomised controlled trial.CC BY-NC · PubMed Central
Figure 2 Range of possible meta-analytic standardised mean differences (SMDs) per index meta-analysis, with median and index meta-analytic SMD.
Figure 2 Range of possible meta-analytic standardised mean differences (SMDs) per index meta-analysis, with median and index meta-analytic SMD.CC BY-NC · PubMed Central
Figure 3 Meta-analysis of differences between the index meta-analytic SMD and median of all its possible meta-analytic SMDs (each calculated using the random-effects model). Differences less than zero indicate that the index meta-analysis SMD is more favourable to the intervention compared with the median meta-analytic SMD. ES, effect size; SMD, standardised mean difference.
Figure 3 Meta-analysis of differences between the index meta-analytic SMD and median of all its possible meta-analytic SMDs (each calculated using the random-effects model). Differences less than zero indicate that the index meta-analysis SMD is more favourable to the intervention compared with the median meta-analytic SMD. ES, effect size; SMD, standardised mean difference.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 25 references

Research works printed but not linked: 1

  1. Fu R, Vandermeer BW, Shamliyan T Chapter 14: handling continuous outcomes in quantitative synthesis. In: Methods guide for effectiveness and comparative effectiveness reviews, AHRQ Publication No. 10(14)-EHC063-EF. Rockville, MD: Agency for Healthcare Research and Quality, 2014 http://www.effectivehealthcare.ahrq.gov 24006546

Cited by

See all 23

Metadata

Authors
Page, M. J., Forbes, A., Chau, M., Green, S. E., & McKenzie, J. E.
Affiliations
  • University of Bristol (GB)
  • Monash University (AU)
  • National Trauma Research Institute (AU)
Corresponding author
McKenzie, Joanne E
Venue
BMJ Open
Publisher
BMJ
Published
2016-04
Volume
6
Issue
4
Pages
e011863
ISSN
2044-6055 (ISSN-L)
OpenAlex
W2344534501
Cited by
23
References
23
Citations by year
2026: 2 · 2025: 2 · 2024: 3 · 2021: 6 · 2020: 3 · 2019: 1 · 2018: 3 · 2017: 2 · 2016: 1
FWCI
2.18
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • BMJ Open (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • Explore Bristol Research (publishedVersion, cc-by-nc)
  • Monash University Research Portal (Monash University) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • HighWire Press Open Archive (submittedVersion)
  • Bristol Research (University of Bristol) (submittedVersion, cc-by-nc)
Funders
  • Monash University
  • National Health and Medical Research Council (1088535, 1072366)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
meta-analysis, systematic reviews, reporting bias
MeSH terms
Cross-Sectional Studies, Humans, Research Design, Meta-Analysis as Topic, Selection Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates: empirical study. https://app.cochat.ai/papers/10.1136/bmjopen-2016-011863
The paper
Page, M. J., Forbes, A., Chau, M., Green, S. E., & McKenzie, J. E. (2016). Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates: empirical study. BMJ Open. https://doi.org/10.1136/bmjopen-2016-011863

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.