Assessing the potential for outcome reporting bias in a review: a tutorial
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Outcome reporting bias (ORB) occurs when variables are selected for publication based on their results. This can impact upon the results of a meta-analysis, biasing the pooled treatment effect estimate.The aim of this paper is to show how to assess a systematic review and corresponding trial reports for ORB using an example review of intravenous and nebulised magnesium in the treatment of asthma. METHODS: The review was assessed for ORB by 1) checking the reasons, when available, for excluding studies to ensure that no studies were excluded because they did not report the outcomes of interest in the review; 2) assessing the eligible studies as to whether the review outcomes of interest were reported. Each study was classified using a system developed in the ORBIT (Outcome Reporting Bias In Trials) project to indicate whether ORB was suspected and a reason for the suspicion. Authors of trials that did not report the outcomes of interest were contacted for information. A sensitivity analysis was performed to assess the robustness of the conclusions of the review to this potential source of bias. RESULTS: Twenty-four studies were included in the review; two studies had been excluded for not reporting either of the two outcomes of interest. Six included studies did not report hospital admission and two did not report pulmonary function. There was high suspicion of outcome reporting bias in four studies. Results from the sensitivity analysis indicate that review conclusions were not overturned. CONCLUSION: This paper demonstrates, with the example of the magnesium review, how to assess a review for outcome reporting bias. A review should not exclude studies if they have not reported the outcomes of interest and should consider the potential for outcome reporting bias in all included studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 1×
“effect size or precision only reported along with sample size or a p-value [27].”
BackgroundSystematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“05) and publication; with studies that report positive or significant results being more likely to be published [4].”
BackgroundSelf-citationKirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133 · cited 4×
“Since both outcomes are important clinical outcomes in Table 2 The ORBIT classification system for missing or incomplete outcome reporting [10]”
BackgroundMethodInfluentialSelf-citationPublication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898 · cited 1×
“Such bias, termed study publication bias, is well recognised as a potential threat to the validity of any meta-analysis and can make the readily available evidence unreliable for decision making [5,6].”
BackgroundReasons or excuses for avoiding meta-analysis in forest plots
Ioannidis, J. P. A., Patsopoulos, N. A., & Rothstein, H. R. (2008) · BMJ · doi:10.1136/bmj.a117 · Cited by 457 · cited 1×
“However, even when there is a large amount of heterogeneity, it may be preferable to quantitatively synthesise the data rather than a qualitative interpretation of the results, as long as the limitations of the methods are properly acknowledged [30].”
BackgroundOutcome selection bias in meta-analysis
Williamson, P. R., Gamble, C., Altman, D. G., & Hutton, J. L. (2005) · Statistical Methods in Medical Research · doi:10.1191/0962280205sm415oa · Cited by 188 · cited 1×
“This is especially important to consider when there is a large proportion of participants in eligible studies which have not been included in the meta-analysis compared to those that are included in the meta-analysis [32].”
ResultSelf-citationIdentification and impact of outcome selection bias in meta‐analysis
Williamson, P. R., & Gamble, C. (2005) · Statistics in Medicine · doi:10.1002/sim.2025 · Cited by 132 · cited 2×
“to assess the robustness of the conclusions of a metaanalysis to ORB [6,8].”
BackgroundSelf-citationApplication and investigation of a bound for outcome reporting bias
Williamson, P. R., & Gamble, C. (2007) · Trials · doi:10.1186/1745-6215-8-9 · Cited by 26 · cited 3×
“A method proposed to consider the potential effect of study publication bias has been adapted for ORB [8] and may be used to simultaneously consider robustness to both ORB and study publication bias.”
BackgroundMethodSelf-citation
Research works printed but not linked: 26
- Jones R: Strength of evidence in qualitative research. J Clin Epidemiol. 2007, 60 (4): 321-3. 10.1016/j.jclinepi.2006.06.001. · DOI
- Higgins JPT, Green S: Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.0. 2008, The Cochrane Collaboration · DOI · Paper page
- Hutton JL, Williamson PR: Bias in meta-analysis due to outcome variable selection within studies. Applied Statistics. 2000, 49 (3): 359-370.
- Higgins JPT, Altman DG: Chapter 8: Assessing risk of bias in included studies. Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.0. Edited by: Higgins JPT, Green S. 2008, The Cochrane Collaboration · DOI · Paper page
- Mohammed S, Goodacre S: Intravenous and nebulised magnesium sulphate for acute asthma: systematic review and meta-analysis. Emerg Med J. 2007, 24 (12): 823-30. 10.1136/emj.2007.052050. · DOI
- Basic Asthma Research Strategy II: The Second Asthma UK Consultation. Clinical and Experimental Allergy. 2006, 1310-
- British Thoracic Society and Scottish Intercollegiate Guidelines Network: British Guideline on the Management of Asthma. A national clinical guideline. Thorax. 2008, 63 (Supplement IV): iv1-iv121. · DOI · Paper page
- Coates JE, Chinn DJ, Miller MR: Lung Function: Physiology, Measurement and Application in Medicine. 2006, Wiley, Sixth · DOI
- Gorelick MH, Stevens MW, Schultz T, Scribano PV: Difficulty in Obtaining Peak Expiratory Flow Measurements in Children With Acute Asthma. Pediatric Emergency Care. 2004, 20 (1): 22-26. 10.1097/01.pec.0000106239.72265.16. · doi:10.1097/01.pec.0000106239.72265.16
- Dadhich P, Vats M, Lokendra D, Gupta RC, Gupta ML, Gupta N: Magnesium Sulphate Nebulization in Acute Severe Asthma. 2003, 107S-
- Santana JC, Barreto SS, Piva JP, Garcia PC: Randomized clinical trial of intravenous magnesium sulphate versus salbutamol in early management of severe acute asthma in children. J Pediatr. 2001, 77 (4): 279-87. · doi:10.2223/jped.235
- Boonyavorakul C, Thakkinstian A, Charoenpan P: Intravenous magnesium sulfate in acute severe asthma. 2000, 221-225.
- Scarfone RJ, Loiselle JM, Joffe MD, Mull CC, Stiller S, Thompson K, Gracely EJ: A randomized trial of magnesium in the emergency department treatment of children with asthma. Ann Emerg Med. 2000, 36 (6): 572-8. 10.1067/mem.2000.111060. · doi:10.1067/mem.2000.111060
- Bessmertny O, DiGregorio RV, Cohen H, Becker E, Looney D, Golden J, Kohl L, Johnson T: A randomized clinical trial of nebulized magnesium sulfate in addition to albuterol in the treatment of acute mild-to-moderate asthma exacerbations in adults. Annals of Emergency Medicine. 2002, 39 (6): 585-591. 10.1067/mem.2002.123300. · doi:10.1067/mem.2002.123300
- Bijani K, Moghadamnia AA, Islami Khalili E: Intravenous magnesium sulphate as an adjunct in the treatment of severe asthmatic patients non-responding to conventional therapy. The Internet Journal of Asthma, Allergy and Immunology. 2002, 2 (1): · doi:10.5580/2381
- Devi PR, Kumar L, Singhi SC, Prasad R, Singh M: Intravenous magnesium sulfate in acute severe asthma not responding to conventional therapy. Indian Pediatr. 1997, 34 (5): 389-97.
- Gürkan F, Haspolat K, Boşnak M, Dikici B, Derman O, Ece A: Intravenous magnesium sulphate in the management of moderate to severe acute asthmatic children nonresponding to conventional therapy. Eur J Emerg Med. 1999, 6 (3): 201-5. · doi:10.1097/00063110-199909000-00005
- Meral A, Coker M, Tanac R: Inhalation therapy with magnesium sulphate and salbutamol sulphate in bronchial asthma. Turkish Journal of Pediatrics. 1996, 38: 169-75.
- Tiffany BR, Berk WA, Todd IK, White SR: Magnesium bolus or infusion fails to improve expiratory flow in acute asthma exacerbations. 1993, 831-834.
- Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG: Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine/Public Library of Science. 2007, 4 (3): e78-
- Higgins JP, Thompson SG, Deeks JJ, Altman DG: Measuring inconsistency in meta-analyses. Bmj. 2003, 327 (7414): 557-60. 10.1136/bmj.327.7414.557. · DOI · Paper page
- Porter RS, Nester , Braitman LE, Geary U, Dalsey WC: Intravenous magnesium is ineffective in adult asthma, a randomized trial. Eur J Emerg Med. 2001, 8 (1): 9-15. 10.1097/00063110-200103000-00003. · doi:10.1097/00063110-200103000-00003
- Sinha I, Jones L, Smyth RL, Williamson PR: A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children. PLoS Med. 2008, 5 (4): e96-10.1371/journal.pmed.0050096. · DOI · Paper page
- Clarke M: Standardising Outcomes in Paediatric Clinical Trials. PLoS Medicine. 2008, 5 (4): e102-10.1371/journal.pmed.0050102. · DOI · Paper page
- Ciarallo L, Sauer AH, Shannon MW: Intravenous magnesium therapy for moderate to severe pediatric asthma: Results of a randomized, placebo-controlled trial. The Journal of Pediatrics. 1996, 129 (6): 809-814. 10.1016/S0022-3476(96)70023-9. · DOI · Paper page
- Ciarallo L, Brousseau D, Reinert S: Higher-Dose Intravenous Magnesium Therapy for Children With Moderate to Severe Acute Asthma. 2000, 979-983.
Cited by
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Interventions for preventing obesity in children
Brown, T., Moore, T. H., Hooper, L., Gao, Y., Zayegh, A., Ijaz, S., Elwenspoek, M., Foxen, S. C., Magee, L., O'Malley, C., et al. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001871.pub4 · Cited by 2,982
Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group (2013) · PLoS ONE · doi:10.1371/journal.pone.0066844 · Cited by 1,127
Self-citationJansen, J. P., Trikalinos, T., Cappelleri, J. C., Daw, J., Andes, S., Eldessouki, R., & Salanti, G. (2014) · Value in Health · doi:10.1016/j.jval.2014.01.004 · Cited by 319
“[20] Dwan K, Gamble C, Kolamunnage-Dona R, et al.”
BackgroundInfluentialPage, M. J., McKenzie, J. E., & Higgins, J. P. T. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019703 · Cited by 278
Metadata
- Authors
- Dwan, K., Gamble, C., Kolamunnage-Dona, R., Mohammed, S., Powell, C., & Williamson, P. R.
- Venue
- Trials
- Published
- 2010-05-12
- Volume
- 11
- Issue
- 1
- Pages
- 52-52
- ISSN
- 1745-6215 (ISSN-L)
- OpenAlex
- W2170245994
- Cited by
- 63
- References
- 9
- FWCI
- 4.63
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/2.0 (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessing the potential for outcome reporting bias in a review: a tutorial. https://app.cochat.ai/papers/10.1186/1745-6215-11-52
- The paper
- Dwan, K., Gamble, C., Kolamunnage-Dona, R., Mohammed, S., Powell, C., & Williamson, P. R. (2010). Assessing the potential for outcome reporting bias in a review: a tutorial. Trials. https://doi.org/10.1186/1745-6215-11-52
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.