The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Bias in Meta-Analysis Due to Outcome Variable Selection Within Studies
Open accessAbstract
SUMMARY: Although bias in meta-analysis arising from selective publication has been studied, within-study selection has received little attention. Chronic diseases often have several possible outcome variables. Methods based on the size of the effect allow results from studies with different outcomes to be combined. However, the possibility of selective reporting of outcomes must be considered. The effect of selective reporting on estimates of the size of the effect and significance levels is presented, and sensitivity analyses are suggested. Substantial publication bias could arise from multiple testing of outcomes in a study, followed by selective reporting. Two meta-analyses, on anthelminth therapy and a treatment for incontinence, are reassessed allowing for within-study selection, as it is clear that more outcomes had been measured than were reported. The sensitivity analyses show that the robustness of the anthelminth results is dependent on what assumption one makes about the reporting strategy for the largest trial. The possible influence of correlation between within-child measurements was such that the conclusions could easily be reversed. The effect of a mild assumption on within-trial selection alone could alter general recommendations about the treatment for incontinence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cox, D. R., & Hinkley, D. V. (1974) · doi:10.1007/978-1-4899-2887-0 · Cited by 3,089
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291
Cited by
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835
Effect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835
(2008) · Cited by 3,790
The COMET Handbook: version 1.0
Williamson, P. R., Altman, D. G., Bagley, H., Barnes, K. L., Blazeby, J. M., Brookes, S. T., Clarke, M., Gargon, E., Gorst, S., Harman, N., et al. (2017) · Trials · doi:10.1186/s13063-017-1978-4 · Cited by 2,011
“Outcomereporting bias has been defined as the selection of a subset of the original recorded outcomes, on the basis of the results, for inclusion in the published reports of trials and other research [18].”
BackgroundSelf-citationSystematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546
“It has been defined as the selection on the basis of the results of a subset of the original variables recorded for inclusion in a publication [9].”
BackgroundSelf-citationThe Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388
“…experiments to decide whether to continue collecting data [15,16]; recording many response variables and deciding which to report postanalysis [16,17], deciding whether to include or drop outliers postanalyses [16], excluding, combining, or splitting treatment groups postanalysis [2], including…”
BackgroundKirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133
“A specific form of bias arising from the selective reporting of the set of study outcomes is outcome reporting bias, which is defined as the selection for publication of a subset of the original recorded outcome variables on the basis of the results.”
BackgroundSelf-citationDwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group (2013) · PLoS ONE · doi:10.1371/journal.pone.0066844 · Cited by 1,127
Self-citationRecent developments in meta‐analysis
Sutton, A. J., & Higgins, J. P. T. (2008) · Statistics in Medicine · doi:10.1002/sim.2934 · Cited by 590
A meta-analysis of 50 years of ruptured abdominal aortic aneurysm repair
Bown, M. J., Sutton, A. J., Bell, P. R. F., & Sayers, R. D. (2002) · British Journal of Surgery · doi:10.1046/j.1365-2168.2002.02122.x · Cited by 565
Aggregate data meta‐analysis with time‐to‐event outcomes
Williamson, P. R., Smith, C. T., Hutton, J. L., & Marson, A. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1303 · Cited by 551
Self-citationEffects of psychotherapy for anxiety in children and adolescents: A meta-analytic review
Reynolds, S., Wilson, C., Austin, J., & Hooper, L. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.01.005 · Cited by 483
Wikkelsø, A., Wetterslev, J., Møller, A. M., & Afshari, A. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007871.pub3 · Cited by 400
Afshari, A., Wikkelsø, A., Brok, J., Møller, A. M., & Wetterslev, J. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007871.pub2 · Cited by 323
Psychological therapies for pathological and problem gambling
Cowlishaw, S., Merkouris, S., Dowling, N., Anderson, C., Jackson, A., & Thomas, S. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008937.pub2 · Cited by 321
Efficacy and safety of sugammadex versus neostigmine in reversing neuromuscular blockade in adults
Hristovska, A.-M., Duch, P., Allingstrup, M., & Afshari, A. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012763 · Cited by 319
Inhaled nitric oxide for acute respiratory distress syndrome (ARDS) in children and adults
Gebistorf, F., Karam, O., Wetterslev, J., & Afshari, A. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002787.pub3 · Cited by 298
Page, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035.pub2 · Cited by 281
Detecting Selection Bias in Meta-Analyses with Multiple Outcomes: A Simulation Study
Fernández-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Van den Noortgate, W. (2021) · The Journal of Experimental Education · doi:10.1080/00220973.2019.1582470 · Cited by 285
Causal Variables, Indicator Variables and Measurement Scales: An Example from Quality of Life
Fayers, P. M., & Hand, D. J. (2002) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/1467-985x.02020 · Cited by 267
Comparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243
Self-citationFrequency and reasons for outcome reporting bias in clinical trials: interviews with trialists
Smyth, R. M. D., Kirkham, J. J., Jacoby, A., Altman, D. G., Gamble, C., & Williamson, P. R. (2011) · BMJ · doi:10.1136/bmj.c7153 · Cited by 220
“An additional and potentially more serious threat to the validity of evidence based medicine is the selection for publication of a subset of the original recorded outcomes on the basis of the results,”
BackgroundSelf-citationSaini, P., Loke, Y. K., Gamble, C., Altman, D. G., Williamson, P. R., & Kirkham, J. J. (2014) · BMJ · doi:10.1136/bmj.g6501 · Cited by 209
Self-citationDwan, K., Altman, D. G., Clarke, M., Gamble, C., Higgins, J. P. T., Sterne, J. A. C., Williamson, P. R., & Kirkham, J. J. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001666 · Cited by 187
“A citation search of a key article [1] was also performed.”
BackgroundSelf-citationObstacles and opportunities in meta-analysis of genetic association studies
Salanti, G., Sanderson, S., & Higgins, J. P. T. (2005) · Genetics in Medicine · doi:10.1097/01.gim.0000151839.12032.1a · Cited by 196
Moreno, S. G., Sutton, A. J., Turner, E. H., Abrams, K. R., Cooper, N. J., Palmer, T. M., & Ades, A. E. (2009) · BMJ · doi:10.1136/bmj.b2981 · Cited by 184
Bias Due to Changes in Specified Outcomes during the Systematic Review Process
Kirkham, J. J., Altman, D. G., & Williamson, P. R. (2010) · PLoS ONE · doi:10.1371/journal.pone.0009810 · Cited by 164
“In an individual RCT, outcome reporting bias (ORB) has been defined as the selection for publication of a subset of the original recorded outcome variables based on the results [5].”
MethodSelf-citationInterventions for helping people recognise early signs of recurrence in bipolar disorder
Morriss, R., Faizal, M. A., Jones, A. P., Williamson, P. R., Bolton, C. A., & McCarthy, J. P. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004854.pub2 · Cited by 166
Self-citationA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
Kirkham, J. J., Altman, D. G., Chan, A.-W., Gamble, C., Dwan, K. M., & Williamson, P. R. (2018) · BMJ · doi:10.1136/bmj.k3802 · Cited by 144
Self-citationMeta-analysis of rare and adverse event data
Sutton, A. J., Cooper, N. J., Lambert, P. C., Jones, D. R., Abrams, K. R., & Sweeting, M. J. (2002) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1586/14737167.2.4.367 · Cited by 141
Identification and impact of outcome selection bias in meta‐analysis
Williamson, P. R., & Gamble, C. (2005) · Statistics in Medicine · doi:10.1002/sim.2025 · Cited by 132
Self-citationYounger, A., Alkon, A., Harknett, K., Jean Louis, R., & Thompson, L. M. (2022) · Environmental Research · doi:10.1016/j.envres.2021.112274 · Cited by 104
Hahn, S., Williamson, P. R., & Hutton, J. L. (2002) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2002.00314.x · Cited by 97
Self-citationSchool-Based Mentoring for Adolescents: A Systematic Review and Meta-Analysis
Wood, S., & Mayo-Wilson, E. (2012) · Research on Social Work Practice · doi:10.1177/1049731511430836 · Cited by 83
Dwan, K., Kirkham, J. J., Williamson, P. R., & Gamble, C. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-002709 · Cited by 55
Self-citationNorris, S. L., Moher, D., Reeves, B. C., Shea, B., Loke, Y., Garner, S., Anderson, L., Tugwell, P., & Wells, G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1062 · Cited by 50
Selective outcome reporting in randomized clinical trials of dental implants
Sendyk, D. I., Rovai, E. S., Souza, N. V., Deboni, M. C. Z., & Pannuti, C. M. (2019) · Journal of Clinical Periodontology · doi:10.1111/jcpe.13128 · Cited by 35
Application and investigation of a bound for outcome reporting bias
Williamson, P. R., & Gamble, C. (2007) · Trials · doi:10.1186/1745-6215-8-9 · Cited by 26
“Within-study selective reporting, or outcome reporting bias, has been defined as 'the selection of a subset of the original variables recorded for inclusion in publication of trials' [4].”
BackgroundSelf-citationCopas, J., Marson, A., Williamson, P., & Kirkham, J. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280217738546 · Cited by 23
Self-citationSchulz, R., Langen, G., Prill, R., Cassel, M., & Weissgerber, T. L. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-059347 · Cited by 23
Pandis, N., Fleming, P. S., Worthington, H., Dwan, K., & Salanti, G. (2015) · PLOS ONE · doi:10.1371/journal.pone.0137667 · Cited by 30
“However, inappropriate practices such as selective inclusion of trials, selective reporting of outcomes and results-driven reporting can bias estimates of treatment effects culminating in compromised patient care, healthcare decisions and service configuration [3,4].”
BackgroundWeston, J., Dwan, K., Altman, D., Clarke, M., Gamble, C., Schroter, S., Williamson, P., & Kirkham, J. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010075 · Cited by 16
Self-citationSchulz, R., Langen, G., Prill, R., Cassel, M., & Weissgerber, T. (2021) · doi:10.1101/2021.07.20.21260565 · Cited by 23
“Outcome reporting bias Only particular outcome variables are included in publications and decisions about which variables to include are based on the statistical significance or direction of the results (23).”
Background
Metadata
- Authors
- Hutton, J. L., & Williamson, P. R.
- Venue
- Journal of the Royal Statistical Society Series C: Applied Statistics
- Publisher
- Oxford University Press (OUP)
- Published
- 2000-09-01
- Volume
- 49
- Issue
- 3
- Pages
- 359-370
- ISSN
- 0035-9254 (ISSN-L)
- 1467-9876
- OpenAlex
- W1997179582
- Cited by
- 194
- References
- 2
- FWCI
- 8.12
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.