Outcome selection bias in meta-analysis
Abstract
Publication bias has been previously identified as a threat to the validity of a meta-analysis. Recently, new evidence has documented an additional threat to validity, the selective reporting of trial outcomes within published studies. Several diseases have several possible measures of outcome. Some articles might report only a selection of those outcomes, perhaps those with statistically significant results. In this article, we review this problem while addressing the questions: what is within-study selective reporting? how common is it? why is it done? how can it mislead? how can it be detected?, and finally, what is the solution? We recommend that both publication bias and selective reporting should be routinely investigated in systematic reviews.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · see citing works
Journal of the American Statistical Association
Fildes, R. A. (1989) · International Journal of Forecasting · doi:10.1016/0169-2070(89)90106-4 · Cited by 3,706 · see citing works
誠一, 市. (2004) · The Journal of AIDS Research · doi:10.11391/aidsr1999.6.104 · Cited by 3,233 · see citing works
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · see citing works
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955 · see citing works
Statistical Methods for Meta-Analysis
(1985) · Elsevier eBooks · doi:10.1016/c2009-0-03396-0 · Cited by 2,549 · see citing works
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · see citing works
Self-citationSystematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693 · see citing works
Cochrane Injuries Group Albumin ReviewersWhy albumin may not work
Berger, A. (1998) · BMJ · doi:10.1136/bmj.317.7153.235 · Cited by 1,080 · see citing works
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · see citing works
Marshall, M., Lockwood, A., Bradley, C., Adams, C., Joy, C., & Fenton, M. (2000) · British Journal of Psychiatry · doi:10.1192/bjp.176.3.249 · Cited by 870 · see citing works
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818 · see citing works
Publication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814 · see citing works
Publication bias and clinical trials
Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987) · Controlled Clinical Trials · doi:10.1016/0197-2456(87)90155-3 · Cited by 715 · see citing works
Chan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613 · see citing works
Self-citationPractical methodology of meta‐analyses (overviews) using updated individual patient data
Stewart, L. A., & Cochrane Working Group On Meta‐Analysis Using Individual Patient Data (1995) · Statistics in Medicine · doi:10.1002/sim.4780141902 · Cited by 486 · see citing works
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Chan, A.-W. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041086 · Cited by 444 · see citing works
Selection Models and the File Drawer Problem
Iyengar, S., & Greenhouse, J. B. (1988) · Statistical Science · doi:10.1214/ss/1177013012 · Cited by 399 · see citing works
Content and quality of 2000 controlled trials in schizophrenia over 50 years
Thornley, B., & Adams, C. E. (1998) · BMJ · doi:10.1136/bmj.317.7167.1181 · Cited by 362 · see citing works
How important is publication bias? A synthesis of available data.
Dickersin, K. (1997) · PubMed · Cited by 312 · see citing works
Callaham, M. L. (1998) · JAMA · doi:10.1001/jama.280.3.254 · Cited by 304 · see citing works
Krleža-Jerić, K., Chan, A., Dickersin, K., Sim, I., Grimshaw, J., & Gluud, C. N. (2005) · BMJ · doi:10.1136/bmj.330.7497.956 · Cited by 236 · see citing works
An Approach for Assessing Publication Bias Prior to Performing a Meta-Analysis
Dear, K., & Begg, C. (1992) · Statistical Science · doi:10.1214/ss/1177011363 · Cited by 202 · see citing works
What Works?: Selectivity Models and Meta-Analysis
Copas, J. B. (1999) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/1467-985x.00123 · Cited by 168 · see citing works
Identification and impact of outcome selection bias in meta‐analysis
Williamson, P. R., & Gamble, C. (2005) · Statistics in Medicine · doi:10.1002/sim.2025 · Cited by 132 · see citing works
Self-citationPublication Bias and Research on Passive Smoking
Misakian, A. L., & Bero, L. A. (1998) · JAMA · doi:10.1001/jama.280.3.250 · Cited by 126 · see citing works
Chalmers, I., & Altman, D. G. (1999) · The Lancet · doi:10.1016/s0140-6736(98)07618-1 · Cited by 108 · see citing works
Self-citationHahn, S., Williamson, P. R., & Hutton, J. L. (2002) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2002.00314.x · Cited by 97 · see citing works
Self-citationPhillips, C. V. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-20 · Cited by 94 · see citing works
Tannock, I. F. (1996) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/88.3-4.206 · Cited by 92 · see citing works
Adjusting for publication bias: modelling the selection process.
Preston, C., Ashby, D., & Smyth, R. (2004) · Journal of Evaluation In Clinical Practice · doi:10.1111/j.1365-2753.2003.00457.x · Cited by 40 · see citing works
Self-citationPublishing raw data and real time statistical analysis on e-journals
Hutchon, D. J. R. (2001) · BMJ · doi:10.1136/bmj.322.7285.530 · Cited by 37 · see citing works
Hahn, S., Garner, P., & Williamson, P. R. (2000) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2000.00230.x · Cited by 10 · see citing works
Self-citationBottom drawer papers: another waste of clinicians' time
Hosking, E.-J., & Albert, T. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01286.x · Cited by 4 · see citing works
Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164 · see citing works
“For example, in a particular disease, if one of two linked outcomes is reported but the other is not, then one should question whether the latter has been selectively omitted [121,122].”
BackgroundSelf-citationCochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648 · see citing works
Self-citationShamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578 · see citing works
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349 · see citing works
“For example, in a particular disease, if one of two linked outcomes is reported but the other is not, then one should question whether the latter has been selectively omitted [121,122].”
BackgroundSelf-citation(2008) · Cited by 3,790 · see citing works
Self-citationAssessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · see citing works
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Italian Journal of Public Health · doi:10.2427/5768 · Cited by 3,374 · see citing works
Self-citationPeters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · see citing works
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · see citing works
“For example, selective reporting of analyses may include intention-to–treat analyses versus per–protocol analyses, endpoint score versus change from baseline, different time points or subgroups [10].”
BackgroundSelf-citationDwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group (2013) · PLoS ONE · doi:10.1371/journal.pone.0066844 · Cited by 1,127 · see citing works
“For example, selective reporting of analyses may include intention-to–treat analyses versus per–protocol analyses, endpoint score versus change from baseline, different time points or subgroups [11].”
BackgroundSelf-citationJansen, J. P., Fleurence, R., Devine, B., Itzler, R. F., Barrett, A., Hawkins, N., Lee, K., Boersma, C., Annemans, L., & Cappelleri, J. C. (2011) · Value in Health · doi:10.1016/j.jval.2011.04.002 · Cited by 1,036 · see citing works
Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK): Explanation and Elaboration
Altman, D. G., McShane, L. M., Sauerbrei, W., & Taube, S. E. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001216 · Cited by 1,019 · see citing works
“Evidence of biased non-publication of whole studies has been accumulating for many years, but recently research has demonstrated evidence of additional within-study selective reporting [205,206].”
ResultSelf-citationSinha, I. P., Smyth, R. L., & Williamson, P. R. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1000393 · Cited by 891 · see citing works
“As a result, the standardisation of outcomes for clinical trials has been proposed as a solution to the problems of inappropriate and non-uniform outcome selection [4,7] and reporting bias [5,8].”
BackgroundSelf-citationStatistical strategies for avoiding false discoveries in metabolomics and related experiments
Broadhurst, D., & Kell, D. B. (2006) · Metabolomics · doi:10.1007/s11306-006-0037-z · Cited by 819 · see citing works
“We note, however, that publication bias – the greater likelihood of publishing a positive than a negative result – can create optimistic inaccuracies in such analyses (Williamson et al., 2005).”
Chapple, C. R., Khullar, V., Gabriel, Z., Muston, D., Bitoun, C. E., & Weinstein, D. L. (2008) · European Urology · doi:10.1016/j.eururo.2008.06.047 · Cited by 768 · see citing works
“Finally, a systematic review is to some extent reliant on the unbiased selection of outcomes in the reviewed studies [52].”
BackgroundOpioid antagonists for alcohol dependence
Srisurapanont, M., & Jarusuraisin, N. (2002) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001867 · Cited by 762 · see citing works
Standardising outcomes for clinical trials and systematic reviews
Clarke, M. (2007) · Trials · doi:10.1186/1745-6215-8-39 · Cited by 483 · see citing works
Acamprosate for alcohol dependence
Rösner, S., Hackl-Herrwerth, A., Leucht, S., Lehert, P., Vecchi, S., & Soyka, M. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004332.pub2 · Cited by 420 · see citing works
Reporting recommendations for tumor marker prognostic studies (REMARK): explanation and elaboration
Altman, D. G., McShane, L. M., Sauerbrei, W., & Taube, S. E. (2012) · BMC Medicine · doi:10.1186/1741-7015-10-51 · Cited by 374 · see citing works
“Evidence of biased non-publication of whole studies has been accumulating for many years, but recently research has demonstrated evidence of additional within-study selective reporting [205,206].”
ResultSelf-citationPeter, J. V., John, P., Graham, P. L., Moran, J. L., George, I. A., & Bersten, A. D. (2008) · BMJ · doi:10.1136/bmj.39537.939039.be · Cited by 351 · see citing works
Page, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035.pub2 · Cited by 281 · see citing works
Tandon, R., Belmaker, R. H., Gattaz, W. F., López-Ibor, J. J., Okasha, A., SINGH, B. S., Stein, D. J., Olié, J., Fleischhacker, W. W., & Moeller, H. (2008) · Schizophrenia Research · doi:10.1016/j.schres.2007.11.033 · Cited by 281 · see citing works
“, focusing on outcomes ‘favorable’ to the sponsor's product) (Heres et al., 2006; Williamson et al., 2005).”
BackgroundOpioid antagonists for alcohol dependence
Rösner, S., Hackl-Herrwerth, A., Leucht, S., Vecchi, S., Srisurapanont, M., & Soyka, M. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001867.pub3 · Cited by 246 · see citing works
Testing for the Presence of Positive-Outcome Bias in Peer Review: A Randomized Controlled Trial
Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.406 · Cited by 245 · see citing works
Comparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243 · see citing works
Self-citationBreckenridge, C. B., Berry, C. L., Chang, E. T., Sielken, R. L., & Mandel, J. S. (2016) · PLoS ONE · doi:10.1371/journal.pone.0151841 · Cited by 239 · see citing works
“In addition, it is important to examine studies to ensure that within-study selection bias is minimized, particularly when the results from a post-hoc subgroup analysis is the main focus of the study [197, 198].”
BackgroundFrequency and reasons for outcome reporting bias in clinical trials: interviews with trialists
Smyth, R. M. D., Kirkham, J. J., Jacoby, A., Altman, D. G., Gamble, C., & Williamson, P. R. (2011) · BMJ · doi:10.1136/bmj.c7153 · Cited by 220 · see citing works
Self-citationAntioxidant supplements for preventing gastrointestinal cancers
Bjelaković, G., Nikolova, D., Simonetti, R. G., & Gluud, C. N. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004183.pub3 · Cited by 217 · see citing works
Johnson, B. T., & Hennessy, E. A. (2019) · Social Science & Medicine · doi:10.1016/j.socscimed.2019.05.035 · Cited by 215 · see citing works
“Unfortunately, other than psychometric artefacts, which attenuate effect sizes (Schmidt and Hunter, 2015), and bias introduced with p-hacking or selective reporting (Kirkham et al., 2010a,b; Rubin, 2017; Williamson et al., 2005), which tend to exaggerate effects, little is known about the effects of other methodologic…”
BackgroundSinha, I. P., Jones, L. V., Smyth, R. L., & Williamson, P. R. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050096 · Cited by 202 · see citing works
“Disease-specific, universally agreed core sets of outcomes that should be measured and reported in all clinical trials of a specific condition, regardless of statistical significance, have been advocated as a solution to this common problem [35].”
Self-citationSchubert, E. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00837 · Cited by 183 · see citing works
Röver, C., Bender, R., Dias, S., Schmid, C. H., Schmidli, H., Sturtz, S., Weber, S., & Friede, T. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1475 · Cited by 192 · see citing works
“This might be compromised by selection effects, for example, publication bias(122) or reporting bias.(123) Especially in the case of only few studies, such effects might be hard to detect from the data, and information on the presence of selection effects may need to come from considerations of the context.”
BackgroundReducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 190 · see citing works
Papier, K., Fensom, G. K., Knuppel, A., Appleby, P. N., Tong, T. Y. N., Schmidt, J. A., Travis, R. C., Key, T. J., & Perez-Cornago, A. (2021) · BMC Medicine · doi:10.1186/s12916-021-01922-9 · Cited by 168 · see citing works
Replication in prevention science
Valentine, J. C., Biglan, A., Boruch, R. F., Castro, F. G., Collins, L. M., Flay, B. R., Kellam, S., Mościcki, E. K., & Schinke, S. P. (2011) · Prevention Science · doi:10.1007/s11121-011-0217-6 · Cited by 166 · see citing works
“If measured outcomes within studies are censored due to the lack of statistical significance, this can also lead to incorrect inferences (Williamson et al. 2005).”
BackgroundInterventions for helping people recognise early signs of recurrence in bipolar disorder
Morriss, R., Faizal, M. A., Jones, A. P., Williamson, P. R., Bolton, C. A., & McCarthy, J. P. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004854.pub2 · Cited by 166 · see citing works
Self-citationWebbe, J., Sinha, I. P., & Gale, C. (2017) · Archives of Disease in Childhood Education & Practice · doi:10.1136/archdischild-2016-312117 · Cited by 152 · see citing works
Joos, S., Miksch, A., Szécsényi, J., Wieseler, B., Grouven, U., Kaiser, T., & Schneider, A. (2008) · Thorax · doi:10.1136/thx.2007.081596 · Cited by 149 · see citing works
“significant treatment differences are reported in published articles, resulting in an overestimation of treatment effects.(40)”
BackgroundDevelopment of a core outcome set for research and audit studies in reconstructive breast surgery
Potter, S., Holcombe, C., Ward, J. A., Blazeby, J. M., Brookes, S. T., Cawthorn, S. J., Harcourt, D. M., Macefield, R. C., Warr, R., Weiler-Mithoff, E. M., et al. (2015) · British journal of surgery · doi:10.1002/bjs.9883 · Cited by 143 · see citing works
Self-citationConn, V. S., Ruppar, T. M., Enriquez, M., & Cooper, P. (2015) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2015.06.001 · Cited by 142 · see citing works
Evidence-Based Practice for Rehabilitation Professionals: Concepts and Controversies
Dijkers, M. P. J. M., Murphy, S. L., & Krellman, J. W. (2012) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2011.12.014 · Cited by 118 · see citing works
Vera-Badillo, F. E., Shapiro, R., Ocana, A., Amir, E., & Tannock, I. F. (2013) · Annals of Oncology · doi:10.1093/annonc/mds636 · Cited by 117 · see citing works
Riley, P., Glenny, A., Worthington, H. V., Littlewood, A., Clarkson, J., & McCabe, M. G. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011552.pub2 · Cited by 116 · see citing works
Under-reporting of harm in clinical trials
Šeruga, B., Templeton, A. J., Badillo, F. E. V., Ocaña, A., Amir, E., & Tannock, I. F. (2016) · The Lancet Oncology · doi:10.1016/s1470-2045(16)00152-2 · Cited by 110 · see citing works
Moran, J. L., Graham, P. L., Rockliff, S., & Bersten, A. D. (2010) · Critical Care · doi:10.1186/cc9182 · Cited by 105 · see citing works
“Secondary outcome analysis was beset by selection bias in reporting [65], as witnessed by study numbers in Table 5; parameter estimates may be biased under such circumstances.”
BackgroundBjelaković, G., Nikolova, D., Simonetti, R. G., & Gluud, C. N. (2008) · Alimentary Pharmacology & Therapeutics · doi:10.1111/j.1365-2036.2008.03785.x · Cited by 103 · see citing works
Future of the Language Models in Healthcare: The Role of Chatgpt
Tustumi, F., Andreollo, N. A., & Aguilar-Nascimento, J. E. D. (2023) · ABCD. Arquivos Brasileiros de Cirurgia Digestiva (São Paulo) · doi:10.1590/0102-672020230002e1727 · Cited by 99 · see citing works
“Another relevant issue in evidence-based guidelines and systematic reviews refers to the risk of selection bias 10,22,23 .”
BackgroundBradley, H. A., Rucklidge, J. J., & Mulder, R. (2016) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12647 · Cited by 91 · see citing works
Vitamin D supplementation for chronic liver diseases in adults
Bjelaković, G., Nikolova, D., Bjelaković, M., & Gluud, C. N. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011564.pub2 · Cited by 93 · see citing works
Röver, C., Bender, R., Dias, S., Schmid, C. H., Schmidli, H., Sturtz, S., Weber, S., & Friede, T. (2021) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 89 · see citing works
Few systematic reviews exist documenting the extent of bias: a systematic review
Tricco, A. C., Tetzlaff, J., Sampson, M., Fergusson, D. A., Cogo, E., Horsley, T., & Moher, D. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.10.017 · Cited by 79 · see citing works
Assessment of chronic post‐surgical pain after knee replacement: Development of a core outcome set
Wylde, V., MacKichan, F., Bruce, J., & Gooberman‐Hill, R. (2014) · European Journal of Pain · doi:10.1002/ejp.582 · Cited by 79 · see citing works
“Standardization of outcomes assessment can reduce heterogeneity between trials, thereby aiding data synthesis (Clarke, 2007) and minimizing the risk of outcome reporting bias (Williamson et al., 2005).”
BackgroundRiley, P., Glenny, A., Worthington, H. V., Littlewood, A., Mauleffinch, L. M. F., Clarkson, J., & McCabe, M. G. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011990.pub2 · Cited by 81 · see citing works
Bürkner, P., Williams, D. R., Simmons, T. C., & Woolley, J. D. (2017) · Schizophrenia Bulletin · doi:10.1093/schbul/sbx053 · Cited by 80 · see citing works
Page, M. J., McKenzie, J. E., & Forbes, A. B. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.10.010 · Cited by 75 · see citing works
Fritz, A., Scherndl, T., & Kühberger, A. (2012) · Theory & Psychology · doi:10.1177/0959354312436870 · Cited by 70 · see citing works
Antioxidant supplements for liver diseases
Bjelaković, G., Gluud, L. L., Nikolova, D., Bjelaković, M., Nagorni, A. V., & Gluud, C. N. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007749.pub2 · Cited by 70 · see citing works
Antioxidant supplements for preventing gastrointestinal cancers
Bjelaković, G., Nikolova, D., Simonetti, R., & Gluud, C. N. (2004) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004183.pub2 · Cited by 70 · see citing works
Riley, P., Glenny, A., Worthington, H. V., Littlewood, A., Clarkson, J., & McCabe, M. G. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011552 · Cited by 71 · see citing works
Milette, K., Roseman, M., & Thombs, B. D. (2010) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2010.09.015 · Cited by 65 · see citing works
“Within-study selective outcome reporting, on the other hand, relates to published studies and refers to the selection of outcomes to report based on statistical significance, including the prioritization of time points for analysis, selective reporting of a subset of measures, and data-driven selection or switching of…”
ResultXu, Z., Zhou, Y., Shao, B., Zhang, J., & Liu, C. (2019) · The American Journal of Chinese Medicine · doi:10.1142/s0192415x19500265 · Cited by 64 · see citing works
Assessing the potential for outcome reporting bias in a review: a tutorial
Dwan, K. M., Gamble, C. L., Kolamunnage‐Dona, R., Mohammed, S., Powell, C., & Williamson, P. R. (2010) · Trials · doi:10.1186/1745-6215-11-52 · Cited by 64 · see citing works
“This is especially important to consider when there is a large proportion of participants in eligible studies which have not been included in the meta-analysis compared to those that are included in the meta-analysis [32].”
ResultSelf-citationMeta‐analysis: antioxidant supplements for liver diseases – the Cochrane Hepato‐Biliary Group
Bjelaković, G., Gluud, L. L., Nikolova, D., Bjelaković, M., Nagorni, A. V., & Gluud, C. N. (2010) · Alimentary Pharmacology & Therapeutics · doi:10.1111/j.1365-2036.2010.04371.x · Cited by 62 · see citing works
Unbiased estimation of odds ratios: combining genomewide association scans with replication studies
Bowden, J., & Dudbridge, F. (2009) · Genetic Epidemiology · doi:10.1002/gepi.20394 · Cited by 62 · see citing works
“In 2007, the Welcome Trust Case Control Consortium published genome scans of seven common diseases [WTCCC, 2007].”
BackgroundResultInfluentialTurner, L., Boutron, I., Hróbjartsson, A., Altman, D. G., & Moher, D. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-79 · Cited by 62 · see citing works
“In addition, some studies have investigated why some outcomes are omitted from published reports [41,88-90] as well as the impact of omission of outcomes on the findings of meta-analyses [91].”
BackgroundSelf-citationWelton, N. J., Cooper, N. J., Ades, A. E., Lu, G., & Sutton, A. J. (2008) · Statistics in Medicine · doi:10.1002/sim.3377 · Cited by 60 · see citing works
Brown, A. W., Aslibekyan, S. W., Bier, D. M., da Silva, R. F., Hoover, A. W., Klurfeld, D. M., Loken, E., Mayo‐Wilson, E., Menachemi, N., Pavela, G., et al. (2021) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2021.1985427 · Cited by 57 · see citing works
“The FFQ, which is most commonly applied in large cohort studies, does not accurately estimate frequency of intake or gauge serving size (Willett et al. 1987).”
MethodSinha, I. P., Williamson, P. R., & Smyth, R. L. (2009) · PLoS ONE · doi:10.1371/journal.pone.0006276 · Cited by 57 · see citing works
“It has also been suggested that arbitrary or inconsistent outcome selection may lead to clinical trials with unnecessarily large sample sizes [17] and reporting biases [17,18].”
BackgroundSelf-citationComparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K. M., Altman, D. G., Blundell, M., Gamble, C. L., & Williamson, P. R. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031 · Cited by 56 · see citing works
Self-citationDwan, K., Kirkham, J. J., Williamson, P. R., & Gamble, C. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-002709 · Cited by 55 · see citing works
Self-citationMoreno, S. G., Sutton, A. J., Ades, A. E., Cooper, N. J., & Abrams, K. R. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.009 · Cited by 55 · see citing works
“Although traditionally publication biases have been seen as the main reason for the observed smallstudy effects in most meta-analyses [9,31,32], the smallstudy effects phenomenon can be induced by a variety of factors besides publication biases [23,33,34].”
BackgroundPeters, J., & Mengersen, K. L. (2008) · Evaluation & the Health Professions · doi:10.1177/0163278708324438 · Cited by 52 · see citing works
Choosing patient-reported outcome measures for shoulder pathology
Padua, R., de Girolamo, L., Grassi, A., & Cucchi, D. (2021) · EFORT Open Reviews · doi:10.1302/2058-5241.6.200109 · Cited by 48 · see citing works
Hedayati, S., Damghanian, H., Farhadinejad, M., & Rastgar, A. A. (2023) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2023.103758 · Cited by 46 · see citing works
“The foremost bias is systematic review of publishing bias in all studies (Williamson et al., 2005).”
Vitamin D supplementation for chronic liver diseases in adults
Bjelaković, M., Nikolova, D., Bjelaković, G., & Gluud, C. N. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011564.pub3 · Cited by 45 · see citing works
on behalf of the ROMIO study group, BROOKES, S. T., Chalmers, K. A., AVERY, K. N. L., Coulman, K. D., & Blazeby, J. M. (2018) · Trials · doi:10.1186/s13063-017-2405-6 · Cited by 44 · see citing works
Sharma, H., & Verma, S. (2019) · Saudi Journal of Anaesthesia · doi:10.4103/sja.sja_124_19 · Cited by 43 · see citing works
Kitsios, G. D., Dahabreh, I. J., Trikalinos, T. A., Schmid, C. H., Huggins, G. S., & Kent, D. M. (2010) · Circulation Cardiovascular Genetics · doi:10.1161/circgenetics.110.957738 · Cited by 37 · see citing works
The use of SSRIs in children and adolescents
Hetrick, S. E., McKenzie, J. E., & Merry, S. (2009) · Current Opinion in Psychiatry · doi:10.1097/yco.0b013e328334bc92 · Cited by 36 · see citing works
Mayo‐Wilson, E., Fusco, N., Hong, H., Li, T., Canner, J. K., & Dickersin, K. (2019) · Trials · doi:10.1186/s13063-019-3581-3 · Cited by 37 · see citing works
Montroy, J. R., Fergusson, N. A., Hutton, B., Lavallée, L. T., MORASH, C. G., Cagiannos, I., Cnossen, S., Fergusson, D. A., & Breau, R. H. (2017) · Transfusion Medicine Reviews · doi:10.1016/j.tmrv.2017.03.002 · Cited by 35 · see citing works
Chong, N. J., & Aziz, Z. (2013) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2013/627182 · Cited by 34 · see citing works
“Selective outcome reporting has been defined as the selective reporting in a publication of only a selection of outcomes, perhaps those based on statistically significant results [53].”
BackgroundGreenberg, L., Jairath, V., Pearse, R. M., & Kahan, B. C. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.05.023 · Cited by 34 · see citing works
Altman, D. G., Furberg, C. D., Grimshaw, J., & Rothwell, P. M. (2006) · Trials · doi:10.1186/1745-6215-7-6 · Cited by 33 · see citing works
“Compounding the problems of non-publication of some studies, especially those with non-significant results, it is clear that authors may report their results selectively, emphasising those that are statistically significant [5].”
BackgroundSelf-citationStandardising Outcomes in Paediatric Clinical Trials
Clarke, M. J. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050102 · Cited by 31 · see citing works
“Reaching agreement on standard outcomes for clinical trials in children is important to enable researchers and clinicians to compare, contrast, and combine the findings of these trials. Such agreement should also help to reduce the possibility of biased reporting. If researchers have a choice of outcomes, they might b…”
BackgroundLacosamide add-on therapy for partial epilepsy
Weston, J., Shukralla, A. A., McKay, A. J., & Marson, A. G. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008841.pub2 · Cited by 30 · see citing works
Du, Y., Zhang, Y., Bai, W., Li, R.-Y., Sun, R., & Wang, N. (2021) · International Journal of Ophthalmology · doi:10.18240/ijo.2022.01.21 · Cited by 30 · see citing works
Page, M. J., McKenzie, J. E., Dwan, K. M., Kirkham, J. J., Green, S., & Forbes, A. B. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035 · Cited by 30 · see citing works
Application and investigation of a bound for outcome reporting bias
Williamson, P. R., & Gamble, C. (2007) · Trials · doi:10.1186/1745-6215-8-9 · Cited by 26 · see citing works
Self-citationCarregaro, R. L., Roscani, A. N. C. P., Raimundo, A. C. S., Ferreira, L., Vanni, T., Salomão, M. D. G., Probst, L. F., & Viscondi, J. Y. K. (2023) · BMC Infectious Diseases · doi:10.1186/s12879-023-08541-0 · Cited by 25 · see citing works
“The quality assessments raised concerns regarding selection bias, suggesting that important studies may have been excluded from the systematic reviews [58].”
BackgroundResultConstitutional mechanisms of vulnerability and resilience to nicotine dependence
Hiroi, N., & Scott, D. (2009) · Molecular Psychiatry · doi:10.1038/mp.2009.16 · Cited by 25 · see citing works
Identification of Additional Trials in Prospective Trial Registers for Cochrane Systematic Reviews
van Enst, W. A., Scholten, R. J. P. M., & Hooft, L. (2012) · PLoS ONE · doi:10.1371/journal.pone.0042812 · Cited by 24 · see citing works
“Empirical evidence consistently suggests that statistically significant and positive findings are more likely to be published than non-significant or negative findings [3,4] and will take shorter time to be submitted and to get published after completion of the study [5–8].”
BackgroundChanging the logic of replication: A case from infant studies
Margoni, F., & Shepperd, M. (2020) · Infant Behavior and Development · doi:10.1016/j.infbeh.2020.101483 · Cited by 20 · see citing works
Bryant, A., Johnson, E. E., Grayling, M. J., Hiu, S. K. W., Elattar, A., Gajjar, K. B., Craig, D., Vale, L. D., & Naik, R. (2022) · American Journal of Therapeutics · doi:10.1097/mjt.0000000000001584 · Cited by 22 · see citing works
Rogozińska, E., Marlin, N., Thangaratinam, S., Khan, K. S., & Zamora, J. (2017) · BMJ Evidence-Based Medicine · doi:10.1136/ebmed-2017-110775 · Cited by 21 · see citing works
Boluyt, N., van der Lee, J. H., Moyer, V. A., Brand, P. L. P., & Offringa, M. (2007) · PEDIATRICS · doi:10.1542/peds.2006-3381 · Cited by 20 · see citing works
Cro, S., Forbes, G., Johnson, N. A., & Kahan, B. C. (2020) · BMC Medicine · doi:10.1186/s12916-020-01590-1 · Cited by 19 · see citing works
Vision Research Literature May Not Represent the Full Intellectual Range of Autism Spectrum Disorder
Brown, A., Chouinard, P. A., & Crewther, S. G. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00057 · Cited by 19 · see citing works
“In brief, it is difficult to disentangle the factors effecting and characterizing visual perception in ASD with-ID both because of the common practice of selection bias and differences in ability to attend and comprehend verbally based task prescriptions (Williamson et al., 2005).”
Background
- Authors
- Williamson, P. R., Gamble, C., Altman, D. G., & Hutton, J. L.
- Affiliations
- University of Liverpool (GB)
- University of Oxford (GB)
- University of Warwick (GB)
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2005-10
- Volume
- 14
- Issue
- 5
- Pages
- 515-524
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W2056439589
- Cited by
- 188
- References
- 35
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 4 · 2023: 10 · 2022: 4 · 2021: 18 · 2020: 7 · 2019: 9 · 2018: 6 · 2017: 9 · 2016: 9 · 2015: 10 · 2014: 8 · 2013: 11 · 2012: 8
- FWCI
- 10.34
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Statistical Methods in Medical Research (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- selective reporting, publication bias, meta-analysis validity, systematic reviews, trial outcomes
- MeSH terms
- Clinical Trials as Topic, United Kingdom, Humans, Publications, Meta-Analysis as Topic, Reproducibility of Results, Bias, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
