The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Outcome selection bias in meta-analysis
Abstract
Publication bias has been previously identified as a threat to the validity of a meta-analysis. Recently, new evidence has documented an additional threat to validity, the selective reporting of trial outcomes within published studies. Several diseases have several possible measures of outcome. Some articles might report only a selection of those outcomes, perhaps those with statistically significant results. In this article, we review this problem while addressing the questions: what is within-study selective reporting? how common is it? why is it done? how can it mislead? how can it be detected?, and finally, what is the solution? We recommend that both publication bias and selective reporting should be routinely investigated in systematic reviews.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
Journal of the American Statistical Association
Fildes, R. A. (1989) · International Journal of Forecasting · doi:10.1016/0169-2070(89)90106-4 · Cited by 3,956
誠一, 市. (2004) · The Journal of AIDS Research · doi:10.11391/aidsr1999.6.104 · Cited by 3,233
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955
Statistical Methods for Meta-Analysis
(1985) · Elsevier eBooks · doi:10.1016/c2009-0-03396-0 · Cited by 2,570
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Self-citationSystematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693
Cochrane Injuries Group Albumin ReviewersWhy albumin may not work
Berger, A. (1998) · BMJ · doi:10.1136/bmj.317.7153.235 · Cited by 1,080
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022
Research works printed but not linked: 1
- Hutton JL, 2000, Applied Statistics, 49, 359
Cited by
Sorted by citations
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
“For example, in a particular disease, if one of two linked outcomes is reported but the other is not, then one should question whether the latter has been selectively omitted [121,122].”
BackgroundSelf-citationCochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Self-citationShamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
“For example, in a particular disease, if one of two linked outcomes is reported but the other is not, then one should question whether the latter has been selectively omitted [121,122].”
BackgroundSelf-citation(2008) · Cited by 3,790
Self-citationAssessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Italian Journal of Public Health · doi:10.2427/5768 · Cited by 3,374
Self-citationPeters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546
“For example, selective reporting of analyses may include intention-to–treat analyses versus per–protocol analyses, endpoint score versus change from baseline, different time points or subgroups [10].”
BackgroundSelf-citationDwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group (2013) · PLoS ONE · doi:10.1371/journal.pone.0066844 · Cited by 1,127
“For example, selective reporting of analyses may include intention-to–treat analyses versus per–protocol analyses, endpoint score versus change from baseline, different time points or subgroups [11].”
BackgroundSelf-citationJansen, J. P., Fleurence, R., Devine, B., Itzler, R. F., Barrett, A., Hawkins, N., Lee, K., Boersma, C., Annemans, L., & Cappelleri, J. C. (2011) · Value in Health · doi:10.1016/j.jval.2011.04.002 · Cited by 1,029
Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK): Explanation and Elaboration
Altman, D. G., McShane, L. M., Sauerbrei, W., & Taube, S. E. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001216 · Cited by 1,034
“Evidence of biased non-publication of whole studies has been accumulating for many years, but recently research has demonstrated evidence of additional within-study selective reporting [205,206].”
ResultSelf-citationSinha, I. P., Smyth, R. L., & Williamson, P. R. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1000393 · Cited by 891
“As a result, the standardisation of outcomes for clinical trials has been proposed as a solution to the problems of inappropriate and non-uniform outcome selection [4,7] and reporting bias [5,8].”
BackgroundSelf-citationStatistical strategies for avoiding false discoveries in metabolomics and related experiments
Broadhurst, D., & Kell, D. B. (2006) · Metabolomics · doi:10.1007/s11306-006-0037-z · Cited by 857
“We note, however, that publication bias – the greater likelihood of publishing a positive than a negative result – can create optimistic inaccuracies in such analyses (Williamson et al., 2005).”
Chapple, C. R., Khullar, V., Gabriel, Z., Muston, D., Bitoun, C. E., & Weinstein, D. L. (2008) · European Urology · doi:10.1016/j.eururo.2008.06.047 · Cited by 768
“Finally, a systematic review is to some extent reliant on the unbiased selection of outcomes in the reviewed studies [52].”
BackgroundOpioid antagonists for alcohol dependence
Srisurapanont, M., & Jarusuraisin, N. (2002) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001867 · Cited by 762
Standardising outcomes for clinical trials and systematic reviews
Clarke, M. (2007) · Trials · doi:10.1186/1745-6215-8-39 · Cited by 483
Acamprosate for alcohol dependence
Rösner, S., Hackl-Herrwerth, A., Leucht, S., Lehert, P., Vecchi, S., & Soyka, M. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004332.pub2 · Cited by 420
Reporting recommendations for tumor marker prognostic studies (REMARK): explanation and elaboration
Altman, D. G., McShane, L. M., Sauerbrei, W., & Taube, S. E. (2012) · BMC Medicine · doi:10.1186/1741-7015-10-51 · Cited by 378
“Evidence of biased non-publication of whole studies has been accumulating for many years, but recently research has demonstrated evidence of additional within-study selective reporting [205,206].”
ResultSelf-citationPeter, J. V., John, P., Graham, P. L., Moran, J. L., George, I. A., & Bersten, A. D. (2008) · BMJ · doi:10.1136/bmj.39537.939039.be · Cited by 351
Page, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035.pub2 · Cited by 281
Tandon, R., Belmaker, R. H., Gattaz, W. F., López-Ibor, J. J., Okasha, A., SINGH, B. S., Stein, D. J., Olié, J., Fleischhacker, W. W., & Moeller, H. (2008) · Schizophrenia Research · doi:10.1016/j.schres.2007.11.033 · Cited by 289
“, focusing on outcomes ‘favorable’ to the sponsor's product) (Heres et al., 2006; Williamson et al., 2005).”
BackgroundOpioid antagonists for alcohol dependence
Rösner, S., Hackl-Herrwerth, A., Leucht, S., Vecchi, S., Srisurapanont, M., & Soyka, M. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001867.pub3 · Cited by 255
Testing for the Presence of Positive-Outcome Bias in Peer Review: A Randomized Controlled Trial
Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.406 · Cited by 245
Comparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243
Self-citationBreckenridge, C. B., Berry, C. L., Chang, E. T., Sielken, R. L., & Mandel, J. S. (2016) · PLoS ONE · doi:10.1371/journal.pone.0151841 · Cited by 262
“In addition, it is important to examine studies to ensure that within-study selection bias is minimized, particularly when the results from a post-hoc subgroup analysis is the main focus of the study [197, 198].”
BackgroundFrequency and reasons for outcome reporting bias in clinical trials: interviews with trialists
Smyth, R. M. D., Kirkham, J. J., Jacoby, A., Altman, D. G., Gamble, C., & Williamson, P. R. (2011) · BMJ · doi:10.1136/bmj.c7153 · Cited by 220
Self-citationAntioxidant supplements for preventing gastrointestinal cancers
Bjelaković, G., Nikolova, D., Simonetti, R. G., & Gluud, C. N. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004183.pub3 · Cited by 219
Johnson, B. T., & Hennessy, E. A. (2019) · Social Science & Medicine · doi:10.1016/j.socscimed.2019.05.035 · Cited by 238
“Unfortunately, other than psychometric artefacts, which attenuate effect sizes (Schmidt and Hunter, 2015), and bias introduced with p-hacking or selective reporting (Kirkham et al., 2010a,b; Rubin, 2017; Williamson et al., 2005), which tend to exaggerate effects, little is known about the effects of other methodologic…”
BackgroundSinha, I. P., Jones, L. V., Smyth, R. L., & Williamson, P. R. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050096 · Cited by 202
“Disease-specific, universally agreed core sets of outcomes that should be measured and reported in all clinical trials of a specific condition, regardless of statistical significance, have been advocated as a solution to this common problem [35].”
Self-citationSchubert, E. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00837 · Cited by 203
Röver, C., Bender, R., Dias, S., Schmid, C. H., Schmidli, H., Sturtz, S., Weber, S., & Friede, T. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1475 · Cited by 209
“This might be compromised by selection effects, for example, publication bias(122) or reporting bias.(123) Especially in the case of only few studies, such effects might be hard to detect from the data, and information on the presence of selection effects may need to come from considerations of the context.”
BackgroundReducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 205
Papier, K., Fensom, G. K., Knuppel, A., Appleby, P. N., Tong, T. Y. N., Schmidt, J. A., Travis, R. C., Key, T. J., & Perez-Cornago, A. (2021) · BMC Medicine · doi:10.1186/s12916-021-01922-9 · Cited by 168
Replication in prevention science
Valentine, J. C., Biglan, A., Boruch, R. F., Castro, F. G., Collins, L. M., Flay, B. R., Kellam, S., Mościcki, E. K., & Schinke, S. P. (2011) · Prevention Science · doi:10.1007/s11121-011-0217-6 · Cited by 166
“If measured outcomes within studies are censored due to the lack of statistical significance, this can also lead to incorrect inferences (Williamson et al. 2005).”
BackgroundInterventions for helping people recognise early signs of recurrence in bipolar disorder
Morriss, R., Faizal, M. A., Jones, A. P., Williamson, P. R., Bolton, C. A., & McCarthy, J. P. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004854.pub2 · Cited by 166
Self-citationWebbe, J., Sinha, I. P., & Gale, C. (2017) · Archives of Disease in Childhood Education & Practice · doi:10.1136/archdischild-2016-312117 · Cited by 159
Joos, S., Miksch, A., Szécsényi, J., Wieseler, B., Grouven, U., Kaiser, T., & Schneider, A. (2008) · Thorax · doi:10.1136/thx.2007.081596 · Cited by 149
“significant treatment differences are reported in published articles, resulting in an overestimation of treatment effects.(40)”
BackgroundDevelopment of a core outcome set for research and audit studies in reconstructive breast surgery
Potter, S., Holcombe, C., Ward, J., Blazeby, J. M., Brookes, S. T., Cawthorn, S. J., Harcourt, D. M., Macefield, R. C., Warr, R., Weiler-Mithoff, E. M., et al. (2015) · British journal of surgery · doi:10.1002/bjs.9883 · Cited by 143
Self-citationConn, V. S., Ruppar, T. M., Enriquez, M., & Cooper, P. (2015) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2015.06.001 · Cited by 142
Evidence-based practice for rehabilitation professionals: concepts and controversies.
Dijkers, M., Murphy, S. L., & Krellman, J. W. (2012) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2011.12.014 · Cited by 112
Vera-Badillo, F. E., Shapiro, R., Ocana, A., Amir, E., & Tannock, I. F. (2013) · Annals of Oncology · doi:10.1093/annonc/mds636 · Cited by 117
Riley, P., Glenny, A., Worthington, H. V., Littlewood, A., Clarkson, J., & McCabe, M. G. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011552.pub2 · Cited by 125
Under-reporting of harm in clinical trials
Šeruga, B., Templeton, A. J., Badillo, F. E. V., Ocaña, A., Amir, E., & Tannock, I. F. (2016) · The Lancet Oncology · doi:10.1016/s1470-2045(16)00152-2 · Cited by 110
Moran, J. L., Graham, P. L., Rockliff, S., & Bersten, A. D. (2010) · Critical Care · doi:10.1186/cc9182 · Cited by 105
“Secondary outcome analysis was beset by selection bias in reporting [65], as witnessed by study numbers in Table 5; parameter estimates may be biased under such circumstances.”
BackgroundBjelaković, G., Nikolova, D., Simonetti, R. G., & Gluud, C. N. (2008) · Alimentary Pharmacology & Therapeutics · doi:10.1111/j.1365-2036.2008.03785.x · Cited by 105
Future of the Language Models in Healthcare: The Role of Chatgpt
Tustumi, F., Andreollo, N. A., & Aguilar-Nascimento, J. E. D. (2023) · ABCD. Arquivos Brasileiros de Cirurgia Digestiva (São Paulo) · doi:10.1590/0102-672020230002e1727 · Cited by 99
“Another relevant issue in evidence-based guidelines and systematic reviews refers to the risk of selection bias 10,22,23 .”
BackgroundBradley, H. A., Rucklidge, J. J., & Mulder, R. (2016) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12647 · Cited by 95
Vitamin D supplementation for chronic liver diseases in adults
Bjelaković, G., Nikolova, D., Bjelaković, M., & Gluud, C. N. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011564.pub2 · Cited by 95
Röver, C., Bender, R., Dias, S., Schmid, C. H., Schmidli, H., Sturtz, S., Weber, S., & Friede, T. (2021) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 89
Few systematic reviews exist documenting the extent of bias: a systematic review
Tricco, A. C., Tetzlaff, J., Sampson, M., Fergusson, D. A., Cogo, E., Horsley, T., & Moher, D. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.10.017 · Cited by 82
Assessment of chronic post‐surgical pain after knee replacement: Development of a core outcome set
Wylde, V., MacKichan, F., Bruce, J., & Gooberman‐Hill, R. (2014) · European Journal of Pain · doi:10.1002/ejp.582 · Cited by 79
“Standardization of outcomes assessment can reduce heterogeneity between trials, thereby aiding data synthesis (Clarke, 2007) and minimizing the risk of outcome reporting bias (Williamson et al., 2005).”
BackgroundRiley, P., Glenny, A., Worthington, H. V., Littlewood, A., Mauleffinch, L. M. F., Clarkson, J., & McCabe, M. G. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011990.pub2 · Cited by 86
Bürkner, P., Williams, D. R., Simmons, T. C., & Woolley, J. D. (2017) · Schizophrenia Bulletin · doi:10.1093/schbul/sbx053 · Cited by 82
Page, M. J., McKenzie, J. E., & Forbes, A. B. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.10.010 · Cited by 75
Fritz, A., Scherndl, T., & Kühberger, A. (2012) · Theory & Psychology · doi:10.1177/0959354312436870 · Cited by 77
Antioxidant supplements for liver diseases
Bjelaković, G., Gluud, L. L., Nikolova, D., Bjelaković, M., Nagorni, A. V., & Gluud, C. N. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007749.pub2 · Cited by 70
Antioxidant supplements for preventing gastrointestinal cancers
Bjelaković, G., Nikolova, D., Simonetti, R., & Gluud, C. N. (2004) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004183.pub2 · Cited by 70
Riley, P., Glenny, A., Worthington, H. V., Littlewood, A., Clarkson, J., & McCabe, M. G. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011552 · Cited by 71
Milette, K., Roseman, M., & Thombs, B. D. (2010) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2010.09.015 · Cited by 67
“Within-study selective outcome reporting, on the other hand, relates to published studies and refers to the selection of outcomes to report based on statistical significance, including the prioritization of time points for analysis, selective reporting of a subset of measures, and data-driven selection or switching of…”
ResultXu, Z., Zhou, Y., Shao, B., Zhang, J., & Liu, C. (2019) · The American Journal of Chinese Medicine · doi:10.1142/s0192415x19500265 · Cited by 64
Assessing the potential for outcome reporting bias in a review: a tutorial
Dwan, K. M., Gamble, C. L., Kolamunnage‐Dona, R., Mohammed, S., Powell, C., & Williamson, P. R. (2010) · Trials · doi:10.1186/1745-6215-11-52 · Cited by 66
“This is especially important to consider when there is a large proportion of participants in eligible studies which have not been included in the meta-analysis compared to those that are included in the meta-analysis [32].”
ResultSelf-citationMeta‐analysis: antioxidant supplements for liver diseases – the Cochrane Hepato‐Biliary Group
Bjelaković, G., Gluud, L. L., Nikolova, D., Bjelaković, M., Nagorni, A. V., & Gluud, C. N. (2010) · Alimentary Pharmacology & Therapeutics · doi:10.1111/j.1365-2036.2010.04371.x · Cited by 63
Unbiased estimation of odds ratios: combining genomewide association scans with replication studies
Bowden, J., & Dudbridge, F. (2009) · Genetic Epidemiology · doi:10.1002/gepi.20394 · Cited by 61
“In 2007, the Welcome Trust Case Control Consortium published genome scans of seven common diseases [WTCCC, 2007].”
BackgroundResultInfluentialTurner, L., Boutron, I., Hróbjartsson, A., Altman, D. G., & Moher, D. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-79 · Cited by 66
“In addition, some studies have investigated why some outcomes are omitted from published reports [41,88-90] as well as the impact of omission of outcomes on the findings of meta-analyses [91].”
BackgroundSelf-citationWelton, N. J., Cooper, N. J., Ades, A. E., Lu, G., & Sutton, A. J. (2008) · Statistics in Medicine · doi:10.1002/sim.3377 · Cited by 60
Brown, A. W., Aslibekyan, S. W., Bier, D. M., da Silva, R. F., Hoover, A. W., Klurfeld, D. M., Loken, E., Mayo‐Wilson, E., Menachemi, N., Pavela, G., et al. (2021) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2021.1985427 · Cited by 63
“The FFQ, which is most commonly applied in large cohort studies, does not accurately estimate frequency of intake or gauge serving size (Willett et al. 1987).”
MethodSinha, I. P., Williamson, P. R., & Smyth, R. L. (2009) · PLoS ONE · doi:10.1371/journal.pone.0006276 · Cited by 59
“It has also been suggested that arbitrary or inconsistent outcome selection may lead to clinical trials with unnecessarily large sample sizes [17] and reporting biases [17,18].”
BackgroundSelf-citationComparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K. M., Altman, D. G., Blundell, M., Gamble, C. L., & Williamson, P. R. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031 · Cited by 59
Self-citationDwan, K., Kirkham, J. J., Williamson, P. R., & Gamble, C. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-002709 · Cited by 55
Self-citationMoreno, S. G., Sutton, A. J., Ades, A. E., Cooper, N. J., & Abrams, K. R. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.009 · Cited by 54
“Although traditionally publication biases have been seen as the main reason for the observed smallstudy effects in most meta-analyses [9,31,32], the smallstudy effects phenomenon can be induced by a variety of factors besides publication biases [23,33,34].”
BackgroundPeters, J., & Mengersen, K. L. (2008) · Evaluation & the Health Professions · doi:10.1177/0163278708324438 · Cited by 52
Choosing patient-reported outcome measures for shoulder pathology
Padua, R., de Girolamo, L., Grassi, A., & Cucchi, D. (2021) · EFORT Open Reviews · doi:10.1302/2058-5241.6.200109 · Cited by 48
Hedayati, S., Damghanian, H., Farhadinejad, M., & Rastgar, A. A. (2023) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2023.103758 · Cited by 46
“The foremost bias is systematic review of publishing bias in all studies (Williamson et al., 2005).”
Vitamin D supplementation for chronic liver diseases in adults
Bjelaković, M., Nikolova, D., Bjelaković, G., & Gluud, C. N. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011564.pub3 · Cited by 49
on behalf of the ROMIO study group, BROOKES, S. T., Chalmers, K. A., AVERY, K. N. L., Coulman, K. D., & Blazeby, J. M. (2018) · Trials · doi:10.1186/s13063-017-2405-6 · Cited by 44
Sharma, H., & Verma, S. (2019) · Saudi Journal of Anaesthesia · doi:10.4103/sja.sja_124_19 · Cited by 44
Kitsios, G. D., Dahabreh, I. J., Trikalinos, T. A., Schmid, C. H., Huggins, G. S., & Kent, D. M. (2010) · Circulation Cardiovascular Genetics · doi:10.1161/circgenetics.110.957738 · Cited by 37
The use of SSRIs in children and adolescents
Hetrick, S. E., McKenzie, J. E., & Merry, S. (2009) · Current Opinion in Psychiatry · doi:10.1097/yco.0b013e328334bc92 · Cited by 36
Mayo-Wilson, E., Fusco, N., Hong, H., Li, T., Canner, J. K., & Dickersin, K. (2019) · Trials · doi:10.1186/s13063-019-3581-3 · Cited by 39
Montroy, J. R., Fergusson, N. A., Hutton, B., Lavallée, L. T., MORASH, C. G., Cagiannos, I., Cnossen, S., Fergusson, D. A., & Breau, R. H. (2017) · Transfusion Medicine Reviews · doi:10.1016/j.tmrv.2017.03.002 · Cited by 35
Chong, N. J., & Aziz, Z. (2013) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2013/627182 · Cited by 34
“Selective outcome reporting has been defined as the selective reporting in a publication of only a selection of outcomes, perhaps those based on statistically significant results [53].”
BackgroundGreenberg, L., Jairath, V., Pearse, R. M., & Kahan, B. C. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.05.023 · Cited by 34
Altman, D. G., Furberg, C. D., Grimshaw, J., & Rothwell, P. M. (2006) · Trials · doi:10.1186/1745-6215-7-6 · Cited by 33
“Compounding the problems of non-publication of some studies, especially those with non-significant results, it is clear that authors may report their results selectively, emphasising those that are statistically significant [5].”
BackgroundSelf-citationStandardising Outcomes in Paediatric Clinical Trials
Clarke, M. J. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050102 · Cited by 31
“Reaching agreement on standard outcomes for clinical trials in children is important to enable researchers and clinicians to compare, contrast, and combine the findings of these trials. Such agreement should also help to reduce the possibility of biased reporting. If researchers have a choice of outcomes, they might b…”
BackgroundLacosamide add-on therapy for partial epilepsy
Weston, J., Shukralla, A. A., McKay, A. J., & Marson, A. G. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008841.pub2 · Cited by 30
Du, Y., Zhang, Y., Bai, W., Li, R.-Y., Sun, R., & Wang, N. (2021) · International Journal of Ophthalmology · doi:10.18240/ijo.2022.01.21 · Cited by 30
Page, M. J., McKenzie, J. E., Dwan, K. M., Kirkham, J. J., Green, S., & Forbes, A. B. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035 · Cited by 31
Application and investigation of a bound for outcome reporting bias
Williamson, P. R., & Gamble, C. (2007) · Trials · doi:10.1186/1745-6215-8-9 · Cited by 26
Self-citationCarregaro, R. L., Roscani, A. N. C. P., Raimundo, A. C. S., Ferreira, L., Vanni, T., Salomão, M. D. G., Probst, L. F., & Viscondi, J. Y. K. (2023) · BMC Infectious Diseases · doi:10.1186/s12879-023-08541-0 · Cited by 25
“The quality assessments raised concerns regarding selection bias, suggesting that important studies may have been excluded from the systematic reviews [58].”
BackgroundResultConstitutional mechanisms of vulnerability and resilience to nicotine dependence
Hiroi, N., & Scott, D. (2009) · Molecular Psychiatry · doi:10.1038/mp.2009.16 · Cited by 26
Identification of Additional Trials in Prospective Trial Registers for Cochrane Systematic Reviews
van Enst, W. A., Scholten, R. J. P. M., & Hooft, L. (2012) · PLoS ONE · doi:10.1371/journal.pone.0042812 · Cited by 24
“Empirical evidence consistently suggests that statistically significant and positive findings are more likely to be published than non-significant or negative findings [3,4] and will take shorter time to be submitted and to get published after completion of the study [5–8].”
BackgroundChanging the logic of replication: A case from infant studies
Margoni, F., & Shepperd, M. (2020) · Infant Behavior and Development · doi:10.1016/j.infbeh.2020.101483 · Cited by 22
Bryant, A., Johnson, E. E., Grayling, M. J., Hiu, S. K. W., Elattar, A., Gajjar, K. B., Craig, D., Vale, L. D., & Naik, R. (2022) · American Journal of Therapeutics · doi:10.1097/mjt.0000000000001584 · Cited by 22
Rogozińska, E., Marlin, N., Thangaratinam, S., Khan, K. S., & Zamora, J. (2017) · BMJ Evidence-Based Medicine · doi:10.1136/ebmed-2017-110775 · Cited by 22
Boluyt, N., van der Lee, J. H., Moyer, V. A., Brand, P. L. P., & Offringa, M. (2007) · PEDIATRICS · doi:10.1542/peds.2006-3381 · Cited by 20
Cro, S., Forbes, G., Johnson, N. A., & Kahan, B. C. (2020) · BMC Medicine · doi:10.1186/s12916-020-01590-1 · Cited by 20
Vision Research Literature May Not Represent the Full Intellectual Range of Autism Spectrum Disorder
Brown, A., Chouinard, P. A., & Crewther, S. G. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00057 · Cited by 21
“In brief, it is difficult to disentangle the factors effecting and characterizing visual perception in ASD with-ID both because of the common practice of selection bias and differences in ability to attend and comprehend verbally based task prescriptions (Williamson et al., 2005).”
Background
Metadata
- Authors
- Williamson, P. R., Gamble, C., Altman, D. G., & Hutton, J. L.
- Affiliations
- University of Liverpool (GB)
- University of Oxford (GB)
- University of Warwick (GB)
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2005-10
- Volume
- 14
- Issue
- 5
- Pages
- 515-524
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W2056439589
- Cited by
- 188
- References
- 35
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 4 · 2023: 10 · 2022: 4 · 2021: 18 · 2020: 7 · 2019: 9 · 2018: 6 · 2017: 9 · 2016: 9 · 2015: 10 · 2014: 8 · 2013: 11 · 2012: 8
- FWCI
- 10.34
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Statistical Methods in Medical Research (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- selective reporting, publication bias, meta-analysis validity, systematic reviews, trial outcomes
- MeSH terms
- Clinical Trials as Topic, United Kingdom, Humans, Publications, Meta-Analysis as Topic, Reproducibility of Results, Bias, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
