Identification and impact of outcome selection bias in meta‐analysis
Abstract
The systematic review community has become increasingly aware of the importance of addressing the issues of heterogeneity and publication bias in meta‐analyses. A potentially bigger threat to the validity of a meta‐analysis appears relatively unnoticed. The within‐study selective reporting of outcomes, defined as the selection of a subset of the original variables recorded for inclusion in publication of trials, can theoretically have a substantial impact on the results. A cohort of meta‐analyses on the Cochrane Library was reviewed to examine how often this form of within‐study publication bias was suspected and explained some of the evident funnel plot asymmetry. In cases where the level of suspicion was high, sensitivity analysis was undertaken to assess the robustness of the conclusion to this bias. Although within‐study selection was evident or suspected in several trials, the impact on the conclusions of the meta‐analyses was minimal. This paper deals with the identification of, sensitivity analysis for, and impact of within‐study selective reporting in meta‐analysis. Copyright © 2004 John Wiley & Sons, Ltd.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
One crucial aim of the protocol is to prevent the selective communication of results, known as results bias.
“One crucial aim of the protocol is to prevent the selective communication of results, known as results bias (Williamson and Gamble 2005).”
From Adherence to the CONSORT statement of randomized clinical trials on dentin hypersensitivity treatment using mouthwashes (2025) · cites this paper as Williamson and Gamble 2005
Reporting all additional analyses conducted, not just those that were statistically significant, helps avoid selective outcome reporting bias within the review.
“This information will help avoid selective outcome reporting bias within the review as has been demonstrated in reports of randomized controlled trials [42],[44],[121],[151],[152].”
From The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration (2009) · cites this paper as [121]
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · see citing works
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955 · see citing works
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · see citing works
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955 · see citing works
Empirical assessment of effect of publication bias on meta-analyses
Sutton, A. J. (2000) · BMJ · doi:10.1136/bmj.320.7249.1574 · Cited by 1,398 · see citing works
Cochrane Injuries Group Albumin ReviewersWhy albumin may not work
Berger, A. (1998) · BMJ · doi:10.1136/bmj.317.7153.235 · Cited by 1,080 · see citing works
The Handbook of Research Synthesis
Hanka, R. (1994) · BMJ · doi:10.1136/bmj.309.6952.488a · Cited by 1,064 · see citing works
Publication bias and clinical trials
Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987) · Controlled Clinical Trials · doi:10.1016/0197-2456(87)90155-3 · Cited by 715 · see citing works
Content and quality of 2000 controlled trials in schizophrenia over 50 years
Thornley, B., & Adams, C. E. (1998) · BMJ · doi:10.1136/bmj.317.7167.1181 · Cited by 362 · see citing works
Is allergen immunotherapy effective in asthma? A meta-analysis of randomized controlled trials.
Abramson, M. J., Puy, R. M., & Weiner, J. M. (1995) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/ajrccm.151.4.7697274 · Cited by 313 · see citing works
Bias in meta‐analysis due to outcome variable selection within studies
Hutton, J. L., & Williamson, P. (2000) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/1467-9876.00197 · Cited by 236 · see citing works
Self-citationHahn, S., Williamson, P. R., & Hutton, J. L. (2002) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2002.00314.x · Cited by 97 · see citing works
Self-citationAn amnesty for unpublished trials
Smith, R. W., & Roberts, I. (1997) · BMJ · doi:10.1136/bmj.315.7109.622 · Cited by 70 · see citing works
Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349 · see citing works
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · see citing works
(2008) · Cited by 3,790 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Italian Journal of Public Health · doi:10.2427/5768 · Cited by 3,374 · see citing works
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · see citing works
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2012) · International Journal of Surgery · doi:10.1016/j.ijsu.2011.10.001 · Cited by 2,345 · see citing works
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · see citing works
“[13] are presented (using the same identifying number for each meta-analysis as in previous articles [13,18]).”
MethodMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.004 · Cited by 1,858 · see citing works
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · see citing works
“Reviewers should scrutinise trials with missing outcome data and ensure that an attempt to contact trialists is always made if the study does not report results. Also, the lack of reporting of specified outcome(s) should not be an automatic reason for exclusion of studies. Statisticians should be involved for the data…”
BackgroundSelf-citationDwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group (2013) · PLoS ONE · doi:10.1371/journal.pone.0066844 · Cited by 1,127 · see citing works
Self-citationKirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133 · see citing works
Self-citationDissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039 · see citing works
Probiotics for the prevention of Clostridium difficile-associated diarrhea in adults and children
Goldenberg, J. Z., Ma, S. S., Saxton, J. D., Martzen, M. R., Vandvik, P. O., Thorlund, K., Guyatt, G. H., & Johnston, B. C. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006095.pub3 · Cited by 741 · see citing works
Hopewell, S., Clarke, M., Moher, D., Wager, E., Middleton, P., Altman, D. G., Schulz, K. F., & and the CONSORT Group (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050020 · Cited by 674 · see citing works
“Nonpublication of entire trials and selective reporting of outcomes within trials has been well-documented [39,41,66].”
BackgroundAnderson, L. M., Quinn, T. A., Glanz, K., Ramirez, G., Kahwati, L. C., Johnson, D. B., Buchanan, L. R., Archer, W. R., Chattopadhyay, S., Kalra, G. P., et al. (2009) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2009.07.003 · Cited by 623 · see citing works
Recent developments in meta‐analysis
Sutton, A. J., & Higgins, J. P. T. (2007) · Statistics in Medicine · doi:10.1002/sim.2934 · Cited by 583 · see citing works
“Statistical methods have been put forward to assess the likely impact of selective outcome [143–145] and subgroup [146] reporting.”
MethodJohnston, B. C., Ma, S. S. Y., Goldenberg, J. Z., Thorlund, K., Vandvik, P. O., Loeb, M., & Guyatt, G. H. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-12-201212180-00563 · Cited by 367 · see citing works
“However, only 17 of 20 trials reported on adverse events, an outcome that would presumably be documented in all probiotics trials, and so we rated the quality of evidence down for selective reporting bias (51).”
BackgroundPeter, J. V., John, P., Graham, P. L., Moran, J. L., George, I. A., & Bersten, A. D. (2008) · BMJ · doi:10.1136/bmj.39537.939039.be · Cited by 351 · see citing works
Assessing risk of bias due to missing results in a synthesis
Page, M. J., Higgins, J. P., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch13 · Cited by 402 · see citing works
Bias Modelling in Evidence Synthesis
Turner, R., Spiegelhalter, D. J., Smith, G. C. S., & Thompson, S. G. (2008) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2008.00547.x · Cited by 322 · see citing works
“We do not here address publication bias or selective reporting of outcomes, for which methods have been discussed elsewhere (Rothstein et al., 2005; Williamson and Gamble, 2005).”
MethodRösner, S., Leucht, S., Lehert, P., & Soyka, M. (2007) · Journal of Psychopharmacology · doi:10.1177/0269881107078308 · Cited by 300 · see citing works
Page, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035.pub2 · Cited by 281 · see citing works
Comparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243 · see citing works
Self-citationCONSORT 2025 explanation and elaboration: updated guideline for reporting randomised trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081124 · Cited by 205 · see citing works
Outcome selection bias in meta-analysis
Williamson, P. R., Gamble, C., Altman, D. G., & Hutton, J. L. (2005) · Statistical Methods in Medical Research · doi:10.1191/0962280205sm415oa · Cited by 188 · see citing works
Self-citationA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165 · see citing works
A simulation study comparing properties of heterogeneity measures in meta‐analyses
Mittlböck, M., & Heinzl, H. (2006) · Statistics in Medicine · doi:10.1002/sim.2692 · Cited by 143 · see citing works
Detecting and adjusting for small‐study effects in meta‐analysis
Rücker, G., Carpenter, J. R., & Schwarzer, G. (2011) · Biometrical Journal · doi:10.1002/bimj.201000151 · Cited by 132 · see citing works
“…Tramèr et al., 1997); selective r 2010 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim www.biometrical-journal.com outcome reporting bias (Chan et al., 2004a,b; Williamson and Gamble, 2005), and citation bias (‘significant’ results are cited more often than studies without a ‘significant’ result,…”
BackgroundHenson, L. A., Gao, W., Higginson, I. J., Smith, M., Davies, J. M., Ellis-Smith, C. I., & Daveson, B. A. (2014) · Journal of Clinical Oncology · doi:10.1200/jco.2014.57.3568 · Cited by 129 · see citing works
“A number of different community services were investigated in relation to ED attendance at the EoL, including homecare services,28,29,34,40 general practitioner house calls,34 nursing care,29,40 and an interdisciplinary cancer support team.60 Results for each study are reported in the Data Supplement.”
MethodSalvo, F., Fourrier‐Réglat, A., Bazin, F., Robinson, P., Riera‐Guàrdia, N., Haag, M. D. M., Caputi, A. P., Moore, N. D., Sturkenboom, M., & Pariente, A. (2011) · Clinical Pharmacology & Therapeutics · doi:10.1038/clpt.2011.45 · Cited by 119 · see citing works
Kirkham, J. J., Riley, R. D., & Williamson, P. R. (2012) · Statistics in Medicine · doi:10.1002/sim.5356 · Cited by 119 · see citing works
“A previous study investigated the impact of selective outcome reporting in a review comparing beta-lactam and beta-lactam–aminoglycoside combination therapy in the treatment of cancer patients with neutropenia [10].”
BackgroundSelf-citationSmall-Study Effects in Meta-Analysis
Schwarzer, G., Carpenter, J., & Rücker, G. (2015) · Use R! · doi:10.1007/978-3-319-21416-0_5 · Cited by 105 · see citing works
Hannink, G., Gooszen, H. G., & Rovers, M. M. (2013) · Annals of Surgery · doi:10.1097/sla.0b013e3182864fa3 · Cited by 101 · see citing works
Heiligenhaus, A., Foeldvari, I., Edelsten, C., Smith, J. R., Saurenmann, R. K., Bodaghi, B., de Boer, J. H., GRAHAM, E. M., Antón, J., Kotaniemi, K., et al. (2012) · Arthritis Care & Research · doi:10.1002/acr.21674 · Cited by 101 · see citing works
“Disease-specific and universally agreed upon outcomes are likely to reduce selective reporting and reporting bias (31).”
BackgroundFuture of the Language Models in Healthcare: The Role of Chatgpt
Tustumi, F., Andreollo, N. A., & Aguilar-Nascimento, J. E. D. (2023) · ABCD. Arquivos Brasileiros de Cirurgia Digestiva (São Paulo) · doi:10.1590/0102-672020230002e1727 · Cited by 99 · see citing works
“Another relevant issue in evidence-based guidelines and systematic reviews refers to the risk of selection bias 10,22,23 .”
BackgroundMethodological quality of pharmacogenetic studies: Issues of concern
Jorgensen, A., & Williamson, P. R. (2008) · Statistics in Medicine · doi:10.1002/sim.3420 · Cited by 83 · see citing works
Self-citationSampath, S., Moran, J. L., Graham, P. L., Rockliff, S., Bersten, A. D., & Abrams, K. R. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000284503.88148.6f · Cited by 80 · see citing works
Page, M. J., McKenzie, J. E., & Forbes, A. B. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.10.010 · Cited by 75 · see citing works
Jorgensen, A., Alfirević, Ž., Smith, C. T., Williamson, P. R., & cerclage IPD Meta‐analysis Group (2007) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2007.01515.x · Cited by 76 · see citing works
Self-citationCleach, L. L., Trinquart, L., Do, G., Maruani, A., Lebrun‐Vignes, B., Ravaud, P., & Chosidow, O. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009036.pub2 · Cited by 72 · see citing works
Wen, X., Xiong, Y., Qu, X., Jin, L., Zhou, C., Zhang, M., & Zhang, Y. (2019) · Gynecological Endocrinology · doi:10.1080/09513590.2019.1590546 · Cited by 72 · see citing works
“First, the number of included studies was small, and only published papers in English were included, which means a part of negative results do not get published or published in non-English journals would not be included in the meta-analysis [60,61].”
BackgroundInterventions for helping people recognise early signs of recurrence in bipolar disorder
Morriss, R., Faizal, M. A., Jones, A. P., Williamson, P. R., Bolton, C. A., & McCarthy, J. P. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004854.pub2 · Cited by 166 · see citing works
Self-citationAvoidable waste of research related to outcome planning and reporting in clinical trials
Yordanov, Y., Dechartres, A., Atal, I., Tran, V.-T., Boutron, I., Créquit, P., & Ravaud, P. (2018) · BMC Medicine · doi:10.1186/s12916-018-1083-x · Cited by 66 · see citing works
Assessing the potential for outcome reporting bias in a review: a tutorial
Dwan, K. M., Gamble, C. L., Kolamunnage‐Dona, R., Mohammed, S., Powell, C., & Williamson, P. R. (2010) · Trials · doi:10.1186/1745-6215-11-52 · Cited by 64 · see citing works
“to assess the robustness of the conclusions of a metaanalysis to ORB [6,8].”
BackgroundSelf-citationBrown, A. W., Aslibekyan, S. W., Bier, D. M., da Silva, R. F., Hoover, A. W., Klurfeld, D. M., Loken, E., Mayo‐Wilson, E., Menachemi, N., Pavela, G., et al. (2021) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2021.1985427 · Cited by 57 · see citing works
“Reviews and meta-analyses (Williamson et al. 2005; Williamson and Gamble 2005; Goodman and Dickersin 2011; Mayo-Wilson, Li, et al. 2017) and scientific theories might be incorrect if they depend on a biased subsample of results and hypothesizing after the results are known (“HARKing”) (Kerr 1998).”
Comparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K. M., Altman, D. G., Blundell, M., Gamble, C. L., & Williamson, P. R. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031 · Cited by 56 · see citing works
Self-citationDwan, K., Kirkham, J. J., Williamson, P. R., & Gamble, C. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-002709 · Cited by 55 · see citing works
Self-citationMoreno, S. G., Sutton, A. J., Ades, A. E., Cooper, N. J., & Abrams, K. R. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.009 · Cited by 55 · see citing works
“Although traditionally publication biases have been seen as the main reason for the observed smallstudy effects in most meta-analyses [9,31,32], the smallstudy effects phenomenon can be induced by a variety of factors besides publication biases [23,33,34].”
BackgroundA systematic review of randomised controlled trials of interventions promoting effective condom use
Free, C., Roberts, I., Abramsky, T., Fitzgerald, M., & Wensley, F. (2009) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2008.085456 · Cited by 54 · see citing works
Moher, D. (2010) · Journal of Chinese Integrative Medicine · doi:10.3736/jcim20100801 · Cited by 53 · see citing works
Rongen, J. J., & Hannink, G. (2016) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.15.00400 · Cited by 53 · see citing works
Peters, J., & Mengersen, K. L. (2008) · Evaluation & the Health Professions · doi:10.1177/0163278708324438 · Cited by 52 · see citing works
“Alternatively, imputing missing data as a form of sensitivity analysis has been used to assess other forms of selective reporting (Williamson & Gamble, 2005).”
MethodAssessing the benefits of FEIBA prophylaxis in haemophilia patients with inhibitors
Valentino, L. A. (2009) · Haemophilia · doi:10.1111/j.1365-2516.2009.02126.x · Cited by 51 · see citing works
Norris, S. L., Moher, D., Reeves, B. C., Shea, B., Loke, Y., Garner, S., Anderson, L., Tugwell, P., & Wells, G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1062 · Cited by 50 · see citing works
van den Berg, I., Bosch, J. L., Jacobs, B. Z., Bouman, I., Duvekot, J. J., & Hunink, M. G. M. (2008) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2008.01.001 · Cited by 43 · see citing works
Synthesizing medical evidence: systematic reviews and metaanalyses
Barker, F. G., & Carter, B. S. (2005) · Neurosurgical FOCUS · doi:10.3171/foc.2005.19.4.6 · Cited by 41 · see citing works
Schmitt, J. M., Lange, T., Kottner, J., Prinsen, C. A. C., Weberschock, T. B., Hahnel, E., Apfelbacher, C., Brandstetter, S., Dreher, A., Stevens, G., et al. (2018) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2018.11.019 · Cited by 41 · see citing works
“Risk of bias Outcome reporting bias, the selection for publication of a subset of the originally recorded outcome variable (Williamson and Gamble, 2005), was more prevalent in our study than in a meta-analysis of general systematic reviews.”
ResultMayo‐Wilson, E., Fusco, N., Hong, H., Li, T., Canner, J. K., & Dickersin, K. (2019) · Trials · doi:10.1186/s13063-019-3581-3 · Cited by 37 · see citing works
Systematic review and meta-analysis of mortality in crop protection product manufacturing workers
Jones, D. R., Sutton, A. J., Abrams, K. R., Fenty, J., Warren, F. C., & Rushton, L. (2008) · Occupational and Environmental Medicine · doi:10.1136/oem.2007.035733 · Cited by 34 · see citing works
Page, M. J., McKenzie, J. E., Dwan, K. M., Kirkham, J. J., Green, S., & Forbes, A. B. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035 · Cited by 30 · see citing works
Useche, S. A., Colomer-Pérez, N., Alonso, F., & Faus, M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0296830 · Cited by 29 · see citing works
“Another potential limitation lies in selection bias, as some studies not have been identified during the search, leading to an incomplete representation of the existing literature [88].”
BackgroundMeier, P. S., & Barrowclough, C. (2009) · Drugs Education Prevention and Policy · doi:10.1080/09687630701741030 · Cited by 27 · see citing works
“As in all reviews, there is the potential problem of publication bias (the ‘bottom-drawer’ problem), and it is also common that authors select only Retention of dual diagnosis clients in drug treatment 33 a subset of the original variables to report in publications (Williamson & Gamble, 2005).”
BackgroundApplication and investigation of a bound for outcome reporting bias
Williamson, P. R., & Gamble, C. (2007) · Trials · doi:10.1186/1745-6215-8-9 · Cited by 26 · see citing works
“Constrained data were imputed for two reviews as follows [8].”
BackgroundMethodInfluentialSelf-citationSmall Study Effects in Diagnostic Imaging Accuracy
Lu, L., Phua, Q. S., Bacchi, S., Goh, R., Gupta, A. K., Kovoor, J. G., Ovenden, C. D., & To, M. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.28776 · Cited by 26 · see citing works
Papageorgiou, S. N., Papadopoulos, M. A. A. P., & Athanasiou, A. E. (2014) · Clinical Oral Investigations · doi:10.1007/s00784-014-1196-3 · Cited by 25 · see citing works
“Although publication bias is often regarded as the main reason for small study effects [5, 11], other factors may also exist [12], such as selective outcome reporting [11], a mathematical artifact due to discordant trial sizes [12], or a random error [13, 14].”
BackgroundPublication Bias & Small-study Effects in Pediatric Dentistry Meta-analyses
Papageorgiou, S. N., Dimitraki, D., Coolidge, T., & Kotsanos, N. (2014) · Journal of Evidence Based Dental Practice · doi:10.1016/j.jebdp.2014.09.001 · Cited by 24 · see citing works
Quality Assessment in Meta-analysis
Dreier, M. (2013) · Springer series on epidemiology and public health · doi:10.1007/978-3-642-37131-8_13 · Cited by 22 · see citing works
Yet another problem with systematic reviews: a living review update
Uttley, L., Weng, Y., & Falzon, L. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111608 · Cited by 23 · see citing works
Rogozińska, E., Marlin, N., Thangaratinam, S., Khan, K. S., & Zamora, J. (2017) · BMJ Evidence-Based Medicine · doi:10.1136/ebmed-2017-110775 · Cited by 21 · see citing works
Pitfalls in systematic reviews
Farquhar, C. M., & Vail, A. (2006) · Current Opinion in Obstetrics & Gynecology · doi:10.1097/01.gco.0000233939.44178.4b · Cited by 21 · see citing works
Liu, P., Qiu, Y., Qian, Y., Chen, X., Wang, Y., Cui, J., & Zhai, X. (2016) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.13591 · Cited by 20 · see citing works
Probiotics for the Prevention of Clostridium difficile–Associated Diarrhea
Johnston, B. C., Goldenberg, J. Z., & Guyatt, G. H. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-9-201305070-00019 · Cited by 20 · see citing works
Selective outcome reporting: telling and detecting true lies. The state of the science
Macura, A., Abraha, I., Kirkham, J., Gensini, G. F., Moja, L., & Iorio, A. (2010) · Internal and Emergency Medicine · doi:10.1007/s11739-010-0371-z · Cited by 19 · see citing works
“When the selection process occurs in relation to outcomes, we refer to it as selective outcome reporting [13].”
MethodNonpharmacological Interventions for Pediatric Migraine: A Network Meta-analysis
Koechlin, H., Kossowsky, J., Lam, T. L., Barthel, J., Gaab, J., Berde, C. B., Schwarzer, G., Linde, K., Meißner, K., & Locher, C. (2021) · PEDIATRICS · doi:10.1542/peds.2019-4107 · Cited by 19 · see citing works
Faus, M., Fernández, C., Alonso, F., & Useche, S. A. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e18775 · Cited by 19 · see citing works
“Consequently, selection bias may have occurred as there is a possibility that the selected advertisements do not fully represent the spectrum of publicity campaigns [82].”
BackgroundClinical and biological data integration for biomarker discovery
Sorani, M. D., Ortmann, W. A., Bierwagen, E. P., & Behrens, T. W. (2010) · Drug Discovery Today · doi:10.1016/j.drudis.2010.06.005 · Cited by 19 · see citing works
Sponsorship-Related Outcome Selection Bias in Published Economic Studies of Triptans
Peura, P. K., Martikainen, J. A., Purmonen, T. T., & Turunen, J. H. O. (2011) · Medical Decision Making · doi:10.1177/0272989x11403834 · Cited by 19 · see citing works
Richardson, M., Kirkham, J. J., Dwan, K. M., Sloan, D. J., Davies, G., & Jorgensen, A. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0533-4 · Cited by 14 · see citing works
Sinha, M. K., & Montori, V. M. (2006) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1586/14737167.6.5.603 · Cited by 14 · see citing works
Ethical considerations in the management of amyotrophic lateral sclerosis
Eisen, A., & Krieger, C. (2013) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2013.05.001 · Cited by 14 · see citing works
Solebo, A. L., Barry, R. J., Keane, P. A., Rahi, J. S., & Denniston, A. K. (2019) · Orphanet Journal of Rare Diseases · doi:10.1186/s13023-019-1108-3 · Cited by 13 · see citing works
Pearson, M., & Peters, J. (2011) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2010.122465 · Cited by 11 · see citing works
“Many limitations of this study relate to the heterogeneity of the studies included in the review, both in terms of their design and their reporting, which hindered (1) the identification of missing or incompletely reported outcomes, (2) using ORBIT to classify instances of missing or incompletely reported outcomes as…”
MethodMethods used to assess outcome consistency in clinical studies: A literature-based evaluation
Rogozińska, E., Gargon, E., Olmedo‐Requena, R., Asour, A., Cooper, N. A. M., Vale, C. L., & Hooft, J. V. ’. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235485 · Cited by 11 · see citing works
Pagel, P. S., & Crystal, G. J. (2021) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2021.04.026 · Cited by 11 · see citing works
Wang, Y., Guo, F., Chen, X., Yu, R., Qin, D., & Hua, F. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.105448 · Cited by 10 · see citing works
Alonso, F., Faus, M., Estéban, C., & Martí, B. (2025) · The European Journal of Psychology Applied to Legal Context · doi:10.5093/ejpalc2025a2 · Cited by 10 · see citing works
Completeness of Reporting in Randomized Controlled Trials of 3 Vaccines
Scott, P., Ott, F., Egger, M., & Low, N. N. (2012) · The Pediatric Infectious Disease Journal · doi:10.1097/inf.0b013e31827032bb · Cited by 9 · see citing works
“CONSORT items 1a: Trial identified as randomized in title Yes 14 (20, 11–31) 4 (19) 10 (20) 6a: Primary outcome nominated At least 1 nominated* 26 (37, 26–50) 6 (29) 20 (41) 8: Method of random allocation sequence generation reported Fully described 24 (34, 23–47) 6 (29) 18 (37) Partially described 9 (13, 6–23) 4 (19)…”
Comments on ‘Empirical vs natural weighting in random effects meta‐analysis’
Thompson, S. G., & Higgins, J. P. T. (2010) · Statistics in Medicine · doi:10.1002/sim.3718 · Cited by 9 · see citing works
Do meta-analyses in nephrology change the way we treat patients?
Mohan, S., & Radhakrishnan, J. (2010) · Kidney International · doi:10.1038/ki.2010.323 · Cited by 8 · see citing works
“(4) Outcome selection bias: This results from the selection of a subset of the original variables recorded for inclusion in publication of trials.(62,63) The selective reporting of data in trials is widespread but this potential bias does not appear to have significantly impacted recent meta-analyses.”
BackgroundWang, A., Menon, R., Li, T., Harris, L. A., Harris, I. A., Naylor, J., & Adie, S. (2023) · ANZ Journal of Surgery · doi:10.1111/ans.18273 · Cited by 8 · see citing works
“Outcome reporting bias, therefore, threatens the validity of systematic reviews and meta-analyses, and skews their conclusions to produce inflated estimates of treatment effect.(2,3) Because of this, clinicians may be compelled to prescribe interventions which offer negligible benefits to their patients.”
BackgroundPooled perspectives: Critical considerations for the modern meta-analysis
Justo, M., Askin, H., Mehta, D., & Benharash, P. (2025) · Surgery Open Science · doi:10.1016/j.sopen.2025.03.005 · Cited by 6 · see citing works
“Researchers can inadvertently introduce inconsistencies by selectively choosing studies based on specific patient populations or outcomes [9,10].”
The methodological quality of systematic reviews regarding the Core Outcome Set (COS) development
Cao, H., Chen, Y., Yang, Z., Lan, J., Kwong, J. S. W., Zhang, R., Zhao, H., Hu, L., Wang, J., Sun, S., et al. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02182-w · Cited by 6 · see citing works
Examining Bias in Published Surgical Glaucoma Clinical Trials
Michael, R., Zhang, H., McIntyre, S., Cape, L., & Toren, A. (2023) · Journal of Glaucoma · doi:10.1097/ijg.0000000000002262 · Cited by 6 · see citing works
“11 Comparing clinical trials to its registry allows the identi fi cation of any changes in primary and secondary outcomes after data collection, which helps reduce ORB.”
BackgroundMcKenzie, J. E., Herbison, P., Roth, P., & Paul, C. (2010) · Trials · doi:10.1186/1745-6215-11-28 · Cited by 6 · see citing works
“In addition to misrepresenting the results from the RCT, bias arising from selective reporting can impact on meta-analyses [14], influencing the conclusions of systematic reviews, and in turn, evidence based clinical practice guidelines [19-21].”
BackgroundLee, S., Khan, T. R., Grindlay, D. J. C., & Karantana, A. (2018) · JBJS Open Access · doi:10.2106/jbjs.oa.17.00065 · Cited by 6 · see citing works
Holper, L. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.00633 · Cited by 6 · see citing works
- Authors
- Williamson, P. R., & Gamble, C.
- Affiliations
- University of Liverpool (GB)
- Corresponding author
- Williamson, P. R.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2004-12-06
- Volume
- 24
- Issue
- 10
- Pages
- 1547-1561
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2108349766
- Cited by
- 132
- References
- 14
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 7 · 2023: 8 · 2022: 3 · 2021: 5 · 2020: 3 · 2019: 5 · 2018: 3 · 2017: 3 · 2016: 4 · 2015: 1 · 2014: 7 · 2013: 9 · 2012: 6
- FWCI
- 5.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Statistics in Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- selective reporting, publication bias, meta-analysis validity, funnel plot asymmetry, sensitivity analysis, heterogeneity, trial reporting
- MeSH terms
- Clinical Trials as Topic, United Kingdom, Humans, Meta-Analysis as Topic, Selection Bias, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
