The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews
Open accessAbstract
OBJECTIVE: To examine the prevalence of outcome reporting bias-the selection for publication of a subset of the original recorded outcome variables on the basis of the results-and its impact on Cochrane reviews. DESIGN: A nine point classification system for missing outcome data in randomised trials was developed and applied to the trials assessed in a large, unselected cohort of Cochrane systematic reviews. Researchers who conducted the trials were contacted and the reason sought for the non-reporting of data. A sensitivity analysis was undertaken to assess the impact of outcome reporting bias on reviews that included a single meta-analysis of the review primary outcome. RESULTS: More than half (157/283 (55%)) the reviews did not include full data for the review primary outcome of interest from all eligible trials. The median amount of review outcome data missing for any reason was 10%, whereas 50% or more of the potential data were missing in 70 (25%) reviews. It was clear from the publications for 155 (6%) of the 2486 assessable trials that the researchers had measured and analysed the review primary outcome but did not report or only partially reported the results. For reports that did not mention the review primary outcome, our classification regarding the presence of outcome reporting bias was shown to have a sensitivity of 88% (95% CI 65% to 100%) and specificity of 80% (95% CI 69% to 90%) on the basis of responses from 62 trialists. A third of Cochrane reviews (96/283 (34%)) contained at least one trial with high suspicion of outcome reporting bias for the review primary outcome. In a sensitivity analysis undertaken for 81 reviews with a single meta-analysis of the primary outcome of interest, the treatment effect estimate was reduced by 20% or more in 19 (23%). Of the 42 meta-analyses with a statistically significant result only, eight (19%) became non-significant after adjustment for outcome reporting bias and 11 (26%) would have overestimated the treatment effect by 20% or more. CONCLUSIONS: Outcome reporting bias is an under-recognised problem that affects the conclusions in a substantial proportion of Cochrane reviews. Individuals conducting systematic reviews need to address explicitly the issue of missing outcome data for their review to be considered a reliable source of evidence. Extra care is required during data extraction, reviewers should identify when a trial reports that an outcome was measured but no results were reported or events observed, and contact with trialists should be encouraged.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Contacting authors or manufacturers may be necessary when publications, conference proceedings, or clinical trials registry records do not provide the complete information needed.
“Contacting authors or manufacturers may also be necessary when publications, conference proceedings, or clinical trials registry records do not provide the complete information needed [63, 81].”
From PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews (2021) · cites this paper as [81]
In “Implications for systematic reviews” (page not provided), the paper advises contacting trialists to obtain outcome results missing from trial reports. It does not address contacting manufacturers or specifically recommend contact when conference proceedings or registry records are incomplete. (LLM, conf=0.85)
Experimental — accuracy not yet measuredFunnel plot asymmetry may be explained by true heterogeneity of effect with respect to study size, poor methodological design (and hence bias of small studies), and selective non-reporting.
“Several explanations may account for funnel plot asymmetry, including true heterogeneity of effect with respect to study size, poor methodological design (and hence bias of small studies), and selective non‐reporting (Kirkham 2010).”
From Mindfulness-based interventions for adults with type 2 diabetes mellitus (2021) · cites this paper as Kirkham 2010
In “Accuracy of classification” (page unavailable), the paper describes a method that assumes smaller studies are more likely than larger studies to omit outcome results, supporting a link between study size and selective non-reporting. The passages do not discuss funnel plot asymmetry or identify true effect heterogeneity and poor methodological design as explanations for it. (LLM, conf=0.76)
Experimental — accuracy not yet measuredWhere outcome misreporting is permitted, it increases the likelihood that reported differences have arisen through chance or are exaggerated.
“Where outcome misreporting is permitted, it increases the likelihood that reported differences have arisen through chance or are exaggerated [2, 3].”
From COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time (2019) · cites this paper as [3]
Selective reporting of outcomes (and predictors) is often a risk.
“However, selective reporting of outcomes (and predictors) is often a risk [51].”
From Reporting and Methods in Clinical Prediction Research: A Systematic Review (2012) · cites this paper as [51]
Outcome reporting bias was assessed by integrating results of Appendix 6 and Appendix 7.
“We assessed outcome reporting bias by integrating the results of Appendix 6, 'Matrix of trial endpoints (publications and trial documents)' (Mathieu 2009), with those of Appendix 7 'High risk of outcome reporting bias according to ORBIT classification' (Kirkham 2010).”
From Diet, physical activity or both for prevention or delay of type 2 diabetes mellitus and its associated complications in people at increased risk of developing type 2 diabetes mellitus (2017) · cites this paper as Kirkham 2010
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Smithson, M. J. (2000) · SAGE Publications Ltd eBooks · doi:10.4135/9781446218525 · Cited by 1,871 · see citing works
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · see citing works
Self-citationEpidemiology and reporting characteristics of systematic reviews.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040078 · Cited by 891 · cited 1× · see citing works
“Cochrane systematic reviews are internationally recognised as among the best sources, if not the best source, of reliable up to date information on health care.”
BackgroundSelf-citationChan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613 · cited 1× · see citing works
“For the majority of trials that were missing outcome data, judgment was needed regarding the potential for outcome reporting bias. We believe we have shown that sufficiently accurate assessments are possible. This conclusion, however, rests on the assumption that the trialists we contacted provided accurate informatio…”
BackgroundSelf-citationImproving Melanoma Classification by Integrating Genetic and Morphologic Features
Virós, A., Fridlyand, J., Bauer, J. M., Lasithiotakis, K., Garbe, C., Pinkel, D., & Bastian, B. C. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050120 · Cited by 376 · see citing works
Human albumin solution for resuscitation and volume expansion in critically ill patients
Roberts, I., Blackhall, K., Alderson, P., Bunn, F., & Schierhout, G. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001208.pub4 · Cited by 364 · see citing works
Bias in meta‐analysis due to outcome variable selection within studies
Hutton, J. L., & Williamson, P. (2000) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/1467-9876.00197 · Cited by 236 · see citing works
Self-citationSinha, I. P., Jones, L. V., Smyth, R. L., & Williamson, P. R. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050096 · Cited by 202 · see citing works
Self-citationMoreno, S. G., Sutton, A. J., Turner, E. H., Abrams, K. R., Cooper, N. J., Palmer, T. M., & Ades, A. E. (2009) · BMJ · doi:10.1136/bmj.b2981 · Cited by 184 · see citing works
Human albumin solution for resuscitation and volume expansion in critically ill patients
Alderson, P. (2002) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001208 · Cited by 164 · see citing works
Compression stockings for preventing deep vein thrombosis in airline passengers
Clarke, M. J., Broderick, C. A., Hopewell, S., Juszczak, E., & Eisinga, A. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004002.pub3 · Cited by 133 · see citing works
Identification and impact of outcome selection bias in meta‐analysis
Williamson, P. R., & Gamble, C. (2005) · Statistics in Medicine · doi:10.1002/sim.2025 · Cited by 132 · see citing works
Self-citationAssociation Between Unreported Outcomes and Effect Size Estimates in Cochrane Meta-analyses
Furukawa, T. A., Watanabe, N., Omori, I. M., Montori, V. M., & Guyatt, G. H. (2007) · JAMA · doi:10.1001/jama.297.5.468-b · Cited by 115 · see citing works
Shea, B., Moher, D., Graham, I. D., Pham, B., & Tugwell, P. (2002) · Evaluation & the Health Professions · doi:10.1177/0163278702025001008 · Cited by 109 · see citing works
Human albumin solution for resuscitation and volume expansion in critically ill patients
The Albumin Reviewers (Alderson P (2004) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001208.pub2 · Cited by 105 · see citing works
Compression stockings for preventing deep vein thrombosis in airline passengers
Clarke, M. J., Hopewell, S., Juszczak, E., Eisinga, A., & Kjeldstrøm, M. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004002.pub2 · Cited by 73 · see citing works
A Bound for Publication Bias Based on the Fraction of Unpublished Studies
Copas, J. B., & Jackson, D. (2004) · Biometrics · doi:10.1111/j.0006-341x.2004.00161.x · Cited by 64 · see citing works
Reporting of clinical trials: a review of research funders' guidelines
Dwan, K. M., Gamble, C. L., Williamson, P. R., & Altman, D. G. (2008) · Trials · doi:10.1186/1745-6215-9-66 · Cited by 43 · see citing works
Self-citationReporting the findings of clinical trials: a discussion paper
Ghersi, D. (2008) · Bulletin of the World Health Organization · doi:10.2471/blt.08.053769 · Cited by 36 · see citing works
Standardising Outcomes in Paediatric Clinical Trials
Clarke, M. J. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050102 · Cited by 31 · see citing works
Application and investigation of a bound for outcome reporting bias
Williamson, P. R., & Gamble, C. (2007) · Trials · doi:10.1186/1745-6215-8-9 · Cited by 26 · see citing works
Self-citationOpiates and Acute Abdominal Pain—Reply
Ranji, S. R., Goldman, L. E., Simel, D. L., & Shojania, K. G. (2007) · JAMA · doi:10.1001/jama.297.5.468-a · Cited by 6 · see citing works
Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223 · see citing works
“Bias related to the selective reporting of outcomes has been characterized as a serious problem in clinical research, including systematic reviews [2-7].”
BackgroundShamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120 · see citing works
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709 · see citing works
Self-citationRethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696 · see citing works
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714 · see citing works
Developing core outcome sets for clinical trials: issues to consider
Williamson, P. R., Altman, D. G., Blazeby, J. M., Clarke, M., Devane, D., Gargon, E., & Tugwell, P. (2012) · Trials · doi:10.1186/1745-6215-13-132 · Cited by 1,661 · see citing works
Self-citationAssessing risk of bias in a randomized trial
Higgins, J. P., Savović, J., Page, M. J., Elbers, R. G., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch8 · Cited by 1,765 · see citing works
Industry sponsorship and research outcome
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub3 · Cited by 1,697 · see citing works
Gray literature: An important resource in systematic reviews
Paez, A. (2017) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12266 · Cited by 1,459 · see citing works
Dobbins, M., Husson, H., DeCorby, K., & LaRocca, R. L. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007651.pub2 · Cited by 1,358 · see citing works
What Is the Evidence for Physical Therapy Poststroke? A Systematic Review and Meta-Analysis
Veerbeek, J. M., van Wegen, E., van Peppen, R., van der Wees, P. J., Hendriks, E., Rietberg, M., & Kwakkel, G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0087987 · Cited by 1,203 · see citing works
Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group (2013) · PLoS ONE · doi:10.1371/journal.pone.0066844 · Cited by 1,127 · see citing works
Self-citationSavović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049 · see citing works
Self-citationIncreasing value and reducing waste: addressing inaccessible research
Chan, A.-W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., Krumholz, H. M., Ghersi, D., & van der Worp, H. B. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62296-5 · Cited by 887 · see citing works
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754 · see citing works
Simera, I., Moher, D., Hirst, A., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-24 · Cited by 590 · see citing works
Self-citationReporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · see citing works
Self-citationGuidelines for the Content of Statistical Analysis Plans in Clinical Trials
Gamble, C., Krishan, A., Stocken, D., Lewis, S., Juszczak, E., Doré, C., Williamson, P. R., Altman, D. G., Montgomery, A., Lim, P., et al. (2017) · JAMA · doi:10.1001/jama.2017.18556 · Cited by 536 · see citing works
Self-citationPRISMA harms checklist: improving harms reporting in systematic reviews
Zorzela, L., Loke, Y. K., Ioannidis, J. P., Golder, S., Santaguida, P., Altman, D. G., Moher, D., Vohra, S., & PRISMA harms group (2016) · BMJ · doi:10.1136/bmj.i157 · Cited by 557 · see citing works
Self-citationInfluence of trial sample size on treatment effect estimates: meta-epidemiological study
Dechartres, A., Trinquart, L., Boutron, I., & Ravaud, P. (2013) · BMJ · doi:10.1136/bmj.f2304 · Cited by 544 · see citing works
An international registry of systematic-review protocols
Booth, A., Clarke, M., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2011) · The Lancet · doi:10.1016/s0140-6736(10)60903-8 · Cited by 512 · see citing works
Core outcome measurement instruments for clinical trials in nonspecific low back pain
Chiarotto, A., Boers, M., Deyo, R. A., Buchbinder, R., Corbin, T. P., Costa, L. O. P., Foster, N. E., Grotle, M., Koes, B. W., Kovacs, F. M., et al. (2018) · Pain · doi:10.1097/j.pain.0000000000001117 · Cited by 461 · see citing works
Why clinical trial outcomes fail to translate into benefits for patients
Heneghan, C., Goldacre, B., & Mahtani, K. R. (2017) · Trials · doi:10.1186/s13063-017-1870-2 · Cited by 404 · see citing works
Ahmed, I., Sutton, A. J., & Riley, R. D. (2012) · BMJ · doi:10.1136/bmj.d7762 · Cited by 404 · see citing works
Core outcome-domains for clinical trials in non-specific low back pain
Chiarotto, A., Deyo, R. A., Terwee, C. B., Boers, M., Buchbinder, R., Corbin, T. P., Costa, L. O. P., Foster, N. E., Grotle, M., Koes, B. W., et al. (2015) · European Spine Journal · doi:10.1007/s00586-015-3892-3 · Cited by 374 · see citing works
Alhelou, H. H., Hamedani-Golshan, M.-E., Zamani, R., Heydarian-Forushani, E., & Siano, P. (2018) · Energies · doi:10.3390/en11102497 · Cited by 349 · see citing works
Hemmingsen, B., Gimenez-Perez, G., Mauricio, D., Roqué i Figuls, M., Metzendorf, M.-I., & Richter, B. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003054.pub4 · Cited by 316 · see citing works
Evans, J. R., & Lawrenson, J. G. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000254.pub4 · Cited by 335 · see citing works
Physical activity and academic achievement in children: a historical perspective
Howie, E. K., & Pate, R. R. (2012) · Journal of Sport and Health Science · doi:10.1016/j.jshs.2012.09.003 · Cited by 297 · see citing works
Page, M. J., McKenzie, J. E., & Higgins, J. P. T. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019703 · Cited by 278 · see citing works
Driving up the Quality and Relevance of Research Through the Use of Agreed Core Outcomes
Williamson, P., Altman, D., Blazeby, J., Clarke, M., & Gargon, E. (2012) · Journal of Health Services Research & Policy · doi:10.1258/jhsrp.2011.011131 · Cited by 288 · see citing works
Self-citationTiming and Completeness of Trial Results Posted at ClinicalTrials.gov and Published in Journals
Riveros, C., Dechartres, A., Perrodeau, E., Haneef, R., Boutron, I., & Ravaud, P. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001566 · Cited by 269 · see citing works
Page, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035.pub2 · Cited by 281 · see citing works
Self-citationGrading the strength of a body of evidence when assessing health care interventions: an EPC update
Berkman, N. D., Lohr, K. N., Ansari, M. T., Balk, E. M., Kane, R., McDonagh, M., Morton, S. C., Viswanathan, M., Bass, E. B., Butler, M., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.023 · Cited by 279 · see citing works
Spielmans, G. I., Berman, M. I., Linardatos, E., Rosenlicht, N. Z., Perry, A., & Tsai, A. C. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001403 · Cited by 275 · see citing works
Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses
Hart, B., Lundh, A., & Bero, L. (2012) · BMJ · doi:10.1136/bmj.d7202 · Cited by 255 · see citing works
COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time
Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R. (2019) · Trials · doi:10.1186/s13063-019-3173-2 · Cited by 233 · see citing works
A practical introduction to multivariate meta-analysis
Mavridis, D., & Salanti, G. (2013) · Statistical Methods in Medical Research · doi:10.1177/0962280211432219 · Cited by 255 · see citing works
Considering bias and conflicts of interest among the included studies
Boutron, I., Page, M. J., Higgins, J. P., Altman, D. G., Lundh, A., Hróbjartsson, A., & on behalf of the Cochrane Bias Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch7 · Cited by 258 · see citing works
Self-citationComparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243 · see citing works
Self-citationAssociation Between Analytic Strategy and Estimates of Treatment Outcomes in Meta-analyses
Dechartres, A., Altman, D. G., Trinquart, L., Boutron, I., & Ravaud, P. (2014) · JAMA · doi:10.1001/jama.2014.8166 · Cited by 233 · see citing works
Self-citationFrequency and reasons for outcome reporting bias in clinical trials: interviews with trialists
Smyth, R. M. D., Kirkham, J. J., Jacoby, A., Altman, D. G., Gamble, C., & Williamson, P. R. (2011) · BMJ · doi:10.1136/bmj.c7153 · Cited by 220 · see citing works
Self-citationSaini, P., Loke, Y. K., Gamble, C., Altman, D. G., Williamson, P. R., & Kirkham, J. J. (2014) · BMJ · doi:10.1136/bmj.g6501 · Cited by 209 · see citing works
Self-citationInterventions to Improve Cortisol Regulation in Children: A Systematic Review
Slopen, N., McLaughlin, K. A., & Shonkoff, J. P. (2014) · Pediatrics · doi:10.1542/peds.2013-1632 · Cited by 211 · see citing works
Hind, D., Cotter, J., Thake, A., Bradburn, M., Cooper, C., Isaac, C., & House, A. (2014) · BMC Psychiatry · doi:10.1186/1471-244x-14-5 · Cited by 180 · see citing works
Halladay, C. W., Trikalinos, T. A., Schmid, I. T., Schmid, C. H., & Dahabreh, I. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.12.017 · Cited by 166 · see citing works
Bias Due to Changes in Specified Outcomes during the Systematic Review Process
Kirkham, J. J., Altman, D. G., & Williamson, P. R. (2010) · PLoS ONE · doi:10.1371/journal.pone.0009810 · Cited by 164 · see citing works
Self-citationA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165 · see citing works
Kirkham, J. J., Altman, D. G., Chan, A.-W., Gamble, C., Dwan, K. M., & Williamson, P. R. (2018) · BMJ · doi:10.1136/bmj.k3802 · Cited by 144 · see citing works
Self-citationWells, G. A., Shea, B., Higgins, J. P., Sterne, J., Tugwell, P., & Reeves, B. C. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1077 · Cited by 141 · see citing works
Mayo-Wilson, E., Fusco, N., Li, T., Hong, H., Canner, J. K., & Dickersin, K. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.007 · Cited by 131 · see citing works
Psychological interventions for diabetes-related distress in adults with type 2 diabetes mellitus
Chew, B. H., Vos, R. C., Metzendorf, M.-I., Scholten, R. J., & Rutten, G. E. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011469.pub2 · Cited by 138 · see citing works
Ensuring the quality and specificity of preregistrations
Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000937 · Cited by 138 · see citing works
Clarke, M., Brice, A., & Chalmers, I. (2014) · PLoS ONE · doi:10.1371/journal.pone.0102670 · Cited by 120 · see citing works
Hodkinson, A., Kirkham, J. J., Tudur-Smith, C., & Gamble, C. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003436 · Cited by 111 · see citing works
Self-citationAnalgesic efficacy of opioids in chronic pain: recent meta‐analyses
Reinecke, H., Weber, C., Lange, K., Simon, M., Stein, C., & Sorgatz, H. (2015) · British Journal of Pharmacology · doi:10.1111/bph.12634 · Cited by 120 · see citing works
Vera-Badillo, F. E., Shapiro, R., Ocana, A., Amir, E., & Tannock, I. F. (2013) · Annals of Oncology · doi:10.1093/annonc/mds636 · Cited by 117 · see citing works
Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy
Mayo-Wilson, E., Li, T., Fusco, N., Bertizzolo, L., Canner, J. K., Cowley, T., Doshi, P., Ehmsen, J., Gresham, G., Guo, N., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.07.014 · Cited by 108 · see citing works
How to reduce unnecessary duplication: use <scp>PROSPERO</scp>
Moher, D., Booth, A., & Stewart, L. (2014) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.12657 · Cited by 113 · see citing works
Sharma, S., Hucker, A., Matthews, T., Grohmann, D., & Laws, K. R. (2021) · BMC Psychology · doi:10.1186/s40359-021-00658-8 · Cited by 109 · see citing works
Empirical evaluation of which trial characteristics are associated with treatment effect estimates
Dechartres, A., Trinquart, L., Faber, T., & Ravaud, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.005 · Cited by 85 · see citing works
Rupp, C., Doebler, P., Ehring, T., & Vossbeck‐Elsebusch, A. N. (2017) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2039 · Cited by 76 · see citing works
Madsen, K. S., Chi, Y., Metzendorf, M.-I., Richter, B., & Hemmingsen, B. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008558.pub2 · Cited by 70 · see citing works
Harms in Systematic Reviews Paper 1: An introduction to research on harms
Qureshi, R., Mayo-Wilson, E., & Li, T. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.10.023 · Cited by 63 · see citing works
de Winter, J. C., & Dodou, D. (2015) · PeerJ · doi:10.7717/peerj.733 · Cited by 78 · see citing works
Rethinking the assessment of risk of bias due to selective reporting: a cross-sectional study
Page, M. J., & Higgins, J. P. T. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0289-2 · Cited by 60 · see citing works
Raghav, K. P. S., Mahajan, S., Yao, J. C., Hobbs, B. P., Berry, D. A., Pentz, R. D., Tam, A., Hong, W. K., Ellis, L. M., Abbruzzese, J., et al. (2015) · Journal of Clinical Oncology · doi:10.1200/jco.2015.62.4148 · Cited by 59 · see citing works
Moelands, S. V., Lucassen, P. L., Akkermans, R. P., De Grauw, W. J., & Van de Laar, F. A. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005061.pub3 · Cited by 57 · see citing works
Chromium picolinate supplementation for overweight or obese adults
Tian, H., Guo, X., Wang, X., He, Z., Sun, R., Ge, S., & Zhang, Z. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010063.pub2 · Cited by 57 · see citing works
Chipperfield, K., Fletcher, J., Millar, J., Brooker, J., Smith, R., Frydenberg, M., & Burney, S. (2013) · Psycho-Oncology · doi:10.1002/pon.3269 · Cited by 56 · see citing works
Dwan, K., Kirkham, J. J., Williamson, P. R., & Gamble, C. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-002709 · Cited by 55 · see citing works
Self-citationCollaborators, T. R., Brady, M. C., Ali, M., VandenBerg, K., Williams, L. J., Williams, L. R., Abo, M., Becker, F., Bowen, A., Brandenburg, C., et al. (2022) · International Journal of Stroke · doi:10.1177/17474930221097477 · Cited by 54 · see citing works
Norris, S. L., Moher, D., Reeves, B. C., Shea, B., Loke, Y., Garner, S., Anderson, L., Tugwell, P., & Wells, G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1062 · Cited by 50 · see citing works
Prospective reporting of statistical analysis plans for randomised controlled trials
Hemming, K., Kearney, A., Gamble, C., Li, T., Jüni, P., Chan, A.-W., & Sydes, M. R. (2020) · Trials · doi:10.1186/s13063-020-04828-8 · Cited by 34 · see citing works
Self-citationSelective outcome reporting in randomized clinical trials of dental implants
Sendyk, D. I., Rovai, E. S., Souza, N. V., Deboni, M. C. Z., & Pannuti, C. M. (2019) · Journal of Clinical Periodontology · doi:10.1111/jcpe.13128 · Cited by 35 · see citing works
Perlmutter, A. S., Tran, V.-T., Dechartres, A., & Ravaud, P. (2017) · Annals of Oncology · doi:10.1093/annonc/mdw682 · Cited by 31 · see citing works
Copas, J., Marson, A., Williamson, P., & Kirkham, J. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280217738546 · Cited by 23 · see citing works
Self-citationSpielmans, G. I., Berman, M. I., Linardatos, E., Rosenlicht, N. Z., Perry, A., & Tsai, A. C. (2016) · Focus · doi:10.1176/appi.focus.140202 · Cited by 20 · see citing works
Pandis, N., Fleming, P. S., Worthington, H., Dwan, K., & Salanti, G. (2015) · PLOS ONE · doi:10.1371/journal.pone.0137667 · Cited by 30 · see citing works
Self-citationMeursinge Reynders, R., Ladu, L., & Di Girolamo, N. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.07.001 · Cited by 18 · see citing works
Selective outcome reporting: telling and detecting true lies. The state of the science
Macura, A., Abraha, I., Kirkham, J., Gensini, G. F., Moja, L., & Iorio, A. (2010) · Internal and Emergency Medicine · doi:10.1007/s11739-010-0371-z · Cited by 19 · see citing works
Self-citationWeston, J., Dwan, K., Altman, D., Clarke, M., Gamble, C., Schroter, S., Williamson, P., & Kirkham, J. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010075 · Cited by 16 · see citing works
Self-citationBashir, R., & Dunn, A. G. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-013048 · Cited by 10 · see citing works
Woodford, J., Farrand, P., Richards, D., & Llewellyn, D. J. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-9 · Cited by 7 · see citing works
da Rosa Oliveira, L., Elagami, R. A., Reis, T. M., Tedesco, T. K., Mendes, F. M., Braga, M. M., Pannuti, C. M., & Raggio, D. P. (2024) · Caries Research · doi:10.1159/000542108 · Cited by 3 · see citing works
Mindfulness-based interventions for adults with type 2 diabetes mellitus
Ee, C. C., Armour, M., Piya, M. K., McMorrow, R., Al-Kanini, I., & Sabag, A. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd014881 · Cited by 2 · see citing works
- Authors
- Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R.
- Affiliations
- University of Liverpool (GB)
- University of Oxford (GB)
- Corresponding author
- Williamson, P. R
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2010-02-15
- Volume
- 340
- Issue
- feb15 1
- Pages
- c365-c365
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- OpenAlex
- W2046391772
- Cited by
- 1,133
- References
- 23
- Citations by year
- 2026: 27 · 2025: 37 · 2024: 53 · 2023: 46 · 2022: 58 · 2021: 77 · 2020: 76 · 2019: 72 · 2018: 66 · 2017: 95 · 2016: 102 · 2015: 111 · 2014: 97 · 2013: 84 · 2012: 70
- FWCI
- 18.91
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- BMJ (publishedVersion)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmaceutical industry and healthcare
- Keywords
- selective reporting, missing outcome data, publication bias, randomized controlled trials, meta-analysis, sensitivity analysis, data extraction, treatment effect estimation
- MeSH terms
- Data Collection, Outcome and Process Assessment, Health Care, Research Design, Review Literature as Topic, Cohort Studies, Bias, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
