Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies
Open accessThe DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To synthesise evidence on the average bias and heterogeneity associated with reported methodological features of randomized trials. DESIGN: Systematic review of meta-epidemiological studies. METHODS: We retrieved eligible studies included in a recent AHRQ-EPC review on this topic (latest search September 2012), and searched Ovid MEDLINE and Ovid EMBASE for studies indexed from Jan 2012-May 2015. Data were extracted by one author and verified by another. We combined estimates of average bias (e.g. ratio of odds ratios (ROR) or difference in standardised mean differences (dSMD)) in meta-analyses using the random-effects model. Analyses were stratified by type of outcome ("mortality" versus "other objective" versus "subjective"). Direction of effect was standardised so that ROR < 1 and dSMD < 0 denotes a larger intervention effect estimate in trials with an inadequate or unclear (versus adequate) characteristic. RESULTS: We included 24 studies. The available evidence suggests that intervention effect estimates may be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (ROR 0.93, 95% CI 0.86 to 0.99; 7 studies) and allocation concealment (ROR 0.90, 95% CI 0.84 to 0.97; 7 studies). For these characteristics, the average bias appeared to be larger in trials of subjective outcomes compared with other objective outcomes. Also, intervention effects for subjective outcomes appear to be exaggerated in trials with lack of/unclear blinding of participants (versus blinding) (dSMD -0.37, 95% CI -0.77 to 0.04; 2 studies), lack of/unclear blinding of outcome assessors (ROR 0.64, 95% CI 0.43 to 0.96; 1 study) and lack of/unclear double blinding (ROR 0.77, 95% CI 0.61 to 0.93; 1 study). The influence of other characteristics (e.g. unblinded trial personnel, attrition) is unclear. CONCLUSIONS: Certain characteristics of randomized trials may exaggerate intervention effect estimates. The average bias appears to be greatest in trials of subjective outcomes. More research on several characteristics, particularly attrition and selective reporting, is needed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Fig 3 Random-effects meta-analysis of RORs associated with inadequate/unclear (versus adequate) sequence generation. The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR is (95% CrI) 0.89 (0.82, 0.96)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.89 (0.75, 1.05)]. The Unverzagt 2013c ROR is based on a multivariable analysis with adjustment for allocation concealment, double blinding, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.98 (0.8, 1.21)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.99 (0.84, 1.16)]. The BRANDO 2012e ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.83 (0.74, 0.94)].](https://figures.cochat.ai/paper-figures/2983b5b33b19494280d6784de9530e9d/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-3.jpg)
![Fig 4 Random-effects meta-analysis of RORs associated with inadequate/unclear (versus adequate) allocation concealment. The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.93 (0.87, 0.99)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.98 (0.88, 1.10)]. The BRANDO 2012c ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.97 (0.85, 1.10)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.85 (0.75, 0.95)].](https://figures.cochat.ai/paper-figures/594b60d931a944d4b7cd0812e7e54cf5/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-4.jpg)
![Fig 5 Random-effects meta-analysis of RORs and dSMDs associated with presence (versus absence) of baseline imbalance. The Unverzagt 2013a ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.92 (0.80, 1.06)].](https://figures.cochat.ai/paper-figures/2117adfe36b54af3aedfc58feb8e5a77/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-5.jpg)


![Fig 8 Estimated RORs and dSMDs associated with any (versus no or minimal) attrition. The boxed section displays the average bias estimates, where available, from the four meta-epidemiological studies contributing to the BRANDO 2012 study. The Abraha 2015a ROR is based on a multivariable analysis with adjustment for use of placebo comparison, sample size, type of centre, items of risk of bias, post-randomisation exclusions, funding, and publication bias [the corresponding univariable ROR (95% CI) is 0.83 (0.71, 0.97)]. The Unverzagt 2013b ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 1.19 (0.98, 1.45)]. The Nuesch 2009bc dSMD is based on a multivariable analysis with adjustment for allocation concealment [the corresponding multivariable dSMD (95% CI) with adjustment for blinding of participants is -0.15 (-0.30, 0.00), and the corresponding univariable dSMD (95% CI) is -0.13 (-0.29, 0.04)].](https://figures.cochat.ai/paper-figures/0e8ce72c7fdb470c826c8be718f13f64/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-8.jpg)

![Fig 10 Random-effects meta-analysis of RORs associated with lack of/unclear double blinding (versus double blinding). The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.87 (0.79, 0.96)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.92 (0.80, 1.04)]. The Unverzagt 2013c ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.84 (0.69, 1.02)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.93 (0.74, 1.18)]. The BRANDO 2012e ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.78 (0.65, 0.92)].](https://figures.cochat.ai/paper-figures/2137b7d01f5c4ed3952ef80ca31122dd/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-10.jpg)
![Fig 11 Random-effects meta-analysis of RORs and dSMDs associated with high/unclear (versus low) risk of bias due to selective reporting. The Unverzagt 2013a ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, attrition, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.73 (0.54, 0.98)].](https://figures.cochat.ai/paper-figures/518f65e477414b18b41b5c661142ba2f/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-11.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical evidence and theoretical considerations suggest that several features of study design are associated with larger intervention effect estimates in studies; these features include inadequate generation and concealment of a random sequence to assign participants to groups, substantial loss to follow-up of participants, and unblinded outcome assessment.”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [80]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“based on meta-epidemiological studies [22][23][24][25] that found little to no effect of blinding on treatment results among randomised trials (including a review of oral health trials 26 )”
From Management of chronic pain secondary to temporomandibular disorders: a systematic review and network meta-analysis of randomised trials (2023) · cites this paper as [23]
Experimental — accuracy not yet measuredIn the citing paper
“There is a suggestion of missing results in the left-hand side of the plot, where results would be unfavourable to SSRIs and in the area of statistical nonsignificance (i.e. the white area where P > 0.10), which adds credence to the possibility that the asymmetry is due to reporting biases.”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [59]
In the citing paper
“There is a suggestion of missing results in the lefthand side of the plot, where results would be unfavourable to SSRIs, and in the area of statistical nonsignificance, which adds credence to the possibility that the asymmetry is due to reporting biases.”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Self-citationPreferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · cited 1×
“We used DerSimonian and Laird’s method of moments estimator to estimate the betweenstudy variance [22].”
MethodThe Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Zhang, Q., Wang, C., Zheng, M., Li, Y., Li, J., Zhang, L., Shang, X., & Yan, C. (2015) · Cerebrovascular Diseases · doi:10.1159/000369778 · Cited by 11,533
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · cited 2×
“[25] combined data from three meta-epidemiological studies [5,26,27] while the BRANDO study [10] combined data from these same three meta-epidemiological studies along with four others [28–31].”
BackgroundMethodDong, Y., Hao, L., Shi, Z., Wang, G., Zhang, Z., & Han, C. (2013) · Urologia Internationalis · doi:10.1159/000351405 · Cited by 4,540
Evidence-Based Medicine: A New Approach to Teaching the Practice of Medicine
Guyatt, G. (1992) · JAMA · doi:10.1001/jama.1992.03490170092032 · Cited by 4,202
(2008) · Cited by 3,790
Self-citationMoher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Self-citationKjaergard, L. L., Villumsen, J., & Gluud, C. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,283 · cited 1×
“[25] combined data from three meta-epidemiological studies [5,26,27] while the BRANDO study [10] combined data from these same three meta-epidemiological studies along with four others [28–31].”
MethodIndustry sponsorship and research outcome
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub3 · Cited by 1,697
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181 · cited 1×
“[25] combined data from three meta-epidemiological studies [5,26,27] while the BRANDO study [10] combined data from these same three meta-epidemiological studies along with four others [28–31].”
MethodSelf-citationSavović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049 · cited 10×
“[10,21], when sufficient information about each was provided);”
BackgroundResultInfluentialSelf-citationSavović, J., Jones, H., Altman, D., Harris, R., Jűni, P., Pildal, J., Als-Nielsen, B., Balk, E., Gluud, C., Gluud, L., et al. (2012) · Health Technology Assessment · doi:10.3310/hta16350 · Cited by 668 · cited 5×
“[10,21], when sufficient information about each was provided);”
BackgroundMethodInfluentialSelf-citationBassler, D. (2010) · JAMA · doi:10.1001/jama.2010.310 · Cited by 611
Industry sponsorship and research outcome
Lundh, A., Sismondo, S., Lexchin, J., Busuioc, O. A., & Bero, L. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub2 · Cited by 605
Influence of trial sample size on treatment effect estimates: meta-epidemiological study
Dechartres, A., Trinquart, L., Boutron, I., & Ravaud, P. (2013) · BMJ · doi:10.1136/bmj.f2304 · Cited by 544 · cited 1×
“industry sponsorship [14], sample size [15], single versus multi-centre status [16,17], stopping trials early for benefit or harm [18], and country of enrolment [19]).”
BackgroundHróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S. (2013) · Canadian Medical Association Journal · doi:10.1503/cmaj.120744 · Cited by 527 · cited 1×
“06; 1 meta-epidemiological study [39]) and time-to-event outcomes (RHR 0.”
Self-citationHrobjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S. (2012) · BMJ · doi:10.1136/bmj.e1119 · Cited by 483 · cited 2×
“Or, a multi-arm trial includes a blinded sub-study (such as experimental versus placebo control) and an unblinded sub-study (such as experimental versus no-treatment control) [11].”
BackgroundSelf-citationBalk, E. M. (2002) · JAMA · doi:10.1001/jama.287.22.2973 · Cited by 478 · cited 6×
“Based on a meta-analysis of three metaepidemiological studies [29,34,35] examining objective and subjective outcomes together, there was little evidence of bias in trials with lack of/unclear blinding of participants (versus blinding of participants) (ROR 0.”
BackgroundMethodInfluentialImpact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal, J., Hrobjartsson, A., Jorgensen, K., Hilden, J., Altman, D., & Gotzsche, P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym087 · Cited by 472
Hróbjartsson, A., Emanuelsson, F., Skou Thomsen, A. S., Hilden, J., & Brorson, S. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyu115 · Cited by 462 · cited 3×
“Inconsistency in average bias was moderate in one meta-epidemiological study (I(2) 60% [41]) and the magnitude of heterogeneity was high in another (tau 0.”
BackgroundSelf-citationHow to use the evidence: assessment and application of scientific evidence
Glasziou, P., Irwig, L., Bain, C., & Colditz, G. A. (2000) · Cited by 395
Sterne, J. A. C., Jüni, P., Schulz, K. F., Altman, D. G., Bartlett, C., & Egger, M. (2002) · Statistics in Medicine · doi:10.1002/sim.1184 · Cited by 382 · cited 3×
“Characteristics Studies (%, n = 24) Type of outcome in included meta-analyses/trials Varied (mortality, other objective or subjective) 18 (75) Mortality only 1 (4) Subjective only 5 (21) Type of outcomemeasure in includedmeta-analyses/trials Binary 16 (67) Continuous 7 (29) Time-to-event 1 (4) Analysis approach used M…”
BackgroundMethodSelf-citationMoher, Cook, Jadad, Tugwell, Moher, Jones, Pham, & Klassen (1999) · Health Technology Assessment · doi:10.3310/hta3120 · Cited by 368
Dechartres, A., Boutron, I., Trinquart, L., Charles, P., & Ravaud, P. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-1-201107050-00006 · Cited by 307 · cited 1×
“industry sponsorship [14], sample size [15], single versus multi-centre status [16,17], stopping trials early for benefit or harm [18], and country of enrolment [19]).”
BackgroundNuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Burgi, E., Scherer, M., Altman, D. G., & Juni, P. (2009) · BMJ · doi:10.1136/bmj.b3244 · Cited by 248
Devereaux, P. J., Choi, P. T.-L., El-Dika, S., Bhandari, M., Montori, V. M., Schünemann, H. J., Garg, A. X., Busse, J. W., Heels-Ansdell, D., Ghali, W. A., et al. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.03.017 · Cited by 209 · cited 1×
“Assessment of characteristics is often entirely based on what is reported in papers, and reported methods do not always reflect actual conduct [56,57].”
BackgroundUnverzagt, S., Prondzinsky, R., & Peinemann, F. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.05.016 · Cited by 172 · cited 1×
“Bias due to participants switching interventions within the trial and being analysed in a group different from the one to which they were randomized was examined in one small metaepidemiological study of 12 meta-analyses in critical care medicine [48].”
BackgroundIndustry sponsorship and research outcome
Lundh, A., Lexchin, J., Sismondo, S., Busuioc, O. A., & Bero, L. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033 · Cited by 166
Abraha, I., Cherubini, A., Cozzolino, F., De Florio, R., Luchetta, M. L., Rimland, J. M., Folletti, I., Marchesi, M., Germani, A., Orso, M., et al. (2015) · BMJ · doi:10.1136/bmj.h2445 · Cited by 165 · cited 1×
“94; 1 meta-epidemiological study [32]), but having a dropout rate>20% (versus 20%) was not (ROR 1.”
BackgroundNüesch, E., Reichenbach, S., Trelle, S., Rutjes, A. W. S., Liewald, K., Sterchi, R., Altman, D. G., & Jüni, P. (2009) · Arthritis Care & Research · doi:10.1002/art.24894 · Cited by 150 · cited 4×
“90; I(2) 0%; 4 meta-epidemiological studies [21,33,37,44]), and in trials of complementary and alternative medicine interventions (CAM) (Dsmd -0.”
BackgroundResultInfluentialHróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Rasmussen, J. V., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyt270 · Cited by 142 · cited 1×
“93; 1 meta-epidemiological study [40]).”
Self-citationBafeta, A., Dechartres, A., Trinquart, L., Yavchitz, A., Boutron, I., & Ravaud, P. (2012) · BMJ · doi:10.1136/bmj.e813 · Cited by 129 · cited 1×
“industry sponsorship [14], sample size [15], single versus multi-centre status [16,17], stopping trials early for benefit or harm [18], and country of enrolment [19]).”
BackgroundBasic study design influences the results of orthodontic clinical investigations
Papageorgiou, S. N., Xavier, G. M., & Cobourne, M. T. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.03.008 · Cited by 108 · cited 3×
“15; I(2) 0%; 4 meta-epidemiological studies [21,33,37,46]), although the 95% CIs overlap.”
BackgroundResultMhaskar, R., Djulbegovic, B., Magazin, A., Soares, H. P., & Kumar, A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.10.016 · Cited by 102 · cited 1×
“To improve the evidence base, future meta-epidemiological studies should report both univariable and multivariable analyses that adjust for potential confounders and, where available, assess risk of bias based on the more detailed methods that are often reported in trial protocols as well as methods reported in public…”
MethodPanagiotou, O. A., Contopoulos-Ioannidis, D. G., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f707 · Cited by 89 · cited 1×
“industry sponsorship [14], sample size [15], single versus multi-centre status [16,17], stopping trials early for benefit or harm [18], and country of enrolment [19]).”
BackgroundGluud, L. L., Thorlund, K., Gluud, C., Woods, L., Harris, R., & Sterne, J. A. C. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-149-3-200808050-00023 · Cited by 85
Self-citationSiersma, V., Als‐Nielsen, B., Chen, W., Hilden, J., Gluud, L. L., & Gluud, C. (2007) · Statistics in Medicine · doi:10.1002/sim.2752 · Cited by 84 · cited 1×
“Characteristics Studies (%, n = 24) Type of outcome in included meta-analyses/trials Varied (mortality, other objective or subjective) 18 (75) Mortality only 1 (4) Subjective only 5 (21) Type of outcomemeasure in includedmeta-analyses/trials Binary 16 (67) Continuous 7 (29) Time-to-event 1 (4) Analysis approach used M…”
MethodArmijo-Olivo, S., Saltaji, H., da Costa, B. R., Fuentes, J., Ha, C., & Cummings, G. G. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008562 · Cited by 84
Chaimani, A., Vasiliadis, H. S., Pandis, N., Schmid, C. H., Welton, N. J., & Salanti, G. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dyt074 · Cited by 83 · cited 3×
“The influence of lack of/unclear blinding of outcome assessors (versus blinding) was negligible in a meta-analysis of four meta-epidemiological studies [29,34,35,37] which analysed objective and subjective outcomes together (ROR 1.”
BackgroundMethodHerbison, P., Hay‐Smith, J., & Gillespie, W. J. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.12.018 · Cited by 56 · cited 1×
“There was only a limited increase in between-trial heterogeneity and limited variation in average bias in the BRANDO study, whereas variation in average bias was high in three smaller meta-epidemiological studies [33,38,44] (Table 3).”
ResultHróbjartsson, A., Boutron, I., Turner, L., Altman, D. G., & Moher, D. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000058 · Cited by 49 · cited 1×
“single-versus multi-centre status, early stopping) represent a specific bias, small-study effects, or spurious findings [54].”
BackgroundSelf-citationVale, C. L., Tierney, J. F., & Burdett, S. (2013) · BMJ · doi:10.1136/bmj.f1798 · Cited by 48
Bialy, L., Vandermeer, B., Lacaze‐Masmonteil, T., Dryden, D. M., & Hartling, L. (2014) · Evidence-Based Child Health: A Cochrane Review Journal · doi:10.1002/ebch.1985 · Cited by 46 · cited 4×
“The influence of lack of/unclear blinding of outcome assessors (versus blinding) was negligible in a meta-analysis of four meta-epidemiological studies [29,34,35,37] which analysed objective and subjective outcomes together (ROR 1.”
BackgroundMethodInfluentialQuantifying Bias in Randomized Controlled Trials in Child Health: A Meta-Epidemiological Study
Hartling, L., Hamm, M. P., Fernandes, R. M., Dryden, D. M., & Vandermeer, B. (2014) · PLoS ONE · doi:10.1371/journal.pone.0088008 · Cited by 43 · cited 8×
“90; I(2) 0%; 4 meta-epidemiological studies [21,33,37,44]), and in trials of complementary and alternative medicine interventions (CAM) (Dsmd -0.”
BackgroundMethodResultInfluentialSample size calculation for meta‐epidemiological studies
Giraudeau, B., Higgins, J. P. T., Tavernier, E., & Trinquart, L. (2016) · Statistics in Medicine · doi:10.1002/sim.6627 · Cited by 40 · cited 1×
“Many meta-epidemiological studies examined a small number of meta-analyses, and so may have had insufficient power to reliably estimate associations [55].”
BackgroundSelf-citationComparison of Large Versus Smaller Randomized Trials for Mental Health-Related Interventions
Contopoulos-Ioannidis, D. G., Gilbody, S. M., Trikalinos, T. A., Churchill, R., Wahlbeck, K., Ioannidis, J. P. A., & The EU-PSI Project (2005) · American Journal of Psychiatry · doi:10.1176/appi.ajp.162.3.578 · Cited by 38
Døssing, A., Tarp, S., Furst, D. E., Gluud, C. N., Beyene, J., Hansen, B. B., Bliddal, H., & Christensen, R. (2014) · BMJ Open · doi:10.1136/bmjopen-2014-005297 · Cited by 22
Mills, E. J., Ayers, D., Chou, R., & Thorlund, K. (2015) · Contemporary Clinical Trials · doi:10.1016/j.cct.2015.07.019 · Cited by 20 · cited 4×
“epidemiological study (as was done in [6,7]), which can be difficult for readers to interpret, we synthesised the average bias estimates for eight characteristics in random-effects meta-analyses.”
BackgroundMethodInfluentialArmijo-Olivo, S., Fuentes, J., Rogers, T., Hartling, L., Saltaji, H., & Cummings, G. G. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-88 · Cited by 15
Feys, F., Bekkering, G. E., Singh, K., & Devroey, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-54 · Cited by 12
Hansen, J. B., Juhl, C. B., Boutron, I., Tugwell, P., Ghogomu, E. A. T., Pardo, J. P., Rader, T., Wells, G. A., Mayhew, A., Maxwell, L. J., et al. (2014) · BMJ Open · doi:10.1136/bmjopen-2014-005491 · Cited by 11
Saltaji, H., Armijo-Olivo, S. L., Cummings, G. G., Amin, M. S., & Flores‐Mir, C. (2014) · BMJ Open · doi:10.1136/bmjopen-2013-004527 · Cited by 9
No evidence for intervention-dependent influence of methodological features on treatment effect
Jacobs, W. C. H., Kruyt, M. C., Moojen, W. A., Verbout, A. J., & Öner, F. C. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.007 · Cited by 7 · cited 1×
“We also reviewed the list of studies included in two other relevant reviews [7,20].”
BackgroundA simple method for converting an odds ratio to effect size for use in meta-analysis
Chinn, S. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001130)19:22<3127::aid-sim784>3.0.co;2-m
Research works printed but not linked: 2
- The Oxford 2011 Levels of Evidence
- Als-Nielsen B, Chen W, Gluud LL, Siersma V, Hilden J, Gluud C. Are trial size and reported methodological quality associated with treatment effects? Observational study of 523 randomised trials [abstract]. 12th Cochrane Colloquium: Bridging the Gaps; 2004 Oct 2–6; Ottawa, Ontario, Canada [Internet]. 2004:[102–3 pp.]. Available: http://onlinelibrary.wiley.com/o/cochrane/clcmr/articles/CMR-6643/frame.html.
Cited by
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787
“…the original tool was used in practice, 6 a systematic review and meta-analysis of meta-epidemiological studies of empirical evidence for biases associated with characteristics of randomised trials 12 and a cross-sectional study of how selective outcome reporting was assessed in Cochrane reviews.”
MethodSelf-citationPage, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Self-citationPage, M. J., Moher, D., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/gwdhk · Cited by 916
Self-citationCONSORT 2010 statement: extension to randomised crossover trials
Dwan, K., Li, T., Altman, D. G., & Elbourne, D. (2019) · BMJ · doi:10.1136/bmj.l4378 · Cited by 785
Why clinical trial outcomes fail to translate into benefits for patients
Heneghan, C., Goldacre, B., & Mahtani, K. R. (2017) · Trials · doi:10.1186/s13063-017-1870-2 · Cited by 404
“In addition, trials with inadequate or unclear sequence generation also biased estimates when outcomes were subjective [20].”
Background
Metadata
- Authors
- Page, M. J., Higgins, J. P. T., Clayton, G., Sterne, J. A. C., Hróbjartsson, A., & Savović, J.
- Affiliations
- University of Bristol (GB)
- University of Southern Denmark (DK)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Corresponding author
- Page, Matthew J.
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-07-11
- Volume
- 11
- Issue
- 7
- Pages
- e0159267
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4939945
- OpenAlex
- W2461177968
- Cited by
- 284
- References
- 64
- Citations by year
- 2026: 24 · 2025: 22 · 2024: 25 · 2023: 29 · 2022: 45 · 2021: 35 · 2020: 33 · 2019: 32 · 2018: 21 · 2017: 14 · 2016: 3
- FWCI
- 7.97
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- University of Southern Denmark Research Portal (University of Southern Denmark) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- Explore Bristol Research (submittedVersion, cc-by)
- Funders
- MRC Network of Hubs for Trials Methodology Research (MR/L004933/1- N61)
- National Health and Medical Research Council (1088535)
- National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Keywords
- study design bias, meta-epidemiology, sequence generation, allocation concealment, blinding, double blinding, subjective outcomes, objective outcomes, attrition, selective attrition, reporting quality, trial heterogeneity, standardized mean difference
- MeSH terms
- Humans, Bias, Epidemiologic Studies, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.