Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies
Open accessThe DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To synthesise evidence on the average bias and heterogeneity associated with reported methodological features of randomized trials. DESIGN: Systematic review of meta-epidemiological studies. METHODS: We retrieved eligible studies included in a recent AHRQ-EPC review on this topic (latest search September 2012), and searched Ovid MEDLINE and Ovid EMBASE for studies indexed from Jan 2012-May 2015. Data were extracted by one author and verified by another. We combined estimates of average bias (e.g. ratio of odds ratios (ROR) or difference in standardised mean differences (dSMD)) in meta-analyses using the random-effects model. Analyses were stratified by type of outcome ("mortality" versus "other objective" versus "subjective"). Direction of effect was standardised so that ROR < 1 and dSMD < 0 denotes a larger intervention effect estimate in trials with an inadequate or unclear (versus adequate) characteristic. RESULTS: We included 24 studies. The available evidence suggests that intervention effect estimates may be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (ROR 0.93, 95% CI 0.86 to 0.99; 7 studies) and allocation concealment (ROR 0.90, 95% CI 0.84 to 0.97; 7 studies). For these characteristics, the average bias appeared to be larger in trials of subjective outcomes compared with other objective outcomes. Also, intervention effects for subjective outcomes appear to be exaggerated in trials with lack of/unclear blinding of participants (versus blinding) (dSMD -0.37, 95% CI -0.77 to 0.04; 2 studies), lack of/unclear blinding of outcome assessors (ROR 0.64, 95% CI 0.43 to 0.96; 1 study) and lack of/unclear double blinding (ROR 0.77, 95% CI 0.61 to 0.93; 1 study). The influence of other characteristics (e.g. unblinded trial personnel, attrition) is unclear. CONCLUSIONS: Certain characteristics of randomized trials may exaggerate intervention effect estimates. The average bias appears to be greatest in trials of subjective outcomes. More research on several characteristics, particularly attrition and selective reporting, is needed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Fig 3 Random-effects meta-analysis of RORs associated with inadequate/unclear (versus adequate) sequence generation. The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR is (95% CrI) 0.89 (0.82, 0.96)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.89 (0.75, 1.05)]. The Unverzagt 2013c ROR is based on a multivariable analysis with adjustment for allocation concealment, double blinding, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.98 (0.8, 1.21)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.99 (0.84, 1.16)]. The BRANDO 2012e ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.83 (0.74, 0.94)].](https://figures.cochat.ai/paper-figures/2983b5b33b19494280d6784de9530e9d/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-3.jpg)
![Fig 4 Random-effects meta-analysis of RORs associated with inadequate/unclear (versus adequate) allocation concealment. The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.93 (0.87, 0.99)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.98 (0.88, 1.10)]. The BRANDO 2012c ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.97 (0.85, 1.10)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.85 (0.75, 0.95)].](https://figures.cochat.ai/paper-figures/594b60d931a944d4b7cd0812e7e54cf5/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-4.jpg)
![Fig 5 Random-effects meta-analysis of RORs and dSMDs associated with presence (versus absence) of baseline imbalance. The Unverzagt 2013a ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.92 (0.80, 1.06)].](https://figures.cochat.ai/paper-figures/2117adfe36b54af3aedfc58feb8e5a77/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-5.jpg)


![Fig 8 Estimated RORs and dSMDs associated with any (versus no or minimal) attrition. The boxed section displays the average bias estimates, where available, from the four meta-epidemiological studies contributing to the BRANDO 2012 study. The Abraha 2015a ROR is based on a multivariable analysis with adjustment for use of placebo comparison, sample size, type of centre, items of risk of bias, post-randomisation exclusions, funding, and publication bias [the corresponding univariable ROR (95% CI) is 0.83 (0.71, 0.97)]. The Unverzagt 2013b ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 1.19 (0.98, 1.45)]. The Nuesch 2009bc dSMD is based on a multivariable analysis with adjustment for allocation concealment [the corresponding multivariable dSMD (95% CI) with adjustment for blinding of participants is -0.15 (-0.30, 0.00), and the corresponding univariable dSMD (95% CI) is -0.13 (-0.29, 0.04)].](https://figures.cochat.ai/paper-figures/0e8ce72c7fdb470c826c8be718f13f64/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-8.jpg)

![Fig 10 Random-effects meta-analysis of RORs associated with lack of/unclear double blinding (versus double blinding). The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.87 (0.79, 0.96)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.92 (0.80, 1.04)]. The Unverzagt 2013c ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.84 (0.69, 1.02)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.93 (0.74, 1.18)]. The BRANDO 2012e ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.78 (0.65, 0.92)].](https://figures.cochat.ai/paper-figures/2137b7d01f5c4ed3952ef80ca31122dd/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-10.jpg)
![Fig 11 Random-effects meta-analysis of RORs and dSMDs associated with high/unclear (versus low) risk of bias due to selective reporting. The Unverzagt 2013a ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, attrition, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.73 (0.54, 0.98)].](https://figures.cochat.ai/paper-figures/518f65e477414b18b41b5c661142ba2f/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-11.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical evidence and theoretical considerations suggest that several features of study design are associated with larger intervention effect estimates in studies; these features include inadequate generation and concealment of a random sequence to assign participants to groups, substantial loss to follow-up of participants, and unblinded outcome assessment.”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [80]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“based on meta-epidemiological studies [22][23][24][25] that found little to no effect of blinding on treatment results among randomised trials (including a review of oral health trials 26 )”
From Management of chronic pain secondary to temporomandibular disorders: a systematic review and network meta-analysis of randomised trials (2023) · cites this paper as [23]
Experimental — accuracy not yet measuredIn the citing paper
“There is a suggestion of missing results in the left-hand side of the plot, where results would be unfavourable to SSRIs and in the area of statistical nonsignificance (i.e. the white area where P > 0.10), which adds credence to the possibility that the asymmetry is due to reporting biases.”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [59]
In the citing paper
“There is a suggestion of missing results in the lefthand side of the plot, where results would be unfavourable to SSRIs, and in the area of statistical nonsignificance, which adds credence to the possibility that the asymmetry is due to reporting biases.”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Self-citationPreferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · cited 1×
“We used DerSimonian and Laird’s method of moments estimator to estimate the betweenstudy variance [22].”
MethodThe Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Zhang, Q., Wang, C., Zheng, M., Li, Y., Li, J., Zhang, L., Shang, X., & Yan, C. (2015) · Cerebrovascular Diseases · doi:10.1159/000369778 · Cited by 11,533
Research works printed but not linked: 3
- OCEBM Levels of Evidence Working Group. " The Oxford 2011 Levels of Evidence". Oxford Centre for Evidence-Based Medicine Available: http://www.cebm.net/index.aspx?o=5653.
- Berkman ND, Santaguida PL, Viswanathan M, Morton SC. AHRQ Methods for Effective Health Care. The Empirical Evidence of Bias in Trials Measuring Treatment Differences. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014. 25392898
- Als-Nielsen B, Chen W, Gluud LL, Siersma V, Hilden J, Gluud C. Are trial size and reported methodological quality associated with treatment effects? Observational study of 523 randomised trials [abstract]. 12th Cochrane Colloquium: Bridging the Gaps; 2004 Oct 2–6; Ottawa, Ontario, Canada [Internet]. 2004:[102–3 pp.]. Available: http://onlinelibrary.wiley.com/o/cochrane/clcmr/articles/CMR-6643/frame.html.
Cited by
Items 101–200 of 283
Zhao, X., Liu, L., & Liu, J. (2022) · Frontiers in Endocrinology · doi:10.3389/fendo.2022.1018450 · Cited by 18
van der Ende, N. A., Roozenbeek, B., Broderick, J. P., Khatri, P., Lingsma, H. F., & Dippel, D. W. (2023) · International Journal of Stroke · doi:10.1177/17474930221131706 · Cited by 14
Xu, C., Zhang, F., Doi, S. A. R., Furuya‐Kanamori, L., Lin, L., Chu, H., Yang, X., Li, S., Zorzela, L., Golder, S., et al. (2024) · BMC Medicine · doi:10.1186/s12916-024-03300-7 · Cited by 17
“The rationale for this approach was that previous studies have shown that objective outcomes are less susceptible to methodological issues involving blinding [32].”
Is the Deductive Falsification Approach a Better Basis for Clinical Trial Appraisal?
Mickenautsch, S. (2019) · Reviews on Recent Clinical Trials · doi:10.2174/1574887114666190313170400 · Cited by 13
The importance of using placebo controls in nonpharmacological randomised trials
Wartolowska, K. A., Hohenschurz‐Schmidt, D., Vase, L., & Aronson, J. K. (2022) · Pain · doi:10.1097/j.pain.0000000000002839 · Cited by 16
“Without adequate blinding, studies with subjective outcomes show larger effect sizes.(35) Blinding of patients and all those involved in the trial and patient care also helps ensure that there will be no deviations from the intended intervention, including patients seeking additional treatments outside the trial and c…”
BackgroundArienti, C., Meeting, P. I. T. 5. C. R. M., Armijo-Olivo, S. L., Ferriero, G., Feys, P., Hoogeboom, T. J., Kiekens, C., Lazzarini, S. G., Minozzi, S., Négrini, S., et al. (2023) · European Journal of Physical and Rehabilitation Medicine · doi:10.23736/s1973-9087.23.08310-7 · Cited by 17
Page, M. J., Boutron, I., Nejstgaard, C. H., Altman, D. G., & Hróbjartsson, A. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.004 · Cited by 17
“There is empirical evidence that flaws in the design and conduct of intervention studies are associated with biased estimates of treatment benefits and harms (1, 2).”
BackgroundSelf-citationPage, M. J. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.022 · Cited by 15
Self-citationDeer, T. R., Kim, C. K., Wahezi, S. E., Qu, H., & Sayed, D. (2021) · Journal of Pain Research · doi:10.2147/jpr.s312573 · Cited by 14
Shiri, R., Väänänen, A., Mattila‐Holappa, P., Kauppi, K., & Borg, P. T. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192013206 · Cited by 18
“The Work Ability Index scores range between 7 and 49.”
MethodLange, K. M., Kasza, J., Sullivan, T., & Yelland, L. N. (2023) · Clinical Trials · doi:10.1177/17407745221146987 · Cited by 38
“Poor and inconsistent reporting of trial results, including unclear randomisation methods, is associated with bias in reported treatment effects.(90,91) This is of particular relevance to partially clustered trials, where there are multiple methods available for randomisation of observational units within pre-existing…”
BackgroundTabatabaei‐Malazy, O., Shadman, Z., Ejtahed, H., Atlasi, R., Abdollahi, M. R., & Larijani, B. A. (2018) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2018.04.004 · Cited by 15
Stewart, C. P., Arnold, C. D., Williams, A. M., Arnold, B. F., Pickering, A. J., Dentz, H. N., Kiprotich, M., Lin, A., Null, C., Colford, J. M., et al. (2024) · Current Developments in Nutrition · doi:10.1016/j.cdnut.2024.103779 · Cited by 13
Laursen, D. R. T., Nejstgaard, C. H., Bjørkedal, E., Frost, A. D., Hansen, M., Paludan‐Müller, A. S., Prosenz, J., Werner, C. P., & Hróbjartsson, A. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000055.pub2 · Cited by 26
Self-citationOliveira, J. A., da Silveira, M. I., Alves, R. D. O., Bezerra, F., de Oliveira, G. J. P. L., & Pigossi, S. C. (2023) · Clinical Oral Investigations · doi:10.1007/s00784-023-05282-x · Cited by 13
“Moreover, a meta-analysis of seven meta-epidemiological studies found that an inadequate method of sequence generation was associated with a small exaggeration of intervention effect estimates, being the bias greater in trials reporting subjective outcomes [62].”
MethodPels, A., Jakobsen, J. C., Ganzevoort, W., Naaktgeboren, C. A., Onland, W., Leemhuis, A. G., & Gluud, C. N. (2019) · Trials · doi:10.1186/s13063-018-3136-z · Cited by 14
Hu, Z., Xie, C., Yang, M., Fu, X., Gao, H., Liu, Y., & Xie, H. (2020) · Phytotherapy Research · doi:10.1002/ptr.6844 · Cited by 15
“These methodological flaws may exaggerate the treatment effect (Page et al., 2016; Page & Higgins, 2016).”
BackgroundArmijo-Olivo, S., de Oliveira-Souza, A. I. S., Mohamad, N., Carletti, E. M. D. C., Fuentes, J., & Ballenberger, N. (2022) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000001947 · Cited by 15
“99).(41) (3) Based on 7 meta-epidemiological studies, trials with inadequate/unclear (vs.”
BackgroundInfluentialNejstgaard, C. H., Laursen, D. R. T., Lundh, A., & Hróbjartsson, A. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1611 · Cited by 18
“97).(30) In our main analysis, we found moderate statistical heterogeneity (I(2) = 46%).”
BackgroundSelf-citationThe effect of disease modifying therapies on fatigue in multiple sclerosis
Rivera, S. C., Aiyegbusi, O. L., Meier, D. P., Dunne, A., Harlow, D. E., Henke, C., Kamudoni, P., & Calvert, M. (2023) · Multiple Sclerosis and Related Disorders · doi:10.1016/j.msard.2023.105065 · Cited by 15
Armijo-Olivo, S. L., Machalicek, W., Dennett, L., & Ballenberger, N. (2021) · European Journal of Physical and Rehabilitation Medicine · doi:10.23736/s1973-9087.20.06428-x · Cited by 14
Meta-research matters: Meta-spin cycles, the blindness of bias, and rebuilding trust
Bero, L. A. (2018) · PLoS Biology · doi:10.1371/journal.pbio.2005972 · Cited by 13
Global systematic review of the effects of suicide prevention interventions in Indigenous peoples
Leske, S., Paul, E., Gibson, M., Little, B., Wenitong, M., & Kõlves, K. (2020) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2019-212368 · Cited by 15
Network meta-analysis on patent foramen ovale: is a stroke or atrial fibrillation worse?
Varotto, L., Bregolin, G., Paccanaro, M., Boni, A., Bonanno, C., & Perini, F. (2020) · Neurological Sciences · doi:10.1007/s10072-020-04922-4 · Cited by 13
Blindsided: challenging the dogma of masking in clinical trials
Drucker, A. M., & Chan, A. (2020) · BMJ · doi:10.1136/bmj.m229 · Cited by 12
Khan, S. A., Sajjad, U., Fawaz, S., Butt, H., Simpson, R. F. G., Ibrahim, A., Robertson, C., Kelly, P. A., Mohdnazri, S. R., Tang, K. H., et al. (2026) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2026.03.027 · Cited by 17
Principles of Systematic Reviewing
Higgins, J. P. T., Smith, G. D., Altman, D. G., & Egger, M. (2022) · Wiley eBooks · doi:10.1002/9781119099369.ch2 · Cited by 14
Self-citationMickenautsch, S., & Yengopal, V. (2024) · Cureus · doi:10.7759/cureus.63581 · Cited by 16
“However, no empirical evidence was established supporting the questions for the bias domains "deviation from intended intervention," "missing outcome data," and "selective reporting" [7].”
BackgroundWang, H., Song, J., Lin, Y., Dai, W., Gao, Y., Qin, L., Chen, Y., Tam, W., Wu, X., & Chung, V. C. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01650-5 · Cited by 14
“Related systematic reviews of ME studies have been published in 2016 with literature search date up to May 2015 [15, 16].”
BackgroundMethodResultInfluentialSoltani, S., Sharifi‐Zahabi, E., Sangsefidi, Z. S., Vasmehjani, A. A., Meshkini, F., Clayton, Z. S., & Abdollahi, S. (2023) · Phytotherapy Research · doi:10.1002/ptr.7719 · Cited by 13
Holper, L., & Hengartner, M. P. (2020) · BMC Psychiatry · doi:10.1186/s12888-020-02839-y · Cited by 16
Bo, A., Hai, A. H., Chen, D., & Hammock, K. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.012 · Cited by 12
Navigating blinding challenges in complex intervention trials: insights from a UK researcher survey
Yonis, A., Dean, S. G., Warren, F. C., Taylor, R. S., Levack, W., & Hay‐Smith, J. (2025) · Trials · doi:10.1186/s13063-025-09223-9 · Cited by 15
“In a large meta-epidemiological study, Page et al. [50] reported that blinding was often not performed in trials with subjective outcomes and lack of blinding was associated with 36% larger treatment effects on average (ratio of odds ratios = 0.64, 95% CI 0.43–0.96).”
InfluentialArmijo-Olivo, S. L., Machalicek, W., de Oliveira‐Souza, A. I. S., Dennett, L., & Ballenberger, N. (2021) · European Journal of Physical and Rehabilitation Medicine · doi:10.23736/s1973-9087.20.06427-8 · Cited by 13
Zeraatkar, D., Pitre, T., Díaz-Martínez, J. P., Chu, D. K., Rochwerg, B. N., Lamontagne, F., Kum, E., Qasim, A., Bartoszko, J. J., & Brignardello‐Petersen, R. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwad131 · Cited by 11
Karlsen, A. P. H., Dahl, J. B., & Mathiesen, O. (2018) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.13072 · Cited by 11
Varotto, L., Bregolin, G., Paccanaro, M., Boni, A., Bonanno, C., & Perini, F. (2020) · Cerebrovascular Diseases · doi:10.1159/000507317 · Cited by 10
Bührer, J., Giovane, C. D., Gencer, B., Adam, L., Lyko, C., Feller, M., da Costa, B. R., Aujesky, D., Bauer, D. C., Rodondi, N., et al. (2023) · Mayo Clinic Proceedings Innovations Quality & Outcomes · doi:10.1016/j.mayocpiqo.2023.04.010 · Cited by 10
Industry Influence on Research: A Cycle of Bias
Bero, L. A. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780197578742.003.0019 · Cited by 11
Efficace, F., Buckstein, R. J., Abel, G. A., Giesinger, J. M., Fenaux, P., Bewersdorf, J. P., Brunner, A. M., Bejar, R. L., Borate, U., Dezern, A. E., et al. (2024) · HemaSphere · doi:10.1002/hem3.69 · Cited by 9
Kolaski, K., Clarke, M. J., & Logan, L. R. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111460 · Cited by 10
Winters, M., Holden, S., Vicenzino, B., Welton, N. J., Caldwell, D. M., Lura, C. B., Weir, A., & Rathleff, M. S. (2018) · BMJ Open · doi:10.1136/bmjopen-2018-022920 · Cited by 11
“We will seek presentation of the study results on national and international conferences, and we will submit the full-text report for ‘open access’ publication in an international peer-reviewed journal.”
Wedderhoff, N., & Bošnjak, M. (2020) · Psychologische Rundschau · doi:10.1026/0033-3042/a000484 · Cited by 10
Wang, Z., Alahdab, F., Farah, M. H., Seisa, M. O., Firwana, M., Rajjoub, R., Saadi, S. M., Jawaid, T., Nayfeh, T., & Murad, M. H. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2021-111667 · Cited by 8
BRAITHWAITE, F., Walters, J. L., Moseley, G. L., Williams, M., & McEvoy, M. P. (2021) · Physiotherapy · doi:10.1016/j.physio.2021.08.007 · Cited by 10
Wewege, M. A., Hansford, H. J., Shah, B., Gilanyi, Y. L., Douglas, S. R. G., Parmenter, B. J., McAuley, J. H., & Jones, M. D. (2022) · Hypertension Research · doi:10.1038/s41440-022-00984-3 · Cited by 11
“Insecure allocation concealment (investigators influencing the allocation of participants to groups) may bias the results [2, 5, 6].”
BackgroundGlossary for randomized clinical trials
Nagendrababu, V., Duncan, H. F., Pulikkotil, S. J., & Dummer, P. M. H. (2020) · International Endodontic Journal · doi:10.1111/iej.13434 · Cited by 11
“This effect has been linked to underpowered studies, reporting bias, lack of blinding and problems with allocation concealment amongst other factors (Page et al. 2016).”
BackgroundInvestigating and Dealing with Publication Bias and Other Reporting Biases
Page, M. J., Sterne, J. A. C., Higgins, J. P. T., & Egger, M. (2022) · Wiley eBooks · doi:10.1002/9781119099369.ch5 · Cited by 14
Self-citationRose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Potrebny, T., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses‐Echavez, J. F., & Berg, R. C. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70048 · Cited by 8
Evaluating the economic effects of the Ebola virus disease in Liberia: A synthetic control approach
Flomo, A. Z. B., Papyrakis, E., & Wagner, N. (2023) · Journal of International Development · doi:10.1002/jid.3736 · Cited by 8
Innocenti, T., Hayden, J. A., Salvioli, S., Giagio, S., Piano, L., Cosentino, C., Brindisino, F., Feller, D., Ogilvie, R., Gianola, S., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.09.001 · Cited by 10
Study design, result posting, and publication of late-stage cardiovascular trials
Kapelios, C. J., Naci, H., Vardas, P. E., & Mossialos, E. A. (2020) · European Heart Journal - Quality of Care and Clinical Outcomes · doi:10.1093/ehjqcco/qcaa080 · Cited by 8
Armijo-Olivo, S. L., da Costa, B. R., Ha, C., Saltaji, H., Cummings, G. G., & Fuentes, J. (2021) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000001837 · Cited by 8
“As stated by Page et al.14 and Armijo-Olivo et al.,12 more meta-epidemiological studies are needed to investigate the influence of attrition biases and related methodological issues in the rehabilitation field.”
MethodInfluentialA mapping review of randomized controlled trials in the spinal cord injury research literature
McIntyre, A., Benton, B., Janzen, S. E., Iruthayarajah, J., Wiener, J., Eng, J. J., & Teasell, R. W. (2018) · Spinal Cord · doi:10.1038/s41393-018-0155-2 · Cited by 8
Armijo-Olivo, S. L., Patrini, M., de Oliveira‐Souza, A. I. S., Dennett, L., Arienti, C., Dahchi, M., & Négrini, S. (2021) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2021.04.007 · Cited by 10
Wang, J., Wu, S., Guo, Q., Lan, H., Janne, E., Wang, L., Zhang, J., Wang, Q., Song, Y., Yang, N., et al. (2021) · Intelligent Medicine · doi:10.1016/j.imed.2021.04.006 · Cited by 8
Giving and taking: ethical treatment assignment in controlled trials
Senn, S. J., & Chalmers, I. (2021) · Journal of the Royal Society of Medicine · doi:10.1177/01410768211049972 · Cited by 7
“Placebos are used in controlled trials to reduce biased inferences about the effects of treatments.(17) Biased inferences can result from health professionals, patients and researchers knowing which patients have received which treatments.”
BackgroundDeer, T. R., Chafin, T. B., Costandi, S. J., Qu, H., Kim, C. K., Jassal, N. S., Patel, K. V., & Calodney, A. K. (2023) · Pain Practice · doi:10.1111/papr.13293 · Cited by 7
Intention-to-Treat Analysis in Clinical Research: Basic Concepts for Clinicians
Armijo-Olivo, S. L., da Silva, J. B., de Castro‐Carletti, E. M., de Oliveira‐Souza, A. I. S., Pelai, E. B., Mohamad, N., Baghbaninaghadehi, F., Dennett, L., Steen, J. P., Kumbhare, D. A., et al. (2024) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000002444 · Cited by 11
“11,73,74 In addition, the way that the influence of ITT has been investigated in the literature has been inconsistent because of a lack of consensus on its definition.”
Person-Centered Research: A novel approach to Randomized Controlled Trials
Badian, R. A., McCormack, B. G., & Sundling, V. (2018) · European Journal for Person Centered Healthcare · doi:10.5750/ejpch.v6i2.1435 · Cited by 7
Danielsen, M. B., Andersen, S., Ryg, J., Bruun, N. H., Madeleine, P., & Jørgensen, M. G. (2023) · Journal of Sports Sciences · doi:10.1080/02640414.2023.2300566 · Cited by 6
“Meta-epidemiological studies have found a small increase in the intervention effect when an inadequate or unclear (versus adequate) methods were reported on these domains (Page et al., 2016).”
MethodRooney, M. L., Lambe, J., O’Connor, A., Dunne, S., Mills, A., Feeney, E. L., & Gibney, E. R. (2025) · European Journal of Nutrition · doi:10.1007/s00394-024-03574-w · Cited by 6
“The protocol for the review was registered with the PROSPERO database of systematic reviews (CRD42023486210) and follows Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines [36].”
Song, Y., & Park, S. (2022) · Journal of Korean Academy of Fundamentals of Nursing · doi:10.7739/jkafn.2022.29.4.399 · Cited by 8
“효과는 평균적으로 더 편향된 결과를 나타낸 것으로 보고하였 다[35].”
Pacheco, R. L., Latorraca, C. D. O. C., Martimbianco, A. L. C., Fontes, L. E. S., Vieira, R. R., Miranda, E., Nunan, D., & Riera, R. (2020) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2019-111329 · Cited by 6
Schwarz, C., Pedraza-Flechas, A. M., Pastor‐Barriuso, R., Lope, V., de Larrea, N. F., Jiménez‐Moleón, J. J., Pollán, M., & Pérez‐Gómez, B. (2021) · Cancers · doi:10.3390/cancers13235952 · Cited by 8
“The asymmetry of the funnel plot may suggest that our estimates can be affected by publication bias or by differences in the quality of smaller studies that may spuriously inflate their effects [72].”
BackgroundSchwendicke, F. (2018) · Evidence-Based Dentistry · doi:10.1038/sj.ebd.6401321 · Cited by 8
Rosenfeld, B. D., Callahan, P. A., Carboni-Jiménez, A., Rice, D. B., Nassar, E., Boruff, J. T., Li, K., & Thombs, B. D. (2024) · International Journal of Forensic Mental Health · doi:10.1080/14999013.2024.2389237 · Cited by 12
“Unclear or inadequate randomization of intervention groups often exaggerates treatment effects (Page et al., 2016), undermining the utility of the RCT design.”
Polo-Ferrero, L., Torres-Alonso, J., González, J. L. S., Hernández-Rubia, S., Pérez-Elvira, R., & Oltra‐Cucarella, J. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14217891 · Cited by 7
Simon‐Tillaux, N., Gerard, A.-L., Rajendrabose, D., Tubach, F., & Dechartres, A. (2022) · Scientific Reports · doi:10.1038/s41598-022-24447-4 · Cited by 6
Kaul, S., & Brophy, J. M. (2025) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2025.04.040 · Cited by 5
Rose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Berg, R. C., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Potrebny, T. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02631-0 · Cited by 5
“Assessing risk of bias (RoB) is important because, for example, treatment effect estimates have been shown to be exaggerated in trials with inadequate or unclear random sequence generation and allocation concealment [1, 2].”
Improving systematic reviews: guidance on guidance and other options and challenges
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.05.008 · Cited by 6
Overgaard, S. H., Moos, C. M., Ioannidis, J. P. A., Luta, G., Berg, J., Nielsen, S. M., Andersen, V. C., & Christensen, R. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1708 · Cited by 5
“These biases have been reported to be sometimes associated with larger effect estimates in other trials 26,27 and were a priori identified as potential confounders.”
Applying GRADE Criteria to Clinical Inputs to Cost-Effectiveness Modeling Studies
Mensch, A., Beck, T., Civello, D., Kunigkeit, C., Lachmann, N., Stock, S., Gandjour, A., & Müller, D. (2018) · PharmacoEconomics · doi:10.1007/s40273-018-0651-4 · Cited by 6
Tack, M. (2021) · Journal of Health Psychology · doi:10.1177/13591053211059391 · Cited by 5
“This is, however, the method used by most meta-epidemiological studies that estimate the impact of bias (Page et al., 2016).”
MethodClayton, G., Elliott, D., Higgins, J. P. T., & Jones, H. E. (2021) · Trials · doi:10.1186/s13063-021-05759-8 · Cited by 4
“infeasibility of blinding the patient or personnel [16, 17], can produce a biasadjusted treatment effect estimate [18], allowing the analyst to assess the robustness of their findings [19].”
BackgroundSelf-citationJia, Y., Gao, G., & Leung, S.-W. (2022) · Journal of Ethnopharmacology · doi:10.1016/j.jep.2022.115996 · Cited by 5
Short- and long-term effects of imatinib in hospitalized COVID-19 patients: A randomized trial
Halme, A. L. E., Laakkonen, S., Rutanen, J., Nevalainen, O., Sinisalo, M., Horstia, S., Mustonen, J. M. J., Pourjamal, N., Vanhanen, A., Ala-Kokko, T., et al. (2024) · Journal of Infection · doi:10.1016/j.jinf.2024.106217 · Cited by 5
Vakuumversiegelungstherapie von Wunden: randomisierte Studien von 2000 bis 2015
Peinemann, F. (2017) · Zentralblatt für Chirurgie - Zeitschrift für Allgemeine Viszeral- Thorax- und Gefäßchirurgie · doi:10.1055/s-0043-104697 · Cited by 4
Gluud, L. L., Jeyaraj, R., & Morgan, M. Y. (2019) · Journal of Clinical and Experimental Hepatology · doi:10.1016/j.jceh.2019.02.004 · Cited by 4
Wang, B. H., Lin, Y., Gao, Y., Song, J., Qin, L., Li, L., Liu, W., Zhong, C. C., Jiang, M. Y., Mao, C., et al. (2024) · Journal of Chinese Integrative Medicine · doi:10.1016/j.joim.2024.04.003 · Cited by 5
Chotalia, R., Mohiaddin, H., Aggarwal, A., Borchardt, E., Ambrogetti, R., Chotalia, M., Squire, I., & Sze, S. (2026) · American Journal of Hospice and Palliative Medicine® · doi:10.1177/10499091261446602 · Cited by 2
Barosa, M., Jamrozik, E., & Prasad, V. K. (2023) · Medicine Health Care and Philosophy · doi:10.1007/s11019-023-10184-6 · Cited by 7
Seventy-five years of insight – the impact of the Ukulinga very long-term grassland experiments
Morris, C. D., & Kirkman, K. P. (2025) · African Journal of Range and Forage Science · doi:10.2989/10220119.2025.2527194 · Cited by 4
Metaepidemiologie und Qualitätssicherung klinischer Evidenzproduktion
Haring, R. (2017) · Springer eBooks · doi:10.1007/978-3-662-55377-0_4 · Cited by 3
Prostate cancer screening: what can we learn from randomised trials?
Auvinen, A. P. (2018) · Translational Andrology and Urology · doi:10.21037/tau.2017.12.13 · Cited by 3
Shaw, R., Ni, M. Y., Pillar, M., & Tejani, A. M. (2018) · Accountability in Research · doi:10.1080/08989621.2018.1486191 · Cited by 3
“…involved a comparison of registered versus unregistered trials for the proportion of trials in each subgroup that reported on several key methodologic characteristics that are known to be important for assessing internal study bias/validity (See Table 1) (Higgins et al. 2011; Page et al. 2016).”
BackgroundEstimating the effect of COVID-19 on trial design characteristics: a registered report
Smith, J. A., DeVito, N., Lee, H., Tiplady, C., Abhari, R. E., & Kartsonaki, C. (2023) · Royal Society Open Science · doi:10.1098/rsos.201543 · Cited by 3
Effects of aquatic exercises in patients after total hip arthroplasty: A systematic review
Centurião, J. H. V. M., Obara, K., Silva, C. T., Paixão, L., Silva, M. F., Dias, J. M., & Cardoso, J. R. (2023) · Physiotherapy Research International · doi:10.1002/pri.2043 · Cited by 5
Youn, S., Wong, S. A., Chrystoja, C. C., TOMLINSON, G. A., Wijeysundera, H. C., Bell, C. M., Gagliardi, A. R., Baxter, N. N., Takata, J. L., Sandhu, L., et al. (2021) · BMC Surgery · doi:10.1186/s12893-021-01278-0 · Cited by 4
Success of Masking 5% Povidone‐Iodine Treatment: The Reducing Adenoviral Patient Infected Days Study
Whiteside, M. M., Shorter, E. S., Margolis, M. S., Alvi, F., Huecker, J. B., Than, T. P., Migneco, M. K., Harthan, J., Morettin, C. E., Hartwick, A. T. E., et al. (2021) · Optometry and Vision Science · doi:10.1097/opx.0000000000001691 · Cited by 4
Mickenautsch, S., Rupf, S., & Yengopal, V. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1201951 · Cited by 3
“The development was further supported by the results of a systematic review of meta-epidemiological studies concerning empirical evidence of bias effect on clinical trial results (2), a systematic review of types of bias in epidemiology (1) and a cross-sectional study on selective outcome reporting (3).”
BackgroundMethodResultInfluentialAssessing the Risk of Bias in Randomized Trials
Page, M. J., Altman, D. G., & Egger, M. (2022) · Wiley eBooks · doi:10.1002/9781119099369.ch4 · Cited by 3
Self-citationRose, C. J., Ringsten, M., Bidonde, J., Glanville, J., Berg, R. C., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Potrebny, T. (2023) · Research Square · doi:10.21203/rs.3.rs-3288515/v1 · Cited by 3
Laursen, D. R. T., Broager, M. I., Damkjær, M. W., Eiset, A. H., Elkjær, M., Faltinsen, E. G., MacLean-Nyegaard, I. R., Nejstgaard, C. H., Paludan‐Müller, A. S., Petersen, L. A. B., et al. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111998 · Cited by 2
Self-citationMickenautsch, S., & Yengopal, V. (2025) · Cureus · doi:10.7759/cureus.94704 · Cited by 2
Karahan, S., Dağ, Z. B., & Özyürek, E. (2026) · Journal of Endodontics · doi:10.1016/j.joen.2025.12.010 · Cited by 3
Early-phase trial efficacy and safety effect estimates regress upon testing in phase 3 trials
Nelson, A., Moyer, H., Vigneault, K., Fernandez, L., Sefranek, L., Mauguen, A., & Kimmelman, J. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112149 · Cited by 3
Gregersen, L., Moos, C. M., Hikmat, Z., Rasmussen, N. F., Petersen, S. R., Heitmann, B. L., Halldorsson, T. I., Andersen, V. C., & Christensen, R. (2026) · Advances in Nutrition · doi:10.1016/j.advnut.2026.100614 · Cited by 2
“Moreover, this study is based on observational data extracted from publications and possible confounding by e.g., resources, expertise, and local research policies may exist [109].”
Stefani, C. M., Rezende, L. V. M. D. L., Honnef, L. R., Oliveira, L. B., Lopes, N. D. C., Massignan, C., Canto, G. D. L., & de Oliveira, J. M. D. (2022) · Journal of Evidence-Based Healthcare · doi:10.17267/2675-021xevidence.2022.e4067 · Cited by 4
Metadata
- Authors
- Page, M. J., Higgins, J. P. T., Clayton, G., Sterne, J. A. C., Hróbjartsson, A., & Savović, J.
- Affiliations
- University of Bristol (GB)
- University of Southern Denmark (DK)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Corresponding author
- Page, Matthew J.
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-07-11
- Volume
- 11
- Issue
- 7
- Pages
- e0159267
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4939945
- OpenAlex
- W2461177968
- Cited by
- 284
- References
- 64
- Citations by year
- 2026: 24 · 2025: 22 · 2024: 25 · 2023: 29 · 2022: 45 · 2021: 35 · 2020: 33 · 2019: 32 · 2018: 21 · 2017: 14 · 2016: 3
- FWCI
- 7.97
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- University of Southern Denmark Research Portal (University of Southern Denmark) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- Explore Bristol Research (submittedVersion, cc-by)
- Funders
- MRC Network of Hubs for Trials Methodology Research (MR/L004933/1- N61)
- National Health and Medical Research Council (1088535)
- National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Keywords
- study design bias, meta-epidemiology, sequence generation, allocation concealment, blinding, double blinding, subjective outcomes, objective outcomes, attrition, selective attrition, reporting quality, trial heterogeneity, standardized mean difference
- MeSH terms
- Humans, Bias, Epidemiologic Studies, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.