Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies
Open accessThe DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To synthesise evidence on the average bias and heterogeneity associated with reported methodological features of randomized trials. DESIGN: Systematic review of meta-epidemiological studies. METHODS: We retrieved eligible studies included in a recent AHRQ-EPC review on this topic (latest search September 2012), and searched Ovid MEDLINE and Ovid EMBASE for studies indexed from Jan 2012-May 2015. Data were extracted by one author and verified by another. We combined estimates of average bias (e.g. ratio of odds ratios (ROR) or difference in standardised mean differences (dSMD)) in meta-analyses using the random-effects model. Analyses were stratified by type of outcome ("mortality" versus "other objective" versus "subjective"). Direction of effect was standardised so that ROR < 1 and dSMD < 0 denotes a larger intervention effect estimate in trials with an inadequate or unclear (versus adequate) characteristic. RESULTS: We included 24 studies. The available evidence suggests that intervention effect estimates may be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (ROR 0.93, 95% CI 0.86 to 0.99; 7 studies) and allocation concealment (ROR 0.90, 95% CI 0.84 to 0.97; 7 studies). For these characteristics, the average bias appeared to be larger in trials of subjective outcomes compared with other objective outcomes. Also, intervention effects for subjective outcomes appear to be exaggerated in trials with lack of/unclear blinding of participants (versus blinding) (dSMD -0.37, 95% CI -0.77 to 0.04; 2 studies), lack of/unclear blinding of outcome assessors (ROR 0.64, 95% CI 0.43 to 0.96; 1 study) and lack of/unclear double blinding (ROR 0.77, 95% CI 0.61 to 0.93; 1 study). The influence of other characteristics (e.g. unblinded trial personnel, attrition) is unclear. CONCLUSIONS: Certain characteristics of randomized trials may exaggerate intervention effect estimates. The average bias appears to be greatest in trials of subjective outcomes. More research on several characteristics, particularly attrition and selective reporting, is needed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Fig 3 Random-effects meta-analysis of RORs associated with inadequate/unclear (versus adequate) sequence generation. The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR is (95% CrI) 0.89 (0.82, 0.96)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.89 (0.75, 1.05)]. The Unverzagt 2013c ROR is based on a multivariable analysis with adjustment for allocation concealment, double blinding, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.98 (0.8, 1.21)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.99 (0.84, 1.16)]. The BRANDO 2012e ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.83 (0.74, 0.94)].](https://figures.cochat.ai/paper-figures/2983b5b33b19494280d6784de9530e9d/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-3.jpg)
![Fig 4 Random-effects meta-analysis of RORs associated with inadequate/unclear (versus adequate) allocation concealment. The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.93 (0.87, 0.99)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.98 (0.88, 1.10)]. The BRANDO 2012c ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.97 (0.85, 1.10)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.85 (0.75, 0.95)].](https://figures.cochat.ai/paper-figures/594b60d931a944d4b7cd0812e7e54cf5/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-4.jpg)
![Fig 5 Random-effects meta-analysis of RORs and dSMDs associated with presence (versus absence) of baseline imbalance. The Unverzagt 2013a ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.92 (0.80, 1.06)].](https://figures.cochat.ai/paper-figures/2117adfe36b54af3aedfc58feb8e5a77/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-5.jpg)


![Fig 8 Estimated RORs and dSMDs associated with any (versus no or minimal) attrition. The boxed section displays the average bias estimates, where available, from the four meta-epidemiological studies contributing to the BRANDO 2012 study. The Abraha 2015a ROR is based on a multivariable analysis with adjustment for use of placebo comparison, sample size, type of centre, items of risk of bias, post-randomisation exclusions, funding, and publication bias [the corresponding univariable ROR (95% CI) is 0.83 (0.71, 0.97)]. The Unverzagt 2013b ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 1.19 (0.98, 1.45)]. The Nuesch 2009bc dSMD is based on a multivariable analysis with adjustment for allocation concealment [the corresponding multivariable dSMD (95% CI) with adjustment for blinding of participants is -0.15 (-0.30, 0.00), and the corresponding univariable dSMD (95% CI) is -0.13 (-0.29, 0.04)].](https://figures.cochat.ai/paper-figures/0e8ce72c7fdb470c826c8be718f13f64/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-8.jpg)

![Fig 10 Random-effects meta-analysis of RORs associated with lack of/unclear double blinding (versus double blinding). The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.87 (0.79, 0.96)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.92 (0.80, 1.04)]. The Unverzagt 2013c ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.84 (0.69, 1.02)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.93 (0.74, 1.18)]. The BRANDO 2012e ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.78 (0.65, 0.92)].](https://figures.cochat.ai/paper-figures/2137b7d01f5c4ed3952ef80ca31122dd/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-10.jpg)
![Fig 11 Random-effects meta-analysis of RORs and dSMDs associated with high/unclear (versus low) risk of bias due to selective reporting. The Unverzagt 2013a ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, attrition, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.73 (0.54, 0.98)].](https://figures.cochat.ai/paper-figures/518f65e477414b18b41b5c661142ba2f/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-11.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical evidence and theoretical considerations suggest that several features of study design are associated with larger intervention effect estimates in studies; these features include inadequate generation and concealment of a random sequence to assign participants to groups, substantial loss to follow-up of participants, and unblinded outcome assessment.”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [80]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“based on meta-epidemiological studies [22][23][24][25] that found little to no effect of blinding on treatment results among randomised trials (including a review of oral health trials 26 )”
From Management of chronic pain secondary to temporomandibular disorders: a systematic review and network meta-analysis of randomised trials (2023) · cites this paper as [23]
Experimental — accuracy not yet measuredIn the citing paper
“There is a suggestion of missing results in the left-hand side of the plot, where results would be unfavourable to SSRIs and in the area of statistical nonsignificance (i.e. the white area where P > 0.10), which adds credence to the possibility that the asymmetry is due to reporting biases.”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [59]
In the citing paper
“There is a suggestion of missing results in the lefthand side of the plot, where results would be unfavourable to SSRIs, and in the area of statistical nonsignificance, which adds credence to the possibility that the asymmetry is due to reporting biases.”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Self-citationPreferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · cited 1×
“We used DerSimonian and Laird’s method of moments estimator to estimate the betweenstudy variance [22].”
MethodThe Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Zhang, Q., Wang, C., Zheng, M., Li, Y., Li, J., Zhang, L., Shang, X., & Yan, C. (2015) · Cerebrovascular Diseases · doi:10.1159/000369778 · Cited by 11,533
Research works printed but not linked: 2
- The Oxford 2011 Levels of Evidence
- Als-Nielsen B, Chen W, Gluud LL, Siersma V, Hilden J, Gluud C. Are trial size and reported methodological quality associated with treatment effects? Observational study of 523 randomised trials [abstract]. 12th Cochrane Colloquium: Bridging the Gaps; 2004 Oct 2–6; Ottawa, Ontario, Canada [Internet]. 2004:[102–3 pp.]. Available: http://onlinelibrary.wiley.com/o/cochrane/clcmr/articles/CMR-6643/frame.html.
Cited by
Items 201–283 of 283
Critical reading of systematic review articles
Laursen, D. R. T., Brorson, S., Lundh, A., Møller, A. M., Nejstgaard, C. H., Rosenberg, J., & Hróbjartsson, A. (2024) · Ugeskrift for Læger · doi:10.61409/v09230616 · Cited by 2
Self-citationShu, M.-Y., Oh, H. S., Jo, Y.-J., Eom, S.-H., Liang, J., Jung, S.-M., Kim, K., Jung, J., Ma, C., & Kim, C. (2025) · Applied Sciences · doi:10.3390/app15147950 · Cited by 7
Meyer, C. L., Bowers, A., Jaiswal, D. D., Checketts, J. X., Engheta, M., Severns, C., Cook, S., Walters, C. G., & Vassar, M. B. (2019) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2019.03.038 · Cited by 3
“While much attention has been devoted to the quality, design, and sample size in RCTs, little is known about the robustness of trial results [14,19-24].”
BackgroundWyer, P. C., & Loughlin, M. (2020) · European Journal for Person Centered Healthcare · doi:10.5750/ejpch.v8i2.1847 · Cited by 4
“characteristics, when present, are associated with smaller, and presumably more plausible, estimates of effect [67,68].”
BackgroundEick, S. M., Goin, D. E., Lam, J., Woodruff, T. J., & Chartres, N. (2022) · Systematic Reviews · doi:10.1186/s13643-022-01894-8 · Cited by 2
“There is empirical evidence that inadequate application of randomization and blinding results in overestimation of efficacy of drug effects [8, 9].”
BackgroundSinnige, J. S., Karbing, D. S., Valk, C. M. A., Schultz, M. J., Rees, S. E., & Paulus, F. (2024) · Respiratory Care · doi:10.4187/respcare.11584 · Cited by 2
“A further limitation of this type of study was the potential for bias in subjective evaluation of the need and success of suctioning 13 and the lack of blinding.”
Leucht, S., Siafis, S., Schneider‐Thoma, J., Tajika, A., Priller, J., Davis, J. M., & Furukawa, T. A. (2024) · Schizophrenia · doi:10.1038/s41537-024-00442-8 · Cited by 3
“…study on the effects of blinding quoted above 1 did not reveal bias, systematic reviews of such meta-epidemiological studies ( “ meta-meta-meta-analyses ” ) suggest that overall, the fi eld is mixed with some studies fi nding and others not fi nding important effects of blinding 28 – 32 .”
Hengartner, M. P., Jakobsen, J. C., Sorensen, A., & Plöderl, M. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/3tuzy · Cited by 2
Mickenautsch, S., Rupf, S., Miletić, I., & Yengopal, V. (2022) · Research Square · doi:10.21203/rs.3.rs-1704764/v2 · Cited by 2
Innocenti, T., Hayden, J. A., Salvioli, S., Giagio, S., Piano, L., Cosentino, C., Brindisino, F., Feller, D., Ogilvie, R., Gianola, S., et al. (2022) · medRxiv · doi:10.1101/2022.07.20.22277703 · Cited by 2
Interpretation von systematischen Übersichtsarbeiten und Metaanalysen
Stengel, D., Mutschler, W., Dubs, L., Kirschner, S., & Renkawitz, T. F. (2022) · Die Unfallchirurgie · doi:10.1007/s00113-022-01244-2 · Cited by 1
“Ein zunehmend populäresMaß für die Unsicherheit von Therapieeffekten ist der Fragilitätsindex (FI).”
BackgroundInfluentialCrocker, T. F., Todd, O. M., & Clegg, A. P. (2025) · Journal of the American Geriatrics Society · doi:10.1111/jgs.19498 · Cited by 4
“Therefore, such reasons will often be associated with the value of a clinical outcome, producing a theoretical risk of bias with unknown direction, and untestable due to the missingness of the data [67–69].”
Bantle, G., Stadelmaier, J., Πετροπούλου, Μ., Meerpohl, J. J., & Schwingshackl, L. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.112038 · Cited by 4
Polo-Ferrero, L., Torres-Alonso, J., Sánchez-González, J. L., Hernández-Rubia, S., Juan, M. A., Pérez-Elvira, R., & Oltra‐Cucarella, J. (2026) · Medicina · doi:10.3390/medicina62010174 · Cited by 2
“For analyses involving the risk of bias scale, studies with low risk served as the reference category, based on prior evidence indicating that trials with higher bias levels may overestimate effect sizes [42,43].”
Swartz, D., Nault, A. J., & Cramer, G. (2026) · Journal of Dairy Science · doi:10.3168/jds.2025-27833 · Cited by 1
Lazzari, C. G., Crawford, P., Ashmore, A., & Kotera, Y. (2026) · Systematic Reviews · doi:10.1186/s13643-026-03093-1 · Cited by 2
“Final synthesis retained the original tool-specific judgments and used narrative integration to ensure transparency and avoid misleading cross-study comparisons [211].”
Jiang, Y., Zhong, C. C., Wang, B. H., Xu, S., Ho, F. F., Kwong, M. H., Ho, L. T. F., Zhou, J., Lam, K., Liu, J.-P., et al. (2026) · Integrative Medicine Research · doi:10.1016/j.imr.2026.101317 · Cited by 1
“17 Finally, incomplete outcome data was known to be associated with inaccurate outcome estimation, or causing failure in detecting differences between groups.”
InfluentialLiu, Y. C., Yan, Q., Zhao, S., Ng, S., Wang, X., & Ning, Z. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1711994 · Cited by 1
“Outcomes may be measured by informant ratings vs. performance-based tasks; assessment timing and reporting practices differ; and risk-of-bias features (such as limited blinding in subjective outcomes) may influence effect estimates (McGuinness and Higgins, 2021; Page et al., 2016; Sterne et al., 2019).”
Agarwal, A., Gao, Y., Lozano, L. E. C., Asif, S., Bakaa, L., Ghadimi, M., Basmaji, J., Das, A., Loeb, M. B., & Guyatt, G. H. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-062428 · Cited by 1
Masri, A. A., Schiffner, U., Mourad, M. S., Schmoeckel, J., Joseph, P., & Splieth, C. H. (2023) · European Archives of Paediatric Dentistry · doi:10.1007/s40368-023-00854-7 · Cited by 1
“This underlines the necessity of improving the quality of study design and reporting of RCTs because biased results of RCTs tend to increase the effect size estimation and raise suspicion about their reliability (Page et al. 2016; Saltaji et al. 2018).”
BackgroundSilva, N. L. D., Hettiarachchi, M., Ranasinghe, C. S., Yahathugoda, T. C., & Ranasinghe, S. (2025) · Asian Pacific Journal of Tropical Medicine · doi:10.4103/apjtm.apjtm_685_24 · Cited by 1
Zhang, J., Wang, J., Han, L., Cao, X., & Shields, L. E. (2016) · Nursing Outlook · doi:10.1016/j.outlook.2016.11.004 · Cited by 2
“Since inclusion of less robust studies can lead to unreliable, often inflat ed estimates of treatment effects[3], a key element in the SR process is appraisal of the r isk of bias or methodological quality of the constituent primary studies to decid whether their results can be trusted and should contribute to eventua…”
BackgroundMasking of Trial Investigators
Howard, G., & Voeks, J. H. (2022) · Springer eBooks · doi:10.1007/978-3-319-52636-2_215 · Cited by 1
A System for Rapidly Yet Rigorously Evaluating the Quality of Randomized Controlled Trials
Smith, E. G., & Grigorian, H. L. (2023) · Journal of Clinical Psychopharmacology · doi:10.1097/jcp.0000000000001724 · Cited by 2
Olaiya, O. R., Huynh, M., & McRae, M. H. (2020) · Plastic & Reconstructive Surgery Global Open · doi:10.1097/gox.0000000000002908 · Cited by 1
Mickenautsch, S., Rupf, S., Miletić, I., & Yengopal, V. (2022) · Research Square · doi:10.21203/rs.3.rs-1704764/v1 · Cited by 1
Zeraatkar, D., Pitre, T., Díaz-Martínez, J. P., Chu, D. K., Rochwerg, B. N., Lamontagne, F., Kum, E., Qasim, A., Bartoszko, J. J., & Brignardello‐Petersen, R. (2022) · medRxiv · doi:10.1101/2022.08.03.22278348 · Cited by 1
Unpacked Cochrane Review on Masks: A Further Look
Kivelä, J. M. (2024) · American Journal of Public Health · doi:10.2105/ajph.2023.307526 · Cited by 1
Steps into the future of childhood disability
Becher, J. G. (2017) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.13400
“In parallel, the results of randomized controlled trials (RCTs) and systematic reviews are more critically received.(2) While RCTs can at times provide clear answers to specific clinical questions, the nature, complexity, and chronicity of childhood neurodisability, and the life-course perspectives needed to assess th…”
BackgroundBernardini, M. (2018) · PSICOBIETTIVO · doi:10.3280/psob2018-001004
The contribution of Douglas Altman’s research to the EVIDENCE Journal
Helal, L. C. A. S. (2018) · Journal of Evidence-Based Healthcare · doi:10.17267/2675-021xevidence.v1i1.2007
Incorporating external evidence syntheses in the design and analysis of trials
Clayton, G. (2019) · Bristol Research (University of Bristol)
Self-citationMasking of Trial Investigators
Howard, G., & Voeks, J. H. (2019) · Springer eBooks · doi:10.1007/978-3-319-52677-5_215-1
How to read a published clinical trial: a practical guide for clinicians
Sonbol, M. B., Firwana, B., Hilal, T., & Murad, M. H. (2020) · Avicenna Journal of Medicine · doi:10.4103/ajm.ajm_186_19 · Cited by 2
“Indeed, the effect of blinding in a study should be assessed for each individual outcome as it is probably less important in objective outcomes as OS (death or alive) compared to progression-free survival.[24]”
BackgroundIntroduction to the Special Issue on Translation
DeWitt, D. S., & Prough, D. S. (2020) · Journal of Neurotrauma · doi:10.1089/neu.2020.7390
“5,6 In addition to these and other limitations in pre-clinical therapy testing, the validity of many clinical trials has been reduced by inadvertent bias, 7 and the relatively large number of single-center 1 and underpowered clinical trials.”
Professionally and Self-Applied Fluorides are Effective in Preventing Dental Root Caries
Cunha‐Cruz, J., & dos Santos, A. P. P. (2020) · Journal of Evidence Based Dental Practice · doi:10.1016/j.jebdp.2020.101523
How to Identify, Interpret and Apply the Scientific Literature to Practice
Carrasco‐Labra, A., Tampi, M. P., Urquhart, O., Howell, S., Booth, A., & Glick, M. (2021) · Wiley eBooks · doi:10.1002/9781119597797.ch29
Shu, M.-J., Chen, S., & Li, J. (2021) · Journal of General Internal Medicine · doi:10.1007/s11606-021-07142-4
“However, trials with inadequate or unclear sequence generation and allocation concealment were found to report higher estimates of effect than those with detailed methods.(5) One weakness of this study is that it is based on published reports rather than study protocols.”
Ding, R., Lu, W., Yi, J., Zhang, L., & Zhao, Z. (2020) · doi:10.21203/rs.3.rs-41578/v1
“It is well recognized that the bias occurring in the design and conduct of studies could impair the internal validity [6], and induce systematic deviations in the estimation of intervention effects of RCTs [7].”
BackgroundHemodynamic interventions and -monitoring in critically ill patients
Koster, G. (2021) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.192215589
Sidani, S., & Braden, C. J. (2021) · Wiley eBooks · doi:10.1002/9781119610113.ch14
Gurusamy, K. S. (2018) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.732099869.793549273
Youn, S., Wong, S. A., Chrystoja, C. C., TOMLINSON, G. A., Wijeysundera, H. C., Bell, C. M., Gagliardi, A. R., Baxter, N. N., Sandhu, L., Takata, J. L., et al. (2020) · Research Square · doi:10.21203/rs.3.rs-71534/v1
Causer, E., Alfawaz, M., Binbraik, Y., Lombardi, J., Lawati, K. A., Sharif, S., Ligori, T., & Oczkowski, S. J. W. (2022) · Trends in Anaesthesia and Critical Care · doi:10.1016/j.tacc.2022.101207
Lee, Z. Y. (2023) · Medical Writing · doi:10.56012/ejyj3404
Compreensão da situação global da mastite em búfalas leiteiras: revisão sistemática
Maia, R. B. M. (2022) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.10.2022.tde-27022023-105235 · Cited by 1
Ten resources for understanding bias in health research: EBM live workshop 2022
Nunan, D., Bashir, K., Bilimoria, K., Birdi, J. S., Campbell, F., Dean, R. S., Downer, M. B., da Costa, G. G., Golob, M. M., Heintzman, A., et al. (2023) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2023-112344
Rayner, C. L. (2024) · Cronfa (Swansea University) · doi:10.23889/suthesis.66311 · Cited by 1
Shashidhar, S. K. M. (2024) · International Journal of Science and Research (IJSR) · doi:10.21275/sr24216013104
“[57] Diverse patient populations: The studies recruited adults and childrens with varied asthama severities (acute exacerbations, slight persistent asthma, hospitalised situations).”
Selection bias risk in “low bias” (RoB 2) rated randomised control trials (Preprint)Preprint
Mickenautsch, S., & Yengopal, V. (2024) · Research Square · doi:10.21203/rs.3.rs-4567677/v1
Monaghan, T. F., Wein, A. J., & Dmochowski, R. R. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-90186-4.00087-0
Xu, C., Fan, S., Furuya‐Kanamori, L., Li, S., Lin, L., Chu, H., Golder, S., Loke, Y. K., & Vohra, S. (2024) · BMC Medicine · doi:10.1186/s12916-024-03621-7 · Cited by 1
“Based on previous studies [29, 30], the following confounders were considered: intervention, control, treatment duration, dosage, allocation concealment, blinding status for participants, blinding status for care providers, blinding status for outcome asses-sors, population age, disease, number of…”
MethodKim, T. (2024) · Perspectives on Integrative Medicine · doi:10.56986/pim.2024.10.009
“The success of blinding has been recognized as a significant factor influencing the effect size in clinical trials, and empirical evidence supports this hypothesis [7].”
BackgroundSources of Bias in Randomised Controlled Trials
Crombie, I. (2021) · Wiley eBooks · doi:10.1002/9781394322183.ch2
Brophy, J. M. (2025) · medRxiv · doi:10.1101/2025.03.11.25323785
Quality Assessment of Computer-Assisted Navigation Knee Arthroplasty Studies
Spooner, A., Helton, A. G., Lee, V., Redden, D. T., Ponce, B. A., Fernicola, J. P., & Ruark, R. J. (2025) · The Journal of Arthroplasty · doi:10.1016/j.arth.2025.05.108
Concordance of Efficacy and Safety in Phase Ib/Ii and Phase Iii Solid Tumour Treatment TrialsPreprint
Nelson, A., Moyer, H., Vigneault, K., Fernandez, L., Sefranek, L., Mauguen, A., & Kimmelman, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5362489
Vasileva, A. V. (2025) · Neurology neuropsychiatry Psychosomatics · doi:10.14412/2074-2711-2025-5-113-120
Rodrigues, J. P., Sá, K. M. M., Lopes, G. C. M., Piovesan, M., Santos, E. M., & Martimbianco, A. L. C. (2025) · Dermatology Practical & Conceptual · doi:10.5826/dpc.1504a5668
“Moreover, systematically identifying recurring methodological limitations can inform improvements in the design, conduct, and reporting of future trials, ultimately contributing to more transparent, rigorous, and trustworthy research that underpins evidence-based clinical practice [1,4].”
Zhang, Y., Wang, B. H., Cheng, B. Z. Z., Zhong, C. C., Ho, F., & Chung, V. C. (2025) · Digestive Surgery · doi:10.1159/000549851
“SRs with flaws in methodological quality could overestimate or un-derestimate the effects of treatments [12, 13]; thus, evidence users, such as policymakers, healthcare practitioners, caregivers, and patients, should evaluate the quality of evidence to avoid misguiding clinical decisions.”
Las publicaciones médicas por dentro
de Antioquia, U., & Jaimes, F. A. (2025) · Editorial Universidad de Antioquia eBooks · doi:10.17533/udea.978-958-501-247-9
Stadelmaier, J., Roux, I., Petropoulou, M., & Schwingshackl, L. (2022) · Figshare · doi:10.6084/m9.figshare.21310965
Stadelmaier, J., Roux, I., Petropoulou, M., & Schwingshackl, L. (2022) · Figshare · doi:10.6084/m9.figshare.21310971
Youn, S., Wong, S. A., Chrystoja, C. C., Tomlinson, G., Wijeysundera, H. C., Bell, C. M., Gagliardi, A. R., Baxter, N. N., Takata, J. L., Sandhu, L., et al. (2021) · Figshare · doi:10.6084/m9.figshare.14746350
Xu, C., Zhang, F., Doi, S. A. R., Furuya‐Kanamori, L., Lin, L., Chu, H., Yang, X., Li, S., Zorzela, L., Golder, S., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26686050
Clayton, G. L., Elliott, D., Higgins, J. P. T., & Jones, H. E. (2021) · Figshare · doi:10.6084/m9.figshare.16960411
Self-citationStadelmaier, J., Roux, I., Petropoulou, M., & Schwingshackl, L. (2022) · Figshare · doi:10.6084/m9.figshare.21310968
Zhuang, S., Zhang, X., Lin, C., & Wang, Z. (2026) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-025-09347-8
“Evidence Evidence suggests that ambiguous randomization reporting and unblinded study designs may increase the treatment effect by approximately 10% and 68%, respectively [62, 63].”
Outcome reporting bias in ophthalmology clinical trials between 2014 and 2023
Li, X., Tien, C.-W., Huang, R. S., Hanna, A., Acharya, S., Lusterio, A., Hsiao, H., Tao, B. K.-L., Xie, J. S., & Margolin, E. (2026) · British Journal of Ophthalmology · doi:10.1136/bjo-2025-328466
Tilborghs, J., de Zoete, A., de Boer, M. R., Innocenti, T., Laguna, J. M., McKenzie, B. J., Ostelo, R., Nim, C. G., Hartvigsen, J., & Rubinstein, S. M. (2026) · MethodsX · doi:10.1016/j.mex.2026.103895
“Since bias cannot be measured directly, its potential must be rigorously assessed, referred to as risk of bias (RoB) [1,2].”
Rehman, N., Guyatt, G. H., Jin, M. J., Silva, L. K., Gu, J., Munir, M., Sadagari, R., Li, M., Xie, D., Rajkumar, S., et al. (2026) · medRxiv · doi:10.64898/2026.04.18.26351146
Villalón, F., & Escaffi‐Schwarz, M. (2026) · Journal of Medical Internet Research · doi:10.2196/87416
“In contrast, earlier studies that relied on self-enrollment or site-based randomization may have introduced uncertainty in sequence generation and allocation concealment, which can modestly inflate effect sizes by 7%10% [30,31].”
Blok, S. L., Geelen, S. J. G., Hulleman, D. N., Scholten-Jaegers, S. M., van Baar, M. E., Pijpe, A., Nieuwenhuis, M. K., Bosma, E., van der Sluis, C. K., Baran, K. C., et al. (2026) · Burns · doi:10.1016/j.burns.2026.108080
How Much Is Enough: A Randomized Non-Inferiority Trial Comparing Three Bodyweight Training Protocols
Aube, J. J., Mendolia, P., Storoschuk, K. L., Wyman, E., Peberdy, M. D., Wu, J. J., Simpson-Stairs, N., Swinton, P. A., & Gurd, B. J. (2026) · Journal of Functional Morphology and Kinesiology · doi:10.3390/jfmk11020240
“Risk of bias is implicated as a cause of augmented treatment effect estimates [42] and the generation of irreproducible data [43].”
Lensen, S. F., Afroz, A., Sadler, L., Farquhar, C. M., & Wilkinson, J. (2026) · Trials · doi:10.1186/s13063-026-09763-8
Predatory Publications Bias the Findings of Orthodontic Meta-analyses
Maroulakos, M. P., Tatas, Z., Al‐Moghrabi, D., Pandis, N., & Fleming, P. S. (2026) · Journal of Dental Research · doi:10.1177/00220345261466209
Gonzalez, F., Olmedo, F. A. M., Chavez, S., & López, V. (2026) · doi:10.2196/preprints.108848
Jelf, D. J., & Jelf, M. (2026) · Open Heart · doi:10.1136/openhrt-2026-004371
“Classification of blinding relied on descriptions reported in trial”
Die IGeL-Bewertung zur Osteopathie bei unspezifischen Kreuzschmerzen
Franke, H. (2026) · Osteopathische Medizin · doi:10.1016/s1615-9071(26)00072-9
IGeL-Bewertung zur Osteopathie bei unspezifischen Kreuzschmerzen
Franke, H. (2026) · DO - Deutsche Zeitschrift für Osteopathie · doi:10.1055/a-2700-2519
Wang, X., Jing, W., Li, Y., Li, H., Li, J., Liu, Y., & Yuan, S. (2026) · Journal of Medical Internet Research · doi:10.2196/95903
“Awareness of treatment allocation may therefore have influenced participant expectations, engagement, or outcome reporting [130].”
Metadata
- Authors
- Page, M. J., Higgins, J. P. T., Clayton, G., Sterne, J. A. C., Hróbjartsson, A., & Savović, J.
- Affiliations
- University of Bristol (GB)
- University of Southern Denmark (DK)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Corresponding author
- Page, Matthew J.
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-07-11
- Volume
- 11
- Issue
- 7
- Pages
- e0159267
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4939945
- OpenAlex
- W2461177968
- Cited by
- 284
- References
- 64
- Citations by year
- 2026: 24 · 2025: 22 · 2024: 25 · 2023: 29 · 2022: 45 · 2021: 35 · 2020: 33 · 2019: 32 · 2018: 21 · 2017: 14 · 2016: 3
- FWCI
- 7.97
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- University of Southern Denmark Research Portal (University of Southern Denmark) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- Explore Bristol Research (submittedVersion, cc-by)
- Funders
- MRC Network of Hubs for Trials Methodology Research (MR/L004933/1- N61)
- National Health and Medical Research Council (1088535)
- National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Keywords
- study design bias, meta-epidemiology, sequence generation, allocation concealment, blinding, double blinding, subjective outcomes, objective outcomes, attrition, selective attrition, reporting quality, trial heterogeneity, standardized mean difference
- MeSH terms
- Humans, Bias, Epidemiologic Studies, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.