The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Compelling evidence from meta-epidemiological studies demonstrates overestimation of effects in randomized trials that fail to optimize randomization and blind patients and outcome assessors
Abstract
Objectives To investigate the impact of potential risk of bias elements on effect estimates in randomized trials.Study design and setting We conducted a systematic survey of meta-epidemiological studies examining the influence of potential risk of bias elements on effect estimates in randomized trials. We included only meta-epidemiological studies that either preserved the clustering of trials within meta-analyses (compared effect estimates between trials with and without the potential risk of bias element within each meta-analysis, then combined across meta-analyses; between-trial comparisons), or preserved the clustering of substudies within trials (compared effect estimates between substudies with and without the element, then combined across trials; within-trial comparisons). Separately for studies based on between- and within-trial comparisons, we extracted ratios of odds ratios (RORs) from each study and combined them using a random-effects model. We made overall inferences and assessed certainty of evidence based on Grading of Recommendations, Assessment, development, and Evaluation and Instrument to assess the Credibility of Effect Modification Analyses.Results Forty-one meta-epidemiological studies (34 of between-, 7 of within-trial comparisons) proved eligible. Inadequate random sequence generation (ROR 0.94, 95% confidence interval [CI] 0.90-0.97) and allocation concealment (ROR 0.92, 95% CI 0.88-0.97) probably lead to effect overestimation (moderate certainty). Lack of patients blinding probably overestimates effects for patient-reported outcomes (ROR 0.36, 95% CI 0.28-0.48; moderate certainty). Lack of blinding of outcome assessors results in effect overestimation for subjective outcomes (ROR 0.69, 95% CI 0.51-0.93; high certainty). The impact of patients or outcome assessors blinding on other outcomes, and the impact of blinding of health-care providers, data collectors, or data analysts, remain uncertain. Trials stopped early for benefit probably overestimate effects (moderate certainty). Trials with imbalanced cointerventions may overestimate effects, while trials with missing outcome data may underestimate effects (low certainty). Influence of baseline imbalance, compliance, selective reporting, and intention-to-treat analysis remain uncertain.Conclusion Failure to ensure random sequence generation or adequate allocation concealment probably results in modest overestimates of effects. Lack of patients blinding probably leads to substantial overestimates of effects for patient-reported outcomes. Lack of blinding of outcome assessors results in substantial effect overestimation for subjective outcomes. For other elements, though evidence for consistent systematic overestimate of effect remains limited, failure to implement these safeguards may still introduce important bias.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Subjective outcomes are more prone to bias from inadequate blinding (ascertainment bias) and allocation concealment (selection bias) than objective outcomes.
“and subjective outcomes are more prone to bias from inadequate blinding (ascertainment bias) and allocation concealment (selection bias) than objective outcomes.271”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [271]
Citations
Cites
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationGRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
Self-citationSchulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Self-citationSchandelmaier, S., Briel, M., Varadhan, R., Schmid, C. H., Devasenapathy, N., Hayward, R. A., Gagnier, J., Borenstein, M., van der Heijden, G. J. M. G., Dahabreh, I. J., et al. (2020) · Canadian Medical Association Journal · doi:10.1503/cmaj.200077 · Cited by 749
Self-citationSavović, J., Jones, H., Altman, D., Harris, R., Jűni, P., Pildal, J., Als-Nielsen, B., Balk, E., Gluud, C., Gluud, L., et al. (2012) · Health Technology Assessment · doi:10.3310/hta16350 · Cited by 668
Self-citationBassler, D. (2010) · JAMA · doi:10.1001/jama.2010.310 · Cited by 613
Self-citationHróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S. (2013) · Canadian Medical Association Journal · doi:10.1503/cmaj.120744 · Cited by 527
Self-citationHróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S. (2012) · BMJ · doi:10.1136/bmj.e1119 · Cited by 505
Self-citationThe Magic of Randomization versus the Myth of Real-World Evidence
Collins, R., Bowman, L. J., Landray, M., & Peto, R. (2020) · New England Journal of Medicine · doi:10.1056/nejmsb1901642 · Cited by 490
Balk, E. M. (2002) · JAMA · doi:10.1001/jama.287.22.2973 · Cited by 478
Hróbjartsson, A., Emanuelsson, F., Skou Thomsen, A. S., Hilden, J., & Brorson, S. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyu115 · Cited by 462
Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change
Cornell, J. E., Mulrow, C. D., Localio, R., Stack, C. B., Meibohm, A. R., Guallar, E., & Goodman, S. N. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2886 · Cited by 453
Sterne, J. A. C., Jüni, P., Schulz, K. F., Altman, D. G., Bartlett, C., & Egger, M. (2002) · Statistics in Medicine · doi:10.1002/sim.1184 · Cited by 382
Self-citationSavović, J., Turner, R. M., Mawdsley, D., Jones, H. E., Beynon, R., Higgins, J. P. T., & Sterne, J. A. C. (2018) · American Journal of Epidemiology · doi:10.1093/aje/kwx344 · Cited by 382
Moher, D., Cook, Jadad, Tugwell, P., Moher, Jones, Pham, & Klassen (1999) · Health Technology Assessment · doi:10.3310/hta3120 · Cited by 364
Moustgaard, H., Clayton, G. L., Jones, H. E., Boutron, I., Jørgensen, L., Laursen, D. R. T., Olsen, M. F., Paludan-Müller, A., Ravaud, P., Savović, J., et al. (2020) · BMJ · doi:10.1136/bmj.l6802 · Cited by 300
Self-citationPage, M. J., Higgins, J. P. T., Clayton, G., Sterne, J. A. C., Hróbjartsson, A., & Savović, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0159267 · Cited by 284
Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Burgi, E., Scherer, M., Altman, D. G., & Juni, P. (2009) · BMJ · doi:10.1136/bmj.b3244 · Cited by 248
Emerson, J. D., Burdick, E., Hoaglin, D. C., Mosteller, F. C., & Chalmers, T. C. (1990) · Controlled Clinical Trials · doi:10.1016/0197-2456(90)90175-2 · Cited by 243
Association Between Analytic Strategy and Estimates of Treatment Outcomes in Meta-analyses
Dechartres, A., Altman, D. G., Trinquart, L., Boutron, I., & Ravaud, P. (2014) · JAMA · doi:10.1001/jama.2014.8166 · Cited by 233
Self-citationUnverzagt, S., Prondzinsky, R., & Peinemann, F. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.05.016 · Cited by 172
Blinding in Physical Therapy Trials and Its Association with Treatment Effects
Armijo-Olivo, S. L., Fuentes, J., da Costa, B. R., Saltaji, H., Ha, C., & Cummings, G. G. (2016) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000000521 · Cited by 166
Self-citationAbraha, I., Cherubini, A., Cozzolino, F., De Florio, R., Luchetta, M. L., Rimland, J. M., Folletti, I., Marchesi, M., Germani, A., Orso, M., et al. (2015) · BMJ · doi:10.1136/bmj.h2445 · Cited by 165
Nüesch, E., Reichenbach, S., Trelle, S., Rutjes, A. W., Liewald, K., Sterchi, R., Altman, D. G., & Jüni, P. (2009) · Arthritis Care & Research · doi:10.1002/art.24894 · Cited by 150
Models for Potentially Biased Evidence in Meta-Analysis using Empirically Based Priors
Welton, N. J., Ades, A. E., Carlin, J. B., Altman, D. G., & Sterne, J. A. C. (2008) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2008.00548.x · Cited by 147
Hróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Rasmussen, J. V., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyt270 · Cited by 142
Self-citationKhan, K. S., Daya, S., Collins, J. A., & Walter, S. D. (1996) · Fertility and Sterility · doi:10.1016/s0015-0282(16)58265-x · Cited by 114
Basic study design influences the results of orthodontic clinical investigations
Papageorgiou, S. N., Xavier, G. M., & Cobourne, M. T. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.03.008 · Cited by 108
Saltaji, H., Armijo-Olivo, S. L., Cummings, G. G., Amin, M. S., da Costa, B. R., & Flores‐Mir, C. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0491-0 · Cited by 90
Self-citationArmijo-Olivo, S. L., Saltaji, H., da Costa, B. R., Fuentes, J., Ha, C., & Cummings, G. G. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008562 · Cited by 87
Self-citationEmpirical evaluation of which trial characteristics are associated with treatment effect estimates
Dechartres, A., Trinquart, L., Faber, T., & Ravaud, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.005 · Cited by 85
Self-citationSiersma, V., Als-Nielsen, B. E. T., Chen, W., Hilden, J., Gluud, L. L., & Gluud, C. N. (2006) · Statistics in Medicine · doi:10.1002/sim.2752 · Cited by 83
Self-citationChaimani, A., Vasiliadis, H. S., Pandis, N., Schmid, C. H., Welton, N. J., & Salanti, G. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dyt074 · Cited by 82
Ndounga‐Diakou, L. A., Trinquart, L., Hróbjartsson, A., Barnes, C., Yavchitz, A., Ravaud, P., & Boutron, I. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000043.pub2 · Cited by 57
Self-citationHerbison, P., Hay‐Smith, J., & Gillespie, W. J. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.12.018 · Cited by 56
Bialy, L. M., Vandermeer, B., Lacaze‐Masmonteil, T., Dryden, D. M., & Hartling, L. (2014) · Evidence-Based Child Health A Cochrane Review Journal · doi:10.1002/ebch.1985 · Cited by 46
Saltaji, H., Armijo-Olivo, S., Cummings, G. G., Amin, M., da Costa, B. R., & Flores-Mir, C. (2018) · Journal of Dental Research · doi:10.1177/0022034517725049 · Cited by 40
Self-citationQuantifying Bias in Randomized Controlled Trials in Child Health: A Meta-Epidemiological Study
Hartling, L., Hamm, M. P., Fernandes, R., Dryden, D., & Vandermeer, B. (2014) · PLoS ONE · doi:10.1371/journal.pone.0088008 · Cited by 39
Mostazir, M., Taylor, G. T., Henley, W., Watkins, E. R., & Taylor, R. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.010 · Cited by 34
Haring, R., Ghannad, M., Bertizzolo, L., & Page, M. J. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.02.004 · Cited by 25
Bolvig, J., Juhl, C. B., Boutron, I., Tugwell, P., Ghogomu, E. A. T., Pardo, J. P., Rader, T., Wells, G. A., Mayhew, A., Maxwell, L. J., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.026 · Cited by 23
Martin, G. L., Trioux, T., Gaudry, S., Tubach, F., Hajage, D., & Dechartres, A. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000005065 · Cited by 22
Wang, Y., Ghadimi, M., Wang, Q., Hou, L., Zeraatkar, D., Iqbal, A., Ho, C., Yao, L., Hu, M., Ye, Z., et al. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.10.018 · Cited by 21
Self-citationRusso, C. D., Cappoli, N., & Navarra, P. (2020) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-020-02895-z · Cited by 20
Stadelmaier, J., Roux, I., Πετροπούλου, Μ., & Schwingshackl, L. (2022) · BMC Medicine · doi:10.1186/s12916-022-02540-9 · Cited by 18
Detection of Associations Between Trial Quality and Effect Sizes
Hempel, S., Miles, J. N. V., Suttorp, M. J., Wang, Z., Johnsen, B., Morton, S. C., Perry, T., Valentine, D., & Shekelle, P. (2012) · Cited by 17
Niederer, D., Weippert, M., & Behrens, M. (2022) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2022.11149 · Cited by 16
A meta-epidemiological study of bias in randomized clinical trials of open and laparoscopic surgery
Amer, M., Herbison, P., Grainger, S., Khoo, C. H., Smith, M. D., & McCall, J. L. (2021) · British journal of surgery · doi:10.1093/bjs/znab035 · Cited by 14
Label‐invariant models for the analysis of meta‐epidemiological data
Rhodes, K. M., Mawdsley, D., Turner, R., Jones, H. E., Savović, J., & Higgins, J. P. T. (2017) · Statistics in Medicine · doi:10.1002/sim.7491 · Cited by 13
Wang, H., Song, J., Lin, Y., Dai, W., Gao, Y., Qin, L., Chen, Y., Tam, W., Wu, X., & Chung, V. C. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01650-5 · Cited by 11
Zeraatkar, D., Pitre, T., Díaz-Martínez, J. P., Chu, D. K., Rochwerg, B. N., Lamontagne, F., Kum, E., Qasim, A., Bartoszko, J. J., & Brignardello‐Petersen, R. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwad131 · Cited by 11
Self-citationWang, Z., Alahdab, F., Farah, M. H., Seisa, M. O., Firwana, M., Rajjoub, R., Saadi, S. M., Jawaid, T., Nayfeh, T., & Murad, M. H. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2021-111667 · Cited by 9
Armijo-Olivo, S. L., da Costa, B. R., Ha, C., Saltaji, H., Cummings, G. G., & Fuentes, J. (2021) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000001837 · Cited by 8
Self-citationA simple method for converting an odds ratio to effect size for use in meta-analysis
Chinn, S. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001130)19:22
Flagged first, then by citations
Cited by
Sorted by citations
SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081660 · Cited by 171
Self-citationWang, Y., Keitz, S., Briel, M., Glasziou, P., Brignardello-Petersen, R., Siemieniuk, R. A. C., Zeraatkar, D., Akl, E. A., Armijo-Olivo, S., Bassler, D., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081199 · Cited by 60
Self-citation
Metadata
- Authors
- Wang, Y., Parpia, S., Couban, R., Wang, Q., Armijo-Olivo, S., Bassler, D., Briel, M., Brignardello-Petersen, R., Gluud, L. L., Keitz, S. A., Letelier, L. M., Ravaud, P., Schulz, K. F., Siemieniuk, R. A. C., Zeraatkar, D., & Guyatt, G. H.
- Affiliations
- McMaster University (CA)
- Capital Medical University (CN)
- University of Alberta (CA)
- University of Zurich (CH)
- University Hospital of Basel (CH)
- Hvidovre Hospital (DK)
- Lahey Hospital and Medical Center (US)
- Inserm (FR)
- University of North Carolina at Chapel Hill (US)
- Corresponding author
- Wang, Ying
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2024-01
- Volume
- 165
- Pages
- 111211
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W4388461776
- Cited by
- 58
- References
- 58
- Citations by year
- 2026: 16 · 2025: 27 · 2024: 15
- FWCI
- 4.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- HAL (submittedVersion)
- Library Osnabrück University (Hochschule Osnabrück) (submittedVersion)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- meta-epidemiology, random sequence generation, allocation concealment, blinding, early trial termination, missing outcome data, selective reporting, intention-to-treat analysis, baseline imbalance, compliance, quality of evidence, GRADE
- MeSH terms
- Humans, Random Allocation, Meta-Analysis as Topic, Bias, Epidemiologic Studies, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
