Impact of Selection Bias on Treatment Effect Size Estimates in Randomized Trials of Oral Health Interventions: A Meta-epidemiological Study
Open accessAbstract
Emerging evidence suggests that design flaws of randomized controlled trials can result in over- or underestimation of the treatment effect size (ES). The objective of this study was to examine associations between treatment ES estimates and adequacy of sequence generation, allocation concealment, and baseline comparability among a sample of oral health randomized controlled trials. For our analysis, we selected all meta-analyses that included a minimum of 5 oral health randomized controlled trials and used continuous outcomes. We extracted data, in duplicate, related to items of selection bias (sequence generation, allocation concealment, and baseline comparability) in the Cochrane Risk of Bias tool. Using a 2-level meta-meta-analytic approach with a random effects model to allow for intra- and inter-meta-analysis heterogeneity, we quantified the impact of selection bias on the magnitude of ES estimates. We identified 64 meta-analyses, including 540 randomized controlled trials analyzing 137,957 patients. Sequence generation was judged to be adequate (at low risk of bias) in 32% ( n = 173) of trials, and baseline comparability was judged to be adequate in 77.8% of trials. Allocation concealment was unclear in the majority of trials ( n = 458, 84.8%). We identified significantly larger treatment ES estimates in trials that had inadequate/unknown sequence generation (difference in ES = 0.13; 95% CI: 0.01 to 0.25) and inadequate/unknown allocation concealment (difference in ES = 0.15; 95% CI: 0.02 to 0.27). In contrast, baseline imbalance (difference in ES = 0.01, 95% CI: –0.09 to 0.12) was not associated with inflated or underestimated ES. In conclusion, treatment ES estimates were 0.13 and 0.15 larger in trials with inadequate/unknown sequence generation and inadequate/unknown allocation concealment, respectively. Therefore, authors of systematic reviews using oral health randomized controlled trials should perform sensitivity analyses based on the adequacy of sequence generation and allocation concealment.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Inadequate or unclear allocation concealment in trial publications has been associated with inflated effect size estimates.
“Similar to failure of random sequence generation, inadequate or unclear allocation concealment in trial publications has been associated with inflated effect size estimates.325 327 328 336 349 350”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [328]
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582 · see citing works
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · see citing works
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560 · see citing works
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538 · see citing works
Random-effects model for meta-analysis of clinical trials: An update
DerSimonian, R., & Kacker, R. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.04.004 · Cited by 2,374 · see citing works
Gluud, L. L., Villumsen, J., & Gluud, C. N. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,262 · see citing works
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · see citing works
Savović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049 · see citing works
Balk, E. M. (2002) · JAMA · doi:10.1001/jama.287.22.2973 · Cited by 478 · see citing works
Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal, J., Hrobjartsson, A., Jorgensen, K., Hilden, J., Altman, D., & Gotzsche, P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym087 · Cited by 472 · see citing works
Sterne, J. A. C., Jüni, P., Schulz, K. F., Altman, D. G., Bartlett, C., & Egger, M. (2002) · Statistics in Medicine · doi:10.1002/sim.1184 · Cited by 382 · see citing works
de Boer, M. R., Waterlander, W. E., Kuijper, L. D., Steenhuis, I. H., & Twisk, J. W. (2015) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-015-0162-z · Cited by 370 · see citing works
Moher, D., Cook, Jadad, Tugwell, P., Moher, Jones, Pham, & Klassen (1999) · Health Technology Assessment · doi:10.3310/hta3120 · Cited by 364 · see citing works
The Concepts of Power Analysis
Cohen, J. (1977) · Statistical Power Analysis for the Behavioral Sciences · doi:10.1016/b978-0-12-179060-8.50006-2 · Cited by 299 · see citing works
Pildal, J., Chan, A.-W., Hróbjartsson, A., Forfang, E., Altman, D. G., & Gøtzsche, P. C. (2005) · BMJ · doi:10.1136/bmj.38414.422650.8f · Cited by 209 · see citing works
Impact of Study Quality on Outcome in Placebo-Controlled Trials of Homeopathy
Linde, K., Scholz, M., Ramı́rez, G., Clausius, N., Melchart, D., & Jonas, W. B. (1999) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00048-7 · Cited by 200 · see citing works
Detecting Selection Bias in Randomized Clinical Trials
Berger, V. W., & Exner, D. V. (1999) · Controlled Clinical Trials · doi:10.1016/s0197-2456(99)00014-8 · Cited by 163 · see citing works
Nüesch, E., Reichenbach, S., Trelle, S., Rutjes, A. W., Liewald, K., Sterchi, R., Altman, D. G., & Jüni, P. (2009) · Arthritis Care & Research · doi:10.1002/art.24894 · Cited by 150 · see citing works
van Tulder, M. W., Suttorp, M. J., Morton, S. C., Bouter, L. M., & Shekelle, P. (2009) · Spine · doi:10.1097/brs.0b013e3181ab6a78 · Cited by 132 · see citing works
Mhaskar, R., Djulbegovic, B., Magazin, A., Soares, H. P., & Kumar, A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.10.016 · Cited by 102 · see citing works
Watzlawick, R., Sena, E. S., Dirnagl, U., Brommer, B., Kopp, M. A., Macleod, M. R., Howells, D. W., & Schwab, J. M. (2013) · JAMA Neurology · doi:10.1001/jamaneurol.2013.4684 · Cited by 97 · see citing works
Armijo-Olivo, S. L., Saltaji, H., da Costa, B. R., Fuentes, J., Ha, C., & Cummings, G. G. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008562 · Cited by 87 · see citing works
Self-citationSiersma, V., Als-Nielsen, B. E. T., Chen, W., Hilden, J., Gluud, L. L., & Gluud, C. N. (2006) · Statistics in Medicine · doi:10.1002/sim.2752 · Cited by 83 · see citing works
Schulz, K. F. (1998) · Clinical Obstetrics & Gynecology · doi:10.1097/00003081-199806000-00005 · Cited by 82 · see citing works
Quantifying Bias in Randomized Controlled Trials in Child Health: A Meta-Epidemiological Study
Hartling, L., Hamm, M. P., Fernandes, R., Dryden, D., & Vandermeer, B. (2014) · PLoS ONE · doi:10.1371/journal.pone.0088008 · Cited by 39 · see citing works
A Descriptive Analysis of Oral Health Systematic Reviews Published 1991–2012: Cross Sectional Study
Saltaji, H., Cummings, G. G., Armijo-Olivo, S. L., Major, M. P., Amin, M. S., Major, P. W., Hartling, L., & Flores‐Mir, C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0074545 · Cited by 38 · see citing works
Self-citationThe effect of bias on the magnitude of clinical outcomes in periodontology: a pilot study
Fenwick, J., Needleman, I. G., & Moles, D. R. (2008) · Journal Of Clinical Periodontology · doi:10.1111/j.1600-051x.2008.01291.x · Cited by 31 · see citing works
Assessing baseline comparability in randomised trials
Elkins, M. R. (2015) · Journal of physiotherapy · doi:10.1016/j.jphys.2015.07.005 · Cited by 29 · see citing works
Design and Validity of Randomized Controlled Dental Restorative Trials
Göstemeyer, G., Blunck, U., Paris, S., & Schwendicke, F. (2016) · Materials · doi:10.3390/ma9050372 · Cited by 23 · see citing works
Faggion, C. M., Wu, Y., Scheidgen, M., & Tu, Y. (2015) · PLoS ONE · doi:10.1371/journal.pone.0139030 · Cited by 14 · see citing works
Saltaji, H., Ospina, M. B., Armijo-Olivo, S. L., Agarwal, S., Cummings, G. G., Amin, M. S., & Flores‐Mir, C. (2016) · The Journal of the American Dental Association · doi:10.1016/j.adaj.2016.03.017 · Cited by 14 · see citing works
Self-citationPandis, N. (2012) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2012.06.011 · Cited by 13 · see citing works
Saltaji, H., Armijo-Olivo, S. L., Cummings, G. G., Amin, M. S., & Flores‐Mir, C. (2014) · BMJ Open · doi:10.1136/bmjopen-2013-004527 · Cited by 9 · see citing works
Self-citationSelection-related bias, an ongoing concern in doing and publishing research
Knottnerus, J. A., & Tugwell, P. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.08.003 · Cited by 8 · see citing works
Baseline comparisons in clinical trials
Altman, D. (1990) · The Lancet · doi:10.1016/0140-6736(90)91515-c · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081660 · Cited by 171 · see citing works
Wang, Y., Parpia, S., Couban, R., Wang, Q., Armijo-Olivo, S., Bassler, D., Briel, M., Brignardello-Petersen, R., Gluud, L. L., Keitz, S. A., et al. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.11.001 · Cited by 58 · see citing works
Self-citationSaltaji, H., Armijo-Olivo, S., Cummings, G. G., Amin, M., & Flores-Mir, C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0190089 · Cited by 43 · see citing works
- Authors
- Saltaji, H., Armijo-Olivo, S., Cummings, G. G., Amin, M., da Costa, B. R., & Flores-Mir, C.
- Affiliations
- University of Alberta (CA)
- Department of Physical Therapy, Florida International University, Miami, Florida, USA, and Institute of Primary Health Care, University of Bern, Bern, Switzerland
- Orthodontic Graduate Program, School of Dentistry, University of Alberta, Edmonton, Canada
- Corresponding author
- Saltaji, H.
- Venue
- Journal of Dental Research
- Publisher
- SAGE Publications
- Published
- 2017-08-16
- Volume
- 97
- Issue
- 1
- Pages
- 5-13
- ISSN
- 0022-0345 (ISSN-L)
- 1544-0591
- OpenAlex
- W2749388036
- Cited by
- 40
- References
- 38
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 4 · 2023: 6 · 2022: 7 · 2021: 6 · 2020: 4 · 2019: 4 · 2018: 3 · 2017: 1
- FWCI
- 0.81
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Journal of Dental Research (publishedVersion)
- PubMed (publishedVersion)
- Bern Open Repository and Information System (University of Bern) (submittedVersion)
- Open Access CRIS of the University of Bern
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- selection bias, treatment effect size, oral health interventions, randomized controlled trials, sequence generation, allocation concealment, risk of bias, meta-epidemiology
- MeSH terms
- Humans, Selection Bias, Randomized Controlled Trials as Topic, Treatment Outcome, Sample Size, Dental Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
