The DOI matches its Crossref record; 27 of the 28 references it cites were found in a registry or an index.
Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study
Open accessAbstract
Flaws in the design of randomized trials may bias intervention effect estimates and increase between-trial heterogeneity. Empirical evidence suggests that these problems are greatest for subjectively assessed outcomes. For the Risk of Bias in Evidence Synthesis (ROBES) Study, we extracted risk-of-bias judgements (for sequence generation, allocation concealment, blinding, and incomplete data) from a large collection of meta-analyses published in the Cochrane Library (issue 4; April 2011). We categorized outcome measures as mortality, other objective outcome, or subjective outcome, and we estimated associations of bias judgements with intervention effect estimates using Bayesian hierarchical models. Among 2,443 randomized trials in 228 meta-analyses, intervention effect estimates were, on average, exaggerated in trials with high or unclear (versus low) risk-of-bias judgements for sequence generation (ratio of odds ratios (ROR) = 0.91, 95% credible interval (CrI): 0.86, 0.98), allocation concealment (ROR = 0.92, 95% CrI: 0.86, 0.98), and blinding (ROR = 0.87, 95% CrI: 0.80, 0.93). In contrast to previous work, we did not observe consistently different bias for subjective outcomes compared with mortality. However, we found an increase in between-trial heterogeneity associated with lack of blinding in meta-analyses with subjective outcomes. Inconsistency in criteria for risk-of-bias judgements applied by individual reviewers is a likely limitation of routinely collected bias assessments. Inadequate randomization and lack of blinding may lead to exaggeration of intervention effect estimates in randomized trials.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The primary limitation of all the available evidence on the effects of antidepressants is that the majority of previous trials assessing the effects of antidepressants have been at high or unclear risk of bias.”
From Should antidepressants be used for major depressive disorder? (2020) · cites this paper as [33]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Trials at high or unclear risk of bias tend to overestimate benefits and underestimate harms.”
From Should antidepressants be used for major depressive disorder? (2020) · cites this paper as [33]
Experimental — accuracy not yet measuredIn the citing paper
“Similar to failure of random sequence generation, inadequate or unclear allocation concealment in trial publications has been associated with inflated effect size estimates.325 327 328 336 349 350”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [327]
In the citing paper
“so there is a great risk that the results of the review overestimate the beneficial effects and underestimate the harmful effects of SSRIs.”
From Should antidepressants be used for major depressive disorder? (2020) · cites this paper as [33]
In the citing paper
“so there is a great risk that the results of the review overestimate the beneficial effects and underestimate the harmful effects of antidepressants.”
From Should antidepressants be used for major depressive disorder? (2020) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationCochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396
Self-citationKjaergard, L. L., Villumsen, J., & Gluud, C. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,283
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181
Self-citationSavović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049
Self-citationTurner, L., Shamseer, L., Altman, D. G., Weeks, L., Peters, J., Kober, T., Dias, S., Schulz, K. F., Plint, A. C., & Moher, D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000030.pub2 · Cited by 740
Research works printed but not linked: 1
- Higgins, 2016, Cochrane Methods, 29
Cited by
Family Caregiving for Older Adults
Schulz, R., Beach, S. R., Czaja, S. J., Martire, L. M., & Monin, J. K. (2020) · Annual Review of Psychology · doi:10.1146/annurev-psych-010419-050754 · Cited by 754
Dunning, D. L., Griffiths, K., Kuyken, W., Crane, C., Foulkes, L., Parker, J., & Dalgleish, T. (2019) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12980 · Cited by 610
Nature-based outdoor activities for mental and physical health: Systematic review and meta-analysis
Coventry, P. A., Brown, J. E., Pervin, J., Brabyn, S., Pateman, R., Breedvelt, J., Gilbody, S., Stancliffe, R., McEachan, R., & White, P. L. (2021) · SSM - Population Health · doi:10.1016/j.ssmph.2021.100934 · Cited by 453
Rubinstein, S. M., de Zoete, A., van Middelkoop, M., Assendelft, W. J. J., de Boer, M. R., & van Tulder, M. W. (2019) · BMJ · doi:10.1136/bmj.l689 · Cited by 381
Moustgaard, H., Clayton, G. L., Jones, H. E., Boutron, I., Jørgensen, L., Laursen, D. R. T., Olsen, M. F., Paludan-Müller, A., Ravaud, P., Savović, J., et al. (2020) · BMJ · doi:10.1136/bmj.l6802 · Cited by 300
Self-citation
Metadata
- Authors
- Savović, J., Turner, R. M., Mawdsley, D., Jones, H. E., Beynon, R., Higgins, J. P. T., & Sterne, J. A. C.
- Affiliations
- National Institute for Health and Care Research (GB)
- MRC Clinical Trials Unit at UCL (GB)
- University of Bristol (GB)
- Corresponding author
- Savović, Jelena
- ORCID
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-10-19
- Volume
- 187
- Issue
- 5
- Pages
- 1113-1122
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- PMC
- PMC5928453
- OpenAlex
- W2767350488
- Cited by
- 382
- References
- 28
- Citations by year
- 2026: 25 · 2025: 33 · 2024: 40 · 2023: 50 · 2022: 46 · 2021: 53 · 2020: 55 · 2019: 47 · 2018: 29 · 2017: 4
- FWCI
- 33.01
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- American Journal of Epidemiology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Apollo (cc-by)
- Funders
- Medical Research Council (MRC) (G0701659/1)
- MRC Methodology Research Panel (MR/K014587/1)
- NIHR CLAHRC West
- MRC (MC-U105260558, MC-UU12023/24)
- National Institute for Health Research (NF-SI-0611-10168)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- risk of bias, meta-epidemiology, robe, allocation concealment, blinding, sequence generation, missing data, between-trial heterogeneity, subjective outcomes, mortality, meta-analysis bias
- MeSH terms
- Bayes Theorem, Humans, Meta-Analysis as Topic, Bias, Odds Ratio, Epidemiologic Studies, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study. https://app.cochat.ai/papers/10.1093/aje/kwx344
- The paper
- Savović, J., Turner, R. M., Mawdsley, D., Jones, H. E., Beynon, R., Higgins, J. P. T., & Sterne, J. A. C. (2018). Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study. American Journal of Epidemiology. https://doi.org/10.1093/aje/kwx344
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
