Disagreements in risk of bias assessment for randomised controlled trials included in more than one Cochrane systematic reviews: a research on research study using cross-sectional design
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Objectives: Assess the frequency and reasons for disagreements in risk of bias assessments for randomised controlled trials (RCTs) included in more than one Cochrane review. Design: Research on research study, using cross-sectional design. Data sources: 2796 Cochrane reviews published between March 2011 and September 2014. Data selection: RCTs included in more than one review. Data extraction: Risk of bias assessment and support for judgement for five key risk of bias items. Data synthesis: For each item, we compared risk of bias assessment made in each review and calculated proportion of agreement. Two reviewers independently analysed 50% of all disagreements by comparing support for each judgement with information from study report to evaluate whether disagreements were related to a difference in information (eg, contact the study author) or a difference in interpretation (same support for judgement but different interpretation). They also identified main reasons for different interpretation. Results: 1604 RCTs were included in more than one review. Proportion of agreement ranged from 57% (770/1348 trials) for incomplete outcome data to 81% for random sequence generation (1193/1466). Most common source of disagreement was difference in interpretation of the same information, ranging from 65% (88/136) for random sequence generation to 90% (56/62) for blinding of participants and personnel. Access to different information explained 32/136 (24%) disagreements for random sequence generation and 38/205 (19%) for allocation concealment. Disagreements related to difference in interpretation were frequently related to incomplete or unclear reporting in the study report (83% of disagreements related to different interpretation for random sequence generation). Conclusions: Risk of bias judgements of RCTs included in more than one Cochrane review differed substantially. Most disagreements were related to a difference in interpretation of an incomplete or unclear description in the study report. A clearer guidance on common causes of incomplete information may improve agreement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Systematic Reviews: Synthesis of Best Evidence for Clinical Decisions
Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-126-5-199703010-00006 · Cited by 2,184
Armijo‐Olivo, S., Stiles, C. R., Hagen, N. A., Biondo, P. D., & Cummings, G. G. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2010.01516.x · Cited by 1,539
Turner, L., Shamseer, L., Altman, D. G., Weeks, L., Peters, J., Kober, T., Dias, S., Schulz, K. F., Plint, A. C., & Moher, D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000030.pub2 · Cited by 740
Turner, L., Shamseer, L., Altman, D. G., Schulz, K. F., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-60 · Cited by 628
Balk, E. M. (2002) · JAMA · doi:10.1001/jama.287.22.2973 · Cited by 478
Jørgensen, L., Paludan-Müller, A. S., Laursen, D. R. T., Savović, J., Boutron, I., Sterne, J. A. C., Higgins, J. P. T., & Hróbjartsson, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0259-8 · Cited by 396
Risk of bias versus quality assessment of randomised controlled trials: cross sectional study
Hartling, L., Ospina, M., Liang, Y., Dryden, D. M., Hooton, N., Krebs Seida, J., & Klassen, T. P. (2009) · BMJ · doi:10.1136/bmj.b4012 · Cited by 348 · cited 1×
“We evaluated a very large and comprehensive collection of Cochrane reviews that spanned multiple specialties and topics, including a number of trials about 10 times larger than the largest study on the topic.(12) We completed our analysis by searching individual study reports to give support to our comments on reasons…”
Savović, J., Weeks, L., Sterne, J. A., Turner, L., Altman, D. G., Moher, D., & Higgins, J. P. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-37 · Cited by 340 · cited 2×
“Comparison with other studies Our findings confirm the importance of issues that were previously identified by Jørgensen et al3 and Savović et al.26 In particular, Savović et al,26 surveying users of the risk of bias tool, reported on the possibility of confusion between random sequence generation and allocation 7Bert…”
ResultPage, M. J., Higgins, J. P. T., Clayton, G., Sterne, J. A. C., Hróbjartsson, A., & Savović, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0159267 · Cited by 284
Hartling, L., Hamm, M. P., Milne, A., Vandermeer, B., Santaguida, P. L., Ansari, M., Tsertsvadze, A., Hempel, S., Shekelle, P., & Dryden, D. M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.07.005 · Cited by 173
Dechartres, A., Trinquart, L., Atal, I., Moher, D., Dickersin, K., Boutron, I., Perrodeau, E., Altman, D. G., & Ravaud, P. (2017) · BMJ · doi:10.1136/bmj.j2490 · Cited by 171 · cited 1×
“This standardisation was done for a previous publication.(23)”
MethodSelf-citationShamseer, L., Hopewell, S., Altman, D. G., Moher, D., & Schulz, K. F. (2016) · Trials · doi:10.1186/s13063-016-1408-z · Cited by 168
Armijo-Olivo, S., Ospina, M., da Costa, B. R., Egger, M., Saltaji, H., Fuentes, J., Ha, C., & Cummings, G. G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0096920 · Cited by 124
Empirical evaluation of which trial characteristics are associated with treatment effect estimates
Dechartres, A., Trinquart, L., Faber, T., & Ravaud, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.005 · Cited by 85
Self-citationHopewell, S., Boutron, I., Altman, D. G., & Ravaud, P. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003342 · Cited by 66
Self-citationda Costa, B. R., Beckett, B., Diaz, A., Resta, N. M., Johnston, B. C., Egger, M., Jüni, P., & Armijo-Olivo, S. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0441-7 · Cited by 62
Research works printed but not linked: 12
- Hrobjartsson, 2013, Assessing risk of bias in randomised clinical trials included in Cochrane Reviews: the why is easy, the how is a challenge, Cochrane Database Syst Rev, 4, ED000058
- 10.1136/bmj.d5928 · DOI · Paper page
- 10.1001/jama.1995.03520290060030 · DOI
- 10.1016/S0140-6736(98)01085-X · DOI · Paper page
- Higgins JPT , Altman DG , Sterne JAC . Chapter 8: Assessing risk of bias in included studies. In: Higgins JP , Green S , eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 (updated March 2011) ed: The Cochrane Collaboration, 2011. www.handbook.cochrane.org.
- Hartling, 2011, Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma, PLoS One, 6, 10.1371/journal.pone.0017242 · DOI · Paper page
- Jordan, 2017, There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review, J Clin Epidemiol, 81, 72, 10.1016/j.jclinepi.2016.08.012 · DOI
- Wilkins, 2017, Risk of bias in assessing Risk of Bias, Ophthalmic Physiol Opt, 37, 107, 10.1111/opo.12333 · DOI
- Review Manager (RevMan) [Computer program] [program]. Version 5.3 version. Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration, 2014.
- deMelo VV . Conference: Advances in Logic Based Intelligent Systems, 2005.
- Propadalo I , Tranfic M , Vuka I , et al . In Cochrane reviews, risk of bias assessments for allocation concealment were frequently not in line with Cochrane’s Handbook guidance. J Clin Epidemiol 2019;106.doi:10.1016/j.jclinepi.2018.10.002 · DOI · Paper page
- 10.1002/sim.5402 · DOI · Paper page
Cited by
Risk of bias assessments for selective reporting were inadequate in the majority of Cochrane reviews
Saric, F., Barcot, O., & Puljak, L. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.04.007 · Cited by 48
Automation of Systematic Reviews with Large Language Models
Cao, C., Arora, R., Cento, P., Budak, A., Manta, K., Farahani, E., Cecere, M., Selemon, A., Sang, J., Gong, L. X., et al. (2025) · medRxiv · doi:10.1101/2025.06.13.25329541 · Cited by 33
Novel AI applications in systematic review: gPT-4 assisted data extraction, analysis, review of bias
Kim, J. K., Chua, M. E., Li, T. G., Rickard, M., & Lorenzo, A. J. (2025) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2024-113066 · Cited by 11
Rose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Berg, R. C., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Potrebny, T. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02631-0 · Cited by 5
“Human–human agreement in RoB assessment has been estimated to range between 16 and 81% [6, 27, 28].”
Metadata
- Authors
- Bertizzolo, L., Bossuyt, P., Atal, I., Ravaud, P., & Dechartres, A.
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2019-04
- Volume
- 9
- Issue
- 4
- Pages
- e028382
- ISSN
- 2044-6055 (ISSN-L)
- OpenAlex
- W2928947675
- Cited by
- 35
- References
- 16
- FWCI
- 0.61
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.