Practicalities of using a modified version of the Cochrane Collaboration risk of bias tool for randomised and non‐randomised study designs applied in a health technology assessment setting
Open accessAbstract
We describe our experience of using a modified version of the Cochrane risk of bias (RoB) tool for randomised and non‐randomised comparative studies. Objectives: To assess time to complete RoB assessment To assess inter‐rater agreement To explore the association between RoB and treatment effect size Methods: Cochrane risk of bias assessment was performed on a sample of full text primary reports included in a systematic review comparing operative techniques for radical prostatectomy. Inter‐rater agreement was assessed using the kappa statistic. Results: Twenty‐four studies were judged as high overall RoB, 13 were judged as low RoB and 11 were unclear. The weighted Kappa value was 0.35 indicating fair agreement. The median (range) time taken to rate each study was 30 min (10–49). The effect estimate for all studies was 0.61 (95% credible interval (CrI) 0.46–0.83) and 0.73 (95% CrI 0.29–1.75) for low risk studies. Conclusions: Although the process was time consuming, using a modified version of the RoB tool proved useful for demonstrating conservative effect estimates. That we only achieved a fair agreement between reviewers demonstrates the urgent need for further validation to improve inter‐rater agreement. We suggest additional RoB levels could improve inter‐rater reliability. © 2013 Crown copyright.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RoB assessment has long been recognized as labor intensive, variably reproducible, and prone to misapplication in specific domains such as selective reporting.
“However, RoB assessment has long been recognized as labor intensive, variably reproducible, and prone to misapplication in specific domains such as selective reporting [14-16].”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [14]
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · see citing works
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · see citing works
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538 · see citing works
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · see citing works
Risk of bias versus quality assessment of randomised controlled trials: cross sectional study
Hartling, L., Ospina, M., Liang, Y., Dryden, D. M., Hooton, N., Krebs Seida, J., & Klassen, T. P. (2009) · BMJ · doi:10.1136/bmj.b4012 · Cited by 348 · see citing works
Choosing between randomised and non-randomised studies: a systematic review.
Britton, McKee, M., Black, McPherson, Sanderson, C. F. B., & Bain (1998) · Health Technology Assessment · doi:10.3310/hta2130 · Cited by 330 · see citing works
Ramsay, C. R., Pickard, R. S., Robertson, C., Close, A. J., Vale, L. D., Armstrong, N., Barocas, D. A., Eden, C., Fraser, C. M., Gurung, T., et al. (2012) · Health Technology Assessment · doi:10.3310/hta16410 · Cited by 259 · see citing works
Self-citationPerioperative Complications of Laparoscopic and Robotic Assisted Laparoscopic Radical Prostatectomy
Hu, J. C., Nelson, R. A., Wilson, T. G., Kawachi, M. H., Ramin, S. A., Lau, C., & Crocitto, L. E. (2006) · The Journal of Urology · doi:10.1016/s0022-5347(05)00156-4 · Cited by 239 · see citing works
Bhayani, S. B., Pavlovich, C. P., HSU, T. H. S., Sullivan, W., & Su, L. I.-M. (2003) · Urology · doi:10.1016/s0090-4295(02)02416-0 · Cited by 149 · see citing works
Guazzoni, G. F., Cestari, A., Naspro, R. L., Riva, M., Centemero, A., Zanoni, M. L., Rigatti, L., & Rigatti, P. (2006) · European Urology · doi:10.1016/j.eururo.2006.02.051 · Cited by 134 · see citing works
Is Laparoscopic Radical Prostatectomy Better Than Traditional Retropubic Radical Prostatectomy?
Artibani, W., Grosso, G., Novara, G., Pecoraro, G., Sidoti, O., Sarti, A., & Ficarra, V. (2003) · European Urology · doi:10.1016/s0302-2838(03)00315-4 · Cited by 86 · see citing works
Hartling, L., Bond, K., Vandermeer, B., Seida, J. C., Dryden, D. M., & Rowe, B. H. (2011) · PLoS ONE · doi:10.1371/journal.pone.0017242 · Cited by 79 · see citing works
Touijer, K., Kuroiwa, K., Eastham, J. A., Vickers, A. J., Reuter, V. E., Scardino, P. T., & Guillonneau, B. D. (2006) · European Urology Supplements · doi:10.1016/s1569-9056(06)61203-7 · Cited by 49 · see citing works
Touijer, K. A., Kuroiwa, K., Eastham, J. A., Vickers, A. J., Reuter, V. E., Scardino, P. T., & Guillonneau, B. D. (2006) · European Urology · doi:10.1016/j.eururo.2006.12.014 · Cited by 40 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400 · see citing works
Effect of treatments for depression on quality of life: a meta-analysis
Hofmann, S. G., Curtiss, J., Carpenter, J. K., & Kind, S. (2017) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2017.1304445 · Cited by 189 · see citing works
Robson, R. C., Pham, B., Hwee, J., Thomas, S. M., Rios, P., Page, M. J., & Tricco, A. C. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.10.003 · Cited by 86 · see citing works
Long-term outcomes of total elbow arthroplasty: a systematic review of studies at 10-year follow-up
Davey, M. S., Hurley, E. T., Gaafar, M., Molony, D. C., Mullett, H., & Pauzenberger, L. (2021) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2020.11.014 · Cited by 57 · see citing works
Virtual Fracture Clinics in Orthopaedic Surgery - A Systematic Review of Current Evidence
Davey, M. S., Coveney, E., Rowan, F. E., Cassidy, J. T., & Cleary, M. S. (2020) · Injury · doi:10.1016/j.injury.2020.11.001 · Cited by 41 · see citing works
Davey, M. S., Davey, M., Hurley, E. T., Galbraith, J. G., Molony, D. C., Mullett, H., & Pauzenberger, L. (2021) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2021.03.142 · Cited by 41 · see citing works
Improving the conduct of systematic reviews: a process mining perspective
Pham, B., Ebrahim Bagheri, A., Rios, P. D. L., Pourmasoumi, A., Robson, R. C., Hwee, J., Isaranuwatchai, W., Darvesh, N., Page, M. J., & Tricco, A. C. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.06.011 · Cited by 40 · see citing works
Ge, D. (2018) · Advances in Skin & Wound Care · doi:10.1097/01.asw.0000542530.71686.5c · Cited by 39 · see citing works
Gao, B., Dwivedi, S., DeFroda, S. F., Bokshan, S. L., Ready, L. V., Cole, B. J., & Owens, B. D. (2019) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2019.02.051 · Cited by 28 · see citing works
Zhu, Y., Xu, Y., Xuan, R., Huang, J., Bíró, I., Fekete, G., & Gu, Y. (2022) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2022.919059 · Cited by 25 · see citing works
Davey, M., Hurley, E., Davey, M., Fried, J. W., Hughes, A., Youm, T. J., & Mccarthy, T. (2021) · American Journal of Sports Medicine · doi:10.1177/03635465211038959 · Cited by 20 · see citing works
Xu, J., Yan, Y., Ma, L., Fu, P., & Peng, H. (2019) · International Urology and Nephrology · doi:10.1007/s11255-019-02271-6 · Cited by 17 · see citing works
Davey, M. S., Davey, M. G., Hurley, E. T., & Mullett, H. (2022) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2022.03.008 · Cited by 15 · see citing works
Xu, Y., Liang, M., Ugbolue, U. C., Fekete, G., & Gu, Y. (2022) · Frontiers in Physiology · doi:10.3389/fphys.2022.707176 · Cited by 14 · see citing works
Davey, M. S., Hurley, E. T., Kilkenny, C. J., Anakwenze, O. A., Klifto, C. S., & Mullett, H. (2022) · Shoulder & Elbow · doi:10.1177/17585732221141062 · Cited by 10 · see citing works
The genetic association between EGF A61G polymorphism (rs4444903) and risk of colorectal cancer
Zhu, Y., Chen, Z., Jiang, H., & Lu, B. (2019) · Medicine · doi:10.1097/md.0000000000014007 · Cited by 10 · see citing works
Kong, Y., Lv, T., Li, M., Zhao, L., Meng, T., Wu, S., Wei, W., Zhang, Q., Chen, S., You, H., et al. (2022) · Hepatology International · doi:10.1007/s12072-022-10369-w · Cited by 9 · see citing works
Davey, M. S., Davey, M. G., Hurley, E. T., Cassidy, J. T., Mullett, H., McInerney, N. M., & Galbraith, J. G. (2021) · Journal of Orthopaedics · doi:10.1016/j.jor.2021.05.004 · Cited by 9 · see citing works
Sharma, Y., Sumanadasa, S., Shahi, R., Woodman, R. J., Mangoni, A. A., Bihari, S., & Thompson, C. H. (2024) · Scientific Reports · doi:10.1038/s41598-024-62571-5 · Cited by 8 · see citing works
Jin, L., Zhou, J., Shao, F., & Yang, F. (2019) · BMC Nephrology · doi:10.1186/s12882-019-1388-9 · Cited by 8 · see citing works
Yang, P., & He, J. (2020) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2020/8627928 · Cited by 8 · see citing works
Birmingham hip resurfacing: a systematic review of outcomes at minimum 10-years follow-up
Davey, M. S., Mohan, K., GAVIN, E. F., Power, F. R., Curtin, W. M., Kaar, K., & Murphy, C. G. (2023) · Acta Orthopaedica Belgica · doi:10.52628/89.4.10045 · Cited by 7 · see citing works
Davey, M. S., Hurley, E. T., Gaafar, M., Galbraith, J. G., Mullett, H., & Pauzenberger, L. (2021) · Journal of ISAKOS Joint Disorders & Orthopaedic Sports Medicine · doi:10.1136/jisakos-2021-000614 · Cited by 7 · see citing works
Return to Play Following COVID-19 Infection—A Systematic Review of Current Evidence
Davey, M. S., Davey, M. G., Hurley, R. W., Hurley, E. T., & Pauzenberger, L. (2021) · Journal of Sport Rehabilitation · doi:10.1123/jsr.2021-0028 · Cited by 6 · see citing works
Sharma, Y., Sumanadasa, S., Shahi, R., & Thompson, C. H. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-082257 · Cited by 5 · see citing works
Matthes, H., & Schad, F. (2019) · Der Onkologe · doi:10.1007/s00761-019-0601-6 · Cited by 3 · see citing works
Meta-analysis of moxibustion in the management of functional dyspepsia
Han, Y., Zhang, H., Xiao-li, P. A. N., & Chen, S. (2019) · 世界针灸杂志：英文版 · doi:10.1016/j.wjam.2019.06.001 · Cited by 3 · see citing works
Greenhalgh, T., Ratnayake, S., Helm, R. K., Poliseli, L., & Williamson, J. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10072 · Cited by 2 · see citing works
Davey, M. S., Davey, M. G., Hurley, E. T., & Mullett, H. (2023) · JBJS Reviews · doi:10.2106/jbjs.rvw.22.00187 · Cited by 2 · see citing works
Yao, Y., Li, N., Li, J., Feng, J. M., Ma, J., Liao, X., & Zhang, Y. (2024) · International Journal of Nursing Practice · doi:10.1111/ijn.13302 · Cited by 1 · see citing works
O’Sullivan, D., Davey, M. S., Hurley, E. T., Dickens, J. F., & Mullett, H. (2024) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2024.10.002 · Cited by 1 · see citing works
Guo, L., Ding, Y., Liu, F., Zhu, W., & Jiang, X. (2017) · Cardiovascular Innovations and Applications · doi:10.15212/cvia.2016.0055 · Cited by 1 · see citing works
Liang, Z., Sheffield, C., Butera, G., & Antani, S. (2026) · JMIR AI · doi:10.2196/93761
Ayodhi, S., Leburu, A., Krishnamurthy, M., Kumar, N., V, & Selvendran, K. (2024) · Journal of Dental Specialities · doi:10.18231/j.jds.2024.016
Davey, M. S., Moore, D., Stanley, C., & Moore, D. P. (2025) · Journal of Orthopaedic Reports · doi:10.1016/j.jorep.2025.100849
Edwards, D. J., Clark, E., Csontos, J. K., Dobbins, M., Gillen, E., Hounsome, J. C., Neil‐Sztramko, S., Lewis, R., Mann, M. K., & Prue, G. (2026) · Research Square · doi:10.21203/rs.3.rs-8724550/v1
Edwards, D. J., Clark, E., Csontos, J. K., Dobbins, M., Gillen, E., Hounsome, J. C., Neil-Sztramko, S. E., Lewis, R., Mann, M. K., & Prue, G. (2026) · Research Square · doi:10.21203/rs.3.rs-8724550/v2
Edwards, D. J., Clark, E., Csontos, J. K., Dobbins, M., Gillen, E., Hounsome, J. C., Neil‐Sztramko, S., Lewis, R. A., Mann, M. K., & Prue, G. (2026) · Research Square · doi:10.21203/rs.3.rs-8724550/v3
- Authors
- Robertson, C., Ramsay, C., Gurung, T., Mowatt, G., Pickard, R., Sharma, P., & The UK Robotic Laparoscopic Prostatectomy HTA Study Group
- Affiliations
- University of Aberdeen (GB)
- Newcastle University (GB)
- Corresponding author
- Robertson, Clare
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2013-11-14
- Volume
- 5
- Issue
- 3
- Pages
- 200-211
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2009327399
- Cited by
- 38
- References
- 15
- Citations by year
- 2026: 5 · 2025: 1 · 2024: 5 · 2023: 2 · 2022: 5 · 2021: 6 · 2020: 2 · 2019: 6 · 2018: 4 · 2017: 2
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- Aberdeen University Research Archive (Aberdeen University) (submittedVersion)
- Newcastle University ePrints (Newcastle Univesity)
- Funders
- NIHR Health Services and Delivery Research (HS&DR) programme (09/14/02)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- risk of bias assessment, Cochrane risk of bias tool, interobserver agreement, health technology assessment, radical prostatectomy, non-randomized studies, tool validation
- MeSH terms
- Research Design, Review Literature as Topic, Sensitivity and Specificity, Software, Technology Assessment, Biomedical, Reproducibility of Results, Observer Variation, Bias, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
