Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials
Open accessThe DOI matches its Crossref record; 60 of the 61 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers. METHODS: Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (κ). RESULTS: In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (κ = 0.02, 95%CI: -0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median κ = 0.19), ranging from κ = -0.04 ("Other bias") to κ = 0.62 ("Sequence generation"). There was also no agreement (κ = -0.29, 95%CI: -0.81, 0.35]) in the overall RoB assessment at the meta-analysis level. CONCLUSIONS: Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, RoB assessment has long been recognized as labor intensive, variably reproducible, and prone to misapplication in specific domains such as selective reporting [14-16].”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [15]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“however, we developed specific decision rules to make decisions as described elsewhere.43”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [43]
In the citing paper
“Risk of bias assessment requires judgement and numerous studies have reported low inter-rater reliability of the RoB1 tool [10, 11].”
From Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system (2022) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 1×
“Statistical analysis Inter-rater reliability of RoB assessments between Cochrane and blinded external reviewers [18–20] and within the panel of external reviewers was assessed using weighted kappa (k) for categorical data.”
MethodThe Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · cited 1×
“Statistical analysis Inter-rater reliability of RoB assessments between Cochrane and blinded external reviewers [18–20] and within the panel of external reviewers was assessed using weighted kappa (k) for categorical data.”
MethodSchulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · cited 6×
“Some studies, for example, have found that inadequate allocation concealment or lack of double-blinding can overestimate treatment effects on average by 18% and 9%, respectively [5,40,42].”
BackgroundMethodInfluential(2008) · Cited by 3,790
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · cited 7×
“The Cochrane Collaboration developed the Risk of Bias tool (RoB) as a method to assess risk of bias based on study design and conduct rather than relying on general reporting issues of trial characteristics [3].”
BackgroundMethodInfluentialWood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538 · cited 4×
“For example, authors of meta-epidemiological studies [8,39,40], have taken information from external sources (Cochrane assessments, or information provided by authors of reviews).”
BackgroundResultInfluentialSelf-citation2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group
Furlan, A. D., Pennick, V., Bombardier, C., & van Tulder, M. (2009) · Spine · doi:10.1097/brs.0b013e3181b1c99f · Cited by 1,663
Cited by
Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2017) · doi:10.23970/ahrqepcmethguide2 · Cited by 448
Armijo-Olivo, S., Pitance, L., Singh, V., Neto, F., Thie, N., & Michelotti, A. (2016) · Physical Therapy · doi:10.2522/ptj.20140548 · Cited by 526
Self-citationJørgensen, L., Paludan-Müller, A. S., Laursen, D. R. T., Savović, J., Boutron, I., Sterne, J. A. C., Higgins, J. P. T., & Hróbjartsson, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0259-8 · Cited by 396
“While there is evidence that some factors are empirically associated with effect estimates, such as single versus multicentre design, early stopping of trials and funding source [14-16], the extent to which these should be considered alongside the main bias domains is still a topic of debate.”
BackgroundSavović, J., Turner, R. M., Mawdsley, D., Jones, H. E., Beynon, R., Higgins, J. P. T., & Sterne, J. A. C. (2018) · American Journal of Epidemiology · doi:10.1093/aje/kwx344 · Cited by 382
Schwingshackl, L., Knüppel, S., Schwedhelm, C., Hoffmann, G., Missbach, B., Stelmach-Mardas, M., Dietrich, S., Eichelmann, F., Kontopanteils, E., Iqbal, K., et al. (2016) · Advances in Nutrition · doi:10.3945/an.116.013052 · Cited by 360
Fox, K. R., Huang, X., Guzmán, E. M., Funsch, K. M., Cha, C. B., Ribeiro, J. D., & Franklin, J. C. (2020) · Psychological Bulletin · doi:10.1037/bul0000305 · Cited by 351
“KMF completed the ratings based on the protocol, with 20% of the studies also independently coded by XH. Interrater agreement ( 0.72) was moderate and consistent with previous studies using the same tool (e.g., Armijo-Olivo et al., 2014; Hartling et al., 2009).”
ResultInfluentialPage, M. J., McKenzie, J. E., & Higgins, J. P. T. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019703 · Cited by 278
Classical psychedelics for the treatment of depression and anxiety: A systematic review
Muttoni, S., Ardissino, M., & John, C. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.07.076 · Cited by 256
Wakefield, S., Kellett, S., Simmonds‐Buckley, M., Stockton, D., Bradbury, A., & Delgadillo, J. (2021) · British Journal of Clinical Psychology · doi:10.1111/bjc.12259 · Cited by 222
Recommendations for assessing the risk of bias in systematic reviews of health-care interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.004 · Cited by 221
Kennedy, C. E., Fonner, V. A., Armstrong, K. A., Denison, J. A., Yeh, P. T., O’Reilly, K. R., & Sweat, M. D. (2019) · Systematic Reviews · doi:10.1186/s13643-018-0925-0 · Cited by 212
Populations and Interventions for Palliative and End-of-Life Care: A Systematic Review
Singer, A. E., Goebel, J. R., Kim, Y. S., Dy, S. M., Ahluwalia, S. C., Clifford, M., Dzeng, E., O'Hanlon, C. E., Motala, A., Walling, A. M., et al. (2016) · Journal of Palliative Medicine · doi:10.1089/jpm.2015.0367 · Cited by 183
Moseley, A. M., Rahman, P., Wells, G. A., Zadro, J. R., Sherrington, C., Toupin-April, K., & Brosseau, L. (2019) · PLOS ONE · doi:10.1371/journal.pone.0222770 · Cited by 174
“With the exception of incomplete outcome data and blinding of participants and personnel, these levels of agreement were the same or better than the agreement observed between pairs of reviewers reported in the literature [6, 7, 33] and our sample size (n = 74) was larger than other studies [6, 33].”
ResultYamato, T. P., Maher, C., Koes, B., & Moseley, A. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.03.002 · Cited by 175
PEDro or Cochrane to Assess the Quality of Clinical Trials? A Meta-Epidemiological Study
Armijo-Olivo, S., da Costa, B. R., Cummings, G. G., Ha, C., Fuentes, J., Saltaji, H., & Egger, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0132634 · Cited by 171
“As described in detail elsewhere, the PEDro and Cochrane training assessments were discussed in a group meeting to determine consistency in ratings, and calibrate assessments [21].”
Self-citationCochrane’s risk of bias tool in the context of psychotherapy outcome research
Munder, T., & Barth, J. (2018) · Psychotherapy Research · doi:10.1080/10503307.2017.1411628 · Cited by 174
“In general, the agreement between RoB coders was found to be moderate to low.3 Meta-epidemiological studies with different raters (Armijo-Olivo et al., 2014; Jordan, Lensen, & Farquhar, 2017) showed the best kappa values (>.70) for rating sequence generation.”
BackgroundIs psychotherapy effective? A re-analysis of treatments for depression
Munder, T., Flückiger, C., Leichsenring, F., Abbass, A. A., Hilsenroth, M. J., Luyten, P., Rabung, S., Steinert, C., & Wampold, B. E. (2019) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796018000355 · Cited by 131
Armijo-Olivo, S., Fuentes, J., da Costa, B. R., Saltaji, H., Ha, C., & Cummings, G. G. (2017) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000000521 · Cited by 160
“Details of the search strategy can be found elsewhere.(25) Metaanalyses and their RCTs were included if (1) the MA included at least 3 RCTs comparing at least 2 interventions, with at least one of the interventions being currently or potentially part of PT practice according to the World Confederation for Physical The…”
Self-citationNeri, S. G., Cardoso, J. R., Cruz, L., Lima, R. M., de Oliveira, R. J., Iversen, M. D., & Carregaro, R. L. (2017) · Clinical Rehabilitation · doi:10.1177/0269215517694677 · Cited by 149
Yoga for Multiple Sclerosis: A Systematic Review and Meta-Analysis
Cramer, H., Lauche, R., Azizi, H., Dobos, G., & Langhorst, J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112414 · Cited by 121
Brosseau, L., Taki, J., Desjardins, B., Thevenot, O., Fransen, M., Wells, G. A., Imoto, A. M., Toupin‐April, K., Westby, M. D., Álvarez‐Gallardo, I. C., et al. (2017) · Clinical Rehabilitation · doi:10.1177/0269215517691083 · Cited by 146
The Effects of Mind-Body Interventions on Sleep Quality: A Systematic Review
Neuendorf, R., Wahbeh, H., Chamine, I., Yu, J., Hutchison, K., & Oken, B. S. (2015) · Evidence-Based Complementary and Alternative Medicine · doi:10.1155/2015/902708 · Cited by 116
“Additionally, the Cochrane ROB has its own limited reliability [132].”
BackgroundYang, X., Wu, J., Ma, Y., Yu, J., Cao, H., Zeng, A., Fu, R., Tang, Y., & Ren, Z. (2025) · Journal of Medical Internet Research · doi:10.2196/60845 · Cited by 123
“Two (XY and JW) researchers independently assessed each study, resolving disagreements through discussion or consulting a third researcher [34].”
Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M. (2021) · PLOS Biology · doi:10.1371/journal.pbio.3001162 · Cited by 103
“Experts’ assessments of trials show that labeling the same trials for different Cochrane reviews resulted in substantial differences [30,31].”
BackgroundHow to avoid bias in systematic reviews of observational studies
de Almeida, C. P. B., & de Goulart, B. N. G. (2017) · Revista CEFAC · doi:10.1590/1982-021620171941117 · Cited by 112
Effects of yoga on chronic neck pain: a systematic review and meta-analysis
Cramer, H., Klose, P., Brinkhaus, B., Michalsen, A., & Dobos, G. (2017) · Clinical Rehabilitation · doi:10.1177/0269215517698735 · Cited by 81
Saltaji, H., Armijo-Olivo, S., Cummings, G. G., Amin, M., da Costa, B. R., & Flores-Mir, C. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0491-0 · Cited by 85
“A similar reviewer training process was conducted in other studies performed by the same research team [29, 30].”
ResultSelf-citationRobson, R. C., Pham, B., Hwee, J., Thomas, S. M., Rios, P., Page, M. J., & Tricco, A. C. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.10.003 · Cited by 86
Page, S. L. L., Bejan‐Angoulvant, T., Angoulvant, D., & Prunier, F. (2015) · Basic Research in Cardiology · doi:10.1007/s00395-015-0467-8 · Cited by 94
“Moreover, although risk of bias was assessed by two authors with consensus reached after discussion and using the same Cochrane tool for all trials included, several studies have shown reliability of the Cochrane tool to be quite low, in particular between pairs of reviewers and primarily due to differing interpretati…”
MethodExercise for rheumatoid arthritis of the hand
Williams, M. A., Srikesavan, C. S., Heine, P., Bruce, J., Brosseau, L., Hoxey-Thomas, N., & Lamb, S. E. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003832.pub3 · Cited by 89
Sievers, S. B., Trembath, D. L., & Westerveld, M. F. (2018) · Augmentative and Alternative Communication · doi:10.1080/07434618.2018.1462849 · Cited by 78
“…it allows for the evaluation of multiple study designs, provides an explicit approach to making judgments about the quality of evidence, and has been used to assess research in order to promote decisions for clinicians and policy makers (Armijo-Olivo et al., 2014; Evans, Lasen, & Tsey, 2015).”
BackgroundMethodJardim, P. S. J., Rose, C. J., Ames, H. M., Echavez, J. F. M., Van de Velde, S., & Muller, A. E. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01649-y · Cited by 68
“Risk of bias assessment requires judgement and numerous studies have reported low inter-rater reliability of the RoB1 tool [10, 11].”
Methodological Issues in Rehabilitation Research: A Scoping Review
Arienti, C., Armijo-Olivo, S. L., Minozzi, S., Tjosvold, L., Lazzarini, S. G., Patrini, M., & Négrini, S. (2021) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2021.04.006 · Cited by 69
Self-citationMachine learning to assist risk-of-bias assessments in systematic reviews
Millard, L. A., Flach, P. A., & Higgins, J. P. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv306 · Cited by 65
da Costa, B. R., Beckett, B., Diaz, A., Resta, N. M., Johnston, B. C., Egger, M., Jüni, P., & Armijo-Olivo, S. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0441-7 · Cited by 62
“We recently found that the reliability of the risk of bias tool might be improved by intensive standardized training of raters [8].”
BackgroundMethodInfluentialSelf-citation(2017) · Europe PMC (PubMed Central) · Cited by 61
Applying the ROBINS-I tool to natural experiments: an example from public health
Thomson, H., Craig, P., Hilton-Boon, M., Campbell, M., & Katikireddi, S. V. (2018) · Systematic Reviews · doi:10.1186/s13643-017-0659-4 · Cited by 60
Gaskell, C., Simmonds‐Buckley, M., Kellett, S., Stockton, C., Somerville, E., Rogerson, E., & Delgadillo, J. (2022) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-022-01225-y · Cited by 70
“1 3 RoB in duplicate is problematic due to risk of imprecise estimates of treatment effect and RoB (Armijo-Olivo et al., 2014).”
BackgroundGates, A., Vandermeer, B., & Hartling, L. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.015 · Cited by 60
Seehra, J., Mortaja, K., Wazwaz, F., Papageorgiou, S. N., Newton, J. T., & Cobourne, M. T. (2023) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2023.01.004 · Cited by 59
A unified framework for bias assessment in clinical research
Stone, J. C., Glass, K., Clark, J., Munn, Z., Tugwell, P., & Doi, S. A. R. (2019) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000165 · Cited by 40
Nursing Staff Time and Care Quality in Long-Term Care Facilities: A Systematic Review
Armijo-Olivo, S. L., Craig, R., Corabian, P., Guo, B., Souri, S., & Tjosvold, L. (2019) · The Gerontologist · doi:10.1093/geront/gnz053 · Cited by 54
Self-citationArmijo-Olivo, S. L., Dennett, L., Arienti, C., Dahchi, M., Arokoski, J. P., Heinemann, A. W., & Malmivaara, A. O. V. (2020) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000001377 · Cited by 49
Self-citationDiabetes self-management education interventions in the WHO African Region: A scoping review
Kumah, E., Otchere, G., Ankomah, S. E., Fusheini, A., Kokuro, C., Aduo-Adjei, K., & Amankwah, J. A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256123 · Cited by 51
“50) as recommended by the EPHPP guidelines [24].”
MethodNeedle size for vaccination procedures in children and adolescents
Beirne, P. V., Hennessy, S., Cadogan, S. L., Shiely, F., Fitzgerald, T., & MacLeod, F. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010720.pub2 · Cited by 49
Saltaji, H., Armijo-Olivo, S., Cummings, G. G., Amin, M., & Flores-Mir, C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0190089 · Cited by 43
Self-citationBonini‐Rocha, A. C., de Andrade, A. L. S., de Moraes, A. M., Matheus, L. B. G., Diniz, L. R., & Martins, W. R. (2017) · PM&R · doi:10.1016/j.pmrj.2017.09.014 · Cited by 42
“10), and the random-effects model was used when heterogeneity was confirmed; this was applied in all analyses [14].”
MethodInter-rater reliability of AMSTAR is dependent on the pair of reviewers
Pieper, D., Jacobs, A., Weikert, B., Fishta, A., & Wegewitz, U. E. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0380-y · Cited by 45
“Another study found that RoB assessments were not consistent when the original RoB assessments were compared with external reviewers [30].”
ResultGarcía‐Perdomo, H. A., Jiménez-Mejías, E., & López‐Ramos, H. (2015) · International braz j urol · doi:10.1590/s1677-5538.ibju.2014.0198 · Cited by 40
“Cystoscopy is the most frequently used and accepted diagnostic tool in urology (1).”
MethodJordan, V., Lensen, S. F., & Farquhar, C. M. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.08.012 · Cited by 41
Bertizzolo, L., Bossuyt, P., Atal, I., Ravaud, P., & Dechartres, A. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-028382 · Cited by 35
Ethics of Outbreaks Position Statement. Part 1: Therapies, Treatment Limitations, and Duty to Treat
Papadimos, T. J., Marcolini, E. G., Hadian, M., Hardart, G. E., Ward, N., Levy, M. M., Stawicki, S. P., & Davidson, J. E. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000003416 · Cited by 36
Inter-review agreement of risk-of-bias judgments varied in Cochrane reviews
Könsgen, N., Barčot, O., Heß, S., Puljak, L., Gooßen, K., Rombey, T., & Pieper, D. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.12.016 · Cited by 37
Comparing machine and human reviewers to evaluate the risk of bias in randomized controlled trials
Armijo‐Olivo, S., Craig, R., & Campbell, S. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1398 · Cited by 31
“13 However, this study used dichotomous scoring (low vs high/unclear RoB), rather than the three categories (low, unclear, and high RoB) used by the Cochrane Collaboration and others, and thus has limited applicability for those following Cochrane methods.”
BackgroundMethodSelf-citationŠuster, S., Baldwin, T., & Verspoor, K. M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1749 · Cited by 40
“The question of annotation reliability has therefore been studied extensively for both RoB1 30,31 and RoB2, 32 – 35 with the latter showing better inter-rater agreement for those authors who are experts in the topic of the review.”
Goldsworthy, S., Palmer, S., Latour, J. M., McNair, H. A., & Cramp, M. (2020) · Radiography · doi:10.1016/j.radi.2020.03.002 · Cited by 30
Eisele‐Metzger, A., Lieberum, J.-L., Toews, M., Siemens, W., Heilmeyer, F., Haverkamp, C., Böhringer, D., & Meerpohl, J. J. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.12 · Cited by 32
Methodological quality and risk of bias in orthodontic systematic reviews using AMSTAR and ROBIS
Hooper, E. J., Pandis, N., Cobourne, M. T., & Seehra, J. (2020) · European Journal of Orthodontics · doi:10.1093/ejo/cjaa074 · Cited by 29
Cihodaru-Ștefanache, Ș., & Podina, I. R. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1625984 · Cited by 31
LAIGNELOT, F., Martin, G. L., OSSMAN, M., Pingeon, O., BOUBAKER, A., PICOVSCHI, E., Jia, K. I. M., Tannier, X., Cohen, J. F., & Dechartres, A. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112221 · Cited by 37
Rogan, S., Hähni, M., Luijckx, E., Dealer, J., Reuteler, S., & Taeymans, J. (2018) · The Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000002658 · Cited by 25
“repeatability) of the RoB tool (1), the Jadad scale (28) was”
MethodNeedle size for vaccination procedures in children and adolescents
Beirne, P. V., Hennessy, S., Cadogan, S. L., Shiely, F., Fitzgerald, T., & MacLeod, F. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010720.pub3 · Cited by 32
Avagyan, A. V., Mkrtchyan, H. H., Shafa, F. A., Mathew, J. A., & Petrosyan, T. (2021) · CoDAS · doi:10.1590/2317-1782/20202020244 · Cited by 27
A systematic review of orthopaedic manual therapy randomized clinical trials quality
Riley, S. P., Swanson, B. T., Brismée, J., & Sawyer, S. F. (2016) · Journal of Manual & Manipulative Therapy · doi:10.1080/10669817.2015.1119372 · Cited by 24
Müggenborg, F., de Castro‐Carletti, E. M., Dennett, L., de Oliveira‐Souza, A. I. S., Mohamad, N., Licht, G., von Piekartz, H., & Armijo-Olivo, S. L. (2023) · Life · doi:10.3390/life13020336 · Cited by 29
“There is a great need for well-designed RCTs considering the limitations highlighted by this review (see section above) to specifically investigate manual trigger point therapy for patients with mTrPs in the orofacial area [23–25].”
MethodSelf-citationSeehra, J., Alshammari, A., Wazwaz, F., Papageorgiou, S. N., Newton, T., & Cobourne, M. T. (2023) · European Journal of Orthodontics · doi:10.1093/ejo/cjad039 · Cited by 26
Trends in Methodological Quality in Controlled Trials of Psychological and Social Interventions
Sundell, K., & Åhsberg, E. (2016) · Research on Social Work Practice · doi:10.1177/1049731516633297 · Cited by 18
Niederer, D., Weippert, M., & Behrens, M. (2022) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2022.11149 · Cited by 17
Effect of Cognitive Stimulation Therapy on Quality of Life: A Critical Review
Chao, I. C. I., Nicpon, K., & Roberts, M. R. (2020) · Physical & Occupational Therapy In Geriatrics · doi:10.1080/02703181.2020.1716915 · Cited by 17
“Also, the intervention effect cannot be identified if the reasons behind participants’ dropouts are unknown.(59) The three CCTs (including Piras et al.”
Backgroundde Oliveira‐Souza, A. I. S., Mohamad, N., de Castro‐Carletti, E. M., Müggenborg, F., Dennett, L., de Oliveira, D. A., & Armijo-Olivo, S. L. (2022) · Disability and Rehabilitation · doi:10.1080/09638288.2022.2127933 · Cited by 18
Self-citationSaltaji, H., Ospina, M. B., Armijo-Olivo, S. L., Agarwal, S., Cummings, G. G., Amin, M. S., & Flores‐Mir, C. (2016) · The Journal of the American Dental Association · doi:10.1016/j.adaj.2016.03.017 · Cited by 17
Self-citationMickenautsch, S., Rupf, S., Miletić, I., & Yengopal, V. (2022) · PLoS ONE · doi:10.1371/journal.pone.0279645 · Cited by 17
Iskander, R., Moyer, H., Fergusson, D. A., McGrath, S., Benedetti, A., & Kimmelman, J. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2515 · Cited by 15
“The authors deemed more labor-intensive methods of assessing risk of bias, such as the Cochrane tool, to be of limited value given that all trial designs would likely have been vetted by the FDA and because of the subjectivity of scoring (23) and our meta-research study objectives.”
Methodda Costa, B. R., Resta, N. M., Beckett, B., Israel-Stahre, N., Diaz, A., Johnston, B. C., Egger, M., Jüni, P., & Armijo-Olivo, S. L. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-144 · Cited by 15
“We recently found that the reliability of the risk of bias tool might be improved by intensive standardized training of raters [8].”
BackgroundMethodSelf-citationChallenges and Opportunities of Using Telenursing During COVID-19 Pandemic: An Integrative Review
Firouzkouhi, M., Abdollahimohammad, A., Arulappan, J., Nouraei, T., & Farzi, J. (2021) · Frontiers in Health Informatics · doi:10.30699/fhi.v10i1.332 · Cited by 14
“All authors approved the internal validity of the article with Cochrane Collaboration High Risk of Bias standards [16].”
MethodWarlo, L. S., Bardai, S. E., de Vries, A., van Veelen, M. C., Moors, S., Rings, E. H. H. M., Legerstee, J. S., & Dierckx, B. (2024) · JMIR Serious Games · doi:10.2196/55034 · Cited by 14
“However, it is worth noting that this tool has been reported to have relatively low reliability [86] and is considered more conservative compared with other risk of bias assessments [87].”
ResultMickenautsch, S., Rupf, S., Miletić, I., & Yengopal, V. (2023) · Reviews on Recent Clinical Trials · doi:10.2174/1574887118666230104152245 · Cited by 11
Song, S., Yang, Y., Geng, C., Tang, Z., Wang, C., & Li, X. F. (2023) · BMC Infectious Diseases · doi:10.1186/s12879-023-08557-6 · Cited by 14
“We suspected this inconsistency may be caused by the low reliability of the risk of bias tool [47].”
Rademaker, M. M., Ramakers, G. G. J., Smit, A. L., Hooft, L., & Stegeman, I. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235535 · Cited by 11
“However there are limitations to the reliability of the Cochrane RoB tool since low agreement rates have been found between Cochrane- and external reviewers [32,33].”
Evidence-Based Psychotherapy in Ischemic Heart Disease: Umbrella Review and Updated Meta-Analysis
Biondi-Zoccai, G., Mazza, M., Roever, L., van Dixhoorn, J., Frati, G., & Abbate, A. (2016) · Psychotherapy for Ischemic Heart Disease · doi:10.1007/978-3-319-33214-7_10 · Cited by 9
Armijo-Olivo, S. L., Patrini, M., de Oliveira‐Souza, A. I. S., Dennett, L., Arienti, C., Dahchi, M., & Négrini, S. (2021) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2021.04.007 · Cited by 10
Self-citationBarčot, O., Borić, M., Došenović, S., & Puljak, L. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01339-1 · Cited by 10
“The concept of adequacy, used in this study, might still be subjective because it was ultimately determined by the authors of this manuscript (although we did our best to follow guidance from the Cochrane Handbook strictly) [14, 15].”
BackgroundMethodda Silva, J. B., Luc, A., de Oliveira‐Souza, A. I. S., de Abreu, R. M., Raposo, J. C. D. S., de Schaetzen, M., Pitance, L., & Armijo-Olivo, S. L. (2025) · Physiotherapy Research International · doi:10.1002/pri.70045 · Cited by 8
“Criteria to determine the overall risk of bias have been used previously (Armijo‐Olivo et al. 2014; Hartling et al. 2012).”
Self-citationArmijo-Olivo, S. L., da Costa, B. R., Ha, C., Saltaji, H., Cummings, G. G., & Fuentes, J. (2021) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000001837 · Cited by 8
“We evaluated biases related to attrition using the domain of “incomplete outcome data” from the RoB tool, following the Cochrane Collaboration guidance and using a three-part answering scheme (high, low, and unclear(27)); in addition, we developed specific decision rules, described elsewhere.(28) We collapsed categori…”
MethodSelf-citationTonin, F. S., Lopes, L. A. A., Rotta, I., Bonetti, A. D. F., Pontarolo, R., Correr, C. J., & Fernández-Llimós, F. (2019) · International Journal of Clinical Pharmacy · doi:10.1007/s11096-019-00818-2 · Cited by 9
Risk of bias in assessing Risk of Bias
Wilkins, A. J. (2016) · Ophthalmic and Physiological Optics · doi:10.1111/opo.12333 · Cited by 7
“Under double-masked conditions, agreement between judges using the Risk of Bias tool is so poor as to suggest that assessments of this kind have questionable validity.(2) In an attempt to be constructive, I should like to offer steps that can be taken to improve Cochrane reviews, and the assessment of bias in particul…”
MethodYao, Y., Shen, J., Luo, J., Li, N., Liao, X., & Zhang, Y. (2024) · Trials · doi:10.1186/s13063-024-08145-2 · Cited by 6
“However, different reviewers might carry out different assessments, leading to unsatis-factory inter-reviewer reliability in RoB assessments [12, 13].”
BackgroundHardt, S., Griefahn, A., Aasdahl, L., & Armijo-Olivo, S. L. (2025) · Disability and Rehabilitation · doi:10.1080/09638288.2025.2462773 · Cited by 5
“Similar decision rules have been used by previous studies when rating the overall risk of bias of RCTs [36].”
Self-citationGuelimi, R., Choudhary, C., Ollivier, C., Beytout, Q., Samaran, Q., Mubuangankusu, A., Chaimani, A., Sbidian, É., Afach, S., & Cleach, L. L. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.112097 · Cited by 8
Guidance on how to efficiently find, choose, and use available systematic reviews was developed
Mathes, T., Prien, P., Klerings, I., Ewald, H., Dressler, C., Harder, T., Scheibler, F., Büchter, R. B., Braun, C., Grummich, K., et al. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111466 · Cited by 6
Kanukula, R., McKenzie, J. E., Cashin, A. G., Korevaar, E., McDonald, S., Mello, A. T., Nguyen, P., Saldanha, I. J., Wewege, M. A., & Page, M. J. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.111244 · Cited by 5
On the need for objective measures of risk of bias
Berger, V. W., & Mickenautsch, S. (2015) · Contemporary Clinical Trials · doi:10.1016/j.cct.2015.02.001 · Cited by 4
“The poor reliability recently found [1] across reviewers assessing the risk of bias should be a cause for serious concern to anyone who wants to base medical decisions on the available medical studies.”
BackgroundBrosseau, L., Toupin‐April, K., Wells, G. A., Smith, C. A. M., Pugh, A. G., Stinson, J., Duffy, C. M., Gifford, W., Moher, D., Sherrington, C., et al. (2015) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2015.11.011 · Cited by 4
Spedding, S. J. (2015) · Advances in Integrative Medicine · doi:10.1016/j.aimed.2015.02.005 · Cited by 4
Young, A. E., Reeves, B. C., Cheng, H., Wasiak, J., Muir, D., Davies, A. K., & Blazeby, J. M. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-033472 · Cited by 3
Bertizzolo, L., Bossuyt, P. M., Atal, I., Ravaud, P., & Dechartres, A. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.008 · Cited by 3
Assessing the Risk of Bias in Randomized Trials
Page, M. J., Altman, D. G., & Egger, M. (2022) · Wiley eBooks · doi:10.1002/9781119099369.ch4 · Cited by 3
Self-citationde Castro‐Carletti, E. M., Müggenborg, F., Dennett, L., de Oliveira‐Souza, A. I. S., Mohamad, N., Pertille, A., Rodrigues-Bigaton, D., & Armijo-Olivo, S. L. (2023) · Clinical Rehabilitation · doi:10.1177/02692155221149350 · Cited by 7
Self-citationFive methodological considerations for validating LLMs in risk of bias assessment
Sahu, V. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10073 · Cited by 2
Nejstgaard, C. H., Lundh, A., Abdi, S., Clayton, G., Gelle, M. H. A., Laursen, D. R. T., Olorisade, B. K., Savović, J., & Hróbjartsson, A. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1527 · Cited by 2
Metadata
- Authors
- Armijo-Olivo, S., Ospina, M., da Costa, B. R., Egger, M., Saltaji, H., Fuentes, J., Ha, C., & Cummings, G. G.
- Affiliations
- University of Alberta (CA)
- Alberta Health Services (CA)
- University of Bern (CH)
- Corresponding author
- Armijo-Olivo, Susan
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-05-13
- Volume
- 9
- Issue
- 5
- Pages
- e96920
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4019638
- OpenAlex
- W2027833822
- Cited by
- 124
- References
- 61
- Citations by year
- 2026: 6 · 2025: 5 · 2024: 10 · 2023: 15 · 2022: 9 · 2021: 9 · 2020: 9 · 2019: 10 · 2018: 12 · 2017: 17 · 2016: 10 · 2015: 10 · 2014: 2
- FWCI
- 12.68
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Florida International University Digital Commons (Florida International University) (submittedVersion)
- Florida International University Digital Commons (Florida International University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- BORIS (University Library Bern) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Open Access CRIS of the University of Bern
- Funders
- Women and Children's Health Research Institute
- Physiotherapy Foundation of Canada
- Alberta Innovates - Health Solutions
- University of Alberta
- Fondation pour la Recherche Médicale
- Killam Trusts
- Alberta Innovates
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- risk of bias assessment, Cochrane risk of bias tool, inter-rater reliability, consensus assessment
- MeSH terms
- Humans, Research Design, Review Literature as Topic, Reproducibility of Results, Bias, Randomized Controlled Trials as Topic, Physical Therapy Modalities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
