Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review
The DOI matches its Crossref record; 184 of the 184 references it cites were found in a registry or an index.
Abstract
Background and Purpose: The methodological quality of randomized controlled trials (RCTs) is commonly evaluated in order to assess the risk of biased estimates of treatment effects. The purpose of this systematic review was to identify scales used to evaluate the methodological quality of RCTs in health care research and summarize the content, construction, development, and psychometric properties of these scales. Methods: Extensive electronic database searches, along with a manual search, were performed. Results: One hundred five relevant studies were identified. They accounted for 21 scales and their modifications. The majority of scales had not been rigorously developed or tested for validity and reliability. The Jadad Scale presented the best validity and reliability evidence; however, its validity for physical therapy trials has not been supported. Discussion and Conclusion: Many scales are used to evaluate the methodological quality of RCTs, but most of these scales have not been adequately developed and have not been adequately tested for validity and reliability. A valid and reliable scale for the assessment of the methodological quality of physical therapy trials needs to be developed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Quality is not well defined, but has been shown to encompass constructs beyond those that may bias the findings, including, for example, imprecision, reporting completeness, ethics, and applicability.”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [79]
Citations
Cites
2 retracted
Interventions for treating chronic pelvic pain in women
Stones, W., Cheong, Y. C., Howard, F. M., & Singh, S. (2005) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000387 · Cited by 165
RetractedInterventions for treating chronic pelvic pain in women
Stones, W., Cheong, Y. C., Howard, F. M., & Singh, S. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000387.pub2 · Cited by 66
RetractedCochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Cochrane Handbook for Systematic Reviews of Interventions
(2019) · doi:10.1002/9781119536604 · Cited by 14,828
Quality criteria were proposed for measurement properties of health status questionnaires
Terwee, C. B., Bot, S. D. M., de Boer, M. R., van der Windt, D. A. W. M., Knol, D. L., Dekker, J., Bouter, L. M., & de Vet, H. C. W. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.012 · Cited by 10,764
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373
Research works printed but not linked: 7
- Khan, 2001, Stage II: conducting the review, phase 5: study quality assessment, 1
- Streiner, 2004, Validity, 172
- Streiner, 2004, Reliability, 126
- Streiner, 2004, Measuring change, 194
- Hrobjartsson, 2001, Is the placebo powerless, N Engl J Med, 344, 1594
- Markes, 2006, Exercise for women receiving adjuvant therapy for breast cancer, Cochrane Database Syst Rev, CD005001
- Spitzer, 1990, Links between passive smoking and disease: a best-evidence synthesis, Clin Invest Med, 13, 17
Cited by
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
OARSI recommendations for the management of hip and knee osteoarthritis
Zhang, W., Nuki, G., Moskowitz, R. W., Abramson, S., Altman, R. D., Arden, N. K., Bierma-Zeinstra, S., Brandt, K. D., Croft, P., Doherty, M., et al. (2010) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2010.01.013 · Cited by 1,700
Armijo‐Olivo, S., Stiles, C. R., Hagen, N. A., Biondo, P. D., & Cummings, G. G. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2010.01516.x · Cited by 1,539
Page, M. J., Moher, D., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/gwdhk · Cited by 916
Turner, L., Shamseer, L., Altman, D. G., Weeks, L., Peters, J., Kober, T., Dias, S., Schulz, K. F., Plint, A. C., & Moher, D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000030.pub2 · Cited by 740
Construção de instrumentos de medida na área da saúde
Coluci, M. Z. O., Alexandre, N. M. C., & Milani, D. (2015) · Ciência & Saúde Coletiva · doi:10.1590/1413-81232015203.04332013 · Cited by 547
Abu Dayyeh, B. K., Kumar, N., Edmundowicz, S. A., Jonnalagadda, S., Larsen, M., Sullivan, S., Thompson, C. C., & Banerjee, S. (2015) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2015.03.1964 · Cited by 469
Tyrer, R. A., & Fazel, M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0089359 · Cited by 459
Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2017) · doi:10.23970/ahrqepcmethguide2 · Cited by 448
Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers
Hartling, L., Milne, A., Hamm, M. P., Vandermeer, B., Ansari, M., Tsertsvadze, A., & Dryden, D. M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.03.003 · Cited by 436
Modini, M., Tan, L., Brinchmann, B., Wang, M.-J., Killackey, E., Glozier, N., Mykletun, A., & Harvey, S. B. (2016) · British Journal of Psychiatry · doi:10.1192/bjp.bp.115.165092 · Cited by 422
Armijo-Olivo, S., Pitance, L., Singh, V., Neto, F., Thie, N., & Michelotti, A. (2016) · Physical Therapy · doi:10.2522/ptj.20140548 · Cited by 526
Armijo-Olivo, S., Warren, S., & Magee, D. (2009) · Physical Therapy Reviews · doi:10.1179/174328809x405928 · Cited by 361
Self-citationMacedo, L. G., Elkins, M. R., Maher, C. G., Moseley, A. M., Herbert, R. D., & Sherrington, C. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.10.005 · Cited by 362
Self-citationHuedo-Medina, T. B., Kirsch, I., Middlemass, J., Klonizakis, M., & Siriwardena, A. N. (2012) · BMJ · doi:10.1136/bmj.e8343 · Cited by 313
Wang, Y., Xue, H., Huang, Y., Huang, L., & Zhang, D. (2017) · Advances in Nutrition · doi:10.3945/an.116.014100 · Cited by 310
Page, M. J., McKenzie, J. E., & Higgins, J. P. T. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019703 · Cited by 278
Beinart, N. A., Goodchild, C. E., Weinman, J. A., Ayis, S., & Godfrey, E. L. (2013) · The Spine Journal · doi:10.1016/j.spinee.2013.08.027 · Cited by 294
“A recent systematic review concluded that no QATs used in systematic reviews were reliable or valid for evaluating the methodological quality of RCTs in physical therapy research [37].”
BackgroundMocking, R. J. T., Harmsen, I., Assies, J., Koeter, M. W. J., Ruhé, H. G., & Schene, A. H. (2016) · Translational Psychiatry · doi:10.1038/tp.2016.29 · Cited by 279
Effects of Resistance Training in Children and Adolescents: A Meta-analysis
Behringer, M., vom Heede, A., Yue, Z., & Mester, J. (2010) · Pediatrics · doi:10.1542/peds.2010-0445 · Cited by 273
Behringer, M., Heede, A. V., Matthews, M., & Mester, J. (2011) · Pediatric Exercise Science · doi:10.1123/pes.23.2.186 · Cited by 269
Rao, G., Lopez-Jimenez, F., Boyd, J., D’Amico, F., Durant, N. H., Hlatky, M. A., Howard, G., Kirley, K., Masi, C., Powell-Wiley, T. M., et al. (2017) · Circulation · doi:10.1161/cir.0000000000000523 · Cited by 269
Velthuis, M. J., Agasi-Idenburg, S. C., Aufdemkampe, G., & Wittink, H. M. (2010) · Clinical Oncology · doi:10.1016/j.clon.2009.12.005 · Cited by 265
A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261
Herman, K., Barton, C., Malliaras, P., & Morrissey, D. (2012) · BMC Medicine · doi:10.1186/1741-7015-10-75 · Cited by 261
Effects of omega-3 fatty acids on cognitive performance: a meta-analysis
Mazereeuw, G., Lanctôt, K. L., Chau, S. A., Swardfager, W., & Herrmann, N. (2012) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2011.12.014 · Cited by 253
Deady, M., Choi, I., Calvo, R. A., Glozier, N., Christensen, H., & Harvey, S. B. (2017) · BMC Psychiatry · doi:10.1186/s12888-017-1473-1 · Cited by 222
Can pain be managed through the Internet? A systematic review of randomized controlled trials
Bender, J. L., Radhakrishnan, A., Diorio, C., Englesakis, M., & Jadad, A. R. (2011) · Pain · doi:10.1016/j.pain.2011.02.012 · Cited by 223
Zou, L., Sasaki, J. E., Wang, H., Xiao, Z., Fang, Q., & Zhang, M. (2017) · Evidence-Based Complementary and Alternative Medicine · doi:10.1155/2017/4548706 · Cited by 229
Lees, C., & Hopkins, J. (2013) · Preventing Chronic Disease · doi:10.5888/pcd10.130010 · Cited by 226
Recommendations for assessing the risk of bias in systematic reviews of health-care interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.004 · Cited by 221
Tan, L., Wang, M.-J., Modini, M., Joyce, S., Mykletun, A., Christensen, H., & Harvey, S. B. (2014) · BMC Medicine · doi:10.1186/1741-7015-12-74 · Cited by 208
Hartling, L., Hamm, M. P., Milne, A., Vandermeer, B., Santaguida, P. L., Ansari, M., Tsertsvadze, A., Hempel, S., Shekelle, P., & Dryden, D. M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.07.005 · Cited by 173
What Works in Falls Prevention After Stroke?: A Systematic Review and Meta-Analysis
Batchelor, F., Hill, K., Mackintosh, S., & Said, C. (2010) · Stroke · doi:10.1161/strokeaha.109.570390 · Cited by 171
High Agreement and High Prevalence: The Paradox of Cohen’s Kappa
Zec, S., Soriani, N., Comoretto, R., & Baldi, I. (2017) · The Open Nursing Journal · doi:10.2174/1874434601711010211 · Cited by 169
Harding, R., List, S., Epiphaniou, E., & Jones, H. (2012) · Palliative Medicine · doi:10.1177/0269216311409613 · Cited by 160
Gomes-Neto, M., Lopes, J. M., Conceição, C. S., Araujo, A., Brasileiro, A., Sousa, C., Carvalho, V. O., & Arcanjo, F. L. (2017) · Physical Therapy in Sport · doi:10.1016/j.ptsp.2016.08.004 · Cited by 156
Gomes-Neto, M., Durães, A. R., Reis, H. F. C. D., Neves, V. R., Martinez, B. P., & Carvalho, V. O. (2017) · European Journal of Preventive Cardiology · doi:10.1177/2047487317728370 · Cited by 151
A meta-regression of the impact of policy on the efficacy of individual placement and support
Brinchmann, B., Widding‐Havneraas, T., Modini, M., Rinaldi, M., Moe, C. F., McDaid, D., Park, A., Killackey, E., Harvey, S. B., & Mykletun, A. (2020) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.13129 · Cited by 134
Armijo-Olivo, S., Ospina, M., da Costa, B. R., Egger, M., Saltaji, H., Fuentes, J., Ha, C., & Cummings, G. G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0096920 · Cited by 124
Self-citationAdamse, C., Dekker-Van Weering, M. G., van Etten-Jamaludin, F. S., & Stuiver, M. M. (2018) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x17716576 · Cited by 115
Mao, Y., Lin, W., Wen, J., & Chen, G. (2020) · BMJ Open Diabetes Research & Care · doi:10.1136/bmjdrc-2020-001225 · Cited by 108
Walser, R. F., Meserve, B. B., & Boucher, T. R. (2009) · Journal of Manual & Manipulative Therapy · doi:10.1179/106698109791352085 · Cited by 98
PEDro's bias: summary quality scores should not be used in meta-analysis
da Costa, B. R., Hilfiker, R., & Egger, M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.08.003 · Cited by 86
Theodorakopoulos, C., Jones, J., Bannerman, E., & Greig, C. A. (2017) · Nutrition Research · doi:10.1016/j.nutres.2017.05.002 · Cited by 86
Samuel, G. O., Hoffmann, S., Wright, R. A., Lalu, M. M., Patlewicz, G., Becker, R. A., DeGeorge, G. L., Fergusson, D., Hartung, T., Lewis, R. J., et al. (2016) · Environment International · doi:10.1016/j.envint.2016.03.010 · Cited by 82
Psychological interventions for migraine: a systematic review
Sullivan, A., Cousins, S., & Ridsdale, L. (2016) · Journal of Neurology · doi:10.1007/s00415-016-8126-z · Cited by 73
“This scale has been deemed to have good construct validity and reliability [15].”
BackgroundRodríguez-Mansilla, J., González López-Arza, M. V., Varela-Donoso, E., Montanero-Fernández, J., González Sánchez, B., & Garrido-Ardila, E. M. (2015) · Clinical Rehabilitation · doi:10.1177/0269215514554240 · Cited by 68
(2017) · Europe PMC (PubMed Central) · Cited by 61
Armijo-Olivo, S., Fuentes, J., Ospina, M., Saltaji, H., & Hartling, L. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-116 · Cited by 60
Self-citationWobma, R., Nijland, R., Ket, J., & Kwakkel, G. (2016) · Journal of Rehabilitation Medicine · doi:10.2340/16501977-2160 · Cited by 48
Saltaji, H., Armijo-Olivo, S., Cummings, G. G., Amin, M., & Flores-Mir, C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0190089 · Cited by 43
Yang, Y.-L., Sui, G.-Y., Liu, G.-C., Huang, D.-S., Wang, S.-M., & Wang, L. (2014) · BMC Cancer · doi:10.1186/1471-2407-14-956 · Cited by 38
Cannabis for Chronic Pain: A Rapid Systematic Review of Randomized Control Trials
Longo, R., Oudshoorn, A., & Befus, D. (2021) · Pain Management Nursing · doi:10.1016/j.pmn.2020.11.006 · Cited by 39
Jackson, K. F., Hodge, D. R., & Vaughn, M. G. (2010) · Journal of Social Service Research · doi:10.1080/01488371003697780 · Cited by 30
van der Vaart, T., Overwater, I. E., Oostenbrink, R., Moll, H. A., & Elgersma, Y. (2015) · JAMA Neurology · doi:10.1001/jamaneurol.2015.0443 · Cited by 17
Tuong, W., Walker, L., & Sivamani, R. K. (2015) · Journal of Dermatological Treatment · doi:10.3109/09546634.2014.991675 · Cited by 17
Estudios Experimentales 1 Parte: El Ensayo Clínico
Manterola, C., & Otzen, T. (2015) · International Journal of Morphology · doi:10.4067/s0717-95022015000100054 · Cited by 15
Carneiro, L., Silva, F., Vasoncelos-Raposo, J., Mota, M. P., Vieira-Coelho, M. A., Deslandes, A., Marinho, H., Amaral, G., & Monteiro-Júnior, R. (2020) · doi:10.21203/rs.3.rs-24600/v1 · Cited by 3
“We analysed randomized controlled trials (RCT) through: randomization, allocation concealment, baseline comparability, eligibility criteria, blinding, descriptive measures for the primary outcome and intention-to-treat analysis [22].”
MethodFujii, R., Miki, T., Nonaka, Y., & Tanaka, S. (2024) · PLOS ONE · doi:10.1371/journal.pone.0297649 · Cited by 1
Metadata
- Authors
- Olivo, S. A., Macedo, L. G., Gadotti, I. C., Fuentes, J., Stanton, T., & Magee, D. J.
- Affiliations
- University of Alberta (CA)
- The University of Sydney (AU)
- IC Gadotti, BScPT, MSc, is a PhD candidate in the Faculty of Rehabilitation Medicine, University of Alberta
- Venue
- Physical Therapy
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-02-01
- Volume
- 88
- Issue
- 2
- Pages
- 156-175
- ISSN
- 0031-9023 (ISSN-L)
- 1538-6724
- OpenAlex
- W2123945981
- Cited by
- 829
- References
- 184
- Citations by year
- 2026: 16 · 2025: 24 · 2024: 35 · 2023: 51 · 2022: 62 · 2021: 63 · 2020: 50 · 2019: 53 · 2018: 43 · 2017: 48 · 2016: 54 · 2015: 53 · 2014: 63 · 2013: 61 · 2012: 44
- FWCI
- 4.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Physical Therapy (publishedVersion)
- PubMed (publishedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- methodological quality assessment, randomized controlled trials, quality assessment scale, Jadad scale, psychometric properties, reliability and validity, bias in treatment effect estimates
- MeSH terms
- Humans, Quality Assurance, Health Care, Research Design, Randomized Controlled Trials as Topic, Physical Therapy Modalities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.