Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: a descriptive analysis
Open accessAbstract
BACKGROUND: Assessing the risk of bias of randomized controlled trials (RCTs) is crucial to understand how biases affect treatment effect estimates. A number of tools have been developed to evaluate risk of bias of RCTs; however, it is unknown how these tools compare to each other in the items included. The main objective of this study was to describe which individual items are included in RCT quality tools used in general health and physical therapy (PT) research, and how these items compare to those of the Cochrane Risk of Bias (RoB) tool. METHODS: We used comprehensive literature searches and a systematic approach to identify tools that evaluated the methodological quality or risk of bias of RCTs in general health and PT research. We extracted individual items from all quality tools. We calculated the frequency of quality items used across tools and compared them to those in the RoB tool. Comparisons were made between general health and PT quality tools using Chi-squared tests. RESULTS: In addition to the RoB tool, 26 quality tools were identified, with 19 being used in general health and seven in PT research. The total number of quality items included in general health research tools was 130, compared with 48 items across PT tools and seven items in the RoB tool. The most frequently included items in general health research tools (14/19, 74%) were inclusion and exclusion criteria, and appropriate statistical analysis. In contrast, the most frequent items included in PT tools (86%, 6/7) were: baseline comparability, blinding of investigator/assessor, and use of intention-to-treat analysis. Key items of the RoB tool (sequence generation and allocation concealment) were included in 71% (5/7) of PT tools, and 63% (12/19) and 37% (7/19) of general health research tools, respectively. CONCLUSIONS: There is extensive item variation across tools that evaluate the risk of bias of RCTs in health research. Results call for an in-depth analysis of items that should be used to assess risk of bias of RCTs. Further empirical evidence on the use of individual items and the psychometric properties of risk of bias tools is needed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Our recent analysis of health scales used to evaluate methodological quality determined that none of these scales are adequate for use alone.
“Our recent analysis of health scales used to evaluate methodological quality determined that none of these scales are adequate for use alone.44,111”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [111]
The Discussion (page unavailable) identifies substantial variation among quality-assessment tools and questions whether many of their items indicate bias. The Conclusion (page unavailable) calls for clearer constructs and thorough validation, but does not determine that none of the scales is adequate for use alone. (LLM, conf=0.84)
Experimental — accuracy not yet measuredAssessment of quality in prior meta-analyses relied on mixed and potentially inadequate tools that included items not linked to any type of trial bias.
“Assessment of quality6,7 relied on mixed and potentially inadequate tools that included items not linked to any type of trial bias (e.g., treatment fidelity)9, thereby potentially confounding the relationship between study quality and treatment effects.”
From The effectiveness of virtual reality based interventions for symptoms of anxiety and depression: A meta-analysis (2018) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 63 · Cited by 60
Research works printed but not linked: 5
- Khan K Ter Riet G Popay J Nixon J Kleijnen J Satge II. Conducting the Review. Phase 5 Study Quality Assessment Undertaking Systematic Reviews of research Effectiveness CRD’s Guidance for those carrying out or commissioning reviews 2001 York: Centre for Reviews and Dissemination 1 20
- Streiner D Norman G Streiner D, Norman G Validity Health Measurements Scales 2004 Oxford: Oxford University Press 172 193
- Streiner D Norman G Streiner D, Norman G Reliability Health Measurements Scales 2004 Oxford: Oxford University Press 126 152
- Streiner D Norman G Streiner D, Norman G Measuring Change Health Measurements Scales 2004 Oxford: Oxford University Press 194 212
- Armijo-Olivo S Stiles CR Hagen NA Biondo PD Cummings GG Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research J Eval Clin Pract 2010 1 7 20698919
- Authors
- Armijo-Olivo, S., Fuentes, J., Ospina, M., Saltaji, H., & Hartling, L.
- Affiliations
- University of Alberta (CA)
- Corresponding author
- Armijo-Olivo, Susan
- Venue
- BMC Medical Research Methodology
- Published
- 2013-09-17
- Volume
- 13
- Issue
- 1
- Pages
- 116-116
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC3848693
- OpenAlex
- W2060971566
- Cited by
- 60
- References
- 63
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 1 · 2022: 5 · 2021: 9 · 2020: 8 · 2019: 6 · 2018: 7 · 2017: 7 · 2016: 3 · 2015: 8 · 2014: 3
- FWCI
- 0.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Children's Health Research Institute
- Women and Children's Health Research Institute
- Physiotherapy Foundation of Canada
- Alberta Innovates - Health Solutions
- University of Alberta
- Killam Trusts
- Alberta Innovates
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- risk of bias assessment, randomized controlled trials, Cochrane risk of bias tool, physiotherapy research, allocation concealment, blinding, intention-to-treat analysis
- MeSH terms
- Health, Humans, Quality Assurance, Health Care, Research Design, Risk, Models, Statistical, Bias, Randomized Controlled Trials as Topic, Physical Therapy Modalities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
