The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Methodological criteria for the assessment of moderators in systematic reviews of randomised controlled trials: a consensus study.
Open accessAbstract
BACKGROUND: Current methodological guidelines provide advice about the assessment of sub-group analysis within RCTs, but do not specify explicit criteria for assessment. Our objective was to provide researchers with a set of criteria that will facilitate the grading of evidence for moderators, in systematic reviews. METHOD: We developed a set of criteria from methodological manuscripts (n = 18) using snowballing technique, and electronic database searches. Criteria were reviewed by an international Delphi panel (n = 21), comprising authors who have published methodological papers in this area, and researchers who have been active in the study of sub-group analysis in RCTs. We used the Research ANd Development/University of California Los Angeles appropriateness method to assess consensus on the quantitative data. Free responses were coded for consensus and disagreement. In a subsequent round additional criteria were extracted from the Cochrane Reviewers' Handbook, and the process was repeated. RESULTS: The recommendations are that meta-analysts report both confirmatory and exploratory findings for sub-groups analysis. Confirmatory findings must only come from studies in which a specific theory/evidence based a-priori statement is made. Exploratory findings may be used to inform future/subsequent trials. However, for inclusion in the meta-analysis of moderators, the following additional criteria should be applied to each study: Baseline factors should be measured prior to randomisation, measurement of baseline factors should be of adequate reliability and validity, and a specific test of the interaction between baseline factors and interventions must be presented. CONCLUSIONS: There is consensus from a group of 21 international experts that methodological criteria to assess moderators within systematic reviews of RCTs is both timely and necessary. The consensus from the experts resulted in five criteria divided into two groups when synthesising evidence: confirmatory findings to support hypotheses about moderators and exploratory findings to inform future research. These recommendations are discussed in reference to previous recommendations for evaluating and reporting moderator studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Both moderators and predictors of change should be measured prior to treatment randomization (Turner et al., 2007; Pincus et al., 2011).”
From Positive psychological wellbeing is required for online self-help acceptance and commitment therapy for chronic pain to be effective (2016) · cites this paper as Pincus et al., 2011
The cited source says what the claim attributes to it.
In the citing paper
“However, our study is in accordance with the methodological requirements of exploratory moderator analyses [68].”
From How and for whom does web-based acceptance and commitment therapy work? Mediation and moderation analyses of web-based ACT for depressive symptoms (2016) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
A comparison of methods to test mediation and other intervening variable effects
MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.83 · Cited by 9,406
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
Edwards, J. R., & Lambert, L. S. (2007) · Psychological Methods · doi:10.1037/1082-989x.12.1.1 · Cited by 4,711
Cole, D. A., & Maxwell, S. E. (2003) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.112.4.558 · Cited by 3,605
Systematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,270
Statistical difficulties of detecting interactions and moderator effects.
McClelland, G. H., & Judd, C. M. (1993) · Psychological Bulletin · doi:10.1037/0033-2909.114.2.376 · Cited by 3,270
Mediators and Moderators of Treatment Effects in Randomized Clinical Trials
Kraemer, H. C., Wilson, G. T., Fairburn, C. G., & Agras, W. S. (2002) · Archives of General Psychiatry · doi:10.1001/archpsyc.59.10.877 · Cited by 2,562 · cited 1×
“We have adopted definitions from Kraemer and colleagues [1], which clearly describes how participant factors affect outcome:”
(2001) · Defense Technical Information Center (DTIC) · Cited by 2,158
2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group
Furlan, A. D., Pennick, V., Bombardier, C., & van Tulder, M. (2009) · Spine · doi:10.1097/brs.0b013e3181b1c99f · Cited by 1,663
Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445
Aguinis, H., Beaty, J. C., Boik, R. J., & Pierce, C. A. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.1.94 · Cited by 1,216 · cited 1×
“[5]); some address the shortcomings and pitfalls inherent in current analytical approaches [6-11], and several focus on the problems inherent in calculating power and adequate sample size for moderator analysis [12-14].”
Subgroup analysis and other (mis)uses of baseline data in clinical trials
Assmann, S. F., Pocock, S. J., Enos, L. E., & Kasten, L. E. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02039-0 · Cited by 1,167
Subgroup analysis in randomised controlled trials: importance, indications, and interpretation
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17709-5 · Cited by 942
The power of statistical tests for moderators in meta-analysis
Hedges, L. V., & Pigott, T. D. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.426 · Cited by 672
Brookes, S. T., Whitley, E., Peters, T. J., Mulheran, P. A., Egger, M., & Davey Smith, G. (2001) · Health Technology Assessment · doi:10.3310/hta5330 · Cited by 580
Form, effect size and power in moderated regression analysis
Champoux, J. E., & Peters, W. S. (1987) · Journal of Occupational Psychology · doi:10.1111/j.2044-8325.1987.tb00257.x · Cited by 494
Moher, D., Hopewell, S., Schulz, K. F., Montori, V. M., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D. R., Egger, M., & Altman, D. G. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.10.006 · Cited by 414
Physician ratings of appropriate indications for six medical and surgical procedures.
Park, R., Fink, A., Brook, R. H., Chassin, M. R., Kahn, K. L., Merrick, N. J., Kosecoff, J., & Solomon, D. H. (1986) · American Journal of Public Health · doi:10.2105/ajph.76.7.766 · Cited by 387
Some methodological and statistical issues in the study of change processes in psychotherapy
Laurenceau, J., Hayes, A. M., & Feldman, G. C. (2007) · Clinical Psychology Review · doi:10.1016/j.cpr.2007.01.007 · Cited by 248
Hancock, M. J., Herbert, R., & Maher, C. G. (2009) · Physical Therapy · doi:10.2522/ptj.20080351 · Cited by 168
Mediation Analysis: A Retrospective Snapshot of Practice and More Recent Directions
Gelfand, L. A., Mensinger, J. L., & TenHave, T. R. (2009) · The Journal of General Psychology · doi:10.3200/genp.136.2.153-178 · Cited by 152
Attention should be given to multiplicity issues in systematic reviews
Bender, R., Bunce, C., Clarke, M. J., Gates, S., Lange, S., Pace, N. L., & Thorlund, K. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.03.004 · Cited by 138
Taylor, W. J., Schumacher, H. R., Baraf, H. S. B., Chapman, P. T., Stamp, L. K., Doherty, M., McQueen, F. M., Dalbeth, N., Schlesinger, N., Furst, D. E., et al. (2007) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2007.079970 · Cited by 78
Toward non‐parametric and clinically meaningful moderators and mediators
KRAEMER, H. C. (2007) · Statistics in Medicine · doi:10.1002/sim.3149 · Cited by 44
Moderators and Mediators of Behavioral Treatment for Headache
Nicholson, R. A., Hursey, K. G., & Nash, J. M. (2005) · Headache The Journal of Head and Face Pain · doi:10.1111/j.1526-4610.2005.05103.x · Cited by 39
Moderator Search in Meta-Analysis: A Review and Cautionary Note on Existing Approaches
Viswesvaran, C., & Sánchez, J. I. (1998) · Educational and Psychological Measurement · doi:10.1177/0013164498058001007 · Cited by 38
Research works printed but not linked: 2
- Higgins JPT, Green S: Cochrane Handbook for Systematic Reviews of Interventions. Version 502 [updated September 2009]. 2009, The Cochrane Collaboration, 2009
- Fitch K: The Rand/UCLA appropriateness method user's manual. Santa Monica: Rand, c2001. 1991
Cited by
Sorted by citations
Using the Delphi expert consensus method in mental health research
Jorm, A. F. (2015) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867415600891 · Cited by 765
Unraveling the mechanisms of manual therapy: modeling an approach
Bialosky, J. E., Beneciuk, J. M., Bishop, M. D., Coronado, R. A., Penza, C. W., Simon, C. B., & George, S. Z. (2018) · Journal of Orthopaedic & Sports Physical Therapy · doi:10.2519/jospt.2018.7476 · Cited by 482
Wertli, M. M., Rasmussen-Barr, E., Held, U., Weiser, S., Bachmann, L. M., & Brunner, F. (2014) · The Spine Journal · doi:10.1016/j.spinee.2014.02.033 · Cited by 343
Mindfulness-Based Programs in the Workplace: a Meta-Analysis of Randomized Controlled Trials
Vonderlin, R., Biermann, M., Bohus, M., & Lyssenko, L. (2020) · Mindfulness · doi:10.1007/s12671-020-01328-3 · Cited by 259
The effects of care bundles on patient outcomes: a systematic review and meta-analysis
Lavallée, J. F., Gray, T. A., Dumville, J. C., Russell, W., & Cullum, N. A. (2017) · Implementation Science · doi:10.1186/s13012-017-0670-0 · Cited by 275
Metadata
- Authors
- Pincus, T., Miles, C., Froud, R., Underwood, M., Carnes, D., & Taylor, S. J.
- Affiliations
- University of London (GB)
- Queen Mary University of London (GB)
- University of Warwick (GB)
- Corresponding author
- Pincus, Tamar
- Venue
- BMC Medical Research Methodology
- Published
- 2011-01-31
- Volume
- 11
- Issue
- 1
- Pages
- 14-14
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC3044921
- OpenAlex
- W2111680408
- Cited by
- 161
- References
- 28
- Citations by year
- 2026: 14 · 2025: 15 · 2024: 9 · 2023: 22 · 2022: 8 · 2021: 14 · 2020: 14 · 2019: 10 · 2018: 2 · 2017: 15 · 2016: 10 · 2015: 9 · 2014: 7 · 2013: 8 · 2012: 2
- FWCI
- 1.48
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ePrints Soton (University of Southampton) (submittedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion, cc-by)
- Royal Holloway Digital Repository (Royal Holloway University of London) (submittedVersion)
- Royal Holloway Digital Repository (Royal Holloway University of London) (submittedVersion, cc-by)
- Funders
- Arizona State University
- University of Washington
- Université Laval
- Queen Mary University of London
- Advantage West Midlands
- National Institute for Health and Care Research (RP-PG-0707-10189)
- Keele University
- Programme Grants for Applied Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- moderators, subgroup analysis, randomized controlled trials, methodological quality criteria, level of evidence, interaction testing, consensus recommendations
- MeSH terms
- Humans, Review Literature as Topic, Sampling Studies, Meta-Analysis as Topic, Effect Modifier, Epidemiologic, Randomized Controlled Trials as Topic, Databases, Bibliographic, Treatment Outcome, Consensus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
