Methodological criteria for the assessment of moderators in systematic reviews of randomised controlled trials: a consensus study.
Open accessAbstract
BACKGROUND: Current methodological guidelines provide advice about the assessment of sub-group analysis within RCTs, but do not specify explicit criteria for assessment. Our objective was to provide researchers with a set of criteria that will facilitate the grading of evidence for moderators, in systematic reviews. METHOD: We developed a set of criteria from methodological manuscripts (n = 18) using snowballing technique, and electronic database searches. Criteria were reviewed by an international Delphi panel (n = 21), comprising authors who have published methodological papers in this area, and researchers who have been active in the study of sub-group analysis in RCTs. We used the Research ANd Development/University of California Los Angeles appropriateness method to assess consensus on the quantitative data. Free responses were coded for consensus and disagreement. In a subsequent round additional criteria were extracted from the Cochrane Reviewers' Handbook, and the process was repeated. RESULTS: The recommendations are that meta-analysts report both confirmatory and exploratory findings for sub-groups analysis. Confirmatory findings must only come from studies in which a specific theory/evidence based a-priori statement is made. Exploratory findings may be used to inform future/subsequent trials. However, for inclusion in the meta-analysis of moderators, the following additional criteria should be applied to each study: Baseline factors should be measured prior to randomisation, measurement of baseline factors should be of adequate reliability and validity, and a specific test of the interaction between baseline factors and interventions must be presented. CONCLUSIONS: There is consensus from a group of 21 international experts that methodological criteria to assess moderators within systematic reviews of RCTs is both timely and necessary. The consensus from the experts resulted in five criteria divided into two groups when synthesising evidence: confirmatory findings to support hypotheses about moderators and exploratory findings to inform future research. These recommendations are discussed in reference to previous recommendations for evaluating and reporting moderator studies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Both moderators and predictors of change should be measured prior to treatment randomization.
“Both moderators and predictors of change should be measured prior to treatment randomization (Turner et al., 2007; Pincus et al., 2011).”
From Positive psychological wellbeing is required for online self-help acceptance and commitment therapy for chronic pain to be effective (2016) · cites this paper as Pincus et al., 2011
The study is in accordance with the methodological requirements of exploratory moderator analyses.
“However, our study is in accordance with the methodological requirements of exploratory moderator analyses [68].”
From How and for whom does web-based acceptance and commitment therapy work? Mediation and moderation analyses of web-based ACT for depressive symptoms (2016) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.51.6.1173 · Cited by 91,542 · see citing works
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
A comparison of methods to test mediation and other intervening variable effects
MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.83 · Cited by 9,406 · see citing works
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · see citing works
Edwards, J. R., & Lambert, L. S. (2007) · Psychological Methods · doi:10.1037/1082-989x.12.1.1 · Cited by 4,711 · see citing works
Cole, D. A., & Maxwell, S. E. (2003) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.112.4.558 · Cited by 3,605 · see citing works
Statistical difficulties of detecting interactions and moderator effects.
McClelland, G. H., & Judd, C. M. (1993) · Psychological Bulletin · doi:10.1037/0033-2909.114.2.376 · Cited by 3,270 · see citing works
Systematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,237 · see citing works
Mediators and Moderators of Treatment Effects in Randomized Clinical Trials
Kraemer, H. C., Wilson, G. T., Fairburn, C. G., & Agras, W. S. (2002) · Archives of General Psychiatry · doi:10.1001/archpsyc.59.10.877 · Cited by 2,562 · cited 1× · see citing works
“We have adopted definitions from Kraemer and colleagues [1], which clearly describes how participant factors affect outcome:”
(2001) · Defense Technical Information Center (DTIC) · Cited by 2,158 · see citing works
2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group
Furlan, A. D., Pennick, V., Bombardier, C., & van Tulder, M. (2009) · Spine · doi:10.1097/brs.0b013e3181b1c99f · Cited by 1,663 · see citing works
Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445 · see citing works
Aguinis, H., Beaty, J. C., Boik, R. J., & Pierce, C. A. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.1.94 · Cited by 1,216 · see citing works
Subgroup analysis and other (mis)uses of baseline data in clinical trials
Assmann, S. F., Pocock, S. J., Enos, L. E., & Kasten, L. E. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02039-0 · Cited by 1,167 · see citing works
Subgroup analysis in randomised controlled trials: importance, indications, and interpretation
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17709-5 · Cited by 942 · see citing works
The power of statistical tests for moderators in meta-analysis
Hedges, L. V., & Pigott, T. D. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.426 · Cited by 672 · see citing works
Brookes, S. T., Whitley, E., Peters, T. J., Mulheran, P. A., Egger, M., & Davey Smith, G. (2001) · Health Technology Assessment · doi:10.3310/hta5330 · Cited by 580 · see citing works
Form, effect size and power in moderated regression analysis
Champoux, J. E., & Peters, W. S. (1987) · Journal of Occupational Psychology · doi:10.1111/j.2044-8325.1987.tb00257.x · Cited by 488 · see citing works
Moher, D., Hopewell, S., Schulz, K. F., Montori, V. M., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D. R., Egger, M., & Altman, D. G. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.10.006 · Cited by 414 · see citing works
Physician ratings of appropriate indications for six medical and surgical procedures.
Park, R., Fink, A., Brook, R. H., Chassin, M. R., Kahn, K. L., Merrick, N. J., Kosecoff, J., & Solomon, D. H. (1986) · American Journal of Public Health · doi:10.2105/ajph.76.7.766 · Cited by 387 · see citing works
Some methodological and statistical issues in the study of change processes in psychotherapy
Laurenceau, J., Hayes, A. M., & Feldman, G. C. (2007) · Clinical Psychology Review · doi:10.1016/j.cpr.2007.01.007 · Cited by 228 · see citing works
Hancock, M. J., Herbert, R., & Maher, C. G. (2009) · Physical Therapy · doi:10.2522/ptj.20080351 · Cited by 161 · see citing works
Mediation Analysis: A Retrospective Snapshot of Practice and More Recent Directions
Gelfand, L. A., Mensinger, J. L., & TenHave, T. R. (2009) · The Journal of General Psychology · doi:10.3200/genp.136.2.153-178 · Cited by 152 · see citing works
Attention should be given to multiplicity issues in systematic reviews
Bender, R., Bunce, C., Clarke, M. J., Gates, S., Lange, S., Pace, N. L., & Thorlund, K. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.03.004 · Cited by 138 · see citing works
Taylor, W. J., Schumacher, H. R., Baraf, H. S. B., Chapman, P. T., Stamp, L. K., Doherty, M., McQueen, F. M., Dalbeth, N., Schlesinger, N., Furst, D. E., et al. (2007) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2007.079970 · Cited by 78 · see citing works
Toward non‐parametric and clinically meaningful moderators and mediators
KRAEMER, H. C. (2007) · Statistics in Medicine · doi:10.1002/sim.3149 · Cited by 44 · see citing works
Moderators and Mediators of Behavioral Treatment for Headache
Nicholson, R. A., Hursey, K. G., & Nash, J. M. (2005) · Headache The Journal of Head and Face Pain · doi:10.1111/j.1526-4610.2005.05103.x · Cited by 39 · see citing works
Moderator Search in Meta-Analysis: A Review and Cautionary Note on Existing Approaches
Viswesvaran, C., & Sánchez, J. I. (1998) · Educational and Psychological Measurement · doi:10.1177/0013164498058001007 · Cited by 38 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Using the Delphi expert consensus method in mental health research
Jorm, A. F. (2015) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867415600891 · Cited by 765 · see citing works
Unraveling the mechanisms of manual therapy: modeling an approach
Bialosky, J. E., Beneciuk, J. M., Bishop, M. D., Coronado, R. A., Penza, C. W., Simon, C. B., & George, S. Z. (2018) · Journal of Orthopaedic & Sports Physical Therapy · doi:10.2519/jospt.2018.7476 · Cited by 482 · see citing works
Wertli, M. M., Rasmussen-Barr, E., Held, U., Weiser, S., Bachmann, L. M., & Brunner, F. (2014) · The Spine Journal · doi:10.1016/j.spinee.2014.02.033 · Cited by 343 · see citing works
Mindfulness-Based Programs in the Workplace: a Meta-Analysis of Randomized Controlled Trials
Vonderlin, R., Biermann, M., Bohus, M., & Lyssenko, L. (2020) · Mindfulness · doi:10.1007/s12671-020-01328-3 · Cited by 259 · see citing works
The effects of care bundles on patient outcomes: a systematic review and meta-analysis
Lavallée, J. F., Gray, T. A., Dumville, J. C., Russell, W., & Cullum, N. A. (2017) · Implementation Science · doi:10.1186/s13012-017-0670-0 · Cited by 262 · see citing works
Influence of Catastrophizing on Treatment Outcome in Patients With Nonspecific Low Back Pain
Wertli, M. M., Burgstaller, J. M., Weiser, S. R., Steurer, J., Kofmehl, R., & Held, U. (2013) · Spine · doi:10.1097/brs.0000000000000110 · Cited by 218 · see citing works
What effect can manual therapy have on a patient’s pain experience
Bishop, M. D., Torres-Cueco, R., Gay, C. W., Lluch-Girbés, E., Beneciuk, J. M., & Bialosky, J. E. (2015) · Pain Management · doi:10.2217/pmt.15.39 · Cited by 197 · see citing works
Knopp, J., Knowles, S., Bee, P., Lovell, K., & Bower, P. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.08.008 · Cited by 164 · see citing works
Prognostic factors for chronic headache: A systematic review
Probyn, K., Bowers, H., Caldwell, F., Mistry, D., Underwood, M., Matharu, M., Pincus, T., & On behalf of the CHESS Team (2017) · Neurology · doi:10.1212/wnl.0000000000004112 · Cited by 129 · see citing works
Milesl, C. L., Pincusl, T., Carnesl, D., Homerl, K. E., Taylorl, S. J. C., Bremnerl, S. A., Rahmanl, A., & Underwoodl, M. (2011) · European Journal of Pain · doi:10.1016/j.ejpain.2011.01.016 · Cited by 119 · see citing works
Self-citationPredictors of treatment outcome in depression in later life: A systematic review and meta-analysis
Tunvirachaisakul, C., Gould, R. L., Coulson, M., Ward, E. V., Reynolds, G., Gathercole, R. L., Grocott, H., Supasitthumrong, T., Tunvirachaisakul, A., Kimona, K., et al. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.10.008 · Cited by 108 · see citing works
Khoury, J. E., Milligan, K., & Girard, T. A. (2015) · Neuropsychology Review · doi:10.1007/s11065-015-9289-6 · Cited by 85 · see citing works
Moderators of response in exercise treatment for depression: A systematic review
Schuch, F. B., Dunn, A. L., Kanitz, A. C., Delevatti, R. S., & Fleck, M. P. D. A. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.01.014 · Cited by 84 · see citing works
Murillo, C., Vo, T., Vansteelandt, S., Harrison, L. E., Cagnie, B. J., Coppieters, I., Chys, M., Timmers, I., & Meeus, M. (2022) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102160 · Cited by 85 · see citing works
Turner, C., O'Gorman, B., Nair, A., & O'Kearney, R. (2018) · Psychiatry Research · doi:10.1016/j.psychres.2017.12.034 · Cited by 73 · see citing works
Identifying potential moderators for response to treatment in low back pain: A systematic review
Gurung, T., Ellard, D. R., Mistry, D., Patel, S., & Underwood, M. R. (2015) · Physiotherapy · doi:10.1016/j.physio.2015.01.006 · Cited by 72 · see citing works
Self-citationMindfulness-based interventions among people of color: A systematic review and meta-analysis
Sun, S., Goldberg, S. B., Loucks, E. B., & Brewer, J. A. (2022) · Psychotherapy Research · doi:10.1080/10503307.2021.1937369 · Cited by 64 · see citing works
“Following the methodological criteria and recommendations for assessment of moderators in systematic reviews of randomized controlled trials (RCTs) (Pincus et al., 2011), moderator analyses are necessary to inform future clinical trials.”
MethodMenchetti, M., Rucci, P., Bortolotti, B., Bombi, A., Scocco, P., KRAEMER, H. C., Berardi, D., & the DEPICS group (2013) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.112.122663 · Cited by 62 · see citing works
Sentinel node biopsy for prostate cancer: report from a consensus panel meeting
van der Poel, H. G., Wit, E. M. K., Acar, C., van den Berg, N. S., van Leeuwen, F. W. B., Olmos, R. A. V., Winter, A., Wawroschek, F., Liedberg, F., MacLennan, S., et al. (2017) · British Journal of Urology · doi:10.1111/bju.13810 · Cited by 57 · see citing works
Beukenhorst, A. L., Schultz, D. M., McBeth, J., Sergeant, J. C., & Dixon, W. G. (2019) · Pain · doi:10.1097/j.pain.0000000000001776 · Cited by 55 · see citing works
Rahman, A., Sikander, S., Malik, A., Ahmed, I., Tomenson, B., & Creed, F. H. (2012) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.112.109207 · Cited by 54 · see citing works
Galdas, P., Fell, J., Bower, P., Kidd, L., Blickem, C., McPherson, K. E., Hunt, K., Gilbody, S. M., & Richardson, G. A. (2015) · BMJ Open · doi:10.1136/bmjopen-2014-006620 · Cited by 53 · see citing works
Reser, M. P., Slikboer, R., & Rossell, S. L. (2019) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867419853348 · Cited by 53 · see citing works
de Zoete, A., Rubinstein, S. M., de Boer, M. R., Ostelo, R., Underwood, M. R., Hayden, J. A., Buffart, L. M., van Tulder, M. W., Brønfort, G., Foster, N. E., et al. (2021) · Physiotherapy · doi:10.1016/j.physio.2021.03.006 · Cited by 52 · see citing works
Self-citationTrompetter, H. R., Bohlmeijer, E. T., Lamers, S. M. A., & Schreurs, K. M. G. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00353 · Cited by 51 · see citing works
“Both moderators and predictors of change should be measured prior to treatment randomization (Turner et al., 2007; Pincus et al., 2011).”
BackgroundInfluentialPots, W. T. M., Trompetter, H. R., Schreurs, K. M. G., & Bohlmeijer, E. T. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-0841-6 · Cited by 46 · see citing works
“However, our study is in accordance with the methodological requirements of exploratory moderator analyses [68].”
ResultShea, S., Kothari, D., Teresi, J. A., Kong, J., Eimicke, J. P., Lantigua, R., Palmas, W., & Weinstock, R. S. (2013) · American Journal of Public Health · doi:10.2105/ajph.2012.300909 · Cited by 46 · see citing works
Patel, S., Friede, T., Froud, R., Evans, D. W., & Underwood, M. R. (2012) · Spine · doi:10.1097/brs.0b013e31827b158f · Cited by 46 · see citing works
Self-citationAverage impact and important features of onboard eco-driving feedback: A meta-analysis
Sanguinetti, A., Queen, E., Yee, C., & Akanesuvan, K. (2020) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2020.02.010 · Cited by 47 · see citing works
Peters, A., Sylvia, L. G., da Silva Magalhães, P. V., Miklowitz, D. J., Frank, E., Otto, M. W., Hansen, N. S., Dougherty, D. D., Berk, M., Nierenberg, A. A., et al. (2014) · Psychological Medicine · doi:10.1017/s0033291714000804 · Cited by 41 · see citing works
Taylor, S. J., Carnes, D., Homer, K., Pincus, T., Kahan, B. C., Hounsome, N., Eldridge, S., Spencer, A., Diaz-Ordaz, K., Rahman, A., et al. (2016) · Programme Grants for Applied Research · doi:10.3310/pgfar04140 · Cited by 44 · see citing works
Self-citationIdentifying Treatment Effect Modifiers in the STarT Back Trial: A Secondary Analysis
Beneciuk, J. M., Hill, J. C., Campbell, P., Afolabi, E., George, S. Z., Dunn, K., & Foster, N. E. (2016) · Journal of Pain · doi:10.1016/j.jpain.2016.10.002 · Cited by 41 · see citing works
Avcı, Ş. K., Çoklar, A. N., & İstanbullu, A. (2019) · TED EĞİTİM VE BİLİM · doi:10.15390/eb.2019.7969 · Cited by 39 · see citing works
A meta-analytic investigation of the impact of curiosity-enhancing interventions
Schutte, N. S., & Malouff, J. M. (2022) · Current Psychology · doi:10.1007/s12144-022-03107-w · Cited by 41 · see citing works
Bruce, J., Hossain, A., Lall, R., Withers, E. J., Finnegan, S., Underwood, M. R., Ji, C., Bojke, C., Longo, R., Hulme, C. T., et al. (2021) · Health Technology Assessment · doi:10.3310/hta25340 · Cited by 39 · see citing works
Self-citationOliveira, A., Fidalgo, A., Farinatti, P., & Monteiro, W. (2024) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2024.105451 · Cited by 36 · see citing works
Hayward, R., & Stynes, S. M. (2020) · Musculoskeletal Care · doi:10.1002/msc.1533 · Cited by 36 · see citing works
Quicke, J. G., Runhaar, J., van der Windt, D., Healey, E. L., Foster, N. E., & Holden, M. A. (2020) · Osteoarthritis and Cartilage Open · doi:10.1016/j.ocarto.2020.100113 · Cited by 33 · see citing works
Kautz‐Turnbull, C., & Petrenko, C. L. M. (2021) · Alcoholism Clinical and Experimental Research · doi:10.1111/acer.14728 · Cited by 33 · see citing works
Physical Exercise for Late‐Life Depression: Customizing an Intervention for Primary Care
Zanetidou, S., Murri, M. B., Menchetti, M., Toni, G., Asioli, F., Bagnoli, L., Zocchi, D., Siena, M., Assirelli, B., Luciano, C., et al. (2016) · Journal of the American Geriatrics Society · doi:10.1111/jgs.14525 · Cited by 33 · see citing works
Mistry, D., Patel, S., Hee, S. W., Stallard, N., & Underwood, M. R. (2014) · Spine · doi:10.1097/brs.0000000000000231 · Cited by 32 · see citing works
Self-citationBruce, J., Lall, R., Withers, E. J., Finnegan, S., Underwood, M. R., Hulme, C. T., Sheridan, R., Skelton, D. A., Martin, F. C., & Lamb, S. E. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-009362 · Cited by 30 · see citing works
Self-citationThe development of CHAMP: a checklist for the appraisal of moderators and predictors
van Hoorn, R., Tummers, M., Booth, A., Gerhardus, A., Rehfuess, E. A., Hind, D., Bossuyt, P. M., Welch, V. A., Debray, T. P. A., Underwood, M. R., et al. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0451-0 · Cited by 28 · see citing works
Self-citationPatel, S., Hee, S. W., Mistry, D., Jordan, J., Brown, S., Dritsaki, M., Ellard, D. R., Friede, T., Lamb, S. E., Lord, J. L., et al. (2016) · Programme Grants for Applied Research · doi:10.3310/pgfar04100 · Cited by 26 · see citing works
Self-citationPirone, C., Pinto, C. M., van der Windt, D., Parker, B., O’Sullivan, M., & Bruce, I. N. (2017) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2017.04.010 · Cited by 27 · see citing works
McDonald, S., Melkonian, M., Karin, E., Dear, B. F., Titov, N., & Wootton, B. M. (2023) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465823000103 · Cited by 26 · see citing works
TAKING PATIENT HETEROGENEITY AND PREFERENCES INTO ACCOUNT IN HEALTH TECHNOLOGY ASSESSMENTS
Kievit, W., Tummers, M., van Hoorn, R., Booth, A., Mozygemba, K., Refolo, P., Sacchini, D., Pfadenhauer, L. M., Gerhardus, A., & van der Wilt, G. J. (2017) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462317000885 · Cited by 26 · see citing works
Meta-Analytic and Meta-Thematic Analysis of Digital Storytelling Method
Talan, T. (2021) · Bartın University Journal of Faculty of Education · doi:10.14686/buefad.706231 · Cited by 26 · see citing works
de Zoete, A., de Boer, M. R., Rubinstein, S. M., van Tulder, M. W., Underwood, M. R., Hayden, J. A., Buffart, L. M., & Ostelo, R. (2020) · Spine · doi:10.1097/brs.0000000000003814 · Cited by 25 · see citing works
Self-citationSchandelmaier, S., Chang, Y., Devasenapathy, N., Devji, T., Kwong, J. S. W., Colunga‐Lozano, L. E., Lee, Y., Agarwal, A., Bhatnagar, N., Ewald, H., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.05.014 · Cited by 24 · see citing works
Cerón, S. C., Bellón, J. Á., Motrico, E., Campos-Paíno, H., Gómez, C. M., Ebert, D. D., Buntrock, C., Gili, M., & Moreno-Peral, P. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101859 · Cited by 25 · see citing works
Roseen, E. J., Gerlovin, H., Felson, D. T., Delitto, A., Sherman, K. J., & Saper, R. B. (2020) · Pain Medicine · doi:10.1093/pm/pnaa153 · Cited by 25 · see citing works
For whom does interpersonal psychotherapy work? A systematic review
Bernecker, S. L., Coyne, A. E., Constantino, M. J., & Ravitz, P. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.07.001 · Cited by 24 · see citing works
Underwood, M., Mistry, D., Lall, R., & Lamb, S. (2011) · Arthritis Care & Research · doi:10.1002/acr.20518 · Cited by 23 · see citing works
Self-citationGaldas, P., Darwin, Z. J., Fell, J., Kidd, L., Bower, P., Blickem, C., McPherson, K. E., Hunt, K., Gilbody, S. M., & Richardson, G. A. (2015) · Health Services and Delivery Research · doi:10.3310/hsdr03340 · Cited by 21 · see citing works
Kenning, C., Coventry, P. A., & Bower, P. (2014) · Journal of Comorbidity · doi:10.15256/joc.2014.4.33 · Cited by 21 · see citing works
A Procedure and Guidelines for Analyzing Groups of Software Engineering Replications
Santos, A., Vegas, S., Oivo, M., & Juristo, N. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2935720 · Cited by 21 · see citing works
Spruill, T. M., Williams, O. A., Teresi, J. A., Lehrer, S., Pezzin, L. E., Waddy, S. P., Lazar, R. M., Williams, S. K., Jean‐Louis, G., Ravenell, J. E., et al. (2015) · Trials · doi:10.1186/s13063-015-0605-5 · Cited by 21 · see citing works
Sextl-Plötz, T., Steinhoff, M., Baumeister, H., Cuijpers, P. J., Ebert, D. D., & Zarski, A. (2024) · Internet Interventions · doi:10.1016/j.invent.2024.100760 · Cited by 20 · see citing works
Effect of comorbid migraine on propranolol efficacy for painful TMD in a randomized controlled trial
Tchivileva, I. E., Ohrbach, R. K., Fillingim, R. B., Lim, P. F., Giosia, M. D., Ribeiro-Dasilva, M., Campbell, J. H., Hadgraft, H., Willis, J., Arbes, S. J., et al. (2021) · Cephalalgia · doi:10.1177/0333102421989268 · Cited by 20 · see citing works
Incidental Vocabulary Acquisition Through Captioned Viewing: A Meta‐Analysis
Kurokawa, S., Hein, A. M., & Uchihara, T. (2024) · Language Learning · doi:10.1111/lang.12697 · Cited by 19 · see citing works
Nee, R. J., Vicenzino, B., Jull, G., Cleland, J. A., & Coppieters, M. (2013) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2013.4490 · Cited by 24 · see citing works
Physical activity levels among people with fibromyalgia: A systematic review and meta‐analysis
Vancampfort, D., Damme, T. V., McGrath, R. L., Machado, V. A., & Schuch, F. B. (2023) · Musculoskeletal Care · doi:10.1002/msc.1771 · Cited by 18 · see citing works
Meltzer, C. J., Irish, J. C., Angelos, P., Busaidy, N. L., Davies, L., Dwojak, S. M., Ferris, R. L., Haugen, B. R., Harrell, R. M., Haymart, M. R., et al. (2018) · Head & Neck · doi:10.1002/hed.25526 · Cited by 17 · see citing works
Shields, C. V., Hultstrand, K. V., West, C. E., Gunstad, J., & Sato, A. F. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192013384 · Cited by 16 · see citing works
Waste not, want not: call to action for spinal manipulative therapy researchers
Aspinall, S. L., Nim, C. G., Hartvigsen, J., Cook, C. E., Skillgate, E., Vogel, S. K., Hohenschurz‐Schmidt, D., Underwood, M. R., & Rubinstein, S. M. (2024) · Chiropractic & Manual Therapies · doi:10.1186/s12998-024-00539-y · Cited by 16 · see citing works
Self-citationMetcalfe, A. J., Mannion, E. G., Parsons, H. M., Brown, J., Parsons, N., Fox, J., Kearney, R. S., Lawrence, T., Bush, H., McGowan, K., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-036829 · Cited by 16 · see citing works
Self-citationOn behalf of the Repository Group, Hee, S. W., Mistry, D., Friede, T., Lamb, S. E., Stallard, N., Underwood, M. R., & Patel, S. (2021) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-021-04028-8 · Cited by 14 · see citing works
Self-citationMoreno-Peral, P., Bellón, J. Á., Motrico, E., Campos-Paíno, H., Gómez, C. M., Ebert, D. D., Buntrock, C., Roca, M., & Cerón, S. C. (2020) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2020.102317 · Cited by 14 · see citing works
Fors, M., Öberg, B. E., Lindbäck, Y., Enthoven, P., & Abbott, A. D. (2020) · Clinical Journal of Pain · doi:10.1097/ajp.0000000000000901 · Cited by 13 · see citing works
Impact of Using History of Mathematics on Students’ Mathematics Attitude: A Meta-Analysis Study
Bütüner, S. Ö. (2015) · European Journal of Science and Mathematics Education · doi:10.30935/scimath/9442 · Cited by 13 · see citing works
Evaluating ACT Processes in Relation to Outcome in Self-Help Treatment for Anxiety-Related Problems.
Ritzert, T. R., Berghoff, C. R., Tifft, E. D., & Forsyth, J. P. (2020) · Behavior Modification · doi:10.1177/0145445519855616 · Cited by 11 · see citing works
Pradhan, S., Chiu, S., Burton, C. L., Forsyth, J. J., Paskins, Z., van der Windt, D., & Babatunde, O. O. (2022) · Physical Therapy · doi:10.1093/ptj/pzac032 · Cited by 12 · see citing works
Li, B., Gao, S., Bao, W., & Li, M. (2022) · Frontiers in Endocrinology · doi:10.3389/fendo.2022.972954 · Cited by 12 · see citing works
Vousoura, E., Gergov, V., Tulbure, B. T., Camilleri, N., Saliba, A., Garcia-Lopez, L., Podina, I. R., Prevendar, T., Löffler-Stastka, H., Chiarenza, G. A., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01788-1 · Cited by 11 · see citing works
Hahne, A. J., Ford, J. J., Richards, M. C., Surkitt, L. D., Chan, A. Y. P., Slater, S. L., & Taylor, N. F. (2017) · Spine · doi:10.1097/brs.0000000000002148 · Cited by 11 · see citing works
Predictors of Pharmacotherapy Response in Generalized Anxiety Disorder: A Systematic Review
Ferreira-Garcia, R., Mochcovitch, M. D., do Cabo, M. C., Nardi, A. E., & Freire, R. C. R. (2017) · Harvard Review of Psychiatry · doi:10.1097/hrp.0000000000000127 · Cited by 11 · see citing works
Edwards, R. C., Vieyra, Y., & Hans, S. L. (2019) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2019.08.001 · Cited by 11 · see citing works
Nielsen, S. M., Rasmussen, M. U., Boers, M., van der Windt, D., de Wit, M. P. T., WOODWORTH, T. G., Flurey, C., Beaton, D., Shea, B., Escorpizo, R. S., et al. (2020) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2020-217895 · Cited by 10 · see citing works
Levels of sedentary behaviour in people with fibromyalgia: a systematic review and meta-analysis
Vancampfort, D., Damme, T. V., Machado, V. A., McGrath, R. L., Stubbs, B., & Schuch, F. B. (2023) · Disability and Rehabilitation · doi:10.1080/09638288.2023.2214379 · Cited by 12 · see citing works
Berg, J., Grant, R., Siervo, M., Stephan, B. C. M., & Tully, P. J. (2025) · Nutrition Reviews · doi:10.1093/nutrit/nuaf155 · Cited by 9 · see citing works
Elias, A. L., Seward, N., & Lund, C. (2024) · Cambridge Prisms Global Mental Health · doi:10.1017/gmh.2024.3 · Cited by 9 · see citing works
Pinto, C. M., Pirone, C., van der Windt, D., Parker, B., & Bruce, I. N. (2017) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2017.01.009 · Cited by 9 · see citing works
McCusker, J., Cole, M. G., Lambert, S., Yaffe⃰, M. J., Ciampi, A., & Belzile, É. (2016) · The Canadian Journal of Psychiatry · doi:10.1177/0706743716648299 · Cited by 9 · see citing works
Goger, P., Guo, R., Jo, S., Cha, C. B., Ribeiro, J. D., Huang, X., & Fox, K. R. (2024) · Behavior Therapy · doi:10.1016/j.beth.2024.10.003 · Cited by 9 · see citing works
Mackenzie, R. M., Ells, L. J., Simpson, S. A., & Logue, J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-025193 · Cited by 8 · see citing works
Roberge, P., Houle, J., Provost, J., Coulombe, S., Beaudin, A., Bower, P., Lemyre, F. C., Drapeau, M., Drouin, M., Hudon, C., et al. (2022) · BMC Psychiatry · doi:10.1186/s12888-021-03675-4 · Cited by 8 · see citing works
Bennett, M. L., McClure, A., Khalesi, H., Vu, M. T. N., Feng, J., Reupert, A., & Yap, M. B. H. (2025) · Journal of Affective Disorders · doi:10.1016/j.jad.2025.120085 · Cited by 7 · see citing works
de Zoete, A., de Boer, M. R., van Tulder, M. W., Rubinstein, S. M., Underwood, M. R., Hayden, J. A., Kalter, J., & Ostelo, R. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0413-y · Cited by 7 · see citing works
Self-citationIndicaciones de la resonancia magnética en la lumbalgia de adultos
Ortuondo, E. M., Zubizarreta, A., Saitua, J. M., Sarabia, C. S., & Arratibel, J. Z. (2013) · Revista de Calidad Asistencial · doi:10.1016/j.cali.2013.10.004 · Cited by 7 · see citing works
Major, A., Paje, D. G., Taxbro, K., McQuilten, Z., Kin, A. D., Alexandrou, E., Hsaiky, L. M., Hill, J. G., Moss, J. G., Kamboj, M., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-25-02523 · Cited by 6 · see citing works
Beneciuk, J. M., George, S. Z., Patterson, C. G., Smith, C., Brennan, G. P., Wegener, S. T., Roseen, E. J., Saper, R. B., & Delitto, A. (2022) · Pain · doi:10.1097/j.pain.0000000000002679 · Cited by 8 · see citing works
Zadro, J. R., Shirley, D., Nilsen, T. I. L., Mork, P. J., & Ferreira, P. H. (2020) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2020.03.019 · Cited by 6 · see citing works
Predictors and moderators of treatment outcome in late-life anxiety: A systematic review
Kyriacou, T., Hodges, J., & Gould, R. L. (2023) · Journal of Affective Disorders · doi:10.1016/j.jad.2023.07.057 · Cited by 6 · see citing works
Rodziewicz, M., Pinto, C. M., Dyball, S., Munguía‐Realpozo, P., Parker, B., & Bruce, I. N. (2023) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2023.152346 · Cited by 5 · see citing works
Dissing, K. B., Vach, W., Hartvigsen, J., Wedderkopp, N., & Hestbæk, L. (2019) · Chiropractic & Manual Therapies · doi:10.1186/s12998-019-0282-7 · Cited by 5 · see citing works
Symptom Dimensions and Trajectories of Functioning Among Bipolar Youth: A Cluster Analysis
Peters, A. T., Weinstein, S. M., Isaia, A., Meter, A. V., Zulauf-McCurdy, C. A., & West, A. E. (2018) · Journal of Psychiatric Practice · doi:10.1097/pra.0000000000000307 · Cited by 4 · see citing works
Holm, L. V., Vach, W., Jarbøl, D. E., Christensen, H. W., Søndergaard, J., & Hestbæk, L. (2021) · Chiropractic & Manual Therapies · doi:10.1186/s12998-021-00373-6 · Cited by 4 · see citing works
Metcalfe, A. J., Arnold, S., Parsons, H. M., Parsons, N., Bhabra, G., Brown, J., Bush, H., Diokno, M., Elliott, M. T., Fox, J., et al. (2023) · Efficacy and Mechanism Evaluation · doi:10.3310/tkjy2101 · Cited by 4 · see citing works
Self-citationEvidence Based Medicine vs. Expert Consensus in Medical Guidelines: An Artificial Conflict
Levy, R. M. (2017) · Neuromodulation Technology at the Neural Interface · doi:10.1111/ner.12578 · Cited by 4 · see citing works
- Authors
- Pincus, T., Miles, C., Froud, R., Underwood, M., Carnes, D., & Taylor, S. J.
- Affiliations
- University of London (GB)
- Queen Mary University of London (GB)
- University of Warwick (GB)
- Corresponding author
- Pincus, Tamar
- Venue
- BMC Medical Research Methodology
- Published
- 2011-01-31
- Volume
- 11
- Issue
- 1
- Pages
- 14-14
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC3044921
- OpenAlex
- W2111680408
- Cited by
- 161
- References
- 28
- Citations by year
- 2026: 14 · 2025: 15 · 2024: 9 · 2023: 22 · 2022: 8 · 2021: 14 · 2020: 14 · 2019: 10 · 2018: 2 · 2017: 15 · 2016: 10 · 2015: 9 · 2014: 7 · 2013: 8 · 2012: 2
- FWCI
- 1.48
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ePrints Soton (University of Southampton) (submittedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion, cc-by)
- Royal Holloway Digital Repository (Royal Holloway University of London) (submittedVersion)
- Royal Holloway Digital Repository (Royal Holloway University of London) (submittedVersion, cc-by)
- Funders
- Arizona State University
- University of Washington
- Université Laval
- Queen Mary University of London
- Advantage West Midlands
- National Institute for Health and Care Research (RP-PG-0707-10189)
- Keele University
- Programme Grants for Applied Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- moderators, subgroup analysis, randomized controlled trials, methodological quality criteria, level of evidence, interaction testing, consensus recommendations
- MeSH terms
- Humans, Review Literature as Topic, Sampling Studies, Meta-Analysis as Topic, Effect Modifier, Epidemiologic, Randomized Controlled Trials as Topic, Databases, Bibliographic, Treatment Outcome, Consensus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
