AHRQ series on complex intervention systematic reviews–paper 2: defining complexity, formulating scope, and questions
Open accessAbstract
BACKGROUND: The early stages of a systematic review set the scope and expectations. This can be particularly challenging for complex interventions given their multidimensional and dynamic nature. RATIONALE: This paper builds on concepts introduced in paper 1 of this series. It describes the methodological, practical, and philosophical challenges and potential approaches for formulating the questions and scope of systematic reviews of complex interventions. Furthermore, it discusses the use of theory to help organize reviews of complex interventions. DISCUSSION: Many interventions in medicine, public health, education, social services, behavioral health, and community programs are complex, and they may not fit neatly within the established paradigm for reviews of straightforward interventions. This paper provides conceptual and operational guidance for these early stages of scope formulation to assist authors of systematic reviews of complex interventions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The results, in particular the wide prediction intervals plus the moderation by study location, suggest that cultural and social differences do determine the extent to which MBPs are beneficial.
“The results obtained, in particular the wide prediction intervals plus the moderation by study location, suggest that cultural and social differences do determine the extent to which MBPs are beneficial [40].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Systematic Reviews in Health Care : Meta-Analysis in Context
Egger, M., Smith, G. D., & Altman, D. G. (2000) · Cited by 2,312 · see citing works
Systematic Reviews in Health Care: Meta‐analysis in Context
Pittler, M. (2010) · Focus on Alternative and Complementary Therapies · doi:10.1111/j.2042-7166.2002.tb05479.x · Cited by 1,639 · see citing works
Successes and Failures in the Implementation of Evidence-Based Guidelines for Clinical Practice
Grol, R. (2001) · Medical Care · doi:10.1097/00005650-200108002-00003 · Cited by 1,233 · see citing works
Epidemiology and the people's health: theory and context
(2011) · Choice Reviews Online · doi:10.5860/choice.49-0897 · Cited by 485 · see citing works
Evidence-based policy: a realist perspective
(2005) · Routledge eBooks · doi:10.4324/9780203624289-9 · Cited by 384 · see citing works
Using logic models to capture complexity in systematic reviews: Logic Models in Systematic Reviews
Anderson, L. M., Petticrew, M., Rehfuess, E., Armstrong, R., Ueffing, E., Baker, P., Francis, D., & Tugwell, P. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.32 · Cited by 372 · see citing works
Integration of mental health/substance abuse and primary care.
(2008) · PubMed Central · Cited by 346 · see citing works
Petticrew, M. P., Rehfuess, E. A., Noyes, J., Higgins, J. P. T., Mayhew, A., Tomás Pantoja, C., Shemilt, I., & Sowden, A. J. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.005 · Cited by 292 · see citing works
Self-citationApplications of text mining within systematic reviews
Thomas, J., McNaught, J., & Ananiadou, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.27 · Cited by 212 · see citing works
Complex interventions and their implications for systematic reviews: a pragmatic approach
Petticrew, M. P., Anderson, L. M., Elder, R. W., Grimshaw, J., Hopkins, D. P., Hahn, R. A., Krause, L. K., Kristjansson, E., Mercer, S. L., Sipe, T., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.004 · Cited by 204 · see citing works
Kelly, M., Morgan, A., Ellis, S., Younger, T., Huntley, J., & Swann, C. (2010) · Social Science & Medicine · doi:10.1016/j.socscimed.2010.06.032 · Cited by 182 · see citing works
Self-citationTime to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’
Petticrew, M. P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0027-1 · Cited by 160 · see citing works
Integration of mental health/substance abuse and primary care
(2008) · PsycEXTRA Dataset · doi:10.1037/e508052009-001 · Cited by 153 · see citing works
Guise, J., Chang, C., Butler, M., Viswanathan, M., & Tugwell, P. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.06.011 · Cited by 125 · see citing works
Self-citationA practice-based tool for engaging stakeholders in future research: a synthesis of current practices
Guise, J., O’Haire, C., McPheeters, M. L., Most, C., LaBrant, L., Lee, K., Cottrell, E., & Graham, E. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.12.010 · Cited by 103 · see citing works
Self-citationBhan, N. (2013) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12026 · Cited by 103 · see citing works
Investigating complexity in systematic reviews of interventions by using a spectrum of methods
Anderson, L. M., Oliver, S. R., Michie, S., Rehfuess, E., Noyes, J., & Shemilt, I. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.014 · Cited by 101 · see citing works
Self-citationA conceptual framework for public health: NICE's emerging approach
Kelly, M. P., Stewart, E., Morgan, A. R., Killoran, A., Fischer, A., Threlfall, A., & Bonnefoy, J. (2008) · Public Health · doi:10.1016/j.puhe.2008.10.031 · Cited by 98 · see citing works
Self-citationSystematic reviews of complex interventions: framing the review question
Squires, J. E., Valentine, J. C., & Grimshaw, J. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.05.013 · Cited by 96 · see citing works
Burford, B. J., Lewin, S. A., Welch, V. A., Rehfuess, E. A., & Waters, E. B. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.017 · Cited by 82 · see citing works
Identifying, documenting, and examining heterogeneity in systematic reviews of complex interventions
Pigott, T., & Shepperd, S. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.013 · Cited by 76 · see citing works
A taxonomy of rapid reviews links report types and methods to specific decision-making contexts
Hartling, L., Guise, J., Kato, E. U., Anderson, J., Belinson, S., Berliner, E., Dryden, D. M., Featherstone, R. M., Mitchell, M. D., Motu’apuaka, M. L., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.036 · Cited by 75 · see citing works
Self-citationNoyes, J., Gough, D., Lewin, S. A., Mayhew, A., Michie, S., Tomás Pantoja, C., Petticrew, M. P., Pottie, K. C., Rehfuess, E. A., Shemilt, I., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.07.003 · Cited by 75 · see citing works
Self-citationAslam, R. W., Hendry, M., Booth, A., Carter, B., Charles, J. M., Craine, N., Edwards, R. T., Noyes, J., Ntambwe, L. I., Pasterfield, D., et al. (2017) · BMC Medicine · doi:10.1186/s12916-017-0904-7 · Cited by 71 · see citing works
Self-citationAnderson, L. M., Petticrew, M. P., Chandler, J., Grimshaw, J., Tugwell, P., O’Neill, J., Welch, V. A., Squires, J. E., Churchill, R. C., & Shemilt, I. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.07.005 · Cited by 69 · see citing works
EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews
Hartling, L., Guise, J., Kato, E. U., Anderson, J., Aronson, N., Belinson, S., Berliner, E., Dryden, D. M., Featherstone, R. M., Foisy, M., et al. (2015) · Cited by 62 · see citing works
Self-citationDoes Integrated Care Improve Treatment for Depression?
Butler, M. E., Kane, R. L., McAlpine, D. D., Kathol, R. G., Fu, S. S., Hagedorn, H., & Wilt, T. J. (2011) · Journal of Ambulatory Care Management · doi:10.1097/jac.0b013e31820ef605 · Cited by 60 · see citing works
Self-citationNoyes, J., Hendry, M., Booth, A., Chandler, J., Lewin, S. A., Glenton, C., & Garside, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.009 · Cited by 58 · see citing works
Self-citationGuise, J., Chang, C., Viswanathan, M., Glick, S., Treadwell, J. R., Umscheid, C. A., Whitlock, E. P., Fu, R. F., Berliner, E., Paynter, R. A., et al. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.06.010 · Cited by 56 · see citing works
Self-citationTurley, R., Saith, R., Bhan, N., Rehfuess, E. A., & Carter, B. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010067 · Cited by 55 · see citing works
The judgement process in evidence-based medicine and health technology assessment
Kelly, M. P., & Moore, T. A. (2011) · Social Theory & Health · doi:10.1057/sth.2011.21 · Cited by 52 · see citing works
Self-citationButler, M., Epstein, R. A., Totten, A., Whitlock, E. P., Ansari, M. T., Damschroder, L. J., Balk, E., Bass, E. B., Berkman, N. D., Hempel, S., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.06.013 · Cited by 51 · see citing works
Self-citationThe Integration of Social, Behavioral, and Biological Mechanisms in Models of Pathogenesis
Kelly, M. P., Kelly, R. S., & Federica Russo, B. (2014) · Perspectives in biology and medicine · doi:10.1353/pbm.2014.0026 · Cited by 44 · see citing works
Self-citationWhitaker, R., Hendry, M., Booth, A., Carter, B., Charles, J. M., Craine, N., Edwards, R., Lyons, M., Noyes, J., Pasterfield, D., et al. (2014) · BMJ Open · doi:10.1136/bmjopen-2013-004733 · Cited by 20 · see citing works
Self-citationBuckley, D., Ansari, M., Butler, M. E., et al. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.10.023 · Cited by 18 · see citing works
Self-citationTurley, R., Saith, R., Bhan, N., Doyle, J., Jones, K., & Waters, E. (2013) · Journal of Public Health · doi:10.1093/pubmed/fdt008 · Cited by 18 · see citing works
Hurley, M., Dickson, K., Walsh, N. E., Hauari, H., Grant, R., Cumming, J., & Oliver, S. R. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010842 · Cited by 17 · see citing works
Excess winter deaths and morbidity and the health risks associated with cold homes
Billeter, J. (2015) · Cited by 12 · see citing works
Guidance for using the iCAT_SR: Intervention Complexity Assessment Tool for Systematic Reviews
Lewin, S. A., Hendry, M., Chandler, J., et al. (2017) · Lirias · Cited by 7 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Kay N, Pennucci A. Early childhood education for low-income students: a review of the evidence and benefit-cost analysis (Doc. No. 14-01-2201). Olympia: Washington State Institute for Public Policy; 2014.
- Nunes V, Neilson J, O'Flynn N, Calvert N, Kuntze S, Smithson H, et al. Clinical Guidelines and Evidence Review for Medicines Adher- ence: involving patients in decisions about prescribed medicines and supporting adherence. London: National Collaborating Centre for Primary Care and Royal College of General Practitioners; 2009.
- Butler M, Epstein RA, Totten A, Whitlock EP, Ansari MT, Damschroder LJ, et al. AHRQ series on complex intervention sys- tematic reviewsdpaper 3: adapting frameworks to develop protocols. J Clin Epidemiol 2017;90:19e27.
Cited by
Sorted by citations
Noyes, J., Booth, A., Moore, G., Flemming, K., Tunçalp, Ö., & Shakibazadeh, E. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000893 · Cited by 497 · see citing works
Self-citationFlemming, K., Booth, A., Garside, R., Tunçalp, Ö., & Noyes, J. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000882 · Cited by 389 · see citing works
Self-citationGalante, J., Friedrich, C., Dawson, A. F., Modrego-Alarcón, M., Gebbing, P., Delgado-Suárez, I., Gupta, R., Dean, L., Dalgleish, T., White, I. R., et al. (2021) · PLOS Medicine · doi:10.1371/journal.pmed.1003481 · Cited by 319 · see citing works
“The results obtained, in particular the wide prediction intervals plus the moderation by study location, suggest that cultural and social differences do determine the extent to which MBPs are beneficial [40].”
MethodResultFormulating questions to explore complex interventions within qualitative evidence synthesis
Booth, A., Noyes, J., Flemming, K., Moore, G., Tunçalp, Ö., & Shakibazadeh, E. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-001107 · Cited by 258 · see citing works
Self-citationvan Eikenhorst, L., Taxis, K., van Dijk, L., & de Gier, H. (2017) · Frontiers in Pharmacology · doi:10.3389/fphar.2017.00891 · Cited by 121 · see citing works
Butler, M., Epstein, R. A., Totten, A., Whitlock, E. P., Ansari, M. T., Damschroder, L. J., Balk, E., Bass, E. B., Berkman, N. D., Hempel, S., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.06.013 · Cited by 51 · see citing works
Self-citation
- Authors
- Kelly, M. P., Noyes, J., Kane, R. L., Chang, C., Uhl, S., Robinson, K. A., Springs, S., Butler, M. E., & Guise, J.-M.
- Affiliations
- University of Cambridge (GB)
- Bangor University (GB)
- University of Minnesota (US)
- Agency for Healthcare Research and Quality (US)
- ECRI Institute (US)
- Johns Hopkins University (US)
- Brown University (US)
- Portland State University (US)
- Corresponding author
- Kelly, Michael P.
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2017-10
- Volume
- 90
- Pages
- 11-18
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2734473338
- Cited by
- 49
- References
- 39
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 1 · 2023: 2 · 2022: 5 · 2021: 13 · 2020: 5 · 2019: 8 · 2018: 1 · 2017: 6
- FWCI
- 1.35
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PDXScholar (Portland State University) (submittedVersion, cc-by-nc-nd)
- Bangor University Research Portal (Bangor University) (submittedVersion, other-oa)
- Apollo (University of Cambridge) (submittedVersion, cc-by-nc-nd)
- Apollo
- Funders
- U.S. Department of Health and Human Services
- University of Minnesota
- Agency for Healthcare Research and Quality (K12HS022998)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Keywords
- complex interventions, research questions, review methodology
- MeSH terms
- Data Interpretation, Statistical, Delphi Technique, Humans, Research Design, Review Literature as Topic, Guidelines as Topic, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
