Investigating complexity in systematic reviews of interventions by using a spectrum of methods
Abstract
Systematic reviews framed by PICOS (Populations, Interventions, Comparisons, Outcomes, and Study designs) have been valuable for synthesizing evidence about the effects of interventions. However, this framework is limited in its utility for exploring the influence of variations within populations or interventions, or about the mechanisms of action or causal pathways thought to mediate outcomes, other contextual factors that might similarly moderate outcomes, or how and when these mechanisms and elements interact. Valuable insights into these issues come from configurative as well as aggregative methods of synthesis. This article considers the range of evidence that can be used in systematic reviews of interventions to investigate complexity in terms of potential sources of variation in interventions and their effects, and presents a continuum of purposes for, and approaches to, evidence synthesis. Choosing an appropriate synthesis method takes into account whether the purpose of the synthesis is to generate, explore, or test theories. Taking complexity into account in a synthesis of economic evidence similarly shifts emphasis from evidence synthesis strategies focused on aggregation toward configurative strategies that aim to develop, explore, and refine (in advance of testing) theories or explanations of how and why interventions are more or less resource intensive, costly or cost-effective in different settings, or when implemented in different ways.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The synthesis methods were not classified as aggregative and configurative.
“the synthesis methods were not classified as aggregative and configurative [45, 46].”
From Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence (2017) · cites this paper as [46]
Cites
Developing and evaluating complex interventions: the new Medical Research Council guidance
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008) · BMJ · doi:10.1136/bmj.a1655 · Cited by 10,995 · see citing works
Self-citationAudit and feedback: effects on professional practice and healthcare outcomes
Ivers, N., Jamtvedt, G., Flottorp, S., Young, J. M., Odgaard-Jensen, J., French, S. D., O'Brien, M. A., Johansen, M., Grimshaw, J., & Oxman, A. D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000259.pub3 · Cited by 5,485 · see citing works
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373 · see citing works
Realist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190 · see citing works
Interventions for enhancing medication adherence
Nieuwlaat, R., Wilczynski, N., Navarro, T., Hobson, N., Jeffery, R., Keepanasseril, A., Agoritsas, T., Mistry, N., Iorio, A., Jack, S., et al. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000011.pub4 · Cited by 2,945 · see citing works
Des Jarlais, D. C., Lyles, C., Crepaz, N., & the TREND Group (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.361 · Cited by 2,817 · see citing works
Developing and Evaluating Complex Interventions: New Guidance
Craig, P., Dieppe, P. A., MacIntyre, S., Michie, S., Nazareth, I., & Petticrew, M. P. (2006) · UCL Discovery (University College London) · Cited by 1,877 · see citing works
Self-citationStandards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination
Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., Mościcki, E. K., Schinke, S., Valentine, J. C., & Ji, P. (2005) · Prevention Science · doi:10.1007/s11121-005-5553-y · Cited by 1,338 · see citing works
Interventions for enhancing medication adherence
Haynes, R. B., Ackloo, E., Sahota, N., McDonald, H. P., & Yao, X. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000011.pub3 · Cited by 1,244 · see citing works
Using meta ethnography to synthesise qualitative research: a worked example
Britten, N., Campbell, R., Pope, C., Donovan, J., Morgan, M., & Pill, R. (2002) · Journal of Health Services Research & Policy · doi:10.1258/135581902320432732 · Cited by 1,110 · see citing works
Reporting standards for research in psychology: why do we need them? What might they be?
APA Publications and Communications Board Working Group on Journal Article Reporting Standards (2008) · American Psychologist · doi:10.1037/0003-066x.63.9.839 · Cited by 571 · see citing works
Integrating qualitative research with trials in systematic reviews
Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R., Brunton, G., & Kavanagh, J. (2004) · BMJ · doi:10.1136/bmj.328.7446.1010 · Cited by 515 · see citing works
Self-citationIntroduction to systematic reviews
O’Connor, E., Whitlock, E. P., & Spring, B. J. (2011) · PsycEXTRA Dataset · doi:10.1037/e657462011-001 · Cited by 463 · see citing works
Carroll, C., Booth, A., & Cooper, K. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-29 · Cited by 461 · see citing works
Generalisability in economic evaluation studies in healthcare: a review and case studies
Sculpher, M. J., Pang, F., Manca, A., Drummond, M., Golder, S., Urdahl, H., Davies, L. M., & Eastwood, A. J. (2004) · Health Technology Assessment · doi:10.3310/hta8490 · Cited by 441 · see citing works
Considering complexity in healthcare systems
Kannampallil, T., Schauer, G. F., Cohen, T., & Patel, V. L. (2011) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.06.006 · Cited by 400 · see citing works
Using logic models to capture complexity in systematic reviews: Logic Models in Systematic Reviews
Anderson, L. M., Petticrew, M., Rehfuess, E., Armstrong, R., Ueffing, E., Baker, P., Francis, D., & Tugwell, P. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.32 · Cited by 372 · see citing works
Self-citationA multidimensional conceptual framework for analysing public involvement in health services research
Oliver, S. R., Rees, R. W., Clarke‐Jones, L., Milne, R., Oakley, A., Gabbay, J., Stein, K., Buchanan, P., & Gyte, G. M. (2008) · Health Expectations · doi:10.1111/j.1369-7625.2007.00476.x · Cited by 344 · see citing works
Self-citationPetticrew, M. P., Rehfuess, E. A., Noyes, J., Higgins, J. P. T., Mayhew, A., Tomás Pantoja, C., Shemilt, I., & Sowden, A. J. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.005 · Cited by 292 · see citing works
Self-citationGardner, B., Whittington, C., McAteer, J., Eccles, M. P., & Michie, S. (2010) · Social Science & Medicine · doi:10.1016/j.socscimed.2010.01.039 · Cited by 225 · see citing works
Self-citationStewart, R. J., van Rooyen, C., Dickson, K., Majoro, M., & de Wet, T. (2010) · IOE EPrints · Cited by 178 · see citing works
Roberts, K. A., Dixon‐Woods, M., Fitzpatrick, R. M., Abrams, K. R., & Jones, D. R. (2002) · The Lancet · doi:10.1016/s0140-6736(02)11560-1 · Cited by 159 · see citing works
Synthesizing Qualitative Research: Choosing the Right Approach
Hannes, K., & Lockwood, C. S. (2011) · Cited by 144 · see citing works
Bhan, N. (2013) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12026 · Cited by 103 · see citing works
Systematic reviews of complex interventions: framing the review question
Squires, J. E., Valentine, J. C., & Grimshaw, J. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.05.013 · Cited by 96 · see citing works
Elaborating on systems thinking in health promotion practice
Naaldenberg, J., Vaandrager, L., Koelen, M. A., Wagemakers, A., Saan, H., & de Hoog, K. (2009) · Global Health Promotion · doi:10.1177/1757975908100749 · Cited by 91 · see citing works
Waters, E., Hall, B. J., Armstrong, R., Doyle, J., Pettman, T. L., & de Silva-Sanigorski, A. (2011) · Journal of Public Health · doi:10.1093/pubmed/fdr064 · Cited by 89 · see citing works
Coren, E., Hossain, R., Pardo, J. P., & Bakker, B. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009823.pub3 · Cited by 87 · see citing works
Water, sanitation and hygiene in developing countries: interventions and diarrhoea—a review
Fewtrell, L., & Colford, J. M. (2005) · Water Science & Technology · doi:10.2166/wst.2005.0244 · Cited by 83 · see citing works
Using qualitative synthesis to explore heterogeneity of complex interventions
Candy, B., King, M., Jones, L., & Oliver, S. R. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-124 · Cited by 78 · see citing works
Self-citationKoehlmoos, T. P., Gazi, R., Hossain, S., et al. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007136.pub2 · Cited by 56 · see citing works
Turley, R., Saith, R., Bhan, N., Rehfuess, E. A., & Carter, B. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010067 · Cited by 55 · see citing works
Self-citationIntroduction to Systematic Reviews
Goodwin, G. M., & Geddes, J. R. (2004) · Journal of Psychopharmacology · doi:10.1177/0269881104042629 · Cited by 46 · see citing works
Evidence‐Based Decisions and Economics
Shemilt, I., Mugford, M., Vale, L. D., et al. (2010) · Wiley eBooks · doi:10.1002/9781444320398 · Cited by 45 · see citing works
Self-citationStakeholder Perspectives and Participation in Reviews
Rees, R., & Oliver, S. R. (2017) · SAGE Publications Ltd eBooks · doi:10.4135/9781036234942.n2 · Cited by 32 · see citing works
Self-citationThe Role of Economic Perspectives and Evidence in Systematic Review
Anderson, R., & Shemilt, I. (2010) · Wiley eBooks · doi:10.1002/9781444320398.ch3 · Cited by 29 · see citing works
Self-citationCoren, E., Hossain, R., Pardo, J. P., Véras, M. M. S., Chakraborty, K., Harris, H., & Martin, A. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009823.pub2 · Cited by 27 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The need for a complex systems model of evidence for public health
Rutter, H., Savona, N., Glonti, K., Bibby, J., Cummins, S., Finegood, D. T., Greaves, F., Harper, L., Hawe, P., Moore, L., et al. (2017) · The Lancet · doi:10.1016/s0140-6736(17)31267-9 · Cited by 1,260 · see citing works
Self-citationA systematic approach to searching: an efficient and complete method to develop literature searches
Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018) · Journal of the Medical Library Association · doi:10.5195/jmla.2018.283 · Cited by 863 · see citing works
Hong, Q. N., Pluye, P., Bujold, M., & Wassef, M. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0454-2 · Cited by 754 · see citing works
“Finally, the synthesis methods were not classified as aggregative and configurative [45, 46].”
MethodHuman resources for health (and rehabilitation): Six Rehab-Workforce Challenges for the century
Jesus, T. S., Landry, M. D., Dussault, G., & Fronteira, I. (2017) · Human Resources for Health · doi:10.1186/s12960-017-0182-7 · Cited by 161 · see citing works
Rohwer, A., Pfadenhauer, L., Burns, J., Brereton, L., Gerhardus, A., Booth, A., Oortwijn, W., & Rehfuess, E. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.06.012 · Cited by 121 · see citing works
Self-citationCzosnek, L., Lederman, O., Cormie, P., Zopf, E., Stubbs, B., & Rosenbaum, S. (2019) · Mental Health and Physical Activity · doi:10.1016/j.mhpa.2018.11.001 · Cited by 94 · see citing works
Meta-Analysis of Complex Interventions
Tanner-Smith, E. E., & Grant, S. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014112 · Cited by 62 · see citing works
Jesus, T. S., & Silva, I. L. (2016) · Clinical Rehabilitation · doi:10.1177/0269215515585133 · Cited by 51 · see citing works
Movsisyan, A., Melendez-Torres, G. J., & Montgomery, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.03.014 · Cited by 50 · see citing works
Kelly, M. P., Noyes, J., Kane, R. L., Chang, C., Uhl, S., Robinson, K. A., Springs, S., Butler, M. E., & Guise, J.-M. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.06.012 · Cited by 49 · see citing works
Self-citationThe wiki toolkit for planning, conducting and reporting mixed studies reviews
Pluye, P., Hong, Q. N., Granikov, V., & Vedel, I. (2018) · Education for Information · doi:10.3233/efi-180220 · Cited by 10 · see citing works
- Authors
- Anderson, L. M., Oliver, S. R., Michie, S., Rehfuess, E., Noyes, J., & Shemilt, I.
- Affiliations
- University of Washington (US)
- University of London (GB)
- University College London (GB)
- Ludwig-Maximilians-Universität München (DE)
- Bangor University (GB)
- University of Cambridge (GB)
- Corresponding author
- Anderson, Laurie M.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2013-11
- Volume
- 66
- Issue
- 11
- Pages
- 1223-1229
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2027121083
- Cited by
- 101
- References
- 37
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 3 · 2023: 4 · 2022: 5 · 2021: 10 · 2020: 5 · 2019: 8 · 2018: 9 · 2017: 19 · 2016: 12 · 2015: 8 · 2014: 8 · 2013: 4
- FWCI
- 2.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- mechanism of action, causal pathways, intervention moderators, cost-effectiveness, theory testing
- MeSH terms
- Humans, Research Design, Review Literature as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review