How to do a good systematic review of effects in international development: a tool kit
Open accessAbstract
We provide a ‘how to’ guide to undertake systematic reviews of effects in international development, by which we mean, synthesis of literature relating to the effectiveness of particular development interventions. Our remit includes determining the review's questions and scope, literature search, critical appraisal, methods of synthesis including meta-analysis, and assessing the extent to which generalisable conclusions can be drawn using a theory-based approach. Our work draws on the experiences of the International Initiative for Impact Evaluation's (3ie's) systematic reviews programme.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This lack of detailed reporting in many publications limited the extent to which a full risk of bias assessment was possible.
“This lack of detailed reporting in many publications limited the extent to which a full risk of bias assessment, for example, based on Waddington and Hombrados (2012), was possible.”
From Interventions to improve the labour market outcomes of youth: A systematic review of training, entrepreneurship promotion, employment services and subsidized employment interventions (2017) · cites this paper as Waddington and Hombrados (2012)
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · see citing works
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799 · see citing works
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · see citing works
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · see citing works
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · see citing works
Hunter, J., & Schmidt, F. (2004) · doi:10.4135/9781412985031 · Cited by 3,023 · see citing works
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955 · see citing works
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · see citing works
Self-citationInterpretation of random effects meta-analyses
Riley, R. D., Higgins, J. P. T., & Deeks, J. J. (2011) · BMJ · doi:10.1136/bmj.d549 · Cited by 2,884 · see citing works
Characterizing Selection Bias Using Experimental Data
Heckman, J., Ichimura, H., Smith, J., & Todd, P. (1998) · Econometrica · doi:10.2307/2999630 · Cited by 2,286 · see citing works
Assessing the impact of planned social change
Campbell, D. T. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90048-x · Cited by 867 · cited 1× · see citing works
“(Campbell 1979, p. 84) What is the evidentiary basis for trends in development policy?”
BackgroundCook, T. D., Shadish, W. R., & Wong, V. C. (2008) · Journal of Policy Analysis and Management · doi:10.1002/pam.20375 · Cited by 604 · see citing works
Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000733 · Cited by 479 · see citing works
Using logic models to capture complexity in systematic reviews: Logic Models in Systematic Reviews
Anderson, L. M., Petticrew, M., Rehfuess, E., Armstrong, R., Ueffing, E., Baker, P., Francis, D., & Tugwell, P. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.32 · Cited by 372 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Systematic review approaches for climate change adaptation research
Berrang-Ford, L., Pearce, T., & Ford, J. D. (2015) · Regional Environmental Change · doi:10.1007/s10113-014-0708-7 · Cited by 474 · see citing works
Viewing systematic reviews and meta-analysis in social research through different lenses
Davis, J., Mengersen, K., Bennett, S., & Mazerolle, L. (2014) · SpringerPlus · doi:10.1186/2193-1801-3-511 · Cited by 364 · see citing works
Influence of land tenure interventions on human well-being and environmental outcomes
Tseng, T.-W. J., Robinson, B. E., Bellemare, M. F., BenYishay, A., Blackman, A., Boucher, T., Childress, M., Holland, M. B., Kroeger, T., Linkow, B., et al. (2020) · Nature Sustainability · doi:10.1038/s41893-020-00648-5 · Cited by 140 · see citing works
Kluve, J., Puerto, S., Robalino, D., Romero, J. M., Rother, F., Stöterau, J., Weidenkaff, F., & Witte, M. (2017) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.4073/csr.2017.12 · Cited by 106 · see citing works
Hennessy, E. A., Johnson, B. T., & Keenan, C. (2019) · Applied Psychology: Health and Well-Being · doi:10.1111/aphw.12169 · Cited by 91 · see citing works
“…explicated in numerous texts (e.g., see Borenstein, Hedges, Higgins, & Rothstein, 2011; Lipsey & Wilson, 2001; The Cochrane Collaboration., 2011; Waddington et al., 2012) and will not be detailed here, nevertheless, there are additional considerations for conducting meta-reviews that should be…”
Background
- Authors
- Waddington, H., White, H., Snilstveit, B., Hombrados, J. G., Vojtkova, M., Davies, P., Bhavsar, A., Eyers, J., Koehlmoos, T. P., Petticrew, M., Valentine, J. C., & Tugwell, P.
- Affiliations
- International Initiative for Impact Evaluation (US)
- London School of Hygiene & Tropical Medicine (GB)
- University of Louisville (US)
- University of Ottawa (CA)
- Corresponding author
- Waddington, Hugh
- ORCID
- Venue
- Journal of Development Effectiveness
- Publisher
- Informa UK Limited
- Published
- 2012-09
- Volume
- 4
- Issue
- 3
- Pages
- 359-387
- ISSN
- 1943-9342 (ISSN-L)
- 1943-9407
- OpenAlex
- W2081468367
- Cited by
- 278
- References
- 16
- Citations by year
- 2026: 9 · 2025: 16 · 2024: 20 · 2023: 31 · 2022: 21 · 2021: 24 · 2020: 13 · 2019: 19 · 2018: 28 · 2017: 36 · 2016: 11 · 2015: 17 · 2014: 15 · 2013: 11 · 2012: 7
- FWCI
- 4.12
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Licence
- cc-by
- Locations
- Journal of Development Effectiveness (publishedVersion)
- Sussex Research Online (University of Sussex) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Figshare (submittedVersion, cc-by)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (submittedVersion)
- Greater South Information System
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- systematic reviews, development interventions, effectiveness evaluation, international development, meta-analysis, critical appraisal, literature searching, review questions, generalizability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
