Assessing ‘what works’ in international development: meta-analysis for sophisticated dummies
Open accessAbstract
Many studies of development interventions are individually unable to provide convincing conclusions because of low statistical significance, small size, limited geographical purview and so forth. Systematic reviews and meta-analysis are forms of research synthesis that combine studies of adequate methodological quality to produce more convincing conclusions. In the social sciences, study designs, types of analysis and methodological quality vary tremendously. Combining these studies for meta-analysis entails more demanding risk of bias assessments to ensure that only studies with largely appropriate methodological characteristics are included, and sensitivity analysis should be performed. In this article, we discuss assessing risk of bias and meta-analysis using such diverse studies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The review team used the framework proposed in Duvendack et al. (2011, 2012) to assess the rigour of the designs in the included primary studies.
“In order to assess the rigour of the designs in the included primary studies, the review team used the framework proposed in Duvendack et al. (2011Duvendack et al. ( , 2012)).”
From Interventions to improve the labour market outcomes of youth: A systematic review of training, entrepreneurship promotion, employment services and subsidized employment interventions (2017) · cites this paper as Duvendack et al., 2012
The approach combined an assessment of both the research design and the method of statistical analysis leading to an implicit hierarchy of study designs, with RCTs as the most rigorous design and cross-section designs at the bottom.
“The approach combined an assessment of both the research design and the method of statistical analysis leading to an implicit hierarchy of study designs, with RCTs as the most rigorous design and cross-section designs at the bottom.”
From Interventions to improve the labour market outcomes of youth: A systematic review of training, entrepreneurship promotion, employment services and subsidized employment interventions (2017) · cites this paper as Duvendack et al., 2012
The review relied on a framework by Duvendack et al. (2012) to assess the analytical and statistical rigour of the included studies based on the studies' design and statistical methodology.
“However, it relied on a framework by Duvendack et al. (2012) to assess the analytical and statistical rigour of the included studies based on the studies' design and statistical methodology.”
From Interventions to improve the labour market outcomes of youth: A systematic review of training, entrepreneurship promotion, employment services and subsidized employment interventions (2017) · cites this paper as Duvendack et al. (2012)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230 · see citing works
Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563 · see citing works
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · see citing works
Interpretation of random effects meta-analyses
Riley, R. D., Higgins, J. P. T., & Deeks, J. J. (2011) · BMJ · doi:10.1136/bmj.d549 · Cited by 2,884 · see citing works
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461 · see citing works
Meta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154 · see citing works
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · see citing works
Counterfactuals and Causal Inference: Methods and Principles for Social Research
Morgan, S. L., & Winship, C. (2007) · doi:10.1017/cbo9780511804564 · Cited by 1,146 · see citing works
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716 · see citing works
Cook, T. D., Shadish, W. R., & Wong, V. C. (2008) · Journal of Policy Analysis and Management · doi:10.1002/pam.20375 · Cited by 604 · see citing works
Grissom, R. J., & Kim, J. J. (2012) · doi:10.4324/9780203803233 · Cited by 470 · cited 1× · see citing works
“It is also important to bear in mind that the effect sizes and their confidence intervals derived from zero-order analyses are reliable only when certain conditions are met, specifically normality and homoscedasticity and independence (see Grissom and Kim 2012 and Wilcox 2012 for an introduction).”
BackgroundNonexperimental Versus Experimental Estimates of Earnings Impacts
Glazerman, S., Levy, D. M., & Myers, D. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716203254879 · Cited by 352 · see citing works
A multivariate mixed linear model for meta-analysis.
Kalaian, H. A., & Raudenbush, S. W. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.3.227 · Cited by 229 · see citing works
Heinsman, D. T., & Shadish, W. R. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.154 · Cited by 125 · see citing works
Flagged first, then by citations
Research works printed but not linked: 47
- Becker, B.J., 2006. Failsafe N or file-drawer number. In: H.R. Rothstein, A.J. Sutton, and M. Borenstein, eds. Publication bias in meta-analysis: prevention, assessment and adjustments. Chichester: Wiley.
- Becker, B.J. and Wu, M.-J., 2007. The synthesis of regression slopes in meta-analysis. Statistical science, 22 (3), 414-429.
- Benson, K. and Hartz, A.J., 2000. A comparison of observational studies and randomized, controlled trials. New England journal of medicine, 342 (25), 1878-1886.
- Borenstein, M., et al., 2009. Introduction to meta-analysis. Chichester: Wiley. Coalition for Evidence-Based Policy, 2010. Checklist for reviewing a randomized controlled trial of a social program or project, to assess whether it produced valid evidence [online]. Available from: http://coalition4evidence.org/wordpress/wp-content/uploads/Checklist-For-Reviewing-a- RCT-Jan10.pdf [Accessed 24 July 2012].
- Chalmers, I., Hedges, L.V., and Cooper, H., 2002. A brief history of research synthesis. Evaluation and the health professions, 25 (1), 12-37.
- Colliver, J.A., Kucera, K., and Verhulst, S.J., 2008. Meta-analysis of quasi-experimental research: are systematic narrative reviews indicated? Medical education, 42 (9), 858-865.
- Cooper, H., 1979. Statistically combining independent studies: a meta-analysis of sex differences in conformity research. Journal of personality and social psychology, 37 (1), 131-146.
- Cumming, G., 2012. Understanding the new statistics: effect sizes, confidence intervals, and meta- analysis. New York: Taylor & Francis.
- Deaton, A., 2010. Instruments, randomization, and learning about development. Journal of economic literature, 48 (2), 424-455.
- Dickersin, K., et al., 1987. Publication bias and clinical trials. Controlled clinical trials, 8 (4), 343-353.
- Duval, S. and Tweedie, R., 2000. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56 (2), 455-463.
- Egger, M., Schneider, M., and Smith, G.D., 1998. Meta-analysis spurious precision? Meta-analysis of observational studies. British medical journal, 316, 140-144.
- Egger, M., et al., 1997. Bias in meta-analysis detected by a simple, graphical test. British medical journal, 315 (7109), 629-634.
- Eysenck, H.J., 1978. An exercise in mega-silliness. American psychologist, 33 (5), 517-517.
- Eysenck, H.J., 1984. Meta-analysis: an abuse of research integration. The journal of special education, 18 (1), 41-59.
- Feinstein, A.R., 1995. Meta-analysis: statistical alchemy for the 21st century. Journal of clinical epidemiology, 48 (1), 71-79.
- Glass, G.V. and Smith, M.L., 1979. Meta-analysis of research on class size and achievement. Educational evaluation and policy analysis, 1 (1), 2-16.
- Grissom, R.J. and Kim, J.J., 2012. Effect size for research. London: Routledge.
- Hansen, B.B. and Bowers, J., 2008. Covariate balance in simple, stratified and clustered comparative studies. Statistical science, 23 (2), 219-236.
- Hedges, L.V. and Olkin, I., 1985. Statistical methods for meta-analysis [online], Academic Press. Available from: http://www.jameslindlibrary.org/illustrating/records/statistical-methods- for-meta-analysis/title_pages [Accessed 24 July 2012].
- Hedges, L.V. and Stock, W., 1983. The effects of class size: an examination of rival hypotheses. American educational research journal, 20, 63-85.
- Jones, D.R., 1992. Meta-analysis of observational epidemiological studies: a review. Journal of the royal society of medicine, 85 (3), 165-168.
- Keef, S.P. and Roberts, L.A., 2004. The meta-analysis of partial effect sizes. The British journal of mathematical and statistical psychology, 57 (1), 97-129.
- King, G., 2007. An introduction to the dataverse network as an infrastructure for data sharing. Sociological methods and research, 36 (2), 173-199.
- Light, R.J. and Pillemer, D.S., 1984. Summing up: the science of reviewing research. Cambridge, MA: Harvard University Press.
- Lipsey, M.W. and Wilson, D.B., 2001. Practical meta-analysis. Thousand Oaks, CA: Sage Publications.
- McGrath, R.E. and Meyer, G.J., 2006. When effect sizes disagree: the case of r and d. Psychological methods, 11 (4), 386-401.
- Nickerson, R.S., 2000. Null hypothesis significance testing: a review of an old and continuing controversy. Psychological methods, 5 (2), 241-301.
- Orwin, R.G., 1983. A fail-safe N for effect size in meta-analysis. Journal of educational statistics, 8 (2), 157-159.
- Petryna, A., 2009. When experiments travel: clinical trials and the global search for human subjects. Princeton, NJ: Princeton University Press.
- Petticrew, M. and Roberts, H., 2006. Systematic reviews in the social sciences: a practical guide. Oxford: Blackwell Publishing.
- Rosenthal, R., 1979. The "file drawer" problem and tolerance for null results. Psychological bulletin, 86, 38-641.
- Rosenthal, R. and Rubin, D.B., 1986. Meta-analytic procedures for combining studies with multiple effect sizes. Psychological bulletin, 99, 400-406.
- Rubin, D., 1974. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational psychology, 66 (5), 688-701.
- Sen, A.K., 1999. Development as freedom. Oxford: Oxford University Press.
- Shapiro, S., 1994. Meta-analysis/shmeta-analysis. American journal of epidemiology, 140 (9), 771-778.
- Sharpe, D., 1997. Of apples and oranges, file drawers and garbage: why validity issues in meta- analysis will not go away. Clinical psychology review, 17 (8), 881-901.
- Slavin, R.E., 1986. Best-evidence synthesis: an alternative to meta-analytic and traditional reviews. Educational researcher, 15 (9), 5-11.
- Smith, G.D. and Egger, M., 1999. Meta-analyses of observational data should be done with due care. British medical journal, 318 (7175), 256.
- Stroup, D., et al., 2000. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Journal of the American medical association, 283 (15), 2008-2012.
- Sutton, A.J., et al., 1998. Systematic reviews of trials and other studies. Health technology assessment, 2 (19), 1-310.
- Valentine, J. and Cooper, H., 2008. A systematic and transparent approach for assessing the method- ological quality of intervention effectiveness research: the study design and implementation assessment device. Psychological methods, 13 (2), 130-149.
- Vandenbroucke, J.P., 1989. Is there a hierarchy of methods in clinical research? Journal of molecular medicine, 67 (10), 515-517.
- Wampold, B.E., Ahn, H.-N., and Kim, D.-M., 2000. Meta-analysis in the social sciences: a useful way to make sense of a series of findings from a large number of studies. Asia Pacific education review, 1 (1), 67-74.
- Wilcox, R.R., 2008. Sample size and statistical power. In: A.M. Nezu and C.M. Nezu, eds. Evidence based outcome research: a practical guide to conducting randomized controlled trials for psychosocial interventions. New York: Oxford University Press, 123-134.
- Wilcox, R.R., 2012. Introduction to robust estimation and hypothesis testing. 3rd ed. San Diego, CA: Academic Press.
- Ziliak, S.T. and McCloskey, D.N., 2008. The cult of statistical significance: how the standard error cost us jobs, justice, and lives. Ann Arbor: University of Michigan Press.
Cited by
Sorted by citations
The Benefits and Challenges of Using Systematic Reviews in International Development Research
Mallett, R., Hagen-Zanker, J., Slater, R., & Duvendack, M. (2012) · Journal of Development Effectiveness · doi:10.1080/19439342.2012.711342 · Cited by 565 · see citing works
Self-citationKluve, J., Puerto, S., Robalino, D., Romero, J. M., Rother, F., Stöterau, J., Weidenkaff, F., & Witte, M. (2017) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.4073/csr.2017.12 · Cited by 106 · see citing works
- Authors
- Duvendack, M., Hombrados, J. G., Palmer-Jones, R., & Waddington, H.
- Affiliations
- Overseas Development Institute (GB)
- International Initiative for Impact Evaluation (US)
- Corresponding author
- Duvendack, Maren
- Venue
- Journal of Development Effectiveness
- Publisher
- Informa UK Limited
- Published
- 2012-09
- Volume
- 4
- Issue
- 3
- Pages
- 456-471
- ISSN
- 1943-9342 (ISSN-L)
- 1943-9407
- OpenAlex
- W2139102553
- Cited by
- 52
- References
- 14
- Citations by year
- 2026: 2 · 2024: 1 · 2023: 1 · 2022: 2 · 2021: 1 · 2020: 3 · 2019: 4 · 2018: 6 · 2017: 9 · 2016: 1 · 2015: 3 · 2014: 9 · 2013: 4 · 2012: 6
- FWCI
- 8.03
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of Development Effectiveness (publishedVersion)
- Sussex Research Online (University of Sussex) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Figshare (submittedVersion, cc-by)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Social Sciences › Development › International Development and Aid
- Keywords
- meta-analysis, international development, risk of bias assessment, development interventions, evidence synthesis, sensitivity analysis, methodological quality, what works
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
