Viewing systematic reviews and meta-analysis in social research through different lenses
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Systematic reviews and meta-analyses are used to combine results across studies to determine an overall effect. Meta-analysis is especially useful for combining evidence to inform social policy, but meta-analyses of applied social science research may encounter practical issues arising from the nature of the research domain. The current paper identifies potential resolutions to four issues that may be encountered in systematic reviews and meta-analyses in social research. The four issues are: scoping and targeting research questions appropriate for meta-analysis; selecting eligibility criteria where primary studies vary in research design and choice of outcome measures; dealing with inconsistent reporting in primary studies; and identifying sources of heterogeneity with multiple confounded moderators. The paper presents an overview of each issue with a review of potential resolutions, identified from similar issues encountered in meta-analysis in medical and biological sciences. The discussion aims to share and improve methodology in systematic reviews and meta-analysis by promoting cross-disciplinary communication, that is, to encourage ‘viewing through different lenses’.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scoping studies: towards a methodological framework
Arksey, H., & O'Malley, L. (2005) · International Journal of Social Research Methodology · doi:10.1080/1364557032000119616 · Cited by 36,742
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
(2002) · Cited by 13,211
Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011) · Child Development · doi:10.1111/j.1467-8624.2010.01564.x · Cited by 10,854 · cited 1×
“ance (Durlak et al. 2011), for example.”
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312
(2000) · Cited by 9,198
Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000412 · Cited by 7,300
Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1. The Cochrane Collaboration
Higgins, J. P. T., Green, S. R., & Higgins, J. P. T. (2008) · Cited by 7,171
(2010) · Cited by 7,034
Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72351 · Cited by 6,277
Evidence-Based Medicine: How to Practice and Teach EBM
Sackett, D. L. (2000) · Cited by 5,950
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
Des Jarlais, D. C., Lyles, C., Crepaz, N., & the TREND Group (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.361 · Cited by 2,817
Dixon-Woods, M., Cavers, D., Agarwal, S., Annandale, E., Arthur, A., Harvey, J., Hsu, R., Katbamna, S., Olsen, R., Smith, L., et al. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-35 · Cited by 2,289
(2000) · Queen Mary Research Online (Queen Mary University of London) · Cited by 1,795
Meta-analysis of individual participant data: rationale, conduct, and reporting
Riley, R. D., Lambert, P. C., & Abo-Zaid, G. (2010) · BMJ · doi:10.1136/bmj.c221 · Cited by 1,701
Evidence-Based Medicine: How to Practice and Teach EBM
Satya-Murti, S. (2000) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.284.18.2382 · Cited by 1,643
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Bulletin of the World Health Organization · doi:10.2471/blt.07.045120 · Cited by 1,632
Publication Manual of the American Psychological Association: BOOK REVIEW
Yao, R. (2011) · Family and Consumer Sciences Research Journal · doi:10.1111/j.1552-3934.2011.02081.x · Cited by 1,451
Systematic Reviews and Meta-Analysis
Littell, J. H., Corcoran, J., & Pillai, V. (2008) · doi:10.1093/acprof:oso/9780195326543.001.0001 · Cited by 1,293
Network meta‐analysis for indirect treatment comparisons
Lumley, T. (2002) · Statistics in Medicine · doi:10.1002/sim.1201 · Cited by 1,206
Handbook of Meta-analysis in Ecology and Evolution
(2013) · doi:10.1515/9781400846184 · Cited by 1,183
A catalogue of reporting guidelines for health research
Simera, I., Moher, D., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · European Journal of Clinical Investigation · doi:10.1111/j.1365-2362.2009.02234.x · Cited by 1,047 · cited 1×
“Online databases, such as the EQUATOR website (Simera et al. 2010) and the MIBBI project (Taylor et al.”
MethodCONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., & Moher, D. (2011) · International Journal of Surgery · doi:10.1016/j.ijsu.2011.09.004 · Cited by 1,044
Boily, M.-C., Baggaley, R. F., Wang, L., Masse, B., White, R. G., Hayes, R. J., & Alary, M. (2009) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(09)70021-0 · Cited by 913 · cited 1×
“This has been identified as an issue in many disciplines including social sciences (Lipsey, 2003), epidemiology (Boily et al. 2009), and ecology (Osenberg et al.”
BackgroundHedges, L. V., Laine, R. D., & Greenwald, R. (1994) · Educational Researcher · doi:10.3102/0013189x023003005 · Cited by 819
Capturing momentary, self-report data: A proposal for reporting guidelines
Stone, A. A., & Shiffman, S. (2002) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm2403_09 · Cited by 807
Bayesian methods in meta-analysis and evidence synthesis
Sutton, A. J., & Abrams, K. R. (2001) · Statistical Methods in Medical Research · doi:10.1177/096228020101000404 · Cited by 710
Taylor, C. F., Field, D., Sansone, S.-A., Aerts, J., Apweiler, R., Ashburner, M., Ball, C. A., Binz, P.-A., Bogue, M., Booth, T., et al. (2008) · Nature Biotechnology · doi:10.1038/nbt.1411 · Cited by 558
Reasons or excuses for avoiding meta-analysis in forest plots
Ioannidis, J. P. A., Patsopoulos, N. A., & Rothstein, H. R. (2008) · BMJ · doi:10.1136/bmj.a117 · Cited by 457
RESOLVING ECOLOGICAL QUESTIONS THROUGH META-ANALYSIS: GOALS, METRICS, AND MODELS
Osenberg, C. W., Sarnelle, O., Cooper, S. D., & Holt, R. D. (1999) · Ecology · doi:10.1890/0012-9658(1999)080[1105:reqtma]2.0.co;2 · Cited by 403 · cited 1×
“In ecology, some researchers argue that each question demands a unique outcome measurement that is defined by the research question and the context of the ecological process (Osenberg et al. 1999).”
BackgroundMethodological Issues in Combining Diverse Study Types in Systematic Reviews
Harden, A., & Thomas, J. (2005) · International Journal of Social Research Methodology · doi:10.1080/13645570500155078 · Cited by 371
Encouraging public cooperation and support for police
Murphy, K., Hinds, L., & Fleming, J. (2008) · Policing and Society · doi:10.1080/10439460802008660 · Cited by 361 · cited 1×
“2005) identified confidence as a key outcome and measured it with seven items asking whether officers were fair, courteous, honest, not intimidating, worked with citizens, treated citizens equally, and showed concern; while a fourth (Murphy et al. 2008) used four items measuring confidence in police, police profession…”
BackgroundRosenblad, A. (2009) · International Statistical Review · doi:10.1111/j.1751-5823.2009.00095_15.x · Cited by 289
Those confounded moderators in meta-analysis: good, bad, and ugly
Lipsey, M. W. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716202250791 · Cited by 283 · cited 2×
“This terminology is adopted for the purposes of the following discussion; specifically, study-level characteristics that vary systematically to some degree are considered “confounded” (Lipsey, 2003).”
BackgroundHow to do a good systematic review of effects in international development: a tool kit
Waddington, H., White, H., Snilstveit, B., Hombrados, J. G., Vojtkova, M., Davies, P., Bhavsar, A., Eyers, J., Koehlmoos, T. P., Petticrew, M., et al. (2012) · Journal of Development Effectiveness · doi:10.1080/19439342.2012.711765 · Cited by 278
Becker, B. J. (2000) · Handbook of Applied Multivariate Statistics and Mathematical Modeling · doi:10.1016/b978-012691360-6/50018-5 · Cited by 273 · cited 1×
“For example, multivariate approaches demonstrated by Becker (2000) provide estimated correlations among outcome measures as well as a correction for missing outcome measures in some trials.”
MethodEvolution of sexual dichromatism: contribution of carotenoid- versus melanin-based coloration
BADYAEV, A. V., & HILL, G. E. (2000) · Biological Journal of the Linnean Society · doi:10.1111/j.1095-8312.2000.tb01196.x · Cited by 263
Gold Standard Myths: Observations on the Experimental Turn in Quantitative Criminology
Sampson, R. J. (2010) · Journal of Quantitative Criminology · doi:10.1007/s10940-010-9117-3 · Cited by 241
Ren, L., Cao, L., Lovrich, N., & Gaffney, M. (2005) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2004.10.003 · Cited by 233 · cited 1×
“A third study (Ren et al. 2005) identified confidence as a key outcome and measured it with seven items asking whether officers were fair, courteous, honest, not intimidating, worked with citizens, treated citizens equally, and showed concern; while a fourth (Murphy et al.”
BackgroundEvolution of sexual dichromatism: contribution of carotenoid- versus melanin-based coloration
BADYAEV, A. (2000) · Biological Journal of the Linnean Society · doi:10.1006/bijl.1999.0350 · Cited by 229
Multivariate Meta-Analysis: The Effect of Ignoring Within-Study Correlation
Riley, R. D. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00593.x · Cited by 225 · cited 1×
“information that is very rarely available (Riley, 2009).”
Legitimacy in Policing: A Systematic Review
Mazerolle, L., Bennett, S., Davis, J., Sargeant, E., & Manning, M. (2013) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.4073/csr.2013.1 · Cited by 224
Self-citationBayesian methods in meta-analysis and evidence synthesis
Sutton, A. J., & Abrams, K. R. (2001) · Statistical Methods in Medical Research · doi:10.1191/096228001678227794 · Cited by 216
Accounting for heterogeneity via random-effects models and moderator analyses in meta-analysis.
Viechtbauer, W. (2007) · Zeitschrift für Psychologie / Journal of Psychology · doi:10.1027/0044-3409.215.2.104 · Cited by 193
Lessons from a systematic review of effects of multisystemic therapy
Littell, J. H. (2004) · Children and Youth Services Review · doi:10.1016/j.childyouth.2004.11.009 · Cited by 179 · cited 2×
“Non-random sampling, differential attrition, and lack of treatment integrity can introduce alternate explanations for treatment effects even in randomised trials (Littell, 2004; Sampson, 2010; Shadish et al. 2002).”
BackgroundGrann, V. R., Jacobson, J. S., Thomason, D., Hershman, D. L., Heitjan, D. F., & Neugut, A. I. (2002) · Journal of Clinical Oncology · doi:10.1200/jco.2002.10.101 · Cited by 154 · cited 1×
“medicine, treatment effectiveness could be measured with survival or quality-adjusted life years (Grann et al. 2002)”
BackgroundProblems in the Design and Reporting of Trials of Antifungal Agents Encountered During Meta-analysis
Johansen, H. K. (1999) · JAMA · doi:10.1001/jama.282.18.1752 · Cited by 128
Current methods for meta-analysis
Schulze, R. (2007) · Zeitschrift für Psychologie / Journal of Psychology · doi:10.1027/0044-3409.215.2.90 · Cited by 79
Reporting guidelines for economic studies
Mason, J. M., & Drummond, M. (1995) · Health Economics · doi:10.1002/hec.4730040202 · Cited by 75
Best-Evidence Synthesis: Why Less Is More
Slavin, R. E. (1987) · Educational Researcher · doi:10.2307/1175827 · Cited by 44
Improving evaluation of anti-crime programs: Summary of a National Research Council report★
Lipsey, M. W., Petrie, C., Weisburd, D. L., & Gottfredson, D. C. (2006) · Journal of Experimental Criminology · doi:10.1007/s11292-006-9009-6 · Cited by 32
RANDOM ASSIGNMENT IN CRIMINAL JUSTICE RESEARCH Some Ethical and Legal Issues
Baunach, P. J. (1980) · Criminology · doi:10.1111/j.1745-9125.1980.tb01307.x · Cited by 18
Purity or pragmatism? Reflecting on the use of systematic review methodology in development
Stewart, R. J., van Rooyen, C., & de Wet, T. (2012) · Journal of Development Effectiveness · doi:10.1080/19439342.2012.711341 · Cited by 15 · cited 2×
“tempt to cover the range of evidence on a topic (Stewart et al. 2012).”
BackgroundHigginson, A., MAZEROLLE, L. G., Davis, J., Bedford, L., Mengersen, K. L., Somerville, A., Thompson, J., Benier, K. H., & Williamson, H. (2013) · Campbell Systematic Reviews · doi:10.1002/cl2.112 · Cited by 4 · cited 1×
“A potential solution, as employed in a systematic review of violent crime interventions in developing countries (Higginson et al. 2013), is to use a series of metaanalyses to test specific outcomes within a broader narrative review of outcomes more broadly defined.”
BackgroundSelf-citation
Research works printed but not linked: 6
- Becker BJ, Hedges LV, Pigott TD: Statistical Analysis Policy Brief Prepared for the Campbell Collaboration Steering Committee. The Campbell Collaboration; 2004. Retrieved fromwww.campbellcollaboration.org
- Hall PA: Neighborhood Watch and Participant Perceptions (Unpublished doctoral dissertation). University of Southern California, California; 1987.
- Higgins JPT, Green S (Eds): In Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration. 2011. Retrieved fromwww.cochrane-handbook.org
- Shadish WR, Myers D: Research Design Policy Brief for the Campbell Collaboration Steering Committee. The Campbell Collaboration; 2004. Retrieved fromwww.campbellcollaboration.org
- Sherman LW, Strang H, Barnes GC, Braithwaite J, Inkpen N, Teh M: Experiments in restorative policing: A progress report on the Canberra Reintegrative Shaming Experiments (RISE). Australian Federal Police and Australian National University, Canberra, Australia; 1998.
- Skogan WG, Steiner L: CAPS at Ten: An Evaluation of Chicago’s Alternative Policing Strategy. Illinois Criminal Justice Information Authority, Chicago, IL; 2004.
Cited by
Literature review as a research methodology: An overview and guidelines
Snyder, H. (2019) · Journal of Business Research · doi:10.1016/j.jbusres.2019.07.039 · Cited by 11,524
Sima, V., Gheorghe, I. G., Subić, J., & Nancu, D. (2020) · Sustainability · doi:10.3390/su12104035 · Cited by 786
“In a SLR targeting research in the field of social sciences, according to [44], the high degree of variability in the studies related to the social sciences area must be considered.”
BackgroundRoss, L. E., Salway, T., Tarasoff, L. A., MacKay, J. M., Hawkins, B. W., & Fehr, C. P. (2018) · The Journal of Sex Research · doi:10.1080/00224499.2017.1387755 · Cited by 617
“However, it has been acknowledged that high heterogeneity values are to be expected in meta-analyses of complex social phenomena (Davis et al., 2014).”
BackgroundSauer, P. C., & Seuring, S. (2023) · Review of Managerial Science · doi:10.1007/s11846-023-00668-3 · Cited by 427
Gupta, S., Modgil, S., Bhattacharyya, S., & Bose, I. (2022) · Annals of Operations Research · doi:10.1007/s10479-020-03856-6 · Cited by 320
Metadata
- Authors
- Davis, J., Mengersen, K., Bennett, S., & Mazerolle, L.
- Affiliations
- The University of Queensland (AU)
- Queensland University of Technology (AU)
- Corresponding author
- Mengersen, Kerrie
- Venue
- SpringerPlus
- Published
- 2014-09-10
- Volume
- 3
- Issue
- 1
- Pages
- 511-511
- ISSN
- 2193-1801 (ISSN-L)
- PMC
- PMC4167883
- OpenAlex
- W2136868285
- Cited by
- 364
- References
- 59
- Citations by year
- 2026: 22 · 2025: 59 · 2024: 80 · 2023: 74 · 2022: 56 · 2021: 27 · 2020: 23 · 2019: 6 · 2018: 7 · 2017: 6 · 2016: 3 · 2015: 1
- FWCI
- 0.92
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- SpringerPlus (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- QUT ePrints (Queensland University of Technology) (submittedVersion, cc-by)
- VU Research Portal (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Science & Engineering Faculty
- Funders
- George Mason University
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Sociology and Political Science › Qualitative Comparative Analysis Research
- Keywords
- systematic reviews, meta-analysis, social research, eligibility criteria, reporting inconsistency, heterogeneity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

