Synthesizing evidence in software engineering research
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Synthesizing the evidence from a set of studies that spans many countries and years, and that incorporates a wide variety of research methods and theoretical perspectives, is probably the single most challenging task of performing a systematic review. In this paper, we perform a tertiary review to assess the types and methods of research synthesis in systematic reviews in software engineering. Almost half of the 31 studies included in our review did not contain any synthesis; of the ones that did, two thirds performed a narrative or a thematic synthesis. The results show that, despite the focus on systematic reviews, there is, currently, limited attention to research synthesis in software engineering. This needs to change and a repertoire of synthesis methods needs to be an integral part of systematic reviews to increase their significance and utility for research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537
Empirical studies of agile software development: A systematic review
Dybå, T., & Dingsøyr, T. (2008) · Information and Software Technology · doi:10.1016/j.infsof.2008.01.006 · Cited by 2,668
Self-citationSynthesising qualitative and quantitative evidence: A review of possible methods
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005) · Journal of Health Services Research & Policy · doi:10.1177/135581960501000110 · Cited by 2,132
Meta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews
Rosenthal, R., & DiMatteo, M. R. (2001) · Annual Review of Psychology · doi:10.1146/annurev.psych.52.1.59 · Cited by 1,780
What are scoping studies? A review of the nursing literature
Davis, K., Drey, N., & Gould, D. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.02.010 · Cited by 855 · cited 1×
“Scoping studies are commonly intended to guide more focused lines of research and development [1], [8].”
BackgroundMeta-Study of Qualitative Health Research
Paterson, B., Thorne, S., Canam, C., & Jillings, C. (2001) · doi:10.4135/9781412985017 · Cited by 664
(2004) · doi:10.5555/998675.999432 · Cited by 656
Self-citationRodgers, M., Sowden, A., Petticrew, M., Arai, L., Roberts, H., Britten, N., & Popay, J. (2009) · Evaluation · doi:10.1177/1356389008097871 · Cited by 561
Motivation in software engineering: A systematic literature review
Beecham, S., Baddoo, N., Hall, T., Robinson, H., & Sharp, H. (2008) · Information and Software Technology · doi:10.1016/j.infsof.2007.09.004 · Cited by 506
Towards Rigor in Reviews of Multivocal Literatures: Applying the Exploratory Case Study Method
Ogawa, R. T., & Malen, B. (1991) · Review of Educational Research · doi:10.3102/00346543061003265 · Cited by 247 · cited 1×
“As SRs evolve and continue to gain popularity, awareness is needed to ensure greater transparency and methodological rigor that will increase the legitimacy of findings and relevance for practice [20].”
Background
Cited by
Recommended Steps for Thematic Synthesis in Software Engineering
Cruzes, D. S., & Dyba, T. (2011) · 2011 International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2011.36 · Cited by 960
Self-citationGarousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831
Research synthesis in software engineering: A tertiary study
Cruzes, D. S., & Dybå, T. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.01.004 · Cited by 346
Self-citationClosing the Gap between Software Engineering Education and Industrial Needs
Garousi, V., Giray, G., Tuzun, E., Catal, C., & Felderer, M. (2020) · IEEE Software · doi:10.1109/ms.2018.2880823 · Cited by 219
“Our research method is meta-analysis [4] which is a form of synthesis that combines the quantitative data from primary studies (the pool of 33 papers in this work).”
MethodCase studies synthesis: a thematic, cross-case, and narrative synthesis worked example
Cruzes, D. S., Dybå, T., Runeson, P., & Höst, M. (2015) · Empirical Software Engineering · doi:10.1007/s10664-014-9326-8 · Cited by 195
“Table 1 outlines some of the methods that are most relevant for synthesizing evidence across case studies (a more complete list is provided in (Cruzes and Dybå 2010, 2011a).”
MethodSelf-citation
Metadata
- Authors
- Cruzes, D. S., & Dybå, T.
- Venue
- Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement
- Publisher
- ACM
- Published
- 2010-09-16
- Pages
- 1-10
- ISSN
- 1938-6451 (ISSN-L)
- OpenAlex
- W2142389202
- Cited by
- 65
- References
- 11
- FWCI
- 2.70
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.