Reporting systematic reviews: Some lessons from a tertiary study
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Abstract Context Many of the systematic reviews published in software engineering are related to research or methodological issues and hence are unlikely to be of direct benefit to practitioners or teachers. Those that are relevant to practice and teaching need to be presented in a form that makes their findings usable with minimum interpretation. Objective We have examined a sample of the many systematic reviews that have been published over a period of six years, in order to assess how well these are reported and identify useful lessons about how this might be done. Method We undertook a tertiary study, performing a systematic review of systematic reviews. Our study found 178 systematic reviews published in a set of major software engineering journals over the period 2010–2015. Of these, 37 provided recommendations or conclusions of relevance to education and/or practice and we used the DARE criteria as well as other attributes related to the systematic review process to analyse how well they were reported. Results We have derived a set of 12 ‘lessons’ that could help authors with reporting the outcomes of a systematic review in software engineering. We also provide an associated checklist for use by journal and conference referees. Conclusion There are several areas where better reporting is needed, including quality assessment, synthesis, and the procedures followed by the reviewers. Researchers, practitioners, teachers and journal referees would all benefit from better reporting of systematic reviews, both for clarity and also for establishing the provenance of any findings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 1×
“Three broad tertiary stud-140 ies summarised all of the systematic reviews published up to the end of 2009 [12, 13, 14].”
Self-citationBrereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.07.009 · Cited by 2,394 · cited 1×
“To do so it employs the same model as we used in an earlier (and widely-cited) paper [8], where the experiences of conducting systematic reviews were codi-95 fied by structuring them as a set of ‘lessons for practice’.”
Self-citationSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 1×
“Three broad tertiary stud-140 ies summarised all of the systematic reviews published up to the end of 2009 [12, 13, 14].”
Self-citationA Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154
A systematic review of systematic review process research in software engineering
Kitchenham, B., & Brereton, P. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.07.010 · Cited by 1,094
Self-citationDoes the technology acceptance model predict actual use? A systematic literature review
Turner, M., Kitchenham, B., Brereton, P., Charters, S., & Budgen, D. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2009.11.005 · Cited by 851
Self-citation(2004) · doi:10.5555/998675.999432 · Cited by 656
Software development in startup companies: A systematic mapping study
Paternoster, N., Giardino, C., Unterkalmsteiner, M., Gorschek, T., & Abrahamsson, P. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.04.014 · Cited by 451
Systematic Reviews to Support Evidence-Based Medicine: How to appraise, conduct and publish reviews
Khan, K. S., & Zamora, J. (2022) · doi:10.1201/9781003220039 · Cited by 151
Measuring and predicting software productivity: A systematic map and review
Petersen, K. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.001 · Cited by 128
Jabangwe, R., Börstler, J., Šmite, D., & Wohlin, C. (2015) · Empirical Software Engineering · doi:10.1007/s10664-013-9291-7 · Cited by 97
A systematic review of comparative evidence of aspect-oriented programming
Ali, M. S., Ali Babar, M., Chen, L., & Stol, K.-J. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.05.003 · Cited by 89
Cited by
Cubric, M. (2020) · Technology in Society · doi:10.1016/j.techsoc.2020.101257 · Cited by 416
Ampatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., & Chatzigeorgiou, A. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.10.006 · Cited by 305
Systematic reviews in sentiment analysis: a tertiary study
Ligthart, A., Catal, C., & Tekinerdogan, B. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09973-3 · Cited by 303
SEGRESS: Software Engineering Guidelines for REporting Secondary Studies
Kitchenham, B., Madeyski, L., & Budgen, D. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3174092 · Cited by 219
Self-citationGuidelines for the Search Strategy to Update Systematic Literature Reviews in Software Engineering
Wohlin, C., Mendes, E., Felizardo, K. R., & Kalinowski, M. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106366 · Cited by 147
“Such call to arms prompted the Software Engineering (SE) community to publish SLRs, leading to more than 430 SLRs published within the period from January 2004 to May 2016 [ 19 , 24 , 54 ] and [4] .”
Metadata
- Authors
- Budgen, D., Brereton, P., Drummond, S., & Williams, N.
- Venue
- Information and Software Technology
- Publisher
- Elsevier BV
- Published
- 2018-03
- Volume
- 95
- Pages
- 62-74
- ISSN
- 0950-5849 (ISSN-L)
- OpenAlex
- W2765565489
- Cited by
- 99
- References
- 12
- FWCI
- 2.38
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.