SEGRESS: Software Engineering Guidelines for REporting Secondary Studies

Kitchenham, Barbara; Madeyski, Lech; Budgen, David

doi:10.1109/tse.2022.3174092

Cited by 219Software Engineering Techniques and PracticesMeta-analysis and systematic reviewsSoftware Engineering Research

The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.

SEGRESS: Software Engineering Guidelines for REporting Secondary Studies

Open access

Kitchenham, Barbara, Madeyski, Lech, Budgen, David·2023·IEEE Transactions on Software Engineering

DOI 10.1109/tse.2022.3174092

Open this paper in CoChat — free account Open PDF

Abstract

Context: Several tertiary studies have criticized the reporting of software engineering secondary studies.Objective: Our objective is to identify guidelines for reporting software engineering (SE) secondary studies which would address problems observed in the reporting of software engineering systematic reviews (SRs).Method: We review the criticisms of SE secondary studies and identify the major areas of concern. We assess the PRISMA 2020 (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement as a possible solution to the need for SR reporting guidelines, based on its status as the reporting guideline recommended by the Cochrane Collaboration whose SR guidelines were a major input to the guidelines developed for SE. We report its advantages and limitations in the context of SE secondary studies. We also assess reporting guidelines for mapping studies and qualitative reviews, and compare their structure and content with that of PRISMA 2020.Results: Previous tertiary studies confirm that reports of secondary studies are of variable quality. However,ad hocrecommendations that amend reporting standards may result in unnecessary duplication of text. We confirm that the PRISMA 2020 statement addresses SE reporting problems, but is mainly oriented to quantitative reviews, mixed-methods reviews and meta-analyses. However, we show that the PRISMA 2020 item definitions can be extended to cover the information needed to report mapping studies and qualitative reviews.Conclusions: In this paper and its Supplementary Material, we present and illustrate an integrated set of guidelines called SEGRESS (Software Engineering Guidelines for REporting Secondary Studies), suitable for quantitative systematic reviews (building upon PRISMA 2020), mapping studies (PRISMA-ScR), and qualitative reviews (ENTREQ and RAMESES), that addresses reporting problems found in current SE SRs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1109/tse.2022.3174092

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

PRISMA accommodates quality assessment for each literature example and assesses the strength of evidence.
“PRISMA accommodates quality assessment for each literature example and assesses the strength of evidence [14].”
From Initial Cybersecurity Framework in the New Capital City of Indonesia: Factors, Objectives, and Technology (2022) · cites this paper as [14]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
Scoping studies: towards a methodological framework
Arksey, H., & O'Malley, L. (2005) · International Journal of Social Research Methodology · doi:10.1080/1364557032000119616 · Cited by 36,742
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Cochrane Handbook for Systematic Reviews of Interventions
(2019) · doi:10.1002/9781119536604 · Cited by 14,828
PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262 · cited 7×
“Page et al. [11] explain the request to “Discuss any limitations of the evidence included in the review” (see item 23b) as follows: “Discussing the completeness, applicability, and uncertainties in the evidence included in the review should help readers interpret the ﬁndings appropriately.”
BackgroundMethodInfluential
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 2×
“The recent GRADE revision speciﬁes four levels of the certainty for a body of evidence related to a given ﬁnding: high, moderate, low and very low (see [23] and [25]).”
Background
GRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714
PRISMA-S: An Extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ
Tong, A., Flemming, K., McInnes, E., Oliver, S., & Craig, J. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-181 · Cited by 3,569
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798 · cited 1×
“The ﬁnal four items relate to ethical scientiﬁc practice, speciﬁcally: protocol The original ideas were presented in [21] and have been revised and reﬁned in a series of articles produced by a technical working group, see ([24], [25], [26], [27], [28], [29], [30], [31]).”
Background
GRADE guidelines 6. Rating the quality of evidence—imprecision
Guyatt, G. H., Oxman, A. D., Kunz, R., Brozek, J., Alonso-Coello, P., Rind, D., Devereaux, P. J., Montori, V. M., Freyschuss, B., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.012 · Cited by 2,693
Empirical studies of agile software development: A systematic review
Dybå, T., & Dingsøyr, T. (2008) · Information and Software Technology · doi:10.1016/j.infsof.2008.01.006 · Cited by 2,668
GRADE guidelines: 5. Rating the quality of evidence—publication bias
Guyatt, G. H., Oxman, A. D., Montori, V., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Djulbegovic, B., Atkins, D., Falck-Ytter, Y., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.011 · Cited by 1,819
RAMESES publication standards: realist syntheses
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-21 · Cited by 1,374
GRADE guidelines: 9. Rating up the quality of evidence
Guyatt, G. H., Oxman, A. D., Sultan, S., Glasziou, P., Akl, E. A., Alonso-Coello, P., Atkins, D., Kunz, R., Brozek, J., Montori, V., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.06.004 · Cited by 1,336 · cited 1×
“The ﬁnal four items relate to ethical scientiﬁc practice, speciﬁcally: protocol The original ideas were presented in [21] and have been revised and reﬁned in a series of articles produced by a technical working group, see ([24], [25], [26], [27], [28], [29], [30], [31]).”
Background
Using Qualitative Evidence in Decision Making for Health and Social Interventions: An Approach to Assess Confidence in Findings from Qualitative Evidence Syntheses (GRADE-CERQual)
Lewin, S., Glenton, C., Munthe-Kaas, H., Carlsen, B., Colvin, C. J., Gülmezoglu, M., Noyes, J., Booth, A., Garside, R., & Rashidian, A. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001895 · Cited by 866
PRISMA for Abstracts: Reporting Systematic Reviews in Journal and Conference Abstracts
Beller, E. M., Glasziou, P. P., Altman, D. G., Hopewell, S., Bastian, H., Chalmers, I., Gøtzsche, P. C., Lasserson, T., Tovey, D., & for the PRISMA for Abstracts Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001419 · Cited by 706
Empirical evidence about the UML: a systematic literature review
Budgen, D., Burn, A. J., Brereton, O. P., Kitchenham, B. A., & Pretorius, R. (2011) · Software: Practice and Experience · doi:10.1002/spe.1009 · Cited by 138
Self-citation
A systematic review of comparative evidence of aspect-oriented programming
Ali, M. S., Ali Babar, M., Chen, L., & Stol, K.-J. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.05.003 · Cited by 89
The Impact of Test-Driven Development on Software Development Productivity — An Empirical Study
Madeyski, L., & Szała, Ł. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-75381-0_18 · Cited by 46 · cited 1×
“The issue of process conformance in the context of Test-Driven Development experiments was discussed, e.g., in [40] and [41].”
MethodSelf-citation

Flagged first, then by citations

Research works printed but not linked: 82

10.1016/j.infsof.2017.10.017 · DOI
10.1109/APSEC.2016.031 · DOI
10.1016/j.infsof.2018.10.006 · DOI · Paper page
10.1016/j.infsof.2020.106397 · DOI
10.1016/j.infsof.2022.106840 · DOI
10.37190/e-Inf200101 · DOI
Shadish, 2002, Experimental and Quasi-Experimental Design for Generalized Causal Inference
10.1145/2745802.2745815 · DOI
Page, 2021, The PRISMA 2020 statement: An updated guideline for reporting systematic reviews, 372
10.1136/bmj.g7647 · DOI · Paper page
Oxman, 2004, Grading quality of evidence and strength of recommendations, Brit. Med. J., 328, 1490, 10.1136/bmj.328.7454.1490 · DOI · Paper page
10.1016/j.jclinepi.2010.09.012 · DOI · Paper page
10.1016/j.jclinepi.2011.03.017 · DOI · Paper page
10.1016/j.jclinepi.2011.04.014 · DOI · Paper page
10.1016/j.jclinepi.2019.11.010 · DOI · Paper page
Page, 2020, Updating guidance for reporting systematic reviews: Development of the PRISMA 2020 statement, 10.31222/osf.io/jb4dx · DOI
10.1109/ESEM.2009.5316026 · DOI
10.1109/TSE.2014.2322358 · DOI
Kitchenham, 1986, The effects of inspections on software quality and productivity, ICL Tech. J., 5, 112
10.1109/32.991322 · DOI
10.1007/978-3-642-04288-1 · DOI
10.1109/TSE.2002.1027796 · DOI
10.1016/j.jss.2015.03.065 · DOI
10.1145/3319008.3319009 · DOI
10.1145/1414004.1414034 · DOI
10.1109/TSE.2007.17 · DOI
10.1109/TSE.2004.43 · DOI
10.1109/ICSE.2005.1553590 · DOI
10.1007/978-3-030-32489-6 · DOI
10.1007/978-1-84800-044-5_11 · DOI · Paper page
10.1145/3241743 · DOI
10.1016/j.infsof.2019.04.005 · DOI
10.1109/TSE.2013.11 · DOI
10.1049/iet-sen.2011.0132 · DOI
10.1109/TSE.2003.1245300 · DOI
10.1109/TSE.2005.58 · DOI
10.1007/s10664-011-9183-7 · DOI
10.1136/bmjopen-2017-019703 · DOI · Paper page
10.1016/j.infsof.2009.11.005 · DOI · Paper page
10.1002/9781119536604.ch21 · DOI
10.1002/pon.4915 · DOI
10.1145/50087.50089 · DOI · Paper page
10.1145/76380.76383 · DOI
10.1002/0470871822.ch27 · DOI
10.1007/978-3-030-32489-6_18 · DOI
10.1016/j.jclinepi.2017.11.029 · DOI
10.1016/j.ijforecast.2007.05.008 · DOI
10.1109/TSE.2007.1001 · DOI
10.1109/ESEM.2017.68 · DOI
10.1109/ESEM.2015.7321184 · DOI
10.1007/s10664-020-09844-5 · DOI
10.1016/j.infsof.2013.07.010 · DOI · Paper page
Booth, 2012, Systematic Approaches to a Successful Literature Review
10.1186/1748-5908-5-69 · DOI · Paper page
10.1016/j.jclinepi.2020.03.016 · DOI · Paper page
10.1016/j.infsof.2011.01.004 · DOI
10.1145/3180155.3180235 · DOI
10.1371/journal.pmed.1000217 · DOI · Paper page
10.1186/1471-2288-11-115 · DOI
10.1186/s13012-017-0688-3 · DOI · Paper page
10.1186/s13012-017-0689-2 · DOI
10.1186/s13012-017-0690-9 · DOI
10.1186/s13012-017-0691-8 · DOI
10.1186/s13012-017-0692-7 · DOI
10.1186/s13012-017-0693-6 · DOI
10.1201/b19467 · DOI
10.1016/j.jclinepi.2017.09.025 · DOI
10.1016/j.jclinepi.2017.10.023 · DOI · Paper page
10.1016/j.jclinepi.2017.06.020 · DOI · Paper page
10.1016/j.jclinepi.2017.11.028 · DOI
10.1016/j.jclinepi.2017.10.022 · DOI
10.1177/1049732312466295 · DOI
Beecham, 2006, Protocol for a systematic literature review of motivation in software engineering
10.1016/j.infsof.2007.09.004 · DOI · Paper page
10.1016/j.infsof.2008.05.009 · DOI
10.1016/j.infsof.2009.02.001 · DOI · Paper page
Kitchenham, 2006, Protocol for systematic review of within- and cross-company estimation models
10.14236/ewic/EASE2006.10 · DOI
10.1109/TSE.2007.1001 · DOI
10.1186/1478-4505-7-S1-S13 · DOI
10.1109/TSE.2022.3165938 · DOI
10.3233/JIFS-169146 · DOI

Sorted by citations

Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Initial Cybersecurity Framework in the New Capital City of Indonesia: Factors, Objectives, and Technology
Sensuse, D. I., Putro, P. A. W., Rachmawati, R., & Sunindyo, W. D. (2022) · Information · doi:10.3390/info13120580 · Cited by 33
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27
Security of Language Models for Code: A Systematic Literature Review
Chen, Y., Sun, W., Fang, C., Chen, Z., Ge, Y., Han, T., Zhang, Q., Liu, Y., Chen, Z., & Xu, B. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735554 · Cited by 10

Authors: Kitchenham, B., Madeyski, L., & Budgen, D.
Affiliations: Keele University (GB)
Wrocław University of Science and Technology (PL)
Durham University (GB)
ORCID: Kitchenham, Barbara
Madeyski, Lech
Budgen, David
Venue: IEEE Transactions on Software Engineering
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2023-03-01
Volume: 49
Issue: 3
Pages: 1273-1298
ISSN: 0098-5589 (ISSN-L)
1939-3520
2326-3881
DOI: 10.1109/tse.2022.3174092
OpenAlex: W4285248702
Cited by: 219
References: 22
Citations by year: 2026: 76 · 2025: 61 · 2024: 51 · 2023: 22
FWCI: 24.93
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: cc-by
creativecommons.org/licenses/by/4.0/legalcode (vor)
Locations: IEEE Transactions on Software Engineering (publishedVersion, cc-by)
Durham Research Online (Durham University) (acceptedVersion, other-oa)
Durham Research Online (Durham University) (submittedVersion, cc-by)
Keele Research Repository (Keele University) (submittedVersion, cc-by)
Topics: Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords: reporting guidelines, PRISMA 2020, systematic reviews, mapping study, PRISMA-ScR, meta-analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

SEGRESS: Software Engineering Guidelines for REporting Secondary Studies

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Metadata

Record

About these checks