Identifying, categorizing and mitigating threats to validity in software engineering secondary studies
Open accessAbstract
Secondary studies are vulnerable to threats to validity. Although, mitigating these threats is crucial for the credibility of these studies, we currently lack a systematic approach to identify, categorize and mitigate threats to validity for secondary studies. In this paper, we review the corpus of secondary studies, with the aim to identify: (a) the trend of reporting threats to validity, (b) the most common threats to validity and corresponding mitigation actions, and (c) possible categories in which threats to validity can be classified. To achieve this goal we employ the tertiary study research method that is used for synthesizing knowledge from existing secondary studies. In particular, we collected data from more than 100 studies, published until December 2016 in top quality software engineering venues (both journals and conference). Our results suggest that in recent years, secondary studies are more likely to report their threats to validity. However, the presentation of such threats is rather ad hoc, e.g., the same threat may be presented with a different name, or under a different category. To alleviate this problem, we propose a classification schema for reporting threats to validity and possible mitigation actions. Both the classification of threats and the associated mitigation actions have been validated by an empirical study, i.e., Delphi rounds with experts. Based on the proposed schema, we provide a checklist, which authors of secondary studies can use for identifying and categorizing threats to validity and corresponding mitigation actions, while readers of secondary studies can use the checklist for assessing the validity of the reported results.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Study inclusion/exclusion bias, quality assessment subjectivity and data extraction bias are common threats to validity in SE secondary studies.
“Study inclusion/exclusion bias, quality assessment subjectivity and data extraction bias are the common threats to validity in SE secondary studies (Ampatzoglou et al., 2019).”
From Identifying the primary dimensions of DevSecOps: A multi-vocal literature review (2024) · cites this paper as Ampatzoglou et al., 2019
Inappropriate construction of search string might return redundant or lacking search results.
“Inappropriate construction of search string might return redundant or lacking search results (Ampatzoglou et al., 2019).”
From Identifying the primary dimensions of DevSecOps: A multi-vocal literature review (2024) · cites this paper as Ampatzoglou et al., 2019
A persistent threat to conclusion validity is the dynamic nature of software development.
“A persistent threat to conclusion validity in the context of software tool reviews is the dynamic nature of software development (Ampatzoglou et al. 2019).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Ampatzoglou et al. 2019
The Related work section defines conclusion validity and discusses researcher bias and statistical analysis as threats, but does not identify the changing nature of software development as one. Searches for “dynamic,” “outdated,” and changing software technologies found no passage addressing that asserted threat in software tool reviews. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThere is no standard way of writing about the threats to validity, in general and specifically for systematic literature reviews.
“as observed by Ampatzoglou et al. [100], there is no standard way of writing about the threats to validity, in general and specifically for SLRs”
From Sustainability in computing education: A systematic literature review (2024) · cites this paper as [100]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 218 · Cited by 305
Research works printed but not linked: 9
- [33]33. Dasanayake, S., Markkula, J., & Oivo, M. (2014, May). Con- cerns in software development: a systematic mapping study. In Proceed- ings of the 18th International Conference on Evaluation and Assessment in Software Engineering (p. 21). ACM.
- [45]45. Engström, E., Runeson, P., and Skoglund, M. A systematic re- view on regression test selection techniques. Information and Software Technology, 52(1), 14-30, 2010.
- [100]100. Misbhauddin, M., and Alshayeb, M. UML model refactoring: a systematic literature review. Empirical Software Engineering, 20(1), 206-251, 2015.
- [120]120. Rafique, Y., & Mi š i ć, V. B. (2013). The effects of test-driven development on external quality and productivity: A meta-analysis. IEEE Transactions on Software Engineering, 39(6), 835-856.
- [121]121. Rashid, M., Anwar, M. W., and Khan, A. M. Toward the tools selection in model based system engineering for embedded systems 𝛽€" A systematic literature review. Journal of Systems and Software, 106, 150-163, 2015.
- [131]131. Senapathi, M. and Srinivasan, A., "Sustained agile usage: a systematic literature review. In Proceedings of the 17th Interna- tional Conference on Evaluation and Assessment in Software Engineer- ing (EASE ' 13). ACM, New York, NY, USA, 119-124.
- [146]146. Sulaman, S. M., Weyns, K., & Höst, M. (2013, April). "A re- view of research on risk analysis methods for IT systems " . In Proceed- ings of the 17th International Conference on Evaluation and Assessment in Software Engineering (pp. 86-96). ACM.
- [162]162. M. Bano , D. Zowghi , N. Ikram , Systematic reviews in requirements engineering: a tertiary study ", Empirical Requirements Engineering (EmpiRE), in: 2014 IEEE Fourth International Workshop on, August 2014, pp. 9-16 .
- [173]173. M. Galster , D. Weyns , D. Tofan , B. Michalik , P. Avgeriou , Variability in software sys- tems -A systematic literature review, IEEE Trans. Softw. Eng. 40 (3) (2014) 282-306 .
- Authors
- Ampatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., & Chatzigeorgiou, A.
- Affiliations
- University of Macedonia (GR)
- University of Western Macedonia (GR)
- University of Groningen (NL)
- Corresponding author
- Ampatzoglou, Apostolos
- Venue
- Information and Software Technology
- Publisher
- Elsevier BV
- Published
- 2019-02
- Volume
- 106
- Pages
- 201-230
- ISSN
- 0950-5849 (ISSN-L)
- OpenAlex
- W2897889469
- Cited by
- 305
- References
- 218
- Citations by year
- 2026: 35 · 2025: 52 · 2024: 66 · 2023: 46 · 2022: 52 · 2021: 27 · 2020: 19 · 2019: 8
- FWCI
- 22.19
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Information and Software Technology (publishedVersion, other-oa)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- threats to validity, secondary studies, software engineering literature, reporting practices, tertiary study, research credibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
