How Reliable Are Systematic Reviews in Empirical Software Engineering?
Open accessAbstract
BACKGROUND-The systematic review is becoming a more commonly employed research instrument in empirical software engineering. Before undue reliance is placed on the outcomes of such reviews it would seem useful to consider the robustness of the approach in this particular research context. OBJECTIVE-The aim of this study is to assess the reliability of systematic reviews as a research instrument. In particular, we wish to investigate the consistency of process and the stability of outcomes. METHOD-We compare the results of two independent reviews undertaken with a common research question. RESULTS-The two reviews find similar answers to the research question, although the means of arriving at those answers vary. CONCLUSIONS-In addressing a well-bounded research question, groups of researchers with similar domain experience can arrive at the same review outcomes, even though they may do so in different ways. This provides evidence that, in this context at least, the systematic review is a robust research method.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The same papers are identified in the snowballing replication as in the previous systematic literature reviews.
“The same papers are identified.”
From Guidelines for snowballing in systematic literature studies and a replication in software engineering (2014) · cites this paper as [11]
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredMacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
From Guidelines for snowballing in systematic literature studies and a replication in software engineering (2014) · cites this paper as [11]
MacDonell et al. conducted two systematic literature reviews in parallel to evaluate the reliability of literature reviews.
“In this paper, the authors conducted two systematic literature reviews in parallel to evaluate the reliability of literature reviews.”
From Guidelines for snowballing in systematic literature studies and a replication in software engineering (2014) · cites this paper as [11]
In this particular case, the research question posed in MacDonell et al. was a good starting point.
“In this particular case, the research question posed in [11] was a good starting point.”
From Guidelines for snowballing in systematic literature studies and a replication in software engineering (2014) · cites this paper as [11]
In MacDonell et al.'s case, the systematic literature review seemed to be robust to differences in process and people, and it produced stable outcomes.
“In their case, the systematic literature review seemed to be robust to differences in process and people, and it produced stable outcomes.”
From Guidelines for snowballing in systematic literature studies and a replication in software engineering (2014) · cites this paper as [11]
Nine studies are in common with both previous systematic literature reviews presented in [11].
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
From Guidelines for snowballing in systematic literature studies and a replication in software engineering (2014) · cites this paper as [11]
If using the papers studied in detail as an efficiency measure, the different approaches require about the same effort.
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
From Guidelines for snowballing in systematic literature studies and a replication in software engineering (2014) · cites this paper as [11]
The paper to exclude is P11.
“However, it is interesting to note that the paper to exclude is P11, which somewhat surprisingly was not cited by P1-P3.”
From Guidelines for snowballing in systematic literature studies and a replication in software engineering (2014) · cites this paper as [11]
The Q1.6 ‘Papers included’ passage says a 2003 Mendes, Mosley, and Counsell study should not have been included, but it does not identify that study as P11. Searches for ‘P11’ and ‘paper to exclude is P11’ found no passage establishing that identification; the copy also does not address whether P1–P3 cited it. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 30 · Cited by 108
Research works printed but not linked: 1
- [10]10. A. Oakley and D. Fullerton, A Systematic Review of Smoking Prevention Programmes for Young People, EPPI Centre, Inst. for Education, 1995.
- Authors
- MacDonell, S., Shepperd, M., Kitchenham, B., & Mendes, E.
- Affiliations
- Auckland University of Technology (NZ)
- Brunel University of London (GB)
- Keele University (GB)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2010-09
- Volume
- 36
- Issue
- 5
- Pages
- 676-687
- ISSN
- 0098-5589 (ISSN-L)
- OpenAlex
- W2112428464
- Cited by
- 108
- References
- 30
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 9 · 2023: 6 · 2022: 8 · 2021: 10 · 2020: 7 · 2019: 4 · 2018: 12 · 2017: 5 · 2016: 7 · 2015: 6 · 2014: 9 · 2013: 6 · 2012: 8
- FWCI
- 3.48
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Tuwhera (Auckland University of Technology) (submittedVersion, public-domain)
- Tuwhera (Auckland University of Technology) (submittedVersion)
- Brunel University Research Archive (BURA) (Brunel University London) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/E046983/, EP/E046983/1)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- systematic reviews, reliability statistics, reproducibility, empirical software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
