The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Experiences using systematic review guidelines
Abstract
A systematic review is a defined and methodical way to identify, assess and analyse published primary studies in order to investigate a specific research question. Kitchenham has recently published guidelines for software engineering researchers performing systematic reviews. The objective of our paper is to critique Kitchenham's guidelines and to comment on systematic review generally with respect to our experiences conducting our first systematic review. Our perspective as neophytes may be particularly illuminating for other software engineering researchers who are also considering conducting their first systematic review. Overall we can recommend Kitchenham's guidelines to other researchers considering systematic reviews. We caution researchers to clearly and narrowly define the research questions they will investigate by systematic review, to reduce the overall effort and to improve the quality of the selection of papers and extraction of data. In particular we recommend defining complementary research questions that are not within the scope of the systematic review in order to clarify the boundaries of the specific research question of interest. An instance of this recommendation is that researchers should clearly define the unit of study for the systematic review.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1984) · Cited by 81,983
Case study research: design and methods
Phelan, S. (2011) · Evaluation & Research in Education · doi:10.1080/09500790.2011.582317 · Cited by 21,763
Hartley, J. (2004) · Essential Guide to Qualitative Methods in Organizational Research · doi:10.4135/9781446280119.n26 · Cited by 15,461
Evidence-Based Medicine: How to Practice and Teach EBM
Sackett, D. L. (2000) · Cited by 5,950
(2004) · Cited by 5,198
Evidence-Based Medicine: How to Practice and Teach EBM
Satya-Murti, S. (2000) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.284.18.2382 · Cited by 1,643
Preliminary guidelines for empirical research in software engineering
Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., El Emam, K., & Rosenberg, J. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1027796 · Cited by 1,510
A rational design process: How and why to fake it
Parnas, D. L., & Clements, P. C. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312940 · Cited by 617
Systematic Review in Software Engineering
Biolchini, J. C. D. A., & Mian, P. G. (2005) · Cited by 561
A rational design process: how and why to fake it
Parnas, D. L., & Clements, P. (2001) · Addison-Wesley Longman Publishing Co., Inc. eBooks · Cited by 200
Cited by
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.07.009 · Cited by 2,394
A systematic review of systematic review process research in software engineering
Kitchenham, B., & Brereton, P. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.07.010 · Cited by 1,094
Software fault prediction metrics: A systematic literature review
Radjenović, D., Heričko, M., Torkar, R., & Živkovič, A. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.02.009 · Cited by 538
On being ‘Systematic’ in Literature Reviews in IS
Boell, S. K., & Cecez-Kecmanovic, D. (2015) · Journal of Information Technology · doi:10.1057/jit.2014.26 · Cited by 485
A Systematic Survey of Program Comprehension through Dynamic Analysis
Cornelissen, B., Zaidman, A., van Deursen, A., Moonen, L., & Koschke, R. (2009) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2009.28 · Cited by 420
Metadata
- Authors
- Staples, M., & Niazi, M.
- Venue
- Journal of Systems and Software
- Publisher
- Elsevier BV
- Published
- 2007-09
- Volume
- 80
- Issue
- 9
- Pages
- 1425-1437
- ISSN
- 0164-1212 (ISSN-L)
- OpenAlex
- W2096598237
- Cited by
- 313
- References
- 14
- FWCI
- 3.45
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Experiences using systematic review guidelines. https://app.cochat.ai/papers/10.1016/j.jss.2006.09.046
- The paper
- Staples, M., & Niazi, M. (2007). Experiences using systematic review guidelines. Journal of Systems and Software. https://doi.org/10.1016/j.jss.2006.09.046
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
