The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Experiences using systematic review guidelines
Abstract
A systematic review is a defined and methodical way to identify, assess and analyse published primary studies in order to investigate a specific research question. Kitchenham has recently published guidelines for software engineering researchers performing systematic reviews. The objective of our paper is to critique Kitchenham's guidelines and to comment on systematic review generally with respect to our experiences conducting our first systematic review. Our perspective as neophytes may be particularly illuminating for other software engineering researchers who are also considering conducting their first systematic review. Overall we can recommend Kitchenham's guidelines to other researchers considering systematic reviews. We caution researchers to clearly and narrowly define the research questions they will investigate by systematic review, to reduce the overall effort and to improve the quality of the selection of papers and extraction of data. In particular we recommend defining complementary research questions that are not within the scope of the systematic review in order to clarify the boundaries of the specific research question of interest. An instance of this recommendation is that researchers should clearly define the unit of study for the systematic review.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1984) · Cited by 81,983
Case study research: design and methods
Phelan, S. (2011) · Evaluation & Research in Education · doi:10.1080/09500790.2011.582317 · Cited by 21,763
Hartley, J. (2004) · Essential Guide to Qualitative Methods in Organizational Research · doi:10.4135/9781446280119.n26 · Cited by 15,461
Evidence-Based Medicine: How to Practice and Teach EBM
Sackett, D. L. (2000) · Cited by 5,950
(2004) · Cited by 5,198
Evidence-Based Medicine: How to Practice and Teach EBM
Satya-Murti, S. (2000) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.284.18.2382 · Cited by 1,643
Preliminary guidelines for empirical research in software engineering
Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., El Emam, K., & Rosenberg, J. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1027796 · Cited by 1,510
A rational design process: How and why to fake it
Parnas, D. L., & Clements, P. C. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312940 · Cited by 617
Systematic Review in Software Engineering
Biolchini, J. C. D. A., & Mian, P. G. (2005) · Cited by 561
A rational design process: how and why to fake it
Parnas, D. L., & Clements, P. (2001) · Addison-Wesley Longman Publishing Co., Inc. eBooks · Cited by 200
Cited by
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.07.009 · Cited by 2,394
A systematic review of systematic review process research in software engineering
Kitchenham, B., & Brereton, P. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.07.010 · Cited by 1,094
Pro-environmental behaviors through the lens of the theory of planned behavior: A scoping review
Yuriev, A., Dahmen, M., Paillé, P., Boiral, O., & Guillaumie, L. (2020) · Resources, Conservation and Recycling · doi:10.1016/j.resconrec.2019.104660 · Cited by 907
Software fault prediction metrics: A systematic literature review
Radjenović, D., Heričko, M., Torkar, R., & Živkovič, A. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.02.009 · Cited by 538
On being ‘Systematic’ in Literature Reviews in IS
Boell, S. K., & Cecez-Kecmanovic, D. (2015) · Journal of Information Technology · doi:10.1057/jit.2014.26 · Cited by 485
A Systematic Survey of Program Comprehension through Dynamic Analysis
Cornelissen, B., Zaidman, A., van Deursen, A., Moonen, L., & Koschke, R. (2009) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2009.28 · Cited by 420
Applying Systematic Reviews to Diverse Study Types: An Experience Report
Dyba, T., Dingsoyr, T., & Hanssen, G. K. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.59 · Cited by 391
Plant diseases recognition on images using convolutional neural networks: A systematic review
Abade, A., Ferreira, P. A., & de Barros Vidal, F. (2021) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2021.106125 · Cited by 342
Overcoming the barriers to pro-environmental behaviors in the workplace: A systematic review
Yuriev, A., Boiral, O., Francoeur, V., & Paillé, P. (2018) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.02.041 · Cited by 335
A Visual Text Mining approach for Systematic Reviews
Malheiros, V., Hohn, E., Pinho, R., Mendonca, M., & Maldonado, J. C. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.21 · Cited by 329
A systematic review on regression test selection techniques
Engström, E., Runeson, P., & Skoglund, M. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2009.07.001 · Cited by 311
Evaluation and Measurement of Software Process Improvement—A Systematic Literature Review
Unterkalmsteiner, M., Gorschek, T., Islam, A. K. M. M., Cheng, C. K., Permadi, R. B., & Feldt, R. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.26 · Cited by 291
Aquilani, B., Silvestri, C., Ruggieri, A., & Gatti, C. (2017) · The TQM Journal · doi:10.1108/tqm-01-2016-0003 · Cited by 273
Mobile technologies and applications towards smart tourism – state of the art
Dorcic, J., Komsic, J., & Markovic, S. (2019) · Tourism Review · doi:10.1108/tr-07-2017-0121 · Cited by 270
Strength of evidence in systematic reviews in software engineering
Dybå, T., & Dingsøyr, T. (2008) · Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/1414004.1414034 · Cited by 261
Variability in Software Systems—A Systematic Literature Review
Galster, M., Weyns, D., Tofan, D., Michalik, B., & Avgeriou, P. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.56 · Cited by 259
Employee green behaviour: A review and recommendations for future research
Tang, G., Ren, S., Wang, M., Li, Y., & Zhang, S. (2023) · International Journal of Management Reviews · doi:10.1111/ijmr.12328 · Cited by 138
On using grey literature and google scholar in systematic literature reviews in software engineering
Yasin, A., Fatima, R., Wen, L., Afzal, W., Azhar, M., & Torkar, R. (2020) · IEEE Access · doi:10.1109/access.2020.2971712 · Cited by 85
Tools to support systematic reviews in software engineering: a feature analysis
Marshall, C., Brereton, P., & Kitchenham, B. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601270 · Cited by 56
Marshall, C., Brereton, P., & Kitchenham, B. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745827 · Cited by 46
Metadata
- Authors
- Staples, M., & Niazi, M.
- Venue
- Journal of Systems and Software
- Publisher
- Elsevier BV
- Published
- 2007-09
- Volume
- 80
- Issue
- 9
- Pages
- 1425-1437
- ISSN
- 0164-1212 (ISSN-L)
- OpenAlex
- W2096598237
- Cited by
- 313
- References
- 14
- FWCI
- 3.45
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.