Using Forward Snowballing to update Systematic Reviews in Software Engineering
The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Background: A Systematic Literature Review (SLR) is a methodology used to aggregate relevant evidence related to one or more research questions. Whenever new evidence is published after the completion of a SLR, this SLR should be updated in order to preserve its value. However, updating SLRs involves significant effort. Objective: The goal of this paper is to investigate the application of forward snowballing to support the update of SLRs. Method: We compare outcomes of an update achieved using the forward snowballing versus a published update using the search-based approach, i.e., searching for studies in electronic databases using a search string. Results: Forward snowballing showed a higher precision and a slightly lower recall. It reduced in more than five times the number of primary studies to filter however missed one relevant study. Conclusions: Due to its high precision, we believe that the use of forward snowballing considerably reduces the effort in updating SLRs in Software Engineering; however the risk of missing relevant papers should not be underrated.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 1×
“Although Webster and Watson [20] have recommended to use database search as the main approach for conducting the search of studies in SLRs, there are other search strategies that can be employed.”
MethodWohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 6×
“Wohlin [23] carried out the comparison using the results from applying snowballing to the results from a previously conducted SLR that employed database searches.”
BackgroundMethodInfluentialBrereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.07.009 · Cited by 2,394 · cited 1×
“Moreover, electronic publication databases were not designed to support SLRs [18][19].”
BackgroundA systematic review of systematic review process research in software engineering
Kitchenham, B., & Brereton, P. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.07.010 · Cited by 1,094
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 1×
“The first step of the snowballing process involves the identification of a set of studies as a starting point (seed set) [21].”
BackgroundAnalyzing the past to prepare for the future
(2002) · MIS Quarterly · doi:10.5555/2017160.2017162 · Cited by 179
Cited by
Wohlin, C., Kalinowski, M., Romero Felizardo, K., & Mendes, E. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106908 · Cited by 432
Self-citationLwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241
Mourão, E., Pimentel, J. F., Murta, L., Kalinowski, M., Mendes, E., & Wohlin, C. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106294 · Cited by 191
“Precision and recall are commonly used in other studies that investigate search strategies [5, 9, 21, 25, [27]].”
BackgroundMethodSelf-citationGuidelines for the Search Strategy to Update Systematic Literature Reviews in Software Engineering
Wohlin, C., Mendes, E., Felizardo, K. R., & Kalinowski, M. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106366 · Cited by 147
“Finally, the second replication of the SLR-update U1 – SLR-update-R2 [8] , was published in Q3 2016 and was co-authored by three of the four co-authors of this paper – Felizardo, Mendes and Kalinowski.”
InfluentialSelf-citationFinding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83
“[16] and Wohlin [62] applied forward snowballing to update SE SLRs and greatly reduced the number studies need to be reviewed comparing to”
Background
Metadata
- Authors
- Felizardo, K. R., Mendes, E., Kalinowski, M., Souza, É. F., & Vijaykumar, N. L.
- Venue
- Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement
- Publisher
- ACM
- Published
- 2016-09-08
- Pages
- 1-6
- ISSN
- 1938-6451 (ISSN-L)
- OpenAlex
- W2522451690
- Cited by
- 102
- References
- 6
- FWCI
- 2.81
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using Forward Snowballing to update Systematic Reviews in Software Engineering. https://app.cochat.ai/papers/10.1145/2961111.2962630
- The paper
- Felizardo, K. R., Mendes, E., Kalinowski, M., Souza, É. F., & Vijaykumar, N. L. (2016). Using Forward Snowballing to update Systematic Reviews in Software Engineering. Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. https://doi.org/10.1145/2961111.2962630
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.