Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 101–200 of 3,934
A Survey on Unmanned Surface Vehicles for Disaster Robotics: Main Challenges and Directions
Jorge, V. A. M., Granada, R., Maidana, R. G., Jurak, D. A., Heck, G., Negreiros, A. P. F., dos Santos, D. H., Gonçalves, L. M. G., & Amory, A. M. (2019) · Sensors · doi:10.3390/s19030702 · Cited by 202
“Then, we performed a snowball search approach [52] to cover references and citing work.”
MethodEmodi, N. V., Lovell, H., Levitt, C., & Franklin, E. (2021) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2021.06.212 · Cited by 203
Alshami, A., Elsayed, M., Ali, E., Eltoukhy, A. E. E., & Zayed, T. (2023) · Systems · doi:10.3390/systems11070351 · Cited by 202
Roos Lindgreen, E., Salomone, R., & Reyes, T. (2020) · Sustainability · doi:10.3390/su12124973 · Cited by 200
“The resulting inventory is complemented by applying the snowballing method to previously conducted CE measurement reviews [38].”
MethodFactors influencing users' adoption and use of conversational agents: A systematic review
Ling, E. C., Tussyadiah, I., Tuomi, A., Stienmetz, J., & Ioannou, A. (2021) · Psychology & Marketing · doi:10.1002/mar.21491 · Cited by 199
Larsen, V. G., Tollin, N., Sattrup, P. A., Birkved, M., & Holmboe, T. (2022) · Journal of Building Engineering · doi:10.1016/j.jobe.2022.104203 · Cited by 199
“Therefore, forward and backwards reference tracking and snowballing methods [65], managed by the setup presented in figure 2, were applied to prioritise the articles and to include a minor number of additional scientific literature cited in the previously identified articles.”
Developing a sustainable smart city framework for developing economies: An Indian context
Yadav, G., Mangla, S. K., Luthra, S., & Rai, D. P. (2019) · Sustainable Cities and Society · doi:10.1016/j.scs.2019.101462 · Cited by 198
Aguilar-Hernandez, G. A., Dias Rodrigues, J. F., & Tukker, A. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.123421 · Cited by 198
Bak, O., Shaw, S., Colicchia, C., & Kumar, V. (2023) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2020.3016988 · Cited by 198
“As far as the “snowballing technique” is concerned, we conducted a backward and forward snowballing following the steps proposed by Wohlin [115]: publications referenced from and referencing to previously selected studies were considered iteratively.”
BackgroundMethodInfluentialHow Far We Have Progressed in the Journey? An Examination of Cross-Project Defect Prediction
Zhou, Y., Yang, Y., Lu, H., Chen, L., Li, Y., Zhao, Y., Qian, J., & Xu, B. (2018) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3183339 · Cited by 196
“”(2) First, we did a forward snowballing search [114] by recursively examining the citations of the “BMW 2002 TSE paper”.”
MethodA review-based comparative study of bad smell detection tools
Fernandes, E., Oliveira, J., Vale, G., Paiva, T., & Figueiredo, E. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2915984 · Cited by 192
“However, after conducting the snowballing step, a search approach that uses citations in papers from a SLR as reference list to identify additional papers not covered by the SLR [99], we found relevant research papers published since 1993.”
MethodSafety in Vehicle Platooning: A Systematic Literature Review
Axelsson, J. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2016.2598873 · Cited by 192
“For the papers selected as relevant from Scopus, a complementary snowballing search [33] was conducted, in which references to and from those papers were followed to check for additional literature.”
MethodMourão, E., Pimentel, J. F., Murta, L., Kalinowski, M., Mendes, E., & Wohlin, C. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106294 · Cited by 191
“On the other hand, Wohlin [2] recommends iteratively identifying papers based on the reference list and the citations of papers that were found by an informal database search.”
BackgroundMethodInfluentialSelf-citationSavaget, P., Geissdoerfer, M., Kharrazi, A., & Evans, S. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.09.208 · Cited by 191
Ullah, U., & Garcia-Zapirain, B. (2024) · IEEE Access · doi:10.1109/access.2024.3353461 · Cited by 188
“By using a snowballing strategy [29], duplicate records were eliminated and additional records were added, which were gathered based on references in the papers and personal readings.”
MethodGiray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Critical risk factors in the application of modular integrated construction: a systematic review
Wuni, I. Y., Shen, G. Q. P., & Mahmud, A. T. (2022) · International Journal of Construction Management · doi:10.1080/15623599.2019.1613212 · Cited by 185
“The snowballing search is an iterative process of locating additional studies based on the reference lists (backward snowballing) and citations (forward snowballing) of the previously identified 29 articles (Wohlin 2014).”
BackgroundMethodThe State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations
Chatzimparmpas, A., Martins, R. M., Jusufi, I., Kucher, K., Rossi, F., & Kerren, A. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14034 · Cited by 182
Smells in software test code: A survey of knowledge in industry and academia
Garousi, V., & Küçük, B. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2017.12.013 · Cited by 186
A systematic review on teaching and learning robotics content knowledge in K-12
Xia, L., & Zhong, B. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.09.007 · Cited by 183
When and what to automate in software testing? A multi-vocal literature review
Garousi, V., & Mäntylä, M. V. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.04.015 · Cited by 182
“Snowballing, in this context, refers to using the reference list of a paper (backward snowballing) or the citations to the paper to identify additional papers (forward) [78].”
BackgroundMethodFrom explainable to interactive AI: A literature review on current trends in human-AI interaction
Raees, M., Meijerink, I., Lykourentzou, I., Khan, V.-J., & Papangelis, K. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103301 · Cited by 180
Lean Startup: a comprehensive historical review
Bortolini, R. F., Nogueira Cortimiglia, M., Danilevicz, A. D. M. F., & Ghezzi, A. (2021) · Management Decision · doi:10.1108/md-07-2017-0663 · Cited by 178
A Systematic Mapping Study on Microservices Architecture in DevOps
Waseem, M., Liang, P., & Shahin, M. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110798 · Cited by 175
“In Phase 2, we inspected the references of the primary studies through the snowballing techniques [30] for further identification of the relevant studies.”
BackgroundMethodInfluentialDriving Behavior Analysis Guidelines for Intelligent Transportation Systems
Nasr Azadani, M., & Boukerche, A. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3076140 · Cited by 173
A systematic mapping study of infrastructure as code research
Rahman, A., Mahdavi-Hezaveh, R., & Williams, L. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.12.004 · Cited by 174
“Jalali and Wohlin [24] studied 77 papers related to the adoption of agile practices in global software engineering and reported that in majority of the papers agile practices were modified with respect to the context and situational requirements.”
MethodInfluentialVat, L. E., Finlay, T., Jan Schuitmaker‐Warnaar, T., Fahy, N., Robinson, P., Boudes, M., Diaz, A., Ferrer, E., Hivert, V., Purman, G., et al. (2020) · Health Expectations · doi:10.1111/hex.12951 · Cited by 173
Virtual Reality for Emotion Elicitation – A Review
Somarathna, R., Bednarz, T., & Mohammadi, G. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3181053 · Cited by 171
“PRISMA [28] and snowballing [29] approach for selecting the papers from 2017-2021. we have witnessed a revolutionary development of VR-related studies in the literature, we have limited the focus of this review to the evaluation of the emotional induction material, which is the primary basis for…”
BackgroundMethodEthics of AI: a systematic literature review of principles and challenges
Khan, A. A., Badshah, S., Liang, P., Waseem, M., Khan, B., Ahmad, A., Fahmideh, M., Niazi, M., & Akbar, M. A. (2022) · The International Conference on Evaluation and Assessment in Software Engineering 2022 · doi:10.1145/3530019.3531329 · Cited by 170
“Moreover, backward snowballing [17] is performed to search the references of the selected 24 studies.”
MethodUnderstanding flaky tests: the developer’s perspective
Eck, M., Palomba, F., Castelluccio, M., & Bacchelli, A. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338945 · Cited by 169
“We also perform forward and backward snowballing [38], inspecting sources referred or that refer to those belonging to the initial set of primary studies.”
On the interplay of Internet of Things and Cloud Computing: A systematic mapping study
Cavalcante, E., Pereira, J., Alves, M. P., Maia, P., Moura, R., Batista, T., Delicato, F. C., & Pires, P. F. (2016) · Computer Communications · doi:10.1016/j.comcom.2016.03.012 · Cited by 167
Neri, A., Cagno, E., Lepri, M., & Trianni, A. (2021) · Sustainable Production and Consumption · doi:10.1016/j.spc.2020.12.018 · Cited by 170
Requirements engineering: A systematic mapping study in agile software development
Curcio, K., Navarro, T., Malucelli, A., & Reinehr, S. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.01.036 · Cited by 169
The effects of educational robotics in STEM education: a multilevel meta-analysis
Ouyang, F., & Xu, W. (2024) · International Journal of STEM Education · doi:10.1186/s40594-024-00469-4 · Cited by 171
“Additionally, snowball-ing approach, was also utilized (Wohlin, 2014) to find the articles that were not extracted in database search through citation checking.”
MethodIntelligent tutoring systems for programming education: a systematic review
Crow, T., Luxton-Reilly, A., & Wuensche, B. (2018) · Proceedings of the 20th Australasian Computing Education Conference · doi:10.1145/3160489.3160492 · Cited by 166
“Backward snowballing as outlined in ’Guidelines for snowballing in systematic literature studies and a replication in software engineer-ing’ [28] was carried out for papers cited within the literature that met the inclusion criteria; this was done to ensure that research and IPTs that were…”
MethodCircular economy in corporate sustainability reporting: A review of organisational approaches
Opferkuch, K., Caeiro, S., Salomone, R., & Ramos, T. B. (2021) · Business Strategy and the Environment · doi:10.1002/bse.2854 · Cited by 168
A systematic review of natural language processing applied to radiology reports.
Casey, A., Davidson, E. M., Poon, M. T. C., Dong, H., Duma, D., Grivas, A., Grover, C., Suárez-Paniagua, V., Tobin, R., Whiteley, W. N., et al. (2021) · UCL Discovery (University College London) · Cited by 160
Deployment and communication patterns in microservice architectures: A systematic literature review
Karabey Aksakalli, I., Çelik, T., Can, A. B., & Teki̇nerdoğan, B. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111014 · Cited by 164
“At the end of the study selection process, we applied backward and forward snowballing activities when selecting the primary studies (Wohlin, 2014).”
BackgroundMethodSecurity Analysis of IoT Devices by Using Mobile Computing: A Systematic Literature Review
Liao, B., Ali, Y., Nazir, S., He, L., & Khan, H. U. (2020) · IEEE Access · doi:10.1109/access.2020.3006358 · Cited by 162
“According to [44], automatic research is better than manual research.”
BackgroundMethodInfluentialClimate change, mental health, and reproductive decision-making: A systematic review
Dillarstone, H., Brown, L. J., & Flores, E. C. (2023) · PLOS Climate · doi:10.1371/journal.pclm.0000236 · Cited by 162
Blockchain-Enabled Information Sharing Within a Supply Chain: A Systematic Literature Review
Wan, P. K., Huang, L., & Holtskog, H. (2020) · IEEE Access · doi:10.1109/access.2020.2980142 · Cited by 160
“A systematic review will provide a clearer picture on current exploration and research work on information sharing within a supply chain [13], [14].”
BackgroundA systematic literature review of literature reviews in software testing
Garousi, V., & Mäntylä, M. V. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.09.002 · Cited by 160
“To decrease the risk of missing relevant studies, similar to previous SM and SLR studies, we conducted selected snowballing [35] (forward and backward) as well, in which we found 6 additional papers as shown in Figure 1.”
MethodImplementing sustainable design theory in business practice: A call to action
Baldassarre, B., Keskin, D., Diehl, J. C., Bocken, N., & Calabretta, G. (2020) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.123113 · Cited by 160
“After identifying the two review papers, we leveraged them to engage in an iterative literature search based on a snowballing approach (Wohlin, 2014).”
MethodVoice Assistants and Smart Speakers in Everyday Life and in Education
TERZOPOULOS, G., & SATRATZEMI, M. (2020) · Informatics in Education · doi:10.15388/infedu.2020.21 · Cited by 161
The ensemble approach to forecasting: A review and synthesis
Wu, H., & Levinson, D. (2021) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2021.103357 · Cited by 160
Lenarduzzi, V., Besker, T., Taibi, D., Martini, A., & Fontana, F. A. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110827 · Cited by 240
“In order to mitigate this threat, we carefully applied the search strategy, performing the search in eight digital libraries in conjunction with the snowballing process (Wohlin, 2014), considering all the references presented in the retrieved papers, and evaluating all the papers that reference the retrieved ones, whi…”
MethodInfluentialIndustrial sustainability performance measurement systems: A novel framework
Cagno, E., Neri, A., Howard, M., Brenna, G., & Trianni, A. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.05.021 · Cited by 158
“Taking inspiration from previous research, we also searched for additional relevant literature looking at references and citations of the initial set of selected contributions using the snowball method (Heckathorn and Cameron, 2017; Skolarus et al., 2017; Wohlin, 2014).”
MethodCantú, A., Aguiñaga, E., & Scheel, C. (2021) · Sustainability · doi:10.3390/su13031529 · Cited by 158
“Finally, a snowball procedure was carried out, in which additional papers were identified using the reference lists [39].”
MethodAdaptive Reuse of Heritage Buildings: From a Literature Review to a Model of Practice
Arfa, F. H., Zijlstra, H., Lubelli, B., & Quist, W. (2022) · The Historic Environment: Policy & Practice · doi:10.1080/17567505.2022.2058551 · Cited by 159
The uses of Patient Reported Experience Measures in health systems: A systematic narrative review
Gilmore, K. J., Corazza, I., Coletta, L., & Allin, S. M. (2022) · Health Policy · doi:10.1016/j.healthpol.2022.07.008 · Cited by 201
Ong, M., Jaumot‐Pascual, N., & Ko, L. T. (2020) · Journal of Engineering Education · doi:10.1002/jee.20345 · Cited by 197
“We compared GS, the engine recommended by Wohlin (2014), to many other engines and directories, including Engineering Village, ERIC, and open source dissertation directories such as oatd.org.”
MethodZieliński, G., Pająk, A., & Wójcicki, M. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13144259 · Cited by 235
“Additionally, the snowballing procedure method was applied according to Wohlin’s recommendations [59] (Figure 1).”
MethodHow to explain AI systems to end users: a systematic literature review and research agenda
Laato, S., Tiainen, M., Najmul Islam, A. K. M., & Mäntymäki, M. (2022) · Internet Research · doi:10.1108/intr-08-2021-0600 · Cited by 155
Applications of deep learning for phishing detection: a systematic literature review
Catal, C., Giray, G., Tekinerdoğan, B., Kumar, S., & Shukla, S. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01672-x · Cited by 225
“T h e i n f e r e n ce s m a x i m i ze t h e li k e li hood C on s : T h e t r a i n i ng p r o ce ss m a y b e e xp e n s i v e , j u s t li k e RB M Forward and backward snowballing followed the database search to identify the primary studies that might have been overlooked [79].”
BackgroundSaade, M. R. M., Yahia, A., & Amor, B. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.118803 · Cited by 186
Including non‐randomized studies on intervention effects
Reeves, B. C., Deeks, J. J., Higgins, J. P., Shea, B., Tugwell, P., Wells, G. A., & on behalf of the Cochrane Non‐Randomized Studies of Interventions Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch24 · Cited by 153
Quality of service approaches in IoT: A systematic mapping
White, G., Nallur, V., & Clarke, S. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.05.125 · Cited by 176
A Survey on Automated Driving System Testing: Landscapes and Trends
Tang, S., Zhang, Z., Zhang, Y., Zhou, J., Guo, Y. R., Liu, S., Guo, S., Li, Y., Ma, L., Xue, Y., et al. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3579642 · Cited by 221
“First, we perform snowballing [36] on each of the remaining papers after abstract analysis.”
MethodHuman Factors in Phishing Attacks: A Systematic Literature Review
Desolda, G., Ferro, L. S., Marrella, A., Catarci, T., & Costabile, M. F. (2022) · ACM Computing Surveys · doi:10.1145/3469886 · Cited by 148
“We intentionally excluded “cybersecurity/cyber security/cyber-security” because these terms are too general for a focused search.”
Soltanian, S., Kalogirou, S. A., Ranjbari, M., Amiri, H., Mahian, O., Khoshnevisan, B., Jafary, T., Nizami, A., Gupta, V. K., Aghaei, S., et al. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.111975 · Cited by 167
Connect API with Blockchain: A Survey on Blockchain Oracle Implementation
Pasdar, A., Lee, Y. C., & Dong, Z. (2022) · ACM Computing Surveys · doi:10.1145/3567582 · Cited by 211
“During the paper collection, snowballing technique [91] is employed to collect relevant studies for the literature review.”
MethodReview of visualising LCA results in the design process of buildings
Hollberg, A., Kiss, B., Röck, M., Soust-Verdaguer, B., Wiberg, A. H., Lasvaux, S., Galimshina, A., & Habert, G. (2020) · Building and Environment · doi:10.1016/j.buildenv.2020.107530 · Cited by 166
Gordon, J. A., Balta‐Ozkan, N., & Nabavi, S. A. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.120850 · Cited by 175
Guidelines for the Search Strategy to Update Systematic Literature Reviews in Software Engineering
Wohlin, C., Mendes, E., Felizardo, K. R., & Kalinowski, M. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106366 · Cited by 147
“He followed his proposed method, as described in [59] .”
Self-citationAkinbi, A., Forshaw, M. J., & Blinkhorn, V. (2021) · Health Information Science and Systems · doi:10.1007/s13755-021-00147-7 · Cited by 178
“Moreover, we implemented forward and backward snowballing iterations [47] to ensure that all the selected publications were relevant and met the inclusion criteria.”
MethodGeopolymer: A Systematic Review of Methodologies
Matsimbe, J. J., Dinka, M. O., Olukanni, D. O., & Musonda, I. (2022) · Materials · doi:10.3390/ma15196852 · Cited by 186
“After clarifying the main inclusion criteria, all the references of the selected articles were subjected to the snowballing technique [54] to gather additional relevant information.”
MethodÇetin, S., Raghu, D., Honic, M., Straub, A., & Gruis, V. (2023) · Sustainable Production and Consumption · doi:10.1016/j.spc.2023.07.011 · Cited by 147
“Following a snowballing procedure (Wohlin, 2014), eight additional papers were added.”
Digital transformation of business model in manufacturing companies: challenges and research agenda
Favoretto, C., Mendes, G. H. D. S., Filho, M. G., de Oliveira, M. G., & Ganga, G. M. D. (2021) · Journal of Business and Industrial Marketing · doi:10.1108/jbim-10-2020-0477 · Cited by 211
Systematic literature review of validation methods for AI systems
Myllyaho, L., Raatikainen, M., Männistö, T., Mikkonen, T., & Nurminen, J. K. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111050 · Cited by 145
“Exclude search hits already found in previous databases Automatic search was chosen over manual search and snow-balling [20] due to the wide range of domains in which AI can be of use.”
BackgroundReengineering legacy applications into software product lines: a systematic mapping
Assunção, W. K. G., Lopez-Herrejon, R. E., Linsbauer, L., Vergílio, S. R., & Egyed, A. F. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9499-z · Cited by 164
“In the snowballing reading, citations and the reference list of found papers are used to identify other relevant studies (Wohlin 2014).”
MethodRecent dynamic vehicle routing problems: A survey
Rios, B. H. O., Xavier, E. C., Miyazawa, F. K., Amorim, P., Curcio, E., & Santos, M. J. (2021) · Computers & Industrial Engineering · doi:10.1016/j.cie.2021.107604 · Cited by 195
den Haan, R., & van der Voort, M. C. (2018) · Sustainability · doi:10.3390/su10124529 · Cited by 176
“To cover for relevant publications missed in the database searches, a snowballing approach was used to identify further relevant publications [46].”
MethodSalehi, S., Arashpour, M., Kodikara, J. K., & Guppy, R. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2021.127936 · Cited by 168
Parry, O., Kapfhammer, G. M., Hilton, M. L., & McMinn, P. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3476105 · Cited by 242
Aros, L. H., Molano, L. X. B., Gutiérrez-Portela, F., Hernandez, J. J. M., & Barrero, M. S. R. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03606-0 · Cited by 325
“Following the strategy proposed by Wohlin (2014), a snowball search was conducted.”
Decomposition of Monolith Applications Into Microservices Architectures: A Systematic Review
Abgaz, Y. M., McCarren, A. L., Elger, P., Solan, D., Lapuz, N., Bivol, M., Jackson, G., Yılmaz, M., Buckley, J., & Clarke, P. M. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3287297 · Cited by 233
“Snowballing: One forward and backward snowballing iteration [88] was then conducted by the first author, on the 33 papers.”
Batı, K. (2021) · Education and Information Technologies · doi:10.1007/s10639-021-10700-2 · Cited by 196
“As previously mentioned, discussions regarding the effectiveness of plugged-in and unplugged applications are ongoing (Kazakoff & Bers, 2012, 2014; Kazakoff et al., 2013).”
BackgroundThirteen years of SysML: a systematic mapping study
Wolny, S., Mazak, A., Carpella, C., Geist, V., & Wimmer, M. (2020) · Software and Systems Modeling · doi:10.1007/s10270-019-00735-y · Cited by 140
“In addition, there are some authors who have adopted surveys from medicine [35] as well as from social sciences [46], or they have applied refined guidelines like introduced in [11,14,60].”
Backgroundvon Wirth, T., Gislason, L., & Seidl, R. (2017) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2017.09.086 · Cited by 139
De Marchi, V., Di Maria, E., Golini, R., & Perri, A. (2020) · International Business Review · doi:10.1016/j.ibusrev.2020.101708 · Cited by 139
Reijers, W., Wright, D., Brey, P. A. E., Weber, K., Rodrigues, R., O’Sullivan, D., & Gordijn, B. (2017) · Science and Engineering Ethics · doi:10.1007/s11948-017-9961-8 · Cited by 180
“Athird selection of the literature was made according to the backwards-snowballing method (Wohlin 2014), looking at the reference lists of all useful sources found in the literature searches.”
MethodFrom monolithic systems to Microservices: An assessment framework
Auer, F., Lenarduzzi, V., Felderer, M., & Taibi, D. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106600 · Cited by 225
“In order to retrieve all relevant papers, we additionally integrated the procedure of also taking into account forward and backward systematic snowballing [15] on the 11 remaining papers.”
MethodInfluentialInternal audit effectiveness: operationalization and influencing factors
Türetken, O., Jethefer, S., & Özkan, B. (2019) · Managerial Auditing Journal · doi:10.1108/maj-08-2018-1980 · Cited by 156
Systemic formalisation of Cyber-Physical-Social System (CPSS): A systematic literature review
Yilma, B. A., Panetto, H., & Naudet, Y. (2021) · Computers in Industry · doi:10.1016/j.compind.2021.103458 · Cited by 160
“After retrieving 705 papers from the the database search, we included 34 additional papers using the “snowball sampling” technique [59].”
MethodTinoco, M. P., de Mendonça, É. M., Fernandez, L. I. C., Caldas, L. R., Reales, O. A. M., & Filho, R. D. T. (2022) · Journal of Building Engineering · doi:10.1016/j.jobe.2022.104456 · Cited by 165
A systematic literature review of innovative technologies adopted in logistics management
Lagorio, A., Zenezini, G., Mangano, G., & Pinto, R. (2020) · International Journal of Logistics Research and Applications · doi:10.1080/13675567.2020.1850661 · Cited by 198
“The approach to going through the papers is similar to in the backward method (Wohlin 2014).”
MethodInteroperability assessment: A systematic literature review
Leal, G. D. S. S., Guédria, W., & Panetto, H. (2019) · Computers in Industry · doi:10.1016/j.compind.2019.01.002 · Cited by 165
“As additional search strategy, we also include 29 papers using the “snowball sampling” technique [39] whereby we consider the referrals of assessment approaches made by experts, as well as the most cited papers in the existing INAS surveys and reviews.”
MethodRequirements engineering for artificial intelligence systems: A systematic mapping study
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107176 · Cited by 132
Execution of UML models: a systematic review of research and practice
Ciccozzi, F., Malavolta, I., & Selić, B. (2018) · Software & Systems Modeling · doi:10.1007/s10270-018-0675-4 · Cited by 170
“With snowballing, performed after the automatic search, we aimed at enlarging the set of potentially relevant solutions by considering each previously selected research study and focusing on those articles either citing or cited by it [49].”
MethodExplainable Student Performance Prediction Models: A Systematic Review
Alamri, R., & Alharbi, B. (2021) · IEEE Access · doi:10.1109/access.2021.3061368 · Cited by 194
“To ensure that our study captures the most number of relevant articles, we applied backward and forward snowballing [34].”
MethodVisual tools for teaching machine learning in K-12: A ten-year systematic mapping
Gresse von Wangenheim, C., Hauck, J. C. R., Pacheco, F. S., & Bertonceli Bueno, M. F. (2021) · Education and Information Technologies · doi:10.1007/s10639-021-10570-8 · Cited by 130
“In order to further minimize the risk of omission, we also included literature found via backward and forward snowballing (Wohlin, 2014).”
MethodMigrating from monolithic architecture to microservices: A Rapid Review
Ponce, F., Márquez, G., & Hernán Astudillo, R. (2019) · International Conference of the Chilean Computer Science Society (SCCC) · doi:10.1109/sccc49216.2019.8966423 · Cited by 199
“In the fourth round, solo research proceeded to perform a snowballing process, as recommended in [14], adding 3 more studies that meet all the criteria.”
MethodHussein, M., & Zayed, T. M. (2020) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.124716 · Cited by 130
“This step aims to filter the retrieved studies and keep only the studies that answer the main research question (Wohlin, 2014).”
Brinkman, D. J., Tichelaar, J., Graaf, S., Otten, R. H. J., Richir, M. C., & van Agtmael, M. A. (2018) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.13491 · Cited by 152
Systems Measures of a Trauma‐Informed Approach: A Systematic Review
Champine, R. B., Lang, J. M., Nelson, A. M., Hanson, R. F., & Tebes, J. K. (2019) · American Journal of Community Psychology · doi:10.1002/ajcp.12388 · Cited by 176
“A snowballing approach (Wohlin, 2014) was used if the resource uncovered in the initial search included references or citations that informed subsequent searches.”
MethodThe evolution of city-scale GHG emissions inventory methods: A systematic review
Arioli, M. S., D’Agosto, M. D. A., Amaral, F. G., & Cybis, H. B. B. (2019) · Environmental Impact Assessment Review · doi:10.1016/j.eiar.2019.106316 · Cited by 152
Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead
Zhou, X., Cao, S., Sun, X., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708522 · Cited by 132
“To avoid omitting any possibly relevant work during our manual and automated search process, we also performed lightweight backward and forward snowballing [81].”
Insights into startup ecosystems through exploration of multi-vocal literature
Tripathi, N., Seppänen, P., Boominathan, G., Oivo, M., & Liukkunen, K. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.08.005 · Cited by 193
“[26], snowballing [27], and multi-vocal literature review [9].”
BackgroundHospitality Industry 4.0 and Climate Change
Youssef, A. B., & Zeqiri, A. (2022) · Circular Economy and Sustainability · doi:10.1007/s43615-021-00141-x · Cited by 154
“Wohlin [50] states that snowballing can be particularly useful for systematic literature reviews:”
BackgroundMachine Learning-Based Models for Prediction of Toxicity Outcomes in Radiotherapy
Isaksson, L. J., Pepa, M., Zaffaroni, M., Marvaso, G., Alterio, D., Volpe, S., Corrao, G., Augugliaro, M., Starzyńska, A., Leonardi, M. C., et al. (2020) · Frontiers in Oncology · doi:10.3389/fonc.2020.00790 · Cited by 151
“Reference lists of selected articles were hand searched for further potential relevant papers and also using the Snowballing technique (9).”
Method
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.