Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 501–600 of 3,934
Brockway, M. L., Daniel, A. I., Reyes, S. M., Gauglitz, J. M., Granger, M. J., McDermid, J. M., Chan, D., Refvik, R., Sidhu, K. K., Musse, S., et al. (2023) · Advances in Nutrition · doi:10.1016/j.advnut.2023.09.015 · Cited by 89
A multivocal literature review on serious games for software process standards education
Calderón, A., Ruiz, M., & O’Connor, R. V. (2017) · Computer Standards & Interfaces · doi:10.1016/j.csi.2017.11.003 · Cited by 74
“In this phase, first forward and backward snowballing (Wohlin, 2014) will be conducted on the scientific primary studies.”
BackgroundJerčić, P., & Sundstedt, V. (2019) · Entertainment Computing · doi:10.1016/j.entcom.2019.01.001 · Cited by 69
“Furthermore, backward and forward snowballing was conducted using Google Scholar in addition to the previously mentioned electronic databases, as outlined in the [84].”
BackgroundMethodThe Mental Health of Elite-Level Coaches: A Systematic Scoping Review
Frost, J., Walton, C. C., Purcell, R., Fisher, K., Gwyther, K., Kocherginsky, M., & Rice, S. M. (2024) · Sports Medicine - Open · doi:10.1186/s40798-023-00655-8 · Cited by 59
“Following data charting, the first author (JF) conducted backward snowballing to ensure all relevant records had been identified within the included studies [71].”
MethodSoftware Engineering for AI-Based Systems: A Survey
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 335
“Therefore, we decided to execute a hybrid search strategy [139], applying both a search string in Scopus3 and a snowballing strategy [209].”
MethodAli, S. H., Foster, T., & Hall, N. L. (2018) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph15122827 · Cited by 57
“A manual search of additional articles was conducted using the ‘snowballing’ technique that investigated relevant citations in the included literature [28].”
On the search for industry-relevant regression testing research
bin Ali, N., Engström, E., Taromirad, M., et al. (2019) · Empirical Software Engineering · doi:10.1007/s10664-018-9670-1 · Cited by 57
Spinos, D., Skarentzos, K., Esagian, S. M., Seymour, K. A., & Economopoulos, K. P. (2021) · Obesity Surgery · doi:10.1007/s11695-020-05188-7 · Cited by 61
“Manual screening of the bibliography of the retrieved papers supplemented our search results, according to the “snowball” methodology [14].”
MethodSecurity testing of web applications: A systematic mapping of the literature
Aydos, M., ALDAN, C., Coşkun, E., & Soydan, A. (2021) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.09.018 · Cited by 106
“We conducted this study using well-known SLR and SLM guidelines in software engineering (B. Kitchenham, 2007; Kai et al., 2008; Petersen et al., 2015; Wohlin, 2014).”
InfluentialSoulé, E., Michonneau, P., Michel, N., & Bockstaller, C. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2021.129291 · Cited by 84
Serious games in future skills development: A systematic review of the design approaches
Gurbuz, S. C., & Çelik, M. (2022) · Computer Applications in Engineering Education · doi:10.1002/cae.22557 · Cited by 87
“Moreover, more studies are found by applying the backward snowball method, which is iteratively reviewing the references of the papers to find additional studies [115].”
MethodBelle, A. B., & Zhao, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119569 · Cited by 94
Natural flood management: Opportunities to implement nature‐based solutions on privately owned land
Thaler, T., Hudson, P., Viavattene, C., & Green, C. (2023) · Wiley Interdisciplinary Reviews Water · doi:10.1002/wat2.1637 · Cited by 75
“Based on the outcome of step 1, we used a snowballing method (Wohlin, 2014), which includes (a) the reference lists as well as (b) citations by the selected papers.”
MethodThe European (Green?) Deal: a systematic analysis of environmental sustainability
Hereu‐Morales, J., Segarra, A., & Valderrama, C. (2023) · Sustainable Development · doi:10.1002/sd.2671 · Cited by 96
Ali, T., Ouni, M. H. E., Qureshi, M. Z., Islam, A. B. M. S., Mahmood, M. S., Ahmed, H. H., & Ajwad, A. (2024) · Construction and Building Materials · doi:10.1016/j.conbuildmat.2024.137370 · Cited by 74
Neri, A., Cagno, E., Susur, E., Urueña, A., Nuur, C., Kumar, V., Franchi, S., & Sorrentino, C. (2024) · R and D Management · doi:10.1111/radm.12715 · Cited by 97
A Systematic Review on Cloud Testing
Bertolino, A., Angelis, G. D., Gallego, M., García, B., Gortázar, F., Lonetti, F., & Marchetti, E. (2019) · ACM Computing Surveys · doi:10.1145/3331447 · Cited by 72
“Theliterature search included an automated query over three popular digital libraries and several iterations of the snowballing approach [155], plus an additional assessment over three more digital libraries of the completeness of results from the above search methodology (as described in Section…”
MethodThe Evolutionary Trajectory of the Agile Concept Viewed from a Management Fashion Perspective
Madsen, D. Ø. (2020) · Social Sciences · doi:10.3390/socsci9050069 · Cited by 68
Internationalization from Central and Eastern Europe: A systematic literature review
Ipsmiller, E., & Dikova, D. (2021) · Journal of International Management · doi:10.1016/j.intman.2021.100862 · Cited by 68
What’s up with Requirements Engineering for Artificial Intelligence Systems?
Ahmad, K., Bano, M., Abdelrazek, M., Arora, C., & Grundy, J. (2021) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00008 · Cited by 95
Zhang, T., Qadrdan, M., Wu, J., Couraud, B., Stringer, M., Walker, S. L., Hawkes, A. D., Allahham, A., Flynn, D., Pudjianto, D., et al. (2024) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2024.114964 · Cited by 70
“The snowballing method [115] was then applied to find additional documents that are relevant to the review topic but were not identified during the literature search.”
Shaikh, M. S., Raj, S., Zheng, G., Xie, S., Wang, C., Dong, X., Lin, Y., Wang, C., & Junejo, N. U. R. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11377-6 · Cited by 100
Open-Source Hardware in Education: A Systematic Mapping Study
Heradio, R., Chacón, J., Vargas, H., Galán, D., Sàenz, J., de la Torre, L., & Dormido, S. (2018) · IEEE Access · doi:10.1109/access.2018.2881929 · Cited by 66
“Those categories with few publications or a lack of empirical evaluations often need further research [7], [19].”
BackgroundMethodInfluentialA systematic examination of knowledge loss in open source software projects
Rashid, M., Clarke, P. M., & O’Connor, R. V. (2018) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.11.015 · Cited by 75
Mitigating risk of failure in information technology projects: Causes and mechanisms
Schmidt, J. (2023) · Project Leadership and Society · doi:10.1016/j.plas.2023.100097 · Cited by 56
Building information modelling (BIM) and the return on investment: a systematic analysis
Sompolgrunk, A., Banihashemi, S., & Mohandes, S. R. (2021) · Construction Innovation · doi:10.1108/ci-06-2021-0119 · Cited by 76
López-Guerrero, R. E., Vera, S., & Carpio, M. (2022) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.112034 · Cited by 66
Zipfel, N., Horreh, B., Hulshof, C. T. J., de Boer, A. G. E. M., & van der Burg-Vermeulen, S. J. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-058630 · Cited by 74
Immersive Learning Frameworks: A Systematic Literature Review
Fernandes, F., Rodrigues, C. S. C., Teixeira, E. N., & Werner, C. M. L. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3242553 · Cited by 108
“Foreacharticle, the one-level backward snowballing technique [23] was carried out in order to identify other studies potentially relevant for this secondary study through bibliography references.”
MethodNLP methods in host-based intrusion detection systems: A systematic review and future directions
Sworna, Z. T., Mousavi, Z., & Babar, M. A. (2023) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2023.103761 · Cited by 87
“Moreover, we complemented the automatic search by extracting more relevant studies using snowballing (Wohlin, 2014).”
MethodFernández, C. S., Garbajosa, J., Yagüe, A., & Pérez, J. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.10.018 · Cited by 66
“Snowballing refers to using the reference list of a paper to identify additional papers [9].”
A public unified bug dataset for java and its assessment regarding metrics and bug prediction
Ferenć, R., Tóth, Z., Ladányi, G., Siket, I., & Gyimóthy, T. (2020) · Software Quality Journal · doi:10.1007/s11219-020-09515-0 · Cited by 68
“We applied a snowballing-like technique (Wohlin 2014) as our data collection process.”
MethodRequirements Engineering in Software Startups: A Systematic Mapping Study
Gupta, V., Fernández-Crehuet, J. M., Hanne, T., & Telesko, R. (2020) · Applied Sciences · doi:10.3390/app10176125 · Cited by 68
“In this stage, their Google citations are extracted, which are then screened by reading their title and abstract (forward snowballing approach as proposed in [9].”
MethodBanijamali, A., Pakanen, O.-P., Kuvaja, P., & Oivo, M. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106271 · Cited by 59
“The study selection process was complemented with a backward snowballing process [44, 45].”
MethodBig data in cybersecurity: a survey of applications and future trends
Alani, M. M. (2021) · Journal of Reliable Intelligent Environments · doi:10.1007/s40860-020-00120-3 · Cited by 81
“After this shortlisting of papers, we applied a Snowballing technique, explained [30].”
Methodde Kwant, C., Rahi, A. F., & Laurenti, R. (2021) · Sustainable Production and Consumption · doi:10.1016/j.spc.2021.03.030 · Cited by 72
Vignau, B., Khoury, R., Hallé, S., & Hamou‐Lhadj, A. (2021) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2021.102143 · Cited by 63
Exploring the consequence of social media usage on firm performance
Hanafizadeh, P., Shafia, S., & Bohlin, E. (2021) · Digital Business · doi:10.1016/j.digbus.2021.100013 · Cited by 82
A Meta-theoretical framework for theory building in project management
Svejvig, P. (2021) · International Journal of Project Management · doi:10.1016/j.ijproman.2021.09.006 · Cited by 73
Fracaro, S. G., Glassey, J., Bernaerts, K., & Wilk, M. (2022) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2022.107691 · Cited by 69
Current approaches for executing big data science projects—a systematic literature review
Saltz, J., & Krasteva, I. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.862 · Cited by 82
“First, the search space could be expanded by using the snowballing technique (Wohlin, 2014) for identification of relevant articles.”
MethodYunus, W. M. A. W. M., Kauhanen, L., Sourander, A., Brown, J. S. L., Peltonen, K., Mishina, K., Lempinen, L., Bastola, K., Gilbert, S., & Gyllenberg, D. (2022) · Child and Adolescent Psychiatry and Mental Health · doi:10.1186/s13034-022-00452-3 · Cited by 48
“Potential papers were also identified using hand searches and the backward snowballing technique [18], which involves looking at the reference lists of the selected papers.”
MethodKnowledge Graphs and Explainable AI in Healthcare
Rajabi, E., & Kafaie, S. (2022) · Information · doi:10.3390/info13100459 · Cited by 86
Cleveland, C. C., Reis, C. R. G., Perakis, S. S., Dynarski, K. A., Batterman, S. A., Crews, T. E., Gei, M., Gundale, M. J., Menge, D. N. L., Peoples, M. B., et al. (2022) · Ecosystems · doi:10.1007/s10021-022-00804-2 · Cited by 67
What makes a code change easier to review: an empirical investigation on code change reviewability
Ram, A., Sawant, A. A., Castelluccio, M., & Bacchelli, A. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236080 · Cited by 80
“We then perform backward snowballing [80] inspecting papers referred to in this initial selection of articles.”
BackgroundMethodFood and grocery retail logistics issues: A systematic literature review
Lagorio, A., & Pinto, R. (2020) · Research in Transportation Economics · doi:10.1016/j.retrec.2020.100841 · Cited by 75
Giannis, D., Sideris, G., Kakos, C. D., Katsaros, I., & Ziogas, I. A. (2020) · Transplantation Reviews · doi:10.1016/j.trre.2020.100570 · Cited by 60
Mølmen, G. N., & Ravndal, J. A. (2021) · Behavioral Sciences of Terrorism and Political Aggression · doi:10.1080/19434472.2021.1993302 · Cited by 91
“Complementing the keyword search, we employed snowballing (Wohlin, 2014, p. 1).”
Deep learning application on code clone detection: A review of current knowledge
Lei, M., Li, H., Li, J., Aundhkar, N., & Kim, D. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111141 · Cited by 79
Data-driven effort estimation techniques of agile user stories: a systematic literature review
Alsaadi, B., & Saeedi, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10132-x · Cited by 78
“One additional level was added, which performed a Backward Snowballing approach (Wohlin 2014).”
MethodEsagian, S. M., Ziogas, I. A., Skarentzos, K., et al. (2022) · Cancers · doi:10.3390/cancers14133177 · Cited by 53
“The reference lists of the included studies were also thoroughly searched to identify eligible, missed studies based on the “snowball” methodology [25].”
MethodA survey on narrative extraction from textual data
Santana, B. S., Campos, R., de Amorim, E. C. F., Jorge, A., Silvano, P., & Nunes, S. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10338-7 · Cited by 98
“From these articles, we apply the snowball procedure (Wohlin 2014) to expand the pool of articles by considering relevant research published in high-quality conferences and journals in the fields of natural language processing and artificial intelligence.”
MethodopenalexR: An R-Tool for Collecting Bibliometric Data from OpenAlex
Aria, M., Le, T. T., Cuccurullo, C., Belfiore, A., & Choe, J. (2024) · The R Journal · doi:10.32614/rj-2023-089 · Cited by 105
Goyal, S., Garg, D., & Luthra, S. (2021) · Environment Development and Sustainability · doi:10.1007/s10668-021-01655-y · Cited by 61
“Further, these 127 research articles were screened by using forward and backward snowball technic (Wohlin, 2014; Yadav et al., 2020a, 2020b), and finally, 47 articles were selected to refer in this work.”
MethodEmbodied Interaction and Spatial Skills: A Systematic Review of Empirical Studies
Lee-Cultura, S., & Giannakos, M. (2020) · Interacting with Computers · doi:10.1093/iwcomp/iwaa023 · Cited by 53
“Wohlin, C. (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, pp. 38. ACM. Wu, H.-K. and Shah, P. (2004) Exploring visuospatial thinking in chemistry lear…”
BackgroundMethodInfluentialMotivational strategies and approaches for single and multi-player exergames: a social perspective
Chan, G., Arya, A., Orji, R., & Zhao, Z. (2019) · PeerJ Computer Science · doi:10.7717/peerj-cs.230 · Cited by 58
“We employed a snowball method ( Wohlin, 2014 ) where we started with some initial set and used their references, both backward and forward iterations, until we no longer found signi fi cant new work.”
Team Formation in Software Engineering: A Systematic Mapping Study
Costa, A., Ramos, F., Perkusich, M., Dantas, E., Dilorenzo, E., Chagas, F., Meireles, A., Albuquerque, D. W., da Silva, L. G. B. A., de Almeida, H. O., et al. (2020) · IEEE Access · doi:10.1109/access.2020.3015017 · Cited by 67
“We executed a Snowballing-based SMS [14], following the guidelines presented by Wohlin [15].”
BackgroundMethodInfluentialBustos, D., Guedes, J. C., Vaz, M. P., Pombo, E., Fernandes, R. J., Costa, J. T., & Baptista, J. S. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18168815 · Cited by 60
“Later, the process continued with backwards citation searching following the snowballing technique [39].”
MethodMelo, A. C. C. D., Fagundes, R., Lenarduzzi, V., & Santos, W. B. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111483 · Cited by 81
Exploring the adoption of technology against delays in construction projects
Gürgün, A. P., Koç, K., & Kunkcu, H. (2022) · Engineering Construction & Architectural Management · doi:10.1108/ecam-06-2022-0566 · Cited by 79
Mazarji, M., Bayero, M. T., Minkina, T., Sushkova, S., Манджиева, С. С., Bauer, T. V., Soldatov, A. V., Sillanpää, M. E. T., & Wong, M. H. (2023) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2023.163330 · Cited by 56
The application of AI techniques in requirements classification: a systematic mapping
Kaur, K., & Kaur, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10667-1 · Cited by 65
“As discussed by Wohlin (2014), the citation of each paper in start set can be identified on Google Scholar.”
Diversity in software engineering
Menezes, Á., & Prikladnicki, R. (2018) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/3195836.3195857 · Cited by 55
“According to Wholin [19] studies of systematic literature review have emerged as a way to synthesize evidence, allowing researchers to arrive at a common understanding of the state of a research area.”
Backgroundde Padua Pieroni, M., McAloone, T., & Pigosso, D. (2019) · Proceedings of the Design Society: International Conference on Engineering Design · doi:10.1017/dsi.2019.258 · Cited by 27
“Finally, the third step envisioned a snowballing approach (Wohlin, 2014) on the basis of initial selected publications (steps one and two) to capture the established and conceptual trends (through cross-reference).”
BackgroundOmar, A., Weerakkody, V., & Daowd, A. (2020) · Government Information Quarterly · doi:10.1016/j.giq.2020.101458 · Cited by 51
Continuous Architecting with Microservices and DevOps: A Systematic Mapping Study
Taibi, D., Lenarduzzi, V., & Pahl, C. (2019) · Communications in Computer and Information Science · doi:10.1007/978-3-030-29193-8_7 · Cited by 51
“– Comprehensiveness : We included results from eight bibliographic sources and papers from the citations of the retrieved papers [19] to increase the paper base.”
BackgroundMethodInfluentialDetermining the source of human-system errors in manual order picking with respect to human factors
Setayesh, A., Grosse, E. H., Glock, C. H., & Neumann, P. (2021) · International Journal of Production Research · doi:10.1080/00207543.2021.1991022 · Cited by 65
“A ‘snowball’ strategy (cf. Wohlin 2014) was implemented by reviewing reference lists (backwards) and citations (forwards) of all relevant papers.”
MethodWoven eTextiles in HCI — a Literature Review
Pouta, E. A. M., & Mikkonen, J. V. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533566 · Cited by 57
“24 remaining papers, using a snowballing technique [119].”
BackgroundCalderón, A., Boubeta-Puig, J., & Ruiz, M. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.11.009 · Cited by 53
Sauerwein, C., Pekaric, I., Felderer, M., & Breu, R. (2018) · Computers & Security · doi:10.1016/j.cose.2018.12.011 · Cited by 70
A taxonomy of attack mechanisms in the automotive domain
Pekaric, I., Sauerwein, C., Haselwanter, S., & Felderer, M. (2021) · Computer Standards & Interfaces · doi:10.1016/j.csi.2021.103539 · Cited by 60
Characterizing Technical Debt and Antipatterns in AI-Based Systems: A Systematic Mapping Study
Bogner, J., Verdecchia, R., & Gerostathopoulos, I. (2021) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1109/techdebt52882.2021.00016 · Cited by 90
“Systematic mapping study process overview bias in favor of any specific publisher [32].”
BackgroundMethodInfluentialA Systematic Mapping Study of Code Quality in Education
Keuning, H., Jeuring, J. T., & Heeren, B. J. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588777 · Cited by 66
“To find additional publications, we have performed snowballing: identifying relevant references from (backwards) and to (forwards) a set of papers [64].”
BackgroundA systematic literature review on the impact of AI models on the security of code generation
Negri-Ribalta, C., Géraud, R., Sergeeva, A., & Lenzini, G. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1386720 · Cited by 109
“To carry out this research, we have followed different SLR guidelines, most notably Wieringa et al. (2006), Kitchenham and Charters (2007), Wohlin (2014), and Petersen et al. (2015).”
BackgroundMethodInfluentialArchitecting systems of systems: A tertiary study
Cadavid, H., Andrikopoulos, V., & Avgeriou, P. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.106202 · Cited by 51
Interconnected business models: present debates and future agenda
Jocevski, M., Arvidsson, N., & Ghezzi, A. (2020) · Journal of Business and Industrial Marketing · doi:10.1108/jbim-06-2019-0292 · Cited by 61
Privacy and Cryptocurrencies—A Systematic Literature Review
Herskind, L., Katsikouli, P., & Dragoni, N. (2020) · IEEE Access · doi:10.1109/access.2020.2980950 · Cited by 66
“We perform forward and backward snowball sampling as described byWohlin [8], to account for vital papers, that may not have been included in our initial search because they were not caught by our queries or may have been published outside the reach of DTU Find-it.”
MethodApplications of model-driven engineering in cyber-physical systems: A systematic mapping study
Mohamed, M. A., Challenger, M., & Kardaş, G. (2020) · Journal of Computer Languages · doi:10.1016/j.cola.2020.100972 · Cited by 53
Angelis, R. D. (2020) · Management Decision · doi:10.1108/md-05-2019-0587 · Cited by 69
Open Source Software Evaluation, Selection, and Adoption: a Systematic Literature Review
Lenarduzzi, V., Taibi, D., Tosi, D., Lavazza, L., & Morasca, S. (2020) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa51224.2020.00076 · Cited by 74
“Snowballing: We performed the snowballing [13] iteratively, until no additional primary studies were identified.”
MethodAnalysis of the Criteria Adopted in Industry to Extract Microservices
Carvalho, L. C. D. S., Garcia, A., Assunção, W. K. G., et al. (2019) · doi:10.1109/cesser-ip.2019.00012 · Cited by 49
A Framework for Evaluating Dashboards in Healthcare
Zhuang, M., Concannon, D., & Manley, E. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3147154 · Cited by 82
“At the same time, we adapted the forward and backward snowballing approach [29] in order to follow”
MethodChallenges of Software Requirements Quality Assurance and Validation: A Systematic Literature Review
Atoum, I., Baklizi, M. K., Alsmadi, I. M., Otoom, A. A., Alhersh, T., Ababneh, J., Almalki, J., & Alshahrani, S. M. (2021) · IEEE Access · doi:10.1109/access.2021.3117989 · Cited by 75
“Following the snowballing guidelines for systematic reviews of [36], the primary studies were expanded.”
Kinally, C., Antoñanzas-Torres, F., Podd, F. J. W., et al. (2022) · Energy Sustainable Development/Energy for sustainable development · doi:10.1016/j.esd.2022.08.014 · Cited by 49
“This involved screening the paper's bibliography for relevant documents (Wohlin, 2014).”
BackgroundGenerating a framework for gender and sexual diversity‐inclusive STEM education
Wright, G. W., & Delgado, C. (2023) · Science Education · doi:10.1002/sce.21786 · Cited by 62
Horani, O. M., Khatibi, A., Alsoud, A. R., Tham, J., & Al‐Adwan, A. S. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7030125 · Cited by 80
Text-based Question Difficulty Prediction: A Systematic Review of Automatic Approaches
AlKhuzaey, S., Grasso, F., Payne, T. R., & Tamma, V. A. M. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00362-1 · Cited by 122
Carraro, G., Tonderski, K. S., & Enrich-Prast, A. (2024) · Journal of Environmental Management · doi:10.1016/j.jenvman.2024.120585 · Cited by 70
“All searches were performed in English, German, and Italian and each reference list was scrutinized via the snowball method (Wohlin, 2014).”
A Critical Survey on Fairness Benefits of Explainable AI
Deck, L., Schoeffer, J., De‐Arteaga, M., & Kühl, N. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658990 · Cited by 144
Quantum computing in bioinformatics: a systematic review mapping
Nałęcz-Charkiewicz, K., Charkiewicz, K., & Nowak, R. M. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae391 · Cited by 74
“In selecting a literature search strategy,we followed the guidelines outlined in [80].”
MethodBlackbox, Five Years On: An Evaluation of a Large-scale Programming Data Collection Project
Brown, N. C. C., Altadmri, A., Sentance, S., & Kölling, M. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3230991 · Cited by 49
“Thus we conducted a snowball search (as per Wohlin [36]), beginning with two early publications on Blackbox [9, 34], and one of the better-known results papers by the administrators of the Blackbox dataset [3].”
MethodBerti, S., Cigala, A., & Sharmahd, N. (2019) · Educational Psychology Review · doi:10.1007/s10648-019-09486-0 · Cited by 83
COVID-19 as Opportunity to Test Digital Nomad Lifestyle
de Almeida, M. A., Correia, A., Schneider, D., & de Souza, J. M. (2021) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd49262.2021.9437685 · Cited by 76
“This literature review was undertaken using a snowballing approach [26].”
BackgroundMethodThe Influence of Robots on Students’ Computational Thinking: A Literature Review
Yang, K., Liu, X., & Chen, G. (2020) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2020.10.8.1435 · Cited by 59
“[6] It refers to searching references and citations of literature that have been collected, screening eligible literature until it is difficult to use backward and forward snowballing in iteration to find new papers and end the snowballing cycle.”
BackgroundGreen Brand Equity—Empirical Experience from a Systematic Literature Review
Górska-Warsewicz, H., Dębski, M., Fabuš, M., & Kováč, M. (2021) · Sustainability · doi:10.3390/su132011130 · Cited by 87
“The forward snowballing consists of identifying new studies citing papers examined in the systematic review [101].”
BackgroundPyNose: A Test Smell Detector For Python
Wang, T., Golubev, Y., Smirnov, O., Li, J., Bryksin, T., & Ahmed, I. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678615 · Cited by 69
“Thus, we conducted backward snowballing [34] (i.”
A decade of research on patterns and architectures for IoT security
Rajmohan, T., Nguyen, P. H., & Ferry, N. (2022) · Cybersecurity · doi:10.1186/s42400-021-00104-7 · Cited by 75
“…sense of the research landscape of methodologies around patterns for security and privacy in IoT, we have conducted a systematic literature review (SLR) following the most popular guidelines from Kitchen-ham et al. (2011), Kitchenham and Charters (2007), Petersen et al. (2015) and Wohlin (2014).”
MethodInfluentialSystematic literature review on software quality for AI-based software
Gezici, B., & Tarhan, A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10105-2 · Cited by 70
“Systematic literature studies can include reviews and synthesizing existing work in a manner that is fair for sustaining researchers in realizing the current state of a research field in software engineering (Wohlin 2014).”
BackgroundMethodJaakkola, N., Karvinen, M., Hakio, K., Wolff, L.-A., Mattelmäki, T., & Friman, M. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.855583 · Cited by 91
“Therefore, the analysis started by a so-called snowball sampling to identify the more relevant articles in relation to the research purpose (Wohlin, 2016).”
MethodNahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00034 · Cited by 87
“of backward snowballing [126], which means that we went”
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.