Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 1,301–1,400 of 3,934
Chapetta, W. A., & Travassos, G. H. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09844-5 · Cited by 22
Smart factories for single-family wooden houses – a practitioner’s perspective
Vestin, A., Säfsten, K., & Löfving, M. (2020) · Construction Innovation · doi:10.1108/ci-10-2019-0114 · Cited by 18
Ziogas, I. A., Kakos, C. D., Wu, W. K., Montenovo, M. I., Matsuoka, L. K., Zarnegar‐Lumley, S., & Alexopoulos, S. P. (2021) · Liver Transplantation · doi:10.1002/lt.25995 · Cited by 18
Impact of COVID-19 on management of urogynaecology patients: a rapid review of the literature
Loganathan, J., & Doumouchtsis, S. K. (2021) · International Urogynecology Journal · doi:10.1007/s00192-021-04704-2 · Cited by 18
“We searched the references of the relevant studies manually using the backward snowballing method [11] in order to identify additional eligible references and studies.”
MethodSustainable Healthcare Facilities: A Scoping Review
Li, Y., Pan, X., Han, Y., & Taylor, J. E. (2021) · Journal of Construction Engineering and Management · doi:10.1061/(asce)co.1943-7862.0002170 · Cited by 25
Vishnubhotla, S. D., Mendes, E., & Lundberg, L. (2018) · International Conference on Software and Computer Applications (ICSCA) · doi:10.1145/3185089.3185096 · Cited by 30
“Besides the primary search process, a secondary search process was executed using backward snowballing technique [41] to identify more relevant studies.”
MethodBrings, J., Daun, M., Kempe, M., & Weyer, T. (2018) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3210459.3210463 · Cited by 27
“A replication by Wohlin [45] supported these results and concluded that snowballing is a good alternative to database searches.”
BackgroundResultSynchronous online instrumental music teaching in cross-cultural learning contexts
Jakobsen, M. L., Hebert, D. G., & Ørngreen, R. (2023) · International Journal of Music Education · doi:10.1177/02557614231201916 · Cited by 26
“…was also used to see if the references in each article contained information of value to the study (backward), or whether others who referenced the article were relevant (forward), but also to identify whether referencing authors have also written other papers of relevance (Wohlin, 2014).”
A Systematic Literature Review of Model-Driven Engineering Using Machine Learning
Marcén, A. C., Iglesias, A., Lapeña, R., Pérez, F., & Cetina, C. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3430514 · Cited by 32
“4) Snowballing: To search for possible missing papers, we applied snowballing to identify additional sources [53].”
MethodMoktadir, M. A., Ren, J., Ayub, Y., & Shi, T. (2024) · Chemical Engineering Journal · doi:10.1016/j.cej.2024.155428 · Cited by 19
The social value of built heritage: an interdisciplinary discourse
Rossitti, M., Oteri, A. M., & Torrieri, F. (2025) · Built Heritage · doi:10.1186/s43238-025-00173-4 · Cited by 25
“Accordingly, in this research, a snowball approach is used to facilitate the screening step in the literature review process (Wohlin 2014).”
Paes, V. M., Ting, A., Masters, J., et al. (2025) · The Bone & Joint Journal · doi:10.1302/0301-620x.107b6.bjj-2024-1255.r1 · Cited by 17
The predictive validity of IELTS scores: a meta-analysis
Gagen, T., & Faez, F. (2023) · Higher Education Research & Development · doi:10.1080/07294360.2023.2280700 · Cited by 25
“Another search method that was utilized was the backwards search or snowball technique (Wohlin, 2014).”
MethodSeizing Requirements Engineering Issues through Supervised Learning Techniques
Gramajo, M. G., Ballejos, L. C., & Ale, M. (2020) · IEEE Latin America Transactions · doi:10.1109/tla.2020.9099757 · Cited by 19
“La estrategia de ”bola de nieve hacia atr´as” [27] se aplic´o posteriormente para detectar estudios asociados con la tem´atica de investigaci´on.”
BackgroundInvited review: Toward a common language in data-driven mastitis detection research
van der Voort, M., Jensen, D. B., Kamphuis, C., et al. (2021) · Journal of Dairy Science · doi:10.3168/jds.2021-20311 · Cited by 17
A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering
Usman, M., bin Ali, N., & Wohlin, C. (2023) · e-Informatica Software Engineering Journal · doi:10.37190/e-inf230105 · Cited by 38
“In Step 1 of the process described in Section 3, we identified and selected four sources (see [17, 19, 20, 21]), in addition to DARE [5], from the EBSE literature to identify the relevant aspects for QAISER.”
BackgroundMethodInfluentialSelf-citationSurvey on test case generation, selection and prioritization for cyber‐physical systems
Sadri‐Moshkenani, Z., Bradley, J. M., & Rothermel, G. (2021) · Software Testing Verification and Reliability · doi:10.1002/stvr.1794 · Cited by 24
“…venues — these include the most important venues that publish papers in the topic area of this survey according to Google Scholar and experts ’ knowledge: { ACM Transactions on Cyber-Physical Systems (cid:129) We added papers that seemed relevant to the starting set using ‘ snowballing ’ [66].”
BackgroundMaghsoudi, A., Harpring, R., Piotrowicz, W., & Heaslip, G. (2021) · Disasters · doi:10.1111/disa.12520 · Cited by 26
“The snowballing technique was administered to identify additional sources by reviewing the reference list of each discovered article (Wohlin, 2014).”
MethodPlant, O. H., van Hillegersberg, J., & Aldea, A. (2021) · IEEE Conference on Business Informatics · doi:10.1109/cbi52690.2021.00025 · Cited by 22
Edelmann, N., & Millard, J. (2021) · International Conference on Theory and Practice of Electronic Governance · doi:10.1145/3494193.3494252 · Cited by 23
“The snowballing literature review method was used [35], rather than a database search based review because 1) it is difficult to formulate a precise search, 2) of the interdisciplinary nature of the topic, 3) snowballing is comparable to the multiple database searches, and 4) it is suitable for…”
MethodContinuous cycles of data-enabled design: reimagining the IoT development process
Lee, B., Cooper, R., Hands, D., & Coulton, P. (2022) · Artificial intelligence for engineering design analysis and manufacturing · doi:10.1017/s0890060421000299 · Cited by 21
“Following the full-text review, two studies were included and using snowballing techniques, that is, a study of “ references to references ” (Wohlin, 2014), and two articles were added which ended up to the total number of four articles in the review.”
MethodInfluentialFregolente, M. V., Neto, A. C. S., Ribeiro, D. R. P., Salerno, M. S., Nakano, D., & de Carvalho, M. M. (2022) · International Journal of Managing Projects in Business · doi:10.1108/ijmpb-05-2021-0123 · Cited by 27
bin Ali, N., & Tanveer, B. (2022) · e-Informatica Software Engineering Journal · doi:10.37190/e-inf220106 · Cited by 25
A Large-Scale Empirical Study of Real-Life Performance Issues in Open Source Projects
Zhao, Y., Xiao, L., Bondi, A. B., Chen, B., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3167628 · Cited by 35
“2nd Round—Snowballing: We apply the snowballing search [50], [51] to further retrieve more relevant literature that are referenced in papers from the 1st Round.”
MethodIndustry-Academy Collaboration in Agile Methodology: a Systematic Literature Review
Marques, D. D. G., Dallegrave, T. L. D. D. A., Barbosa, L. E. L., Rodrigues, C. M. D. O., & Santos, W. B. (2022) · Iberian Conference on Information Systems and Technologies (CISTI) · doi:10.23919/cisti54924.2022.9820166 · Cited by 19
An Overview of Software Architecture Education
Oliveira, B. R. N., Garcés, L., Lyra, K. T., dos Santos, D. S., Isotani, S., & Nakagawa, E. Y. (2022) · Congresso Ibero-Americano em Engenharia de Software (CIbSE) · doi:10.5753/cibse.2022.20964 · Cited by 25
“For the remaining 48 studies, we extracted all the pieces of information that were registered in our extraction form, including 2 RESEARCH METHOD the references flagged as potential candidates for snowballing [Wohlin 2014].”
MethodIntelligent Quran Recitation Recognition and Verification: Research Trends and Open Issues
Alrumiah, S. S., & Al-Shargabi, A. A. (2022) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-022-07273-8 · Cited by 37
“Moreover, forward and backward snowballing [50] was performed to cover papers not included in the search results using the defined search string.”
MethodA Systematic Review on Continuing Education of Teachers for Educational Robotics
Oliveira, D. S., Garcia, L. T. D. S., & Gonçalves, L. M. G. (2023) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-022-01804-z · Cited by 17
“Other approaches as snowballing appear in the literature [33].”
BackgroundPredictive Analytics and Software Defect Severity: A Systematic Review and Future Directions
Olaleye, T. O., Arogundade, O. '., Misra, S., Abayomi-Alli, A. A., & Köse, U. (2023) · Scientific Programming · doi:10.1155/2023/6221388 · Cited by 30
Schöne, N., & Heinz, B. (2023) · Energies · doi:10.3390/en16041658 · Cited by 19
“During snowballing, we consulted the reference sections of selected papers and searched in the citations for potentially eligible work [40].”
Habib, A. S., Khan, S. U. R., & Felix, E. A. (2023) · IET Software · doi:10.1049/sfw2.12104 · Cited by 23
Moslehian, A. S., Warner, E., & Andrews, F. J. (2023) · Journal of Housing and the Built Environment · doi:10.1007/s10901-023-10027-z · Cited by 30
“Furthermore, backward and forward “snowball” searches were also conducted on the articles returned from full-text assessment, resulting in the addition of 279 articles (see Greenhalgh & Peacock, 2005; Wohlin, 2014).”
MethodBrom, C., Yaghobová, A., Drobná, A., & Urban, M. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11775-9 · Cited by 22
“Therefore, we resorted to additional snowball sampling (Wohlin, 2014).”
MethodFernandes, N. D. S., Leite, A. J. M., Marçal, E., & Viana, W. (2023) · Universal Access in the Information Society · doi:10.1007/s10209-023-00994-z · Cited by 36
“For the design of the SLR, the guidelines proposed by Kitchenham et al. [16] and the steps of the Snowballing method were followed [17, 18].”
Hoedemakers, J., Vanderstukken, A., & Stoffers, J. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1092865 · Cited by 28
“The first and second authors also independently used backward snowballing (Wohlin, 2014) to identify additional articles based on the 16 found during the initial searches and using forward snowballing.”
BackgroundMethodInfluentialProvenance Data Management in Health Information Systems: A Systematic Literature Review
Sembay, M. J., de Macedo, D. D. J., Pioli, L., Braga, R. M. M., & Cabezuelo, A. S. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13060991 · Cited by 36
“Soon after, four verifications were carried out following the recommendations of [88]: (1) title verification in the reference list; (2) reference location verification; (3) reading of the abstract of the referenced study; and (4) verification of the complete references of the referenced study.”
BackgroundMethodInfluentialBig Code Search: A Bibliography
Kim, K., Ghatpande, S., Kim, D., Zhou, X., Liu, K., Bissyandé, T. F., Klein, J., & Traon, Y. L. (2023) · ACM Computing Surveys · doi:10.1145/3604905 · Cited by 21
“To further ensure that we cover all the big code search [72] papers and avoid confusion from missing papers, we conducted snowballing on each paper found by text searching following a well-known guideline [293].”
Machine Learning-Based Automated Grading and Feedback Tools for Programming: A Meta-Analysis
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588822 · Cited by 29
Nevsky, A., Neate, T., Simperl, E. P. B., & Vatavu, R.-D. (2023) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3573381.3596159 · Cited by 24
“Additionally, we conducted reference snowballing following guidelines by Wohlin [227].”
MethodTechnology to Automatically Record Eating Behavior in Real Life: A Systematic Review
Hiraguchi, H., Perone, P., Toet, A., Camps, G., & Brouwer, A.-M. (2023) · Sensors · doi:10.3390/s23187757 · Cited by 17
Critical Success Factors for Implementing Cloud ERP in SMEs: A Systematic Review
Barbieri, L. M., Sott, M. K., & Monticelli, J. M. (2023) · International Journal of Innovation and Technology Management · doi:10.1142/s0219877024300039 · Cited by 34
Towards a context-sensitive conceptualisation of digital transformation
Mühlburger, M., & Krumay, B. (2023) · Journal of Information Technology · doi:10.1177/02683962231219514 · Cited by 31
“For the development of the taxonomy, we relied on a minimal information sampling approach (Van Rijnsoever, 2017) based on a starting point (Wohlin, 2014).”
Sports recommender systems: overview and research directions
Felfernig, A., Wundara, M., Tran, T. N. T., Le, V.-M., Lubos, S., & Erdeniz, S. P. (2024) · Journal of Intelligent Information Systems · doi:10.1007/s10844-024-00857-w · Cited by 43
“Based on the snowballing technique (Wohlin, 2014), we have analyzed and selected further topic-relevant scientific contributions.”
Mapping the individual, social and biospheric impacts of Foundation Models
Hernández, A. D., Krishna, S., Perini, A., Katell, M. A., Bennett, S., Borda, A., Hashem, Y., Hadjiloizou, S., Mahomed, S., Jayadeva, S., et al. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658939 · Cited by 48
Chatbots for Sexual Health Improvement: A Systematic Review
Fetrati, H., Chan, G., & Orji, R. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2376808 · Cited by 39
“This method involves identifying key articles or references within existing literature and then tracing their citations forward or back-ward to uncover additional relevant sources (Wohlin, 2014).”
A Survey on Hyperparameters Optimization of Deep Learning for Time Series Classification
Fristiana, A. H., Alfarozi, S. A. I., Permanasari, A. E., Pratama, M., & Wibirama, S. (2024) · IEEE Access · doi:10.1109/access.2024.3516198 · Cited by 40
“Additionally, the snowballing technique [43] was employed, in which further relevant articles were identified through the references and citations of the initially included studies.”
Rashid, N., & Khan, S. U. (2017) · Journal of Software Evolution and Process · doi:10.1002/smr.1927 · Cited by 25
“57 The same approach has been used by other researchers.”
MethodRefactoring-Aware Code Review: A Systematic Mapping Study
Coelho, F., Massoni, T. L., & Alves, E. L. G. (2019) · IEEE/ACM International Workshop on Refactoring (IWoR) · doi:10.1109/iwor.2019.00019 · Cited by 25
“At this point, these ten manuscripts were used as a starting set for a backward snowballing procedure [17], that led to three (3) new conference papers [18]–[20].”
MethodCosting Secure Software Development: A Systematic Mapping Study
Venson, E., Guo, X., Yan, Z., & BOEHM, B. W. (2019) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3339252.3339263 · Cited by 32
“snowballing) to identify additional related papers[81].”
MethodSearch Strategy to Update Systematic Literature Reviews in Software Engineering
Mendes, E., Felizardo, K. R., Wohlin, C., & Kalinowski, M. (2019) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2019.00061 · Cited by 18
“He followed his proposed method, as described in [44].”
Self-citationKey challenges of climate change adaptation in the building sector
Kristl, Ž., Senior, C., & Salaj, A. T. (2020) · Urbani izziv · doi:10.5379/urbani-izziv-en-2020-31-01-004 · Cited by 22
Esagian, S. M., Ziogas, I. A., Kosmidis, D., Hossain, M. D., Tannir, N. M., & Msaouel, P. (2021) · Cancers · doi:10.3390/cancers13040695 · Cited by 17
“In accordance with the snowball methodology, references from all included articles and previously published systematic reviews/meta-analyses were also manually searched to identify any potentially missed but otherwise eligible for inclusion studies [19].”
MethodThe Relation of Test-Related Factors to Software Quality: A Case Study on Apache Systems
Pecorelli, F., Palomba, F., & Lucia, A. D. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09891-y · Cited by 22
“At this point, the inspectors adopted a backward and forward snowballing approach as recommended by Wohlin [102].”
BackgroundMethodInfluentialA Multicriteria Decision Making Taxonomy of IOT Security Challenging Factors
Akbar, M. A., Alsanad, A., Mahmood, S., & Alothaim, A. (2021) · IEEE Access · doi:10.1109/access.2021.3104527 · Cited by 21
“Forward snowballing refers to explore the related literature in which a particular study is used; and the backward snowballing refers to the literature cited in a particular study [35].”
BackgroundMethodTowards a systematic approach to manual annotation of code smells
Сливка, Ј., Luburić, N., Prokić, S., Grujić, K.-G., Kovačević, A., Sladić, G., & Vidaković, D. (2023) · Science of Computer Programming · doi:10.1016/j.scico.2023.102999 · Cited by 16
Toward a Flipped Classroom Instructional Model for History education: A Call for Research
Lo, C. K. (2017) · International Journal of Culture and History (EJournal) · doi:10.18178/ijch.2017.3.1.075 · Cited by 30
On Challenges in Engineering IoT Software Systems
Motta, R. C., de Oliveira, K. M., & Travassos, G. H. (2019) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2019.15 · Cited by 20
Hosseinian, S. M., Mirzahossein, H., & Guzik, R. (2024) · Sustainability · doi:10.3390/su16146239 · Cited by 25
SoK: Access Control Policy Generation from High-level Natural Language Requirements
Jayasundara, S. H., Arachchilage, N. A. G., & Russello, G. (2024) · ACM Computing Surveys · doi:10.1145/3706057 · Cited by 34
“Additional papers were identified by Backward Snowballing Search [ 101 ] in Phase 3.”
The Effects of AI on Street-Level Bureaucracy: A Scoping Review
Gillingham, C., Morley, J., & Floridi, L. (2025) · Digital Society · doi:10.1007/s44206-025-00178-7 · Cited by 26
“To ensure all of the relevant literature was included in the review (including that which does not use the term ‘artificial intelligence’), we then went through a process of backward snowballing adapted from the technique described by Wohlin (2014).”
Review: Multiobject tracking in livestock − from farm animal management to state-of-the-art methods
Nidhi, M. H., Liu, K., & Flay, K. J. (2025) · animal · doi:10.1016/j.animal.2025.101503 · Cited by 22
Exploring the role of trust in AI-driven decision-making: a systematic literature review
Montealegre-López, N. (2025) · Management Review Quarterly · doi:10.1007/s11301-025-00526-4 · Cited by 36
Detecting Misuse of Security APIs: A Systematic Review
Mousavi, Z., Islam, C., Babar, M. A., Abuadbba, A., & Moore, K. (2025) · ACM Computing Surveys · doi:10.1145/3735968 · Cited by 43
“Furthermore, we used forward and backward snowballing techniques [25] to ensure the maximum number of relevant papers were included in our review.”
Götz, S., Tichy, M., & Groner, R. (2020) · Software & Systems Modeling · doi:10.1007/s10270-020-00815-4 · Cited by 16
“The snowballing procedures followed the guidelines laid out by Wohlin [67].”
MethodAgent-Based Software Testing: A Definition and Systematic Mapping Study
Kumaresen, P. P., Frasheri, M., & Enoiu, E. P. (2020) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c51114.2020.00016 · Cited by 21
“example of such a method is the snowballing (backward and forward) search [14].”
MethodA Scoping Review on Open Educational Resources to Support Interactions of Learners with Disabilities
Moon, J., & Park, Y. (2021) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v22i1.5110 · Cited by 24
“To complement the search results, we conducted snowball sampling to identify additional literature (see Wohlin, 2014).”
MethodRecent Advances in Intelligent Source Code Generation: A Survey on Natural Language Based Studies
Yang, C., Liu, Y., & Yin, C. (2021) · Entropy · doi:10.3390/e23091174 · Cited by 24
“To collect studies related to intelligent code generation and its similar tasks scientifically and effectively, we utilize the snowballing approach [76].”
BackgroundMethodRisks and opportunities of assisted colonization: the perspectives of experts
Kracke, I., Essl, F., Zulka, K. P., & Schindler, S. (2021) · Nature Conservation · doi:10.3897/natureconservation.45.72554 · Cited by 16
“In addition, “snowballing” was used (Wohlin 2014).”
Noël, R., Taramasco, C., & Márquez, G. (2022) · Journal of Medical Internet Research · doi:10.2196/26577 · Cited by 28
Mouchaers, I., Verbeek, H., Kempen, G. I. J. M., van Haastregt, J. C. M., Vlaeyen, E., Goderis, G., & Metzelthin, S. (2022) · European Journal of Ageing · doi:10.1007/s10433-021-00668-w · Cited by 25
Kremeike, K., Ullrich, A., Schulz, H., Rosendahl, C., Boström, K., Kaur, S., Oubaid, N., Plathe-Ignatz, C., Leminski, C., Hower, K. I., et al. (2022) · BMC Palliative Care · doi:10.1186/s12904-022-00960-1 · Cited by 17
“We conduct a systematic literature review for all measures (evaluated or not) on optimisation of care in the dying phase comprising a scoping review, in which the research question is based on the SPIDERscheme [35] with a specific search syntax and a manual search for “grey literature” using keyword search and a snowb…”
MethodChun, H.-R., Cheon, E., & Hwang, J. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-055179 · Cited by 19
Student-Generated Teaching Materials: A Scoping Review Mapping the Research Field
Ribosa, J., & Durán, D. (2022) · Education in the Knowledge Society (EKS) · doi:10.14201/eks.27443 · Cited by 29
A study of approaches to answering complex questions over knowledge bases
Gomes, J., de Mello, R. C., Ströele, V., & de Souza, J. F. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01737-x · Cited by 17
“It is expected that an in-depth searching with more levels may add new references to our final list, however, this process is time-consuming and the one-level forward snowballing is more effective [35, 103].”
BackgroundEnd-User Development for Artificial Intelligence: A Systematic Literature Review
Esposito, A., Calvano, M., Curci, A., Desolda, G., Lanzilotti, R., Lorusso, C., & Piccinno, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34433-6_2 · Cited by 24
“…the Google Scholar digital library using the search strings described in Section 2.1; • Phase 2 - Backward and forward snowballing search : we checked references and citations of the publications resulting from the previous phase, as well as publications that cited publications from Phase 1 [10].”
MethodInfluentialBlockchain Performance in Industry 4.0: Drivers, use cases, and future directions
Bovenzi, G., Aceto, G., Persico, V., & Pescapè, A. (2023) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2023.100513 · Cited by 19
Educational Data Comics: What can Comics do for Education in Visualization?
Boucher, M., Bach, B., Stoiber, C., Wang, Z., & Aigner, W. (2023) · IEEE VIS Workshop on Visualization Education, Literacy, and Activities (EduVis) · doi:10.1109/eduvis60792.2023.00012 · Cited by 24
“To further identify important and possibly older work in the respective areas, we used the reference snowballing method described by Wohlin [58], including and excluding papers according to the same aforementioned criteria.”
MethodReadiness and maturity models for Industry 4.0: A systematic literature review
Ünlü, H., Demirörs, O., & Garousi, V. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2641 · Cited by 32
“Wohlin 47 defined validity threats in the context of four types: internal validity, construct validity, conclusion validity, and external validity.”
Quradaa, F. H., Shahzad, S., & Almoqbily, R. S. (2024) · PLoS ONE · doi:10.1371/journal.pone.0296858 · Cited by 27
From Microservice to Monolith: A Multivocal Literature Review
Su, R., Li, X., & Taibi, D. (2024) · Electronics · doi:10.3390/electronics13081452 · Cited by 47
A systematic review of the biomarkers associated with cognition and mood state in bipolar disorder
Pérez‐Ramos, A., Romero-López-Alberca, C., Hidalgo‐Figueroa, M., Berrocoso, E., & Pérez‐Revuelta, J. I. (2024) · International Journal of Bipolar Disorders · doi:10.1186/s40345-024-00340-z · Cited by 20
Vishwakarma, L. P., Singh, R. K., Mishra, R., & Mani, V. (2024) · Journal of Enterprise Information Management · doi:10.1108/jeim-11-2023-0606 · Cited by 23
The illusion of the shared electric automated mobility transition
Milakis, D., & Seibert, D. (2024) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2024.101171 · Cited by 38
Women’s Journey in STEM Education in Brazil: A Rapid Review on Engineering and Computer Science
Costa, L. F. C., Nascimento, L. M. A., Lima, Y., Santos, A. M., Barbosa, C. E., Xexeo, G. B., & de Souza, J. M. (2024) · IEEE Access · doi:10.1109/access.2024.3442879 · Cited by 25
“After filtering the papers from the second search, we performed a third search through backward snowballing [42].”
MethodLeavy, S., Allegretti, G., Presotto, E., Montoya, M. A., & Talamini, E. (2024) · Sustainability · doi:10.3390/su16208727 · Cited by 31
Changes as First-Class Citizens: A Research Perspective on Modern Software Tooling
Soetens, Q. D., Robbes, R., & Demeyer, S. (2017) · ACM Computing Surveys · doi:10.1145/3038926 · Cited by 16
“…the ACM Digital Library 1 and the IEEE Xplore Digital Library, 2 and, second, we use the possibly relevant articles from those queries as a starting point for forward snowballing (i.e., recursively check the references of the relevant articles for other possibly relevant articles) [Wohlin 2014].”
MethodCombining Agile Development and Software Product Lines in Automotive: Challenges and Recommendations
Hohl, P., Stupperich, M., Münch, J., et al. (2018) · IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) · doi:10.1109/ice.2018.8436277 · Cited by 16
“According to Wohlin et al. [24], we address the four categories of validity concerns: Internal, external, conclusion and construct validity.”
BackgroundMethodInfluentialLessons Learned on Communication Channels and Practices in Agile Software Development
Ahmad, M. O., Lenarduzzi, V., Oivo, M., & Taibi, D. (2018) · Annals of Computer Science and Information Systems · doi:10.15439/2018f72 · Cited by 26
“We identified and how we mitigated them based on SMS guidelines [2-5].”
MethodMethodologies, Methods, Techniques and Tools Used on SLR Elaboration: A Mapping Study
Palomino, M., Dávila, A., & Meléndez, K. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-01171-0_2 · Cited by 17
A Comparative Study of Android Repackaged Apps Detection Techniques
Zhan, X., Zhang, T., & Tang, Y. (2019) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8667975 · Cited by 21
“Based on the systematic literature review methodology proposed in [24–26], we search for related work published from 2007 to 2018 through IEEE Explore, ACM Digital Library, Springer Link and Google Scholar.”
MethodA Review on Software Architectures for Heterogeneous Platforms
Andrade, H., & Crnković, I. (2018) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2018.00035 · Cited by 21
“In order to minimize the threat of missing relevant studies in the field, we also conducted the snowballing procedure [7] and obtained an additional 16 studies, resulting in a total of 116 studies.”
ResultObrecht, T. P., Röck, M., Hoxha, E., & Passer, A. (2020) · IOP Conference Series Earth and Environmental Science · doi:10.1088/1755-1315/588/3/032024 · Cited by 24
Challenges and Recommendations in DevOps Education: A Systematic Literature Review
Fernandes, M. A. C., Ferino, S., Kulesza, U., & Aranha, E. H. D. S. (2020) · Brazilian Symposium on Software Engineering · doi:10.1145/3422392.3422496 · Cited by 23
“Foram utilizadas duas estratégias para busca de artigos: (i) a pes-quisa em bibliotecas digitais relevantes; e (ii) a utilização da técnica snowballing [16].”
BackgroundRosberg, T., Cavalcanti, T., Thorslund, B., et al. (2021) · Journal of Rail Transport Planning & Management · doi:10.1016/j.jrtpm.2021.100240 · Cited by 16
Sauerwein, C., Fischer, D., Rubsamen, M., Rosenberger, G., Stelzer, D., & Breu, R. (2021) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3465481.3470048 · Cited by 33
“Our literature review was based on the methodologies proposed by [74] and [76].”
MethodThe emergence of lean construction in the Norwegian AEC industry
Lohne, J., Torp, O., Andersen, B. S., et al. (2021) · Construction Management and Economics · doi:10.1080/01446193.2021.1975041 · Cited by 16
“Identified articles were utilized in the search using snowballing techniques (backwards and forwards), according to the prescriptions of Wohlin (2014).”
MethodAlipour, M., Dupuy-Chessa, S. S., & Céret, É. (2021) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii52823.2021.9597415 · Cited by 16
CONSERVE: A framework for the selection of techniques for monitoring containers security
Jolak, R., Rosenstatter, T., Mohamad, M., Strandberg, K., Sangchoolie, B., Nowdehi, N., & Scandariato, R. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111158 · Cited by 30
“To complement our review and cover more relevant work, we use the snowballing approach (Wohlin, 2014).”
MethodInfluentialBuilding Condition Indicators Analysis for BIM-FM Integration
Matos, R., Rodrigues, H., Costa, A. G., & Rodrigues, F. (2022) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-022-09719-6 · Cited by 20
“In the selected articles, the respective references were checked to find older articles not detected in the initial survey [40].”
BackgroundOzretich, R. W., Wood, C. L., Allan, F., Koumi, A. R., Norman, R., Brierley, A. S., Leo, G. A. D., & Little, D. C. (2022) · Reviews in Fisheries Science & Aquaculture · doi:10.1080/23308249.2022.2039096 · Cited by 18
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.