Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 2,901–3,000 of 3,934
Elmarkez, A., Mesli-Kesraoui, S., Berruet, P., & Oquendo, F. (2025) · Machines · doi:10.3390/machines13070538 · Cited by 6
“Moreover, to achieve maximal coverage, we used another search technique which is snowballing (Section 3.3.5) [32].”
InfluentialJin, W., Sun, M., Zhou, J., Shang, J., Huang, Z., & Liu, T. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3744921 · Cited by 3
A Systematic Survey on Large Language Models for Static Code Analysis
Salih, H. A. M., & Sarhan, Q. I. (2025) · ARO-The Scientific Journal of Koya University · doi:10.14500/aro.12082 · Cited by 3
Martins, M. A. B., de Almeida, L. F. C., Cappellaro, A. P., et al. (2025) · Updates in Surgery · doi:10.1007/s13304-025-02302-1 · Cited by 1
“The references from all included studies, previous systematic reviews were also manually searched for additional studies [8].”
Concentius, S., & Mantau, M. J. (2025) · Simpósio Brasileiro de Sistemas Colaborativos (SBSC) · doi:10.5753/sbsc.2025.6547 · Cited by 1
Christian perspectives on palliative sedation: a literature study
Lambaerts, J., & Broeckaert, B. (2025) · BMC Palliative Care · doi:10.1186/s12904-025-01817-z · Cited by 3
A systematic literature review on the use of large language models for optimization algorithms
Morell, Z. S., Ferreira, T., Yackley, J. J., & Ksontini, E. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108240 · Cited by 1
Ma, W., Fan, M., Shi, Y., & Kowalska, P. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2531269 · Cited by 2
“Finally, following Wohlin ’ s guidelines for snowballing in systematic reviews (Wohlin, 2014), we selected potentially relevant articles from the references of the articles we reviewed.”
Advances in Automotive Digital Forensics: Recent Trends and Future Directions
Strandberg, K., & Eldefrawy, M. H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-00627-1_13 · Cited by 1
Exploring ethical concerns of mobile applications from App Reviews: A literature survey
Sorathiya, A., & Ginde, G. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112900 · Cited by 1
Integration of Critical Information Literacy Skills in Academic Libraries in Africa
Bouaamri, A., & Otike, F. W. (2025) · Open Information Science · doi:10.1515/opis-2025-0020 · Cited by 5
Shayegani, I., Bucks, G., Awartani, I., et al. (2025) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--55351 · Cited by 1
Muttillo, V., Eramo, R., Cederbladh, J., Strandberg, P. E., & Ashraf, A. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112579 · Cited by 3
A Systematic Literature Review on Disruptions in Construction Supply Chain: Some Stylized Trends
Castelblanco, G., Mangano, G., & Mahdavishari, M. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-03542-4_8 · Cited by 1
AI-led design innovation: understanding design-centric AI methods and assistance types
Lee, B., & Kim, J. (2025) · Proceedings of the Design Society · doi:10.1017/pds.2025.10061 · Cited by 2
“In addition to this, 17 studies were added through snowball sampling, a widely adopted literature search strategy that allows for the iterative inclusion of related papers either cited by or referencing the selected works (Wohlin, 2014).”
Alizada, M., Lampinen, A., Idestam-Almquist, P., & Westin, T. (2025) · Springer eBooks · doi:10.1007/978-3-032-01628-7_14 · Cited by 3
DIRT: a Literature-Based Benchmark Suite for Grounders
Laer, L. V., Vandevelde, S., & Vennekens, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04587-4_21 · Cited by 3
Endurant Stents in Abdominal Aortic Aneurysm Repair: A Systematic Review and Meta-Analysis
Loufopoulos, G., Nana, P., Dakis, K., et al. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14186453 · Cited by 1
“Potentially eligible studies were also hand-searched using the snowball methodology [13].”
Peters, D. G. M., Hulsker, C. C. C., Looijenga, L. H. J., et al. (2025) · EJC Paediatric Oncology · doi:10.1016/j.ejcped.2025.100322 · Cited by 1
Community mosaic models – a tool for defining communities within disaster risk management
Durrant, L. J., Vadher, A. N., & Teller, J. (2025) · Disaster Prevention and Management An International Journal · doi:10.1108/dpm-06-2024-0141 · Cited by 1
Januário, J., Nicolau, M. I., Felizardo, K. R., & Pereira, J. A. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.11559 · Cited by 3
“When forward snowballing is performed manually, the researcher uses predefined selection criteria in the protocol to select relevant studies [22].”
InfluentialKwok, B. W. J., Goodge, T. A., Sim, R. Y. H., Chen, K., & Lee, J. S. A. (2025) · International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI) · doi:10.1145/3744335.3758516 · Cited by 2
“To broaden the initial set of selected papers, both forward and backward snowballing techniques were applied [60, 62], resulting in the identification of 59 additional records.”
Bahlmann, J., Yoon, Z., Franzese, V., Stoll, E., & Hein, A. M. (2025) · Advances in Space Research · doi:10.1016/j.asr.2025.10.024 · Cited by 3
Navigating Digital Twins: Challenges and Implementation Techniques through the lens of MBSE
Islam, R., Akundi, A., & Luna, S. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.08.201 · Cited by 1
Díaz, M. Y., Aguirre, V. G., & Watkins, L. (2025) · Sexual and Reproductive Health Matters · doi:10.1080/26410397.2025.2576365 · Cited by 2
González, L. M., Márquez, G., Parra-Soto, S. L., Cardona-Valencia, M., & Taramasco, C. (2026) · Journal of Medical Internet Research · doi:10.2196/79187 · Cited by 3
Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead
Yang, G., Zheng, W., Liang, D., Hu, P., Yang, Y., Peng, S., Li, Z., Feng, J., Wei, X., Sun, K., et al. (2026) · ACM Computing Surveys · doi:10.1145/3841636 · Cited by 25
“To ensure comprehensive literature coverage and minimize the risk of overlooking relevant studies during our manual and automated search processes, we conducted systematic forward and backward snowballing procedures [34].”
Baptista, A., Fontaínhas‐Fernandes, A., & Sequeira, T. (2025) · Discover Sustainability · doi:10.1007/s43621-025-02085-w · Cited by 2
“This strategy, according to Wohlin [33], is useful for expanding the coverage of literature on dispersed or interdisciplinary topics.”
A Multi-Factor Framework for Cold-Climate Campus Design and Student Health
Li, C., Maruthaveeran, S., Shahidan, M. F., Tao, Z., & Wang, Z. (2025) · Buildings · doi:10.3390/buildings15224133 · Cited by 3
Systematic review and meta-analysis of endovascular repair for traumatic subclavian artery injury
Coutinho, M. L., Oliveira, R. E. N. D. N., Delgado, L. M., et al. (2025) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2025.11.016 · Cited by 1
Explainable AI for Decision-Making in Decentralized Energy Trading
Sharma, V. C., & Babber, K. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-5540-5.ch005 · Cited by 2
Hosting Capacity: A Systematic Review from an Honduran Perspective
Fajardo, A. R., Martinez, M. L., Tabora, J. M., Paye, J. C. H., & Rivera, D. A. (2025) · IEEE International Conference on Industry Applications · doi:10.1109/induscon66435.2025.11241822 · Cited by 1
“In addition, 11 articles were incorporated through the reference review (snowball technique) [14], resulting in a total of 24 studies considered in the systematic review.”
Sadrnezhaad, M., López, J. A. H., Mårtensson, T., & Varró, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-12089-2_13 · Cited by 2
“Following Wohlin’s guidelines for snowballing in literature studies [19], we began by establishing the research goal and scope, focusing on the application of generative AI in modeling and simulation workflows for cyber-physical systems.”
Hansen, R., Williamson, S. D., Larsen, R. G., & Ravn, S. L. (2025) · Journal of Applied Physiology · doi:10.1152/japplphysiol.00891.2025 · Cited by 1
A development framework for human work integrated AI systems in manufacturing
Yamamoto, Y., Muñoz, Á. A., & Sandström, K. (2025) · Frontiers in Manufacturing Technology · doi:10.3389/fmtec.2025.1601903 · Cited by 1
“For Class A articles, a limited backwards and forward snowball search (Wohlin, 2014) was conducted as a complement to identify potentially relevant articles.”
Tsilivigkos, C., Micco, R. D., Vitkos, E. N., & Warnecke, A. (2025) · Audiology Research · doi:10.3390/audiolres16010001 · Cited by 5
“Furthermore, a manual search of the reference sections of included articles was conducted to identify any additional potentially eligible studies [25].”
FREA-XR: An Evaluation Framework for Extended Reality User Authentication Mechanisms
Katsini, C., Epiphaniou, G., & Maple, C. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2558033 · Cited by 1
“This technique, as recommended by Wohlin (2014), aimed to ensure comprehensive coverage of the literature.”
Microservices versus Monoliths: Identifying Challenges and Proposing Practical Recommendations
Krug, D. D. S., Chanin, R. M., & de Sales, A. H. C. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-032-08570-2_15 · Cited by 1
Cares, R. A., Bond, A., & Franco, A. M. A. (2025) · Environmental Management · doi:10.1007/s00267-025-02350-8 · Cited by 1
“Full texts of the selected articles were then reviewed, with a snowballing approach (Wohlin 2014) used to identify additional relevant references.”
XSec Companions: Exploring the Design of Cybersecurity Companions
Rodriguez, S. D., Winterberg, S. E., Bumiller, F., Dietz, F., Alt, F., & Hassib, M. E. (2025) · European Symposium on Usable Security (EuroUSEC) · doi:10.1109/eurousec69254.2025.00028 · Cited by 1
“To ensure a meaningful sample, we performed a three-round snowball-sampling literature search [42].”
Azhari, M. F., Syah, G. M., Nurdiati, S., Khatizah, E., & Najib, M. K. (2025) · Turkish Journal of Mathematics and Computer Science · doi:10.47000/tjmcs.1689564 · Cited by 3
Antifragility and Growth Through Adversity: A Scoping Review
Holton, N., Cottin, M., Wright, A., Michael, M., Antonio, D. S., & Bigliassi, M. (2026) · Psychological Reports · doi:10.1177/00332941261416041 · Cited by 5
UX Design in Digital Game Development: A Systematic Literature Review and Research Agenda
da Silva, F. F., Machado, L. D. S., Leal, G. C. L., & Balancieri, R. (2026) · IEEE Access · doi:10.1109/access.2026.3652449 · Cited by 6
“After the initial selection, the snowballing technique was applied, as recommended by Wohlin and Claes [21].”
He, C., Elmqvist, N., Bellucci, A., Munshi, M., & Jacucci, G. (2026) · ACM Computing Surveys · doi:10.1145/3788869 · Cited by 4
“We also performed backward and forward citation searches [140] to capture relevant work that fell outside of the search queries.”
AI systems’ negative social impact and factors
Ahmad, N. H., Stigholt, L., Duboc, L., & Penzenstadler, B. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108038 · Cited by 6
Capone, V., Maggiore, G., & Presti, L. L. (2026) · Journal of Strategic Marketing · doi:10.1080/0965254x.2026.2618066 · Cited by 3
“During the identification phase, the snowballing technique was also adopted, which uses article citations to identify additional contributions, thereby improving literature coverage (Wohlin, 2014).”
Alrohaeem, N., Gallage, N., & Islam, S. M. N. (2025) · Journal of Entrepreneurship in Emerging Economies · doi:10.1108/jeee-05-2025-0280 · Cited by 7
RealisticCodeBench: Towards More Realistic Evaluation of Large Language Models for Code Generation
Yu, X., Chen, H., Liu, L., Hu, X., Keung, J., & Xia, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00248 · Cited by 3
“Literature Search: To understand the progress of code generation benchmarks, we conduct a literature search covering publications from 2021 to 2025 by using a forward snowballing approach [35] 1 .”
Harwood, M., Broom, C. J., van Wyk, A., Castañeda, R. A., Wong, M. Y. L., Bernard, A. T. F., Stebbing, P., Dunn, A. M., & South, J. (2026) · Knowledge and Management of Aquatic Ecosystems · doi:10.1051/kmae/2026001 · Cited by 1
An integrated intuitive and deliberative model of expert decision-making in air traffic control
Gyles, D., & Bearman, C. (2026) · Cognition Technology & Work · doi:10.1007/s10111-025-00850-5 · Cited by 3
“The literature reviews used an iterative abductive search strategy that combined structured data-based searches with back - ward and forward citation chaining (“snowballing”) (Hirt et al. 2023; Wohlin 2014 ).”
Pedagogical approaches for climate justice orientations to teaching history and social studies
McGregor, H. E., & Evans, R. S. (2026) · History Education Research Journal · doi:10.14324/herj.23.1.03 · Cited by 2
Esmaeili, S. V., Fard, Z. S. M., & Pouyakian, M. (2026) · ACS Chemical Health & Safety · doi:10.1021/acs.chas.5c00161 · Cited by 1
Devcich, B. G., Ang, L.-M., Wang, M., & Larue, G. S. (2026) · Journal of Safety Research · doi:10.1016/j.jsr.2026.01.021 · Cited by 3
Frascaria, G., Jaramillo-Dent, D., & Latzer, M. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2625258 · Cited by 2
“Consequently, to ensure that all relevant studies were included, a complementary snow-balling technique was employed (Wohlin, 2014).”
Ingham, B., Connelly-Akturk, S., Mitchell, H., et al. (2026) · International Journal of Developmental Disabilities · doi:10.1080/20473869.2026.2628966 · Cited by 1
Biophilic designs: a systematic literature review of current research and future research directions
Kumar, S., & Taneja, N. (2026) · Facilities · doi:10.1108/f-05-2025-0093 · Cited by 1
A systematic mapping study on the research landscape of LLM-based code clone detection
Jiang, B., Ruffolo, M., Kulkarni, A., Canini, T., & AlOmar, E. A. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108096 · Cited by 4
Miciuła, I., Wojtaszek, H., Mastalerz, M. W., Wysocki, W., Plecka, P., Czaplewski, M., Doskocz, J., & Raba-Schulze, A. (2026) · Energies · doi:10.3390/en19051139 · Cited by 2
Psychosocial issues of neonatal screening in the context of its major expansion: a scoping review
Sébert, A.-L., Gargiulo, M., de Lonlay, P., Arnoux, J., Vaiman, D., Bensimon, C., & Araneda, M. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1564032 · Cited by 2
“Backward and forward citation tracking, key components of the snowballing method (Wohlin, 2014), were used to complemented the database searches, allowing the identification of additional relevant references and ensuring comprehensive coverage of the literature.”
Akbarpoor, F., Delgado, L. M., Aakef, K., Mokhtar, J., Masiero, B. B., Alshehhi, D. A., Aakef, A. J., de Oliveira, H. M., Rowaiaee, R., Khokar, M. A., et al. (2026) · The American Surgeon · doi:10.1177/00031348261423926 · Cited by 1
Olaiya, O., Chan, G., & Orji, R. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2632155 · Cited by 1
Changing Policy Narratives in Addressing Indonesia's Deforestation
Wibisono, H., Wulan, P. Y. N., Rahmaningtyas, A. N., et al. (2026) · Review of Policy Research · doi:10.1111/ropr.70082 · Cited by 1
Data structures and algorithms misconceptions in concept inventories: a systematic literature review
Tilanterä, A., & Korhonen, A. (2026) · Computer Science Education · doi:10.1080/08993408.2026.2633989 · Cited by 1
Alkadhi, B., Hendley, R., & Bahsoon, R. (2026) · Interacting with Computers · doi:10.1093/iwc/iwag003 · Cited by 3
Material passports for circular economy in the construction industry: state-of-art-review
Qazi, Z., Wuni, I. Y., & Akinkuowo, F. (2026) · International Journal of Construction Management · doi:10.1080/15623599.2026.2642423 · Cited by 4
Pino, M. (2026) · International Entrepreneurship and Management Journal · doi:10.1007/s11365-026-01208-7 · Cited by 1
“In this way, we can identify other contributions that might be relevant but that did not arise in the search results for some reason (Wohlin, 2014).”
Garrab, S., Boughriou, S., & BenSassi, M. (2026) · Applied Intelligence · doi:10.1007/s10489-026-07208-y · Cited by 2
“Data Augmentation [30, 31] - Expand dataset artificially (over-sampling, transformations).”
Negative pressure wound therapy in primary closure surgeries for pilonidal disease: a meta-analysis
Martins, M. A. B., Amador, W. F. O., Oliveira, R. E. N. D. N., et al. (2026) · Updates in Surgery · doi:10.1007/s13304-026-02536-7 · Cited by 1
Addressing Visual Impairments with Model-Driven Engineering: A Systematic Literature Review
Michael, J., Netz, L., Rumpe⋆, B., Mueller, I., Grundy, J., Wickramathilaka, S., & Khalajzadeh, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3803865 · Cited by 1
“All selected studies underwent forward and backward snowballing as per Wohlin [96]: • backward snowballing “means using the reference list [of a candidate study] to identify new study to include” • forward snowballing “refers to identifying new studies based on those studies citing the study being…”
The carbon footprint of procreation: a critical review
Pont, E. (2026) · Environmental Research Letters · doi:10.1088/1748-9326/ae57c3 · Cited by 1
“The research strategy combined keyword searches for scientific articles with backward and forward snowballing (Wohlin 2014).”
Linåker, J., Olsson, T., & Papatheocharous, E. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10846-y · Cited by 5
Ntanasis‐Stathopoulos, I., Manganas, S., Filippatos, C., et al. (2026) · American Journal of Hematology · doi:10.1002/ajh.70312 · Cited by 1
“Additionally, a comprehensive and systematic snowball approach was used to capture all relevant records by reviewing the reference lists of included studies, ensuring comprehensive coverage and minimizing the risk of omitting previously cited literature [21].”
A Survey of Algorithm Debt in Machine and Deep Learning Systems: Definition, Smells, and Future Work
Simon, E. I.-O., Hettiarachchi, C., Fard, F. H., Potanin, A., & Suominen, H. (2026) · ACM Computing Surveys · doi:10.1145/3806391 · Cited by 1
“To identify additional relevant studies, we performed forward and backward snowballing on Google Scholar [44, 52, 90], examining the citations and references of our primary studies.”
Trigo, A. (2026) · Journal of Documentation · doi:10.1108/jd-01-2026-0013 · Cited by 2
Industrial MLOps: a systematic review of architectures and implementation challenges
Rajashekarappa, M., Bekar, E. T., Karlsson, A., Bokrantz, J., Subramaniyan, M., & Skoogh, A. (2026) · Production & Manufacturing Research · doi:10.1080/21693277.2026.2658878 · Cited by 4
“…‘Manufacturing-MLOps challenges,’ and ‘Architecture and framework to MLOps’ (33 assessed, 21 excluded) • Quality Assessment: Final selection based on real-world deployment validation or implementation readiness, resulting in 12 studies (10 selected plus 2 via backward snowballing (Wohlin, 2014))”
Explicit and Implicit Learning Mechanisms in AI Educational Assistants: A Systematic Review
Alqarni, F., Alhirabi, N., RANA, O. F., & Perera, C. (2026) · AI · doi:10.3390/ai7050160 · Cited by 2
Fonseca, P., Amador, W. F. O., Scremin, G. M., et al. (2026) · Pediatric Surgery International · doi:10.1007/s00383-026-06449-9 · Cited by 1
Towards an Explainable Framework for Arabic Fake News Detection: A Systematic Review
AlZaid, A., Alkhayyal, M. K., Dajam, T. A., et al. (2026) · IEEE International Conference on Communication Systems and Network Technologies (CSNT) · doi:10.1109/csnt69054.2026.11502406 · Cited by 1
“XAI taxonomies require users to document explanation properties together with their boundaries and application scope when they deploy systems responsibly [27].”
Zaczyk, M., & Semrau, J. (2026) · Sustainability · doi:10.3390/su18104877 · Cited by 7
Zhang, L., MA, Y., Kłoczko-Gajewska, A., & Turebekova, B. (2026) · Sustainability · doi:10.3390/su18104993 · Cited by 1
A Conceptual Model for Born-Accessible Web Accessibility Regeneration Using Large Language Models
Vera-Amaro, G., & Rojano-Cáceres, J. R. (2026) · Computers · doi:10.3390/computers15060343 · Cited by 2
“To mitigate the risk of missing relevant publications, backward and forward snow-balling were conducted following the guidelines proposed in [32].”
Bukowski, L. A., & Werbińska-Wojciechowska, S. (2026) · Sustainability · doi:10.3390/su18115630 · Cited by 3
Mayassah, M., & Koch, E. (2026) · Applied Sciences · doi:10.3390/app16115572 · Cited by 1
Río, M. Á. M. D., & Cornejo, V. M. R. (2026) · Innovation The European Journal of Social Science Research · doi:10.1080/13511610.2026.2677779 · Cited by 3
Delgado, L. M., Pompeu, B. F., Peñaranda, L. M., Martins, G. H. A., Viana, P., Theis, C., de Figueiredo, S. M. P., & Formiga, F. B. (2026) · Techniques in Coloproctology · doi:10.1007/s10151-026-03355-z · Cited by 1
“We also searched the references of the included studies and previous systematic reviews and meta-analyses aiming for the inclusion of additional studies [13].”
Bennin, K. E., van Teeffelen, D., Hurst, W., & Babur, Ö. (2026) · Precision Agriculture · doi:10.1007/s11119-026-10398-7 · Cited by 1
“Besides these, forward and backward snowballing were used to identify additional relevant data (Wohlin, 2014).”
Jiang, J., Rahbarimanesh, A., & Maqbool, R. (2026) · Project Leadership and Society · doi:10.1016/j.plas.2026.100233 · Cited by 4
Unveiling the power of code pre-trained models in automated program repair: A systematic review
Zhan, S., Wang, X. J., Wei, D., Xiang, X., Lin, Y., Wu, H., & Lu, J. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113015 · Cited by 3
Bucaioni, A., Weyssow, M., He, J., Lyu, Y., & Lo, D. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3828546 · Cited by 20
“Subsequently, we conducted closed recursive backward and forward snowballing [26], but identified no additional sources resulting in a total of 76 studies from the automatic search and selection process.”
Towards artificial intelligence for the public sector: framing and bridging academia and practice
Mintz, Z. W., & Ma, J. (2026) · Global Public Policy and Governance · doi:10.1007/s43508-026-00148-w · Cited by 1
“First, each is a peer-reviewed review or agenda-setting article rather than a single-case study, which makes them well-suited for snowball sampling: review articles aggregate dispersed primary literature and accumulate dense reference lists (Lecy et al., 2012 ; Wohlin, 2014).”
Wu, C.-S., Hsu, Y., Lai, K., & Chiang, H.-W. (2026) · Applied System Innovation · doi:10.3390/asi9070149 · Cited by 1
“Wohlin [16] Snowballing/citation chasing Backward and forward snowballing can complement database search and improve coverage.”
Li, S., Wu, L., & Li, Y. (2026) · Buildings · doi:10.3390/buildings16153010 · Cited by 1
Explanation of Similarities in Temporal Case-Based Reasoning by Visualization
Schander, R., Schultheis, A., & Bergmann, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33865-5_11 · Cited by 1
Yajnik, A., & Pushkala, S. A. (2026) · IEEE Access · doi:10.1109/access.2026.3719291 · Cited by 1
“Backward and forward citation chaining [22] began from 20 canonical papers across the five target domains.”
A Systematic Review of MLOps Tools: Tool Adoption, Lifecycle Coverage, and Critical Insights
Micallef, Z., Rajenthiram, K., & Gerostathopoulos, I. (2026) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3793653.3793785 · Cited by 1
“We apply both backward and forward snowballing to expand the results obtained through our initial search, following the guidelines of Wohlin et al. [47].”
Features of conceptual blending in the context of visualisation
Featherstone, C., & van der Poel, E. (2017) · IST-Africa Week Conference · doi:10.23919/istafrica.2017.8102296 · Cited by 1
“Various sources were consulted on the recommended procedure for conducting a critical literature review [5]–[8].”
MethodEffective collection and selection of research articles for a systematic review
Rožanc, I. (2018) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.23919/mipro.2018.8400251 · Cited by 1
Categories of Change Triggers in Business Processes
Kaplan, A., Busch, K., Koziolek, A., et al. (2018) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00049 · Cited by 1
“papers examined” [17].”
Automatic geometry alteration when designing for metal additive manufacturing
Bonde, J. M., Borgue, O., Panarotto, M., et al. (2020) · doi:10.35199/norddesign2020.8 · Cited by 1
“The publication list was then complemented with additional entries through backward snowballing (Wohlin, 2014) to reach publications outside the range of Google Scholar.”
Software Developers Sentiment Analysis: A Systematic Mapping
Coelho, A. A. N., Silva, T. T. O., de Oliveira, A. M., et al. (2018) · doi:10.1109/clei.2018.00017 · Cited by 1
“A partir disso, espera-se obter um conhecimento das pesquisas existentes até o momento, através de um mapeamento sistemático [8] [9].”
Background
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.