Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 3,801–3,900 of 3,934
Young Women and Heritage Management: Insights from a Systematic Literature Review
Acharya, I., & Sharma, J. P. (2026) · Heritage & Society · doi:10.1080/2159032x.2026.2687526
Iqbal, F., Khan, M. A., Weerakoon, O., Oyedeji, S., & Porras, J. (2026) · IEEE/ACM International Workshop on Green and Sustainable Software (GREENS) · doi:10.1145/3786148.3788622
“The systematic literature review (SLR) followed the guidelines established by Brereton et al. [13] and the snowballing protocol proposed by Wohlin [53].”
Can I Give You My Opinion? Designing an Accessible Online Survey Tool
Pereira, L. S., Modesto, C., & Duarte, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29653-5_8
Delgado, L. M., Pompeu, B. F., de Figueiredo, V. L. P., et al. (2026) · Cureus · doi:10.7759/cureus.112138
Quayson, F., & Dadson, I. Y. (2026) · Research Square · doi:10.21203/rs.3.rs-8913300/v2
Meraviglia, M., Costa, F., & Nebuloni, A. (2026) · Advances in Science, Technology & Innovation/Advances in science, technology & innovation · doi:10.1007/978-3-032-11306-1_35
THINKING ISSUES: Combining Practitioner and Academic Views: Planning a Multi-Vocal Literature Review
Clear, T. (2026) · ACM Inroads · doi:10.1145/3811411
“As a further strategy for identifying relevant sources, Varela-Vaca and colleagues have recommended snowballing [15, 31] as one way to identify such grey literature sources from trawling through the citation lists provided in the ‘white literature’ sources [29].”
Domain-specific languages for functional testing: A systematic mapping study
Lima, Y. A., Rodrigues, E., & Assunção, W. K. G. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108239
Policy Influence on Perceived Risks in Adopting the Circular Economy in the Construction Industry
Kadirappulige, I., Karunasena, G., Gurmu, A. T., & Udawatta, N. (2026) · Journal of Legal Affairs and Dispute Resolution in Engineering and Construction · doi:10.1061/jladah.ladr-1578
Technical Lag as Latent Technical Debt: A Rapid Review
Panter, S. K., & Eisty, N. U. (2026) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1145/3794915.3795784 · Cited by 3
“We conducted a Rapid Review following Cartaxo et al. [2] and Wohlin’s snowballing guidelines [19] to synthesize existing research on TL. Searches were conducted in the ACM Digital Library, IEEE Xplore, Scopus, and Springer.”
Huynh, D.-M.-Q., & Poelmans, S. (2026) · Information Systems and e-Business Management · doi:10.1007/s10257-026-00733-w
Rackard, S. M., Guttin, T., Cashman, D., Nagar, P., GORAN, G. V., & Hoque, M. A. (2026) · Journal of Veterinary Medical Education · doi:10.3138/jvme-2025-0173
Naveen, M., Srinivas, S. S., & Singh, A. (2026) · Research in Transportation Business & Management · doi:10.1016/j.rtbm.2026.101812
Lopes, M. C., Alves, R. R. N., Schiel, N., et al. (2026) · Human Ecology · doi:10.1007/s10745-026-00715-z
“Then, using the Snowballing method (Wohlin, 2014), we checked the reference lists of all selected articles to identify potential studies for inclusion in the review.”
Trombose venosa cerebral em infecções não bacterianas: uma revisão sistemática
Medeiros, M. D. C. (2025) · Arca - Repositório Institucional da Fiocruz
Recent advances in active flutter suppression: a comprehensive review
Cobo-González, Á., Rodríguez-Robles, R., & Cuerno-Rejado, C. (2026) · The Aeronautical Journal · doi:10.1017/aer.2026.10212
Kadırhan, Z. (2026) · Çukurova Üniversitesi Sosyal Bilimler Enstitüsü Dergisi · doi:10.35379/cusosbil.1696829
Aduko, J., Sumani, J. B. B., & Anokye, K. (2026) · Agroforestry Systems · doi:10.1007/s10457-026-01580-x
Beyond residuals: informality as an enduring feature of developing economy, the Ghanaian experience
Agyabeng, A. N. (2026) · International Journal of Development Issues · doi:10.1108/ijdi-01-2026-0010 · Cited by 1
Williamson, S. D., Andersen, M. M., & Ravn, S. L. (2026) · Journal of Spinal Cord Medicine · doi:10.1080/10790268.2026.2696692
Pompeu, B. F., Alves, F. H. N. N., Micherino, B. V., Silva, A. L. C. E., Barone, G., Delgado, L. M., Theis, C., de Figueiredo, S. M. P., & Formiga, F. B. (2026) · The American Surgeon · doi:10.1177/00031348261471467
Human-Centered AI Design and Implementation for Intelligence Augmentation
Oguz, C. K., Meyer, M., & Ramos, I. (2026) · SpringerBriefs in information systems · doi:10.1007/978-3-032-30186-4_3
Xiang, S., Koch, H., Lilla, N., Nieuwenboom, J. W., & Schüpbach, M. (2026) · AI Brain and Child · doi:10.1007/s44436-026-00039-1
A Conceptual Model to Enhance Process Mining Analysis with Provenance
Caunedo, Á. J. B., Franceschetti, M., Völzer, H., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-40795-5_31
Turrero, J. M. C., & Zabaleta, A. P. (2026) · Circular Economy and Sustainability · doi:10.1007/s43615-026-00790-w
“To increase the sample, we employed the backward snowball sampling method [ 65 ], which yielded two additional articles [ 8 , 19 ], mainly reviewing existing Literature Reviews related to CPPPM, and two other articles found over Google Scholar search and other cross-references, reaching the final…”
I can’t recognize (yet): Delayed Rendering to Defeat Visual Phishing Detectors
Yuan, Y., Rado, C. A., Apruzzese, G., et al. (2026) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp68448.2026.00026
“To this end, we considered five recent seminal works [13, 19, 20, 22, 26] and applied the snowball method [47] (as also done, e.g., in a recent SoK [48]); we complemented this search by looking for papers mentioning “phishing” on DBLP.”
Oliveira, R. E. N. D. N., Delgado, L. M., Gonçalves, L. B., et al. (2026) · The American Surgeon · doi:10.1177/00031348261472868
Markonda, L. P., & Spatz, D. L. (2026) · Breastfeeding Medicine · doi:10.1177/15568253261473733
SoK: A Defense-Oriented Evaluation of Software Supply Chain Security
Ishgair, E. A., Gomez, J., Melara, M. S., Cárdenas, Á. A., & Torres-Arias, S. (2026) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp68448.2026.00059 · Cited by 1
“Fundamentally, a software supply chain can be described in terms of four core elements: 1) Principals: Broadly, any organization, institution or individual involved with any aspect of the software supply chain.”
Camilo, H. T., & Hauck, J. C. R. (2026) · Anais do Workshop sobre Educação em Computação (WEI) · doi:10.5753/wei.2026.20394
Chaudhary, P. (2026) · International journal of organizational analysis · doi:10.1108/ijoa-11-2025-6207
Deepfakes e plataformização: a violência digital de gênero como fenômeno transnacional
de Carvalho, F. C., do Amaral, D. G., & Ribeiro, J. P. G. (2026) · Ação Midiática – Estudos em Comunicação Sociedade e Cultura · doi:10.5380/am.v32i1.104014
Generative Language Models for Disease Treatment Recommendations: A Systematic Literature Review
Neto, L. M., da Silva, M. H. L. F., Monteiro, K. H. D. C., Simão, E. P., Neto, S. R. D. S., de Vasconcelos, L. F. S., Pimenta, J. P., Rodrigues, I. R., & Endo, P. T. (2026) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-026-00242-y · Cited by 1
Narbaiza, X., Diez-Cañamero, B., & Santamaria, I. C. (2026) · Circular Economy and Sustainability · doi:10.1007/s43615-026-01102-y
“Snowballing is particularly valuable in emerging and interdisciplinary research domains, where relevant contributions may be dispersed across different research communities and may not always be retrieved through keyword-based database searches alone (Webster and Watson 2002 ; Wohlin 2014 ).”
Xie, M., Yang, Z., Li, Y., Liu, H., Milani, A., & Bonaiuto, M. (2026) · Behavioral Sciences · doi:10.3390/bs16081343
“This involved examining the reference lists of key articles and identifying studies that cited these articles, thereby helping identify additional relevant studies that might not have been retrieved through the database searches (Wohlin, 2014).”
Towards Circularity in Multi-Storey Wood Buildings: a Systematic Literature Review
Quarcoo, F., Myhren, J. A., Binder, E., & Dodoo, A. (2026) · Circular Economy and Sustainability · doi:10.1007/s43615-026-01082-z
Lerner, A., Araújo, C. A. S., & da Silva, M. F. (2026) · doi:10.5753/sbcas_estendido.2026.26501
“Estratégias complementares de snowballing backward (2 incorporados) e forward (8 incorporados) [Wohlin 2014] elevaram o corpus final a 15 estudos publicados entre 2024 e 2025.”
Low-Cost and Transportable CAVE Systems
Beji, F., Pivotto, I. D., Matias, V., & Ferreira, W. D. (2026) · Springer proceedings in business and economics · doi:10.1007/978-3-032-23282-3_34 · Cited by 1
A Systematic Review on Applications of Deep Learning in Supply Chain Risk Management
Vankayalapati, R. H., & Banerjee, A. (2026) · Operations Research Forum · doi:10.1007/s43069-026-00682-7 · Cited by 1
“To complement the search, a snowball approach [8] was used, where reference lists of highly cited articles were examined to identify seminal works and other relevant contributions.”
The Role of Organizational Theories in Understanding Circular Ecosystems
Ferrari, A. G., Armellini, F., & Jugend, D. (2026) · Springer proceedings in business and economics · doi:10.1007/978-3-032-23282-3_20
Requirements-driven database evolution: A systematic literature review
Litavská, H., Suchánek, M., & Kroha, P. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108276
Factors Influencing Consumer Adoption of Smart Parcel Lockers: A Scoping Review
Alamoudi, L. H., & Salam, M. A. (2026) · Logistics · doi:10.3390/logistics10080180
“Backward citation tracking involved examining reference lists of relevant studies, while forward citation tracking was used to identify later studies citing key publications [20].”
Human in the Loop for Fuzz Testing: Literature Review and the Road Ahead
Yu, J., Wen, X., Cheng, S., Xie, X., Hu, Q., & Wang, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3833416 · Cited by 2
“These directions can be organized into three stages, each reflecting a distinct role that humans may assume in the fuzzing process: These stages outline a promising research agenda for the evolution of HITL for fuzzing.”
The Impact of Artificial Intelligence on Designers' Creative Thinking in the Ideation Phase
Jiang, S., Xie, J., Li, P., pang, X., & Liu, Z. (2026) · International Conference on Intelligent Engineering and Next-Gen Healthcare Systems (IEHNS) · doi:10.1109/iehns68708.2026.11606882
Ghanim, R. (2026) · Al-Noor Journal of Engineering Management and Computer Science · doi:10.71229/8chpd013
Private equity and innovation: A systematic review and future outlook
Rollmann, S., & Schiereck, D. (2026) · Corporate Ownership and Control · doi:10.22495/cocv23i2art12
Sanchis, E. S., Ortí, J. E. D. L. R., Sancho-Cantus, D., Cunha-Pérez, C., & Mohedo, J. C. (2026) · Pathophysiology · doi:10.3390/pathophysiology33030059
Mitsanis, C., Fortuna, N. Z., & Beveridge, C. A. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.08.07.743620 · Cited by 1
An Integrated Product Service System Framework for On-Site Digital Human Guide Systems
Liu, Z., Zhu, T., Wang, F., Yan, J., Xiong, W., Osmani, M., & Demian, P. (2026) · Systems · doi:10.3390/systems14080964
Uncertainty in Process Mining: Survey and Conceptual Framework
Lepsien, A., Aleknonytė‐Resch, M., Weidlich, M., & Koschmider, A. (2026) · ACM Computing Surveys · doi:10.1145/3839360
“As a secondary information source, we used reference searching in an iterative snowball sampling process [57, 61, 164, 169].”
Marchetti, E., Nikghadam-Hojjati, S., Calabrò, A., & Barata, J. (2026) · Information · doi:10.3390/info17080771
Giunta, B., Burnay, C., & Faulkner, A. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-35043-5_11
Šmite, D., Jekabsone, I., & Gaile-Sarkane, E. (2026) · Business and management · doi:10.3846/bm.2026.1635
“…guidance for integrative literature reviews, snowballing was used to systematically identify relevant studies, by screening the reference lists of included articles (backward snowball-ing) and examining articles citing the included studies (forward snowballing) (Wohlin, 2014; Kraus et al., 2022).”
Global Disparities in the Recreational Fishery Economy: A Systematic Review
Sha, W., Chen, G., Han, B., et al. (2026) · Fishes · doi:10.3390/fishes11080474
“In addition to the database search, a snowball sampling method [28,29] was employed to check the reference lists of selected papers for further relevant studies.”
Kubiak, T. (2026) · American Impact Review · doi:10.66308/air.e2026068 · Cited by 2
Zoupas, I., Covell, J. M., Naranjo, J. C., Podgorsek, B., Chen, P., LaPar, D. J., & Greenleaf, C. E. (2026) · Perfusion · doi:10.1177/02676591261477954
Renewable Energy Investments Under Cannibalization: A Semi‐Systematic Review
Pustějovský, Z., & Hudík, M. (2026) · Journal of Economic Surveys · doi:10.1111/joes.70153
Interoperability Mechanisms for System-of-Systems: A Systematic Literature Review
Vieira, T. F., Goldschmidt, R. R., Cavalcanti, M. C., et al. (2026) · IEEE Access · doi:10.1109/access.2026.3725151
“Following the identification of the 51 primary studies from the search, we performed a manual search phase using backward and forward snowballing to capture relevant works potentially missed by the digital libraries’ indexing [15].”
Storti, R., Tuccillo, D., & Agliata, F. (2026) · Corporate Ownership and Control · doi:10.22495/cocv23i3art1
Tatari, K. K., Latif, S., Rehman, S. U., et al. (2026) · Information · doi:10.3390/info17080798
“The backward and forward snowballing method was used [75].”
Smith, J. M., McGill, M. M., & Degener, R. M. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--60607
Idrees, R., & Amin, M. B. (2026) · Cluster Computing · doi:10.1007/s10586-026-06416-y
Ajah, J., & Jamison, C. S. E. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--59978
Interventions in Circuit Education in K-12 Worldwide: A Systematic Review
Raut, M., Shayegani, I., & Yoon, S. Y. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--60194
Assessing the awareness support in collaborative software development tools
Mantau, M. J., & Concentius, S. (2026) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2026.6287
“[2012]; Wohlin [2014], using the Awareness Assessment Model presented by Mantau and Benitti [2023a] as its foundation.”
Risk assessment of relief food delivery delays from a cross-disaster perspective
Yang, M., Qu, S. J., Qu, Z., Dubey, R., Zhao, G., Mohsendokht, M., Ji, Y., & Yang, Z. (2026) · International Journal of Production Economics · doi:10.1016/j.ijpe.2026.110192 · Cited by 1
Delgado, L. M., & Pimpalwar, A. P. (2026) · Pediatric Surgery International · doi:10.1007/s00383-026-06592-3
“We also searched for the references of the included studies and previous systematic reviews and meta-analyses aiming for the inclusion of additional studies [13].”
Pajkanović, R., Koscher, R., & Stoeglehner, G. (2026) · Research Square · doi:10.21203/rs.3.rs-10528836/v1
Chen, S., Cao, R., & Cheng, S. (2026) · Behavioral Sciences · doi:10.3390/bs16091486
“…and that emerging studies often debut as conference papers, a supplementary strategy of citation-based snowballing was introduced beyond the primary database searches to mitigate publication bias and capture significant literature potentially missed by conventional retrieval methods (Wohlin, 2014).”
Automated Refactoring Adoption—Results from an Adapted Multivocal Literature Review
Siddik, M., Oloruntola, S., Ahmmed, A., Yousif, M., Marks, G., Yılmaz, M., McCarren, A. L., & Mccarren, A. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-36816-4_15
Interaction Capability (Usability) Measurement - Results from an Adapted MLR
Houshidari, S. D. F., Farrell, C., Gillespie, A., Murphy, S., Marks, G., Yılmaz, M., & McCarren, A. L. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-36816-4_19
Large Language Models for Fuzz Testing in Microservices: A Systematic Literature Review
Song, Y., Ping, K., Wang, Y., & Li, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36587-3_23
Tsilivigkos, C., Sakellaris, N., Vitkos, E. N., et al. (2026) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-026-10557-6
AI Literacy in STEAM Education: A Systematic Review
Chasanidou, D., Kalogiannakis, M., Ράικου, Ν., Stavropoulou, G., & Beazidou, E. (2026) · Computers · doi:10.3390/computers15090565
Evaluating DevOps, DevSecOps, MLOps and CloudOps–Results from an Adapted MLR
Yakushenko, A., McCann, R., Hughes, C., McElligott, S., Yılmaz, M., MacMahon, S. T., Mccarren, A., & McCarren, A. L. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-36813-3_6
The UX grammar: a standardized tool to design UX
Dabouis, B., Boccara, V., & Yannou, B. (2026) · Cognition Technology & Work · doi:10.1007/s10111-026-00894-1
Brito, N. L. H., Ramos, J. S., Santos, T. A., et al. (2026) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2026.106054
Rāhui as living Indigenous marine governance
Muntoni, M., Parsons, M., & Fisher, K. (2026) · MAST. Maritime studies/Maritime studies · doi:10.1007/s40152-026-00519-3 · Cited by 1
“A backward snowballing approach was also used, whereby the reference lists of key publications were reviewed to identify additional relevant sources (Wohlin 2014).”
A Systematic Literature Review of Code Smell Detection Tools for JavaScript Systems
Souza, S., Ribeiro, F., Fernandes, E., et al. (2026) · SBC Reviews on Computer Science · doi:10.5753/reviews.2026.6187
“Table 2 presents the data extracted from the 22 tools collected by executing the SLR protocol (Kitchenham and Charters, 2007; Wohlin, 2014).”
InfluentialHu, P., Hanafiah, M. H., Omar, A. R. C., Osman, L. H., & Chen, L. (2026) · Multidisciplinary Reviews · doi:10.31893/multirev.2027060
“This snowballing approach is a recognized and widely recommended complement to database-driven SLRs, particularly for ensuring comprehensive coverage in fields with rapidly expanding literatures (Wohlin, 2014; Greenhalgh & Peacock, 2005).”
Riemer, P., & Greefrath, G. (2026) · Digital Experiences in Mathematics Education · doi:10.1007/s40751-026-00201-7
Te Whare Māwhaiwhai: Stitching an integrated Indigenous framework of support addressing FASD
McLachlan, A., Kingi, T., Fitzpatrick, M., Rona, S., Karanui-Golf, T., Jones, H., & Martell, H. (2026) · Child Protection and Practice · doi:10.1016/j.chipro.2026.100346
A Guided Framework for Designing an Interactive Dashboard of Local E-Participation
Dua, A., & Wimmer, M. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36676-4_9
Sixty years of research on districting problems: Models, algorithms, and applications
Campos-Valdés, C., Álvarez‐Miranda, E., Pereira, J., et al. (2026) · Omega · doi:10.1016/j.omega.2026.103654
Artificial intelligence for oral history and disaster experience: a systematic mapping studyPreprint
Azarsa, A., Asgary, A., Nayebi, M., & Holman, S. (2026) · Research Square · doi:10.21203/rs.3.rs-8751395/v1
A Strategy-based Framework for Exploratory Playtesting in Open-World Games
Duarte, Y. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787657
“The multivocal approach includes both academic sources and gray literature (e.g., developer blogs, technical reports, and postmortems), following recognized SE guidelines [21, 30].”
AI-Supported lecturer decision-making in higher education: a socio-technical perspective
Köroğlu, M. N., Mayrhofer, J., Steinmaurer, A., Wintersberger, P., Dennerlein, S., & Weinhandl, R. (2026) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-026-00623-8
“The snow-balling process continued iteratively until 20 consecutive reference lists yielded no new relevant studies, indicating saturation (Wohlin, 2014).”
Games Designed to Teach or Practice Technical Debt Management: A Systematic Mapping Study
Radnejad, M., & Seaman, C. B. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3795679
“Our study methodology was conducted following guidelines from Kitchenham [14] and Petersen [17], as well as Wohlin [23] for “snowballing” to ensure comprehensive coverage of relevant studies.”
Falkeman, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35747-2_2
Sotolani, R. S., Parziale, A., Voria, G., Catolino, G., Palomba, F., & Spinola, R. O. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7442691
The economic dimensions of cloud security: A systematic literature review
Filiopoulou, E., Δέδε, Γ., Evangelatos, S., Kamalakis, T., Michalakelis, C., & Νικολαϊδου, Μ. (2026) · Computers & Security · doi:10.1016/j.cose.2026.105139 · Cited by 1
Augmenting Expertise in Organizational Foresight: The Role of GenAI in Collaborative Settings
Zumbrunn, L. (2026) · Futures & Foresight Science · doi:10.1002/ffo2.70052 · Cited by 2
Makkass, M., Laghouaouta, Y., & Anwar, A. (2026) · Computers · doi:10.3390/computers15090619
Cultural factors influencing creativity in product development: an exploratory study
Bastian, A., Kazmouz, N., Willerscheid, J., & Albers, A. (2026) · International Journal of Design Creativity and Innovation · doi:10.1080/21650349.2026.2728155
Queiroz, L. F., Lemos, M. J., Curi-Pesantes, B., et al. (2026) · Journal of Neonatal-Perinatal Medicine · doi:10.1177/19345798261490120
Influencer Authenticity in Marketing: A Systematic Literature Review and Future Research Agenda
Çelik, F., Ibrahim, B., Polat, E., Moulard, J. G., & Akbulut, M. (2026) · International Journal of Consumer Studies · doi:10.1111/ijcs.70258
Kwok, B. W. J., Goodge, T. A., Sim, R. Y. H., Chen, K., & Lee, J. S. A. (2026) · International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI) · doi:10.1145/3828157.3828787
Alam, M. A., Mimi, A., Islam, M. M., Sharmin, K., & Bhuiyan, M. R. I. (2026) · SAGE Open · doi:10.1177/21582440261489866
Analyzing the Adoption of LLMs to Support Software Architecture: A Systematic Mapping Study
Neumann, E., Guerra, E., & Farias, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39143-8_20
Toward Reliable Agentic AI for Software Architectural Analysis
Liljas, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39143-8_4
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.