The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Successful combination of database search and snowballing for identification of primary studies in systematic literature studies
Open accessAbstract
A good search strategy is essential for a successful systematic literature study. Historically, database searches have been the norm, which was later complemented with snowball searches. Our conjecture is that we can perform even better searches if combining these two search approaches, referred to as a hybrid search strategy. Our main objective was to compare and evaluate a hybrid search strategy. Furthermore, we compared four alternative hybrid search strategies to assess whether we could identify more cost-efficient ways of searching for relevant primary studies. To compare and evaluate the hybrid search strategy, we replicated the search procedure in a systematic literature review (SLR) on industry–academia collaboration in software engineering. The SLR used a more “traditional” approach to searching for relevant articles for an SLR, while our replication was executed using a hybrid search strategy. In our evaluation, the hybrid search strategy was superior in identifying relevant primary studies. It identified 30% more primary studies and even more studies when focusing only on peer-reviewed articles. To embrace individual viewpoints when assessing research articles and minimise the risk of missing primary studies, we introduced two new concepts, wild cards and borderline articles, when performing systematic literature studies. The hybrid search strategy is a strong contender for being used when performing systematic literature studies. Furthermore, alternative hybrid search strategies may be viable if selected wisely in relation to the start set for snowballing. Finally, the two new concepts were judged as essential to cater for different individual judgements and to minimise the risk of excluding primary studies that ought to be included.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 12×
“Once a start set was identified, back-ward and forward snowballing were performed to determine further articles to include, using the guidelines in Wohlin (2014).”
InfluentialSelf-citationSystematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659
Self-citationIdentifying relevant studies in software engineering
Zhang, H., Babar, M. A., & Tell, P. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.010 · Cited by 557
A Model for Technology Transfer in Practice
Gorschek, T., Garre, P., Larsson, S., & Wohlin, C. (2006) · IEEE Software · doi:10.1109/ms.2006.147 · Cited by 350
Self-citationThe role of replications in Empirical Software Engineering
Shull, F. J., Carver, J. C., Vegas, S., & Juristo, N. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9060-1 · Cited by 315
Garousi, V., Petersen, K., & Ozkan, B. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.07.006 · Cited by 241
Experiences from using snowballing and database searches in systematic literature studies
Badampudi, D., Wohlin, C., & Petersen, K. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745818 · Cited by 221
Self-citationOn the reliability of mapping studies in software engineering
Wohlin, C., Runeson, P., da Mota Silveira Neto, P. A., Engström, E., do Carmo Machado, I., & de Almeida, E. S. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.04.076 · Cited by 206
Self-citationDeveloping search strategies for detecting relevant experiments
Dieste, O., Grimán, A., & Juristo, N. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9091-7 · Cited by 200
Mourão, E., Pimentel, J. F., Murta, L., Kalinowski, M., Mendes, E., & Wohlin, C. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106294 · Cited by 191
Self-citation
Research works printed but not linked: 5
- Punter, 2010, Industrial impact and lessons learned, 279, 10.1007/978-90-481-9849-8_17 · doi:10.1007/978-90-481-9849-8_17
- Rombach, 2008, Impact of research on practice in the field of inspections, reviews and walkthroughs: Learning from successful industrial uses, SIGSOFT Softw. Eng. Notes, 33, 26, 10.1145/1449603.1449609 · doi:10.1145/1449603.1449609
- S.J. Lamprecht, G.-J. van Rooyen, Models for technology research collaboration between industry and academia in South Africa, in: Proceedings IEEE Software Engineering Colloquium, SE, 2012, pp. 11–17. · doi:10.1109/se.2012.6242350
- W. Raschke, M. Zilli, J. Loinig, R. Weiss, C. Steger, C. Kreiner, Embedding research in the industrial field: A case of a transition to a software product line, in: Proceedings International Workshop on Long–Term Industrial Collaboration on Software Engineering, 2014, pp. 3–8. · doi:10.1145/2647648.2647649
- P. Krishnan, K. Ross, P. Pari-Salas, Industry academia collaboration: an experience report at a small university, in: Proceedings Conference on Software Engineering Education and Training, 2009, pp. 117–121. · doi:10.1109/cseet.2009.22
Cited by
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
“The hybrid approach is the combination of search strategy and snowballing [32, 33].”
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
Synergies and gaps between circularity assessment and Life Cycle Assessment (LCA)
Samani, P. (2023) · Science of The Total Environment · doi:10.1016/j.scitotenv.2023.166611 · Cited by 81
Sustainable software engineering: Reflections on advances in research and practice
Venters, C. C., Capilla, R., Nakagawa, E. Y., Betz, S., Penzenstadler, B., Crick, T., & Brooks, I. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107316 · Cited by 59
Identifying the primary dimensions of DevSecOps: A multi-vocal literature review
Zhao, X., Clear, T., & Lal, R. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112063 · Cited by 40
Migratory grief: a systematic review
Renner, A., Schmidt, V., & Kersting, A. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1303847 · Cited by 25
Technology to Automatically Record Eating Behavior in Real Life: A Systematic Review
Hiraguchi, H., Perone, P., Toet, A., Camps, G., & Brouwer, A.-M. (2023) · Sensors · doi:10.3390/s23187757 · Cited by 17
The ‘Paternal body’: Reviewing the corporeal impact of new fatherhood on employed men
Gatrell, C. (2025) · International Journal of Management Reviews · doi:10.1111/ijmr.12377 · Cited by 11
The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9
Metadata
- Authors
- Wohlin, C., Kalinowski, M., Romero Felizardo, K., & Mendes, E.
- Venue
- Information and Software Technology
- Publisher
- Elsevier BV
- Published
- 2022-07
- Volume
- 147
- Pages
- 106908
- ISSN
- 0950-5849 (ISSN-L)
- OpenAlex
- W4220698983
- Cited by
- 432
- References
- 29
- FWCI
- 44.84
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Funders
- Conselho Nacional de Desenvolvimento Científico e Tecnológico (312827/2020-2)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.