Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 432

The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.

Successful combination of database search and snowballing for identification of primary studies in systematic literature studies

Open access
Wohlin, Claes, Kalinowski, Marcos, Romero Felizardo, Katia, Mendes, Emilia·2022·Information and Software Technology

Abstract

A good search strategy is essential for a successful systematic literature study. Historically, database searches have been the norm, which was later complemented with snowball searches. Our conjecture is that we can perform even better searches if combining these two search approaches, referred to as a hybrid search strategy. Our main objective was to compare and evaluate a hybrid search strategy. Furthermore, we compared four alternative hybrid search strategies to assess whether we could identify more cost-efficient ways of searching for relevant primary studies. To compare and evaluate the hybrid search strategy, we replicated the search procedure in a systematic literature review (SLR) on industry–academia collaboration in software engineering. The SLR used a more “traditional” approach to searching for relevant articles for an SLR, while our replication was executed using a hybrid search strategy. In our evaluation, the hybrid search strategy was superior in identifying relevant primary studies. It identified 30% more primary studies and even more studies when focusing only on peer-reviewed articles. To embrace individual viewpoints when assessing research articles and minimise the risk of missing primary studies, we introduced two new concepts, wild cards and borderline articles, when performing systematic literature studies. The hybrid search strategy is a strong contender for being used when performing systematic literature studies. Furthermore, alternative hybrid search strategies may be viable if selected wisely in relation to the start set for snowballing. Finally, the two new concepts were judged as essential to cater for different individual judgements and to minimise the risk of excluding primary studies that ought to be included.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.infsof.2022.106908

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 29 references

Research works printed but not linked: 5

  1. Punter, 2010, Industrial impact and lessons learned, 279, 10.1007/978-90-481-9849-8_17 · doi:10.1007/978-90-481-9849-8_17
  2. Rombach, 2008, Impact of research on practice in the field of inspections, reviews and walkthroughs: Learning from successful industrial uses, SIGSOFT Softw. Eng. Notes, 33, 26, 10.1145/1449603.1449609 · doi:10.1145/1449603.1449609
  3. S.J. Lamprecht, G.-J. van Rooyen, Models for technology research collaboration between industry and academia in South Africa, in: Proceedings IEEE Software Engineering Colloquium, SE, 2012, pp. 11–17. · doi:10.1109/se.2012.6242350
  4. W. Raschke, M. Zilli, J. Loinig, R. Weiss, C. Steger, C. Kreiner, Embedding research in the industrial field: A case of a transition to a software product line, in: Proceedings International Workshop on Long–Term Industrial Collaboration on Software Engineering, 2014, pp. 3–8. · doi:10.1145/2647648.2647649
  5. P. Krishnan, K. Ross, P. Pari-Salas, Industry academia collaboration: an experience report at a small university, in: Proceedings Conference on Software Engineering Education and Training, 2009, pp. 117–121. · doi:10.1109/cseet.2009.22

Cited by

Metadata

Authors
Wohlin, C., Kalinowski, M., Romero Felizardo, K., & Mendes, E.
Venue
Information and Software Technology
Publisher
Elsevier BV
Published
2022-07
Volume
147
Pages
106908
ISSN
0950-5849 (ISSN-L)
OpenAlex
W4220698983
Cited by
432
References
29
FWCI
44.84
Type
journal article
Language
en
Open access
hybrid
Licence
Funders
Conselho Nacional de Desenvolvimento Científico e Tecnológico (312827/2020-2)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.