CoChat
Sign Up
Cited by 71Meta-analysis and systematic reviewsscientometrics and bibliometrics researchInformation Retrieval and Search Behavior

Reference-based search strategies in systematic reviews

Open access
Skoglund, Mats, Runeson, Per·2009·Electronic Workshops in Computing
DOI 10.14236/ewic/ease2009.4
Open this paper in CoChat — free accountOpen PDF

Abstract

In systematic reviews, the number of articles found by search strings tend to be very large. In order to limit the number of articles to handle manually, we investigate a search strategy based on references between papers. We first identify a “take-off paper” which is the starting point for the search and then we follow the references from that paper. We also investigate “cardinal papers”, i.e. papers that are referenced by many authors, and let the references to those papers guide the selection in the systematic review. We evaluate the search strategies on three published systematic reviews. The results vary greatly between the three studied systematic reviews, from 88% reduction to 92% extension of the original paper set.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.14236/ewic/ease2009.4

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Despite that the proposed method increased the precision without missing too many relevant papers for the technically focused reviews, its results were not satisfactory when the search area was wide or the searches included general terms.

    “Despite that the proposed method increased the precision without missing too many relevant papers for the technically focused reviews, its results were not satisfactory when the search area was wide or the searches included general terms.”

    From Guidelines for snowballing in systematic literature studies and a replication in software engineering (2014) · cites this paper as [16]

    The Conclusions report increased precision without much loss of recall for the technically focused regression-testing review, but unsatisfactory results for the more general agile-methods review and the review of experiments. The paper does not specifically say that including general search terms caused the unsatisfactory results; it points instead to the many and diverse references to agile methods. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Skoglund and Runeson studied a reference-based search approach with the primary objective to reduce the number of initial articles found in systematic literature studies.

    “Skoglund and Runeson [16] studied a reference-based search approach with the primary objective to reduce the number of initial articles found in systematic literature studies.”

    From Guidelines for snowballing in systematic literature studies and a replication in software engineering (2014) · cites this paper as [16]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 11

  1. Bahli B. and Zeid, E.S.A. (2005) The role of knowledge creation in adopting extreme programming model: an empirical study. Proceedings of ITI 3rd International Conference on Information and Communications Technology: Enabling Technologies for the New Knowledge Society, Cairo, Egypt, 5-6 December, pp. 75-87.
  2. Brereton P., Kitchenham B.A., Budgen D., Turner M. and Khalil M. (2007) Lessons from applying the systematic literature review process within the software engineering domain. Journal of Systems and Software, 80, 571-583.
  3. Engström E., Skoglund M. and Runeson P. (2008) Empirical Evaluations of Regression Test Selection Techniques: A Systematic Review. Proceedings of International Symposium on Empirical Software Engineering and Measurement, Kaiserslauten, Germany, pp. 22-31.
  4. Jørgensen A. and Sheppard M (2007) A systematic review of software development cost estimation studies. IEEE Transactions of software engineering, 33, 33-53.
  5. Martyn J. (1964) Bibliographic coupling. Journal of documentation, 20, 236.
  6. Rothermel G and Harrold M. (1996) Analyzing Regression test selection techniques. IEEE Transactions on Software Engineering, 22, 529-551.
  7. Rothermel G. and Harrold M. (1997) A Safe Efficient Regression Test Selection Technique. ACM Transactions on Software Engineering and Methodology, 6, 173-210.
  8. Sjøberg D., Hannay J., Hansen O., Kampenes V., Karahasanovic, Liborg N. and Rekdal A. (2005) A survey of controlled experiments in software engineering, IEEE Transactions of software engineering. 31, 733-753.
  9. Thelin, T., Runeson, P., Wohlin, C., Olsson, T. and Andersson, C. (2002) How much information is needed for usage-based reading? A series of experiments. Proceedings of International Symposium on Empirical Software Engineering, Nara, Japan, pp. 127-138.
  10. Tsay M., Xu H. and Wu C. (2003) Journal co-citation analysis of semiconductor literature. Scientometrics, 57, 7-25.
  11. Zheng J., Robinson B., Williams L. and Smiley K. (2006) Applying Regression Test Selection for COTS-based Applications. Proceedings of International Conference on Software Engineering, Shanghai, China. pp. 512-522.

Cited by

Sorted by citations

Authors
Skoglund, M., & Runeson, P.
Affiliations
Lund University (SE)
ORCID
Runeson, Per
Venue
Electronic Workshops in Computing
Publisher
BCS Learning & Development
Published
2009
ISSN
1477-9358 (ISSN-L)
DOI
10.14236/ewic/ease2009.4
OpenAlex
W1741569902
Cited by
71
References
3
Citations by year
2025: 4 · 2024: 6 · 2022: 4 · 2021: 5 · 2020: 2 · 2018: 6 · 2017: 1 · 2016: 3 · 2015: 7 · 2014: 3 · 2013: 6 · 2012: 19
FWCI
0.61
Type
journal article
Language
en
Open access
Licence
Locations
  • Electronic workshops in computing (publishedVersion, cc-by)
  • Lund University Publications (Lund University) (submittedVersion, other-oa)
Funders
  • VINNOVA
  • Vetenskapsrådet
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
systematic reviews, citation analysis, literature searching, bibliometric indicators

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review