CoChat

The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature

Garousi, Vahid, Felderer, Michael, Mäntylä, Mika V. · 2016 · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.

Abstract

Systematic Literature Reviews (SLR) may not provide insight into the "state of the practice" in SE, as they do not typically include the "grey" (non-published) literature. A Multivocal Literature Review (MLR) is a form of a SLR which includes grey literature in addition to the published (formal) literature. Only a few MLRs have been published in SE so far. We aim at raising the awareness for MLRs in SE by addressing two research questions (RQs): (1) What types of knowledge are missed when a SLR does not include the multivocal literature in a SE field? and (2) What do we, as a community, gain when we include the multivocal literature and conduct MLRs? To answer these RQs, we sample a few example SLRs and MLRs and identify the missing and the gained knowledge due to excluding or including the grey literature. We find that (1) grey literature can give substantial benefits in certain areas of SE, and that (2) the inclusion of grey literature brings forward certain challenges as evidence in them is often experience and opinion based. Given these conflicting viewpoints, the authors are planning to prepare systematic guidelines for performing MLRs in SE.

Overview

Integrity signals

Registry

Crossref · 10.1145/2915970.2916008

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Research works printed but not linked: 3

  1. 10.3102/00346543061003287 · doi:10.3102/00346543061003287 · Paper page
  2. Yasin A., 2012, Sweden
  3. D. J. Anderson Kanban: Blue Hole Press 2010. D. J. Anderson Kanban : Blue Hole Press 2010.

Cited by

See all 210

Metadata

Authors
Garousi, V., Felderer, M., & Mäntylä, M. V.
Affiliations
  • Hacettepe University (TR)
  • Universität Innsbruck (AT)
  • University of Oulu (FI)
Venue
Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering
Publisher
ACM
Published
2016-06
Pages
1-6
OpenAlex
W2407772516
Cited by
210
References
37
Citations by year
2026: 5 · 2025: 20 · 2024: 17 · 2023: 16 · 2022: 25 · 2021: 28 · 2020: 25 · 2019: 21 · 2018: 34 · 2017: 13 · 2016: 6
FWCI
20.95
Type
conference paper
Language
en
Open access
closed
Locations
International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Optics and Image Analysis
Keywords
systematic reviews, grey literature, software engineering, state of practice, research guidelines, knowledge gaps

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.