The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Systematic Literature Reviews (SLR) may not provide insight into the "state of the practice" in SE, as they do not typically include the "grey" (non-published) literature. A Multivocal Literature Review (MLR) is a form of a SLR which includes grey literature in addition to the published (formal) literature. Only a few MLRs have been published in SE so far. We aim at raising the awareness for MLRs in SE by addressing two research questions (RQs): (1) What types of knowledge are missed when a SLR does not include the multivocal literature in a SE field? and (2) What do we, as a community, gain when we include the multivocal literature and conduct MLRs? To answer these RQs, we sample a few example SLRs and MLRs and identify the missing and the gained knowledge due to excluding or including the grey literature. We find that (1) grey literature can give substantial benefits in certain areas of SE, and that (2) the inclusion of grey literature brings forward certain challenges as evidence in them is often experience and opinion based. Given these conflicting viewpoints, the authors are planning to prepare systematic guidelines for performing MLRs in SE.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, SLRs focus mainly on research contribution and do not include grey literature from practice (GL) [8].”
From Measuring Software Delivery Performance Using the Four Key Metrics of DevOps (2021) · cites this paper as [8]
In the citing paper
“Furthermore, the very current perspective, the relevance for practitioners and the low volume of published research indicates that not only formal literature should be used to cover a topic [8].”
From Measuring Software Delivery Performance Using the Four Key Metrics of DevOps (2021) · cites this paper as [8]
In the citing paper
“The multivocal literature review was conducted according to the guideline of Garousi et al. [8] which considers also the popular SLR guidelines by Kitchenham and Charters [12].”
From Measuring Software Delivery Performance Using the Four Key Metrics of DevOps (2021) · cites this paper as [8]
In the citing paper
“Within Software Engineering (SE), MLRs are being undertaken as exploratory studies [46] [47] and are seen as beneficial due to the practical nature of the field [43].”
From DevSecOps practices and tools (2025) · cites this paper as [43]
In the citing paper
“Therefore, to mitigate the challenges of working with GL sources, this research will adopt systematic guidelines for performing MLR in SE [43] that provide the research with a structured process to search, collect, and extract data from GL.”
From DevSecOps practices and tools (2025) · cites this paper as [43]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Extreme Programming Explained: Embrace Change
Beck, K. L., & Andres, C. (2000) · Cited by 5,902
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 1×
“We aim at raising the awareness for MLRs in SE by addressing two research questions (RQs): (1) What types of knowledge are missed when a SLR does not include the multivocal literature in a SE field? and (2) What do we, as a community, gain when we include the multivocal literature and conduct MLRs?”
Background(2008) · Encyclopedia of Epidemiology · doi:10.4135/9781412953948.n339 · Cited by 3,229
Guidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669 · cited 1×
“Many such SLRs and SMs appear regularly in various SE venues (journals and conferences) each year [1, 2].”
BackgroundExperimentation in Software Engineering: An Introduction
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2011) · Lund University Publications (Lund University) · Cited by 2,214
Experimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2000) · The Kluwer International Series in Software Engineering · doi:10.1007/978-1-4615-4625-2 · Cited by 1,776
Lean Software Development: An Agile Toolkit
Poppendieck, M., & Poppendieck, T. (2003) · International Linear Collider · Cited by 1,359
Extreme Programming Explained: Embrace Change (2nd Edition)
Beck, K. L., & Andres, C. (2004) · Addison-Wesley Professional eBooks · Cited by 929
Whyte, W. (1991) · Sage eBooks · doi:10.4135/9781412985383 · Cited by 917
Searching for grey literature for systematic reviews: challenges and benefits
Mahood, Q., Van Eerd, D., & Irvin, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1106 · Cited by 814 · cited 1×
“[ 10] discusses the challenges and benefits of including for grey literature in SLRs. Hopewell et al. [12] conducted a review of five studies, in the area of evidence-based medicine, comparing the effect of the inclusion or exclusion of ‘grey’ literature in meta-analyses of randomized medical…”
Background
Research works printed but not linked: 3
- 10.3102/00346543061003287 · doi:10.3102/00346543061003287 · Paper page
- Yasin A., 2012, Sweden
- D. J. Anderson Kanban: Blue Hole Press 2010. D. J. Anderson Kanban : Blue Hole Press 2010.
Cited by
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831
Self-citationSmart contract applications within blockchain technology: a systematic mapping study
Macrinici, D., Cartofeanu, C., & Gao, S. (2018) · Telematics and Informatics · doi:10.1016/j.tele.2018.10.004 · Cited by 458
“Garousi et al. (2016) underlined the importance of such literature in their work.”
BackgroundThe pains and gains of microservices: A Systematic grey literature review
Soldani, J., Tamburri, D. A., & Van Den Heuvel, W.-J. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.09.082 · Cited by 435
Garousi, V., Petersen, K., & Ozkan, B. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.07.006 · Cited by 241
Self-citationLwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241
Metadata
- Authors
- Garousi, V., Felderer, M., & Mäntylä, M. V.
- Affiliations
- Hacettepe University (TR)
- Universität Innsbruck (AT)
- University of Oulu (FI)
- Venue
- Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2016-06
- Pages
- 1-6
- OpenAlex
- W2407772516
- Cited by
- 210
- References
- 37
- Citations by year
- 2026: 5 · 2025: 20 · 2024: 17 · 2023: 16 · 2022: 25 · 2021: 28 · 2020: 25 · 2019: 21 · 2018: 34 · 2017: 13 · 2016: 6
- FWCI
- 20.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Optics and Image Analysis
- Keywords
- systematic reviews, grey literature, software engineering, state of practice, research guidelines, knowledge gaps
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature. https://app.cochat.ai/papers/10.1145/2915970.2916008
- The paper
- Garousi, V., Felderer, M., & Mäntylä, M. V. (2016). The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature. Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering. https://doi.org/10.1145/2915970.2916008
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.