The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Systematic Literature Reviews (SLR) may not provide insight into the "state of the practice" in SE, as they do not typically include the "grey" (non-published) literature. A Multivocal Literature Review (MLR) is a form of a SLR which includes grey literature in addition to the published (formal) literature. Only a few MLRs have been published in SE so far. We aim at raising the awareness for MLRs in SE by addressing two research questions (RQs): (1) What types of knowledge are missed when a SLR does not include the multivocal literature in a SE field? and (2) What do we, as a community, gain when we include the multivocal literature and conduct MLRs? To answer these RQs, we sample a few example SLRs and MLRs and identify the missing and the gained knowledge due to excluding or including the grey literature. We find that (1) grey literature can give substantial benefits in certain areas of SE, and that (2) the inclusion of grey literature brings forward certain challenges as evidence in them is often experience and opinion based. Given these conflicting viewpoints, the authors are planning to prepare systematic guidelines for performing MLRs in SE.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, SLRs focus mainly on research contribution and do not include grey literature from practice (GL) [8].”
From Measuring Software Delivery Performance Using the Four Key Metrics of DevOps (2021) · cites this paper as [8]
In the citing paper
“Furthermore, the very current perspective, the relevance for practitioners and the low volume of published research indicates that not only formal literature should be used to cover a topic [8].”
From Measuring Software Delivery Performance Using the Four Key Metrics of DevOps (2021) · cites this paper as [8]
In the citing paper
“The multivocal literature review was conducted according to the guideline of Garousi et al. [8] which considers also the popular SLR guidelines by Kitchenham and Charters [12].”
From Measuring Software Delivery Performance Using the Four Key Metrics of DevOps (2021) · cites this paper as [8]
In the citing paper
“Within Software Engineering (SE), MLRs are being undertaken as exploratory studies [46] [47] and are seen as beneficial due to the practical nature of the field [43].”
From DevSecOps practices and tools (2025) · cites this paper as [43]
In the citing paper
“Therefore, to mitigate the challenges of working with GL sources, this research will adopt systematic guidelines for performing MLR in SE [43] that provide the research with a structured process to search, collect, and extract data from GL.”
From DevSecOps practices and tools (2025) · cites this paper as [43]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Extreme Programming Explained: Embrace Change
Beck, K. L., & Andres, C. (2000) · Cited by 5,902
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 1×
“We aim at raising the awareness for MLRs in SE by addressing two research questions (RQs): (1) What types of knowledge are missed when a SLR does not include the multivocal literature in a SE field? and (2) What do we, as a community, gain when we include the multivocal literature and conduct MLRs?”
Background(2008) · Encyclopedia of Epidemiology · doi:10.4135/9781412953948.n339 · Cited by 3,229
Guidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669 · cited 1×
“Many such SLRs and SMs appear regularly in various SE venues (journals and conferences) each year [1, 2].”
BackgroundExperimentation in Software Engineering: An Introduction
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2011) · Lund University Publications (Lund University) · Cited by 2,214
Experimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2000) · The Kluwer International Series in Software Engineering · doi:10.1007/978-1-4615-4625-2 · Cited by 1,776
Lean Software Development: An Agile Toolkit
Poppendieck, M., & Poppendieck, T. (2003) · International Linear Collider · Cited by 1,359
Extreme Programming Explained: Embrace Change (2nd Edition)
Beck, K. L., & Andres, C. (2004) · Addison-Wesley Professional eBooks · Cited by 929
Whyte, W. (1991) · Sage eBooks · doi:10.4135/9781412985383 · Cited by 917
Searching for grey literature for systematic reviews: challenges and benefits
Mahood, Q., Van Eerd, D., & Irvin, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1106 · Cited by 814 · cited 1×
“[ 10] discusses the challenges and benefits of including for grey literature in SLRs. Hopewell et al. [12] conducted a review of five studies, in the area of evidence-based medicine, comparing the effect of the inclusion or exclusion of ‘grey’ literature in meta-analyses of randomized medical…”
BackgroundA systematic mapping study on technical debt and its management
Li, Z., Avgeriou, P., & Liang, P. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2014.12.027 · Cited by 743 · cited 2×
“Whereas, in Table 3 we saw the most recent study (SLR2 [47] published in 2015) had the most comprehensive the results, we see in Table 4 that this is not the case for technical debt ‘causes’.”
ResultGrey literature in meta-analyses of randomized trials of health care interventions
Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000010.pub3 · Cited by 598
(2012) · Software in 30 Days · doi:10.1002/9781119203278.app2 · Cited by 445 · cited 1×
“The SLR collected agile and lean metrics only from the primary books presenting the methodologies for Scrum [35], XP [36], Kanban [37] and Lean [38] but this part could have been extended with more systematic search.”
MethodAn exploration of technical debt
Tom, E., Aurum, A., & Vidgen, R. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2012.12.052 · Cited by 438 · cited 7×
“Other authors have recognized similar difficulties, e.g., [13]: “ there are apparent issues of reliability and validity associated with these writings due to their diversity”.”
BackgroundMethodInfluentialState‐of‐the‐Evidence Reviews: Advantages and Challenges of Including Grey Literature
Benzies, K. M., Premji, S., Hayden, K. A., & Serrett, K. (2006) · Worldviews on Evidence-Based Nursing · doi:10.1111/j.1741-6787.2006.00051.x · Cited by 408 · cited 1×
“Due to potential value and the problems related to grey literature, the authors are planning to prepare a set of systematic guidelines for conducting MLRs in SE based on our own experiences and by adopting guidelines of MLRs in other areas, e.g., education [5] and nursing [9].”
BackgroundClarke, P., & O’Connor, R. V. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2011.12.003 · Cited by 332 · cited 1×
“The importance of context (the 6 th element in the rubric) has been extensively been discussed in the SE literature, e.g., [48, 49].”
BackgroundContext in industrial software engineering research
Petersen, K., & Wohlin, C. (2009) · 2009 3rd International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2009.5316010 · Cited by 270 · cited 1×
“The importance of context (the 6 th element in the rubric) has been extensively been discussed in the SE literature, e.g., [48, 49].”
BackgroundTowards Rigor in Reviews of Multivocal Literatures: Applying the Exploratory Case Study Method
Ogawa, R. T., & Malen, B. (1991) · Review of Educational Research · doi:10.3102/00346543061003265 · Cited by 247 · cited 4×
“MLRs have become popular in several other fields, e.g., education, e.g., [5, 6].”
BackgroundMethodInfluentialKupiainen, E., Mäntylä, M. V., & Itkonen, J. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.02.005 · Cited by 211 · cited 3×
“Two of these SLRs [17, 18] are studies that the authors have co-authored in recent years.”
BackgroundMethodSelf-citationPentinmaki, I. (2004) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/979743.979773 · Cited by 189
The financial aspect of managing technical debt: A systematic literature review
Ampatzoglou, A., Ampatzoglou, A., Chatzigeorgiou, A., & Avgeriou, P. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.04.001 · Cited by 186
Refactoring for Software Design Smells: Managing Technical Debt
Suryanarayana, G., Samarthyam, G., & Sharma, T. (2014) · Cited by 170
Graphical user interface (GUI) testing: Systematic mapping and repository
Banerjee, I., Nguyen, B., Garousi, V., & Memon, A. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.03.004 · Cited by 168 · cited 3×
“Two of these SLRs [17, 18] are studies that the authors have co-authored in recent years.”
BackgroundMethodSelf-citationGrey literature in meta-analyses of randomized trials of health care interventions
Hopewell, S., McDonald, S., Clarke, M., & Egger, M. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000010.pub2 · Cited by 137
Grey Literature and Systematic Reviews
Hopewell, S., Clarke, M., & Mallett, S. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch4 · Cited by 128 · cited 1×
“, [9-11].”
Suryanarayana, G., Samarthyam, G., & Sharma, T. (2014) · Elsevier eBooks · doi:10.1016/b978-0-12-801397-7.00001-1 · Cited by 62
A Systematic Literature Review of Software Process Improvement in Small and Medium Web Companies
Sulayman, M., & Mendes, E. (2009) · Communications in computer and information science · doi:10.1007/978-3-642-10619-4_1 · Cited by 57 · cited 1×
“Many other SLRs have also included the grey literature in their reviews and have not used the ‘multivocal’ terminology, e.g., [15].”
BackgroundManaging Software Debt: Building for Inevitable Change
Sterling, C. (2010) · Cited by 54
Scaling Lean: Mastering the Key Metrics for Startup Growth
Maurya, A. (2016) · Cited by 42
Towards Utility in Reviews of Multivocal Literatures
Patton, M. Q. (1991) · Review of Educational Research · doi:10.2307/1170631 · Cited by 33 · cited 2×
“MLRs have become popular in several other fields, e.g., education, e.g., [5, 6].”
BackgroundA CONSOLIDATED UNDERSTANDING OF TECHNICAL DEBT
Tom, E., Aurum, A., & Vidgen, R. T. (2012) · Journal of the Association for Information Systems · Cited by 20
Software Engineering Research under the Lamppost.
Wohlin, C. (2013) · International Conference on Software and Data Technologies · Cited by 18
Kuļešovs, I. (2015) · Environment Technology Resources Proceedings of the International Scientific and Practical Conference · doi:10.17770/etr2015vol3.187 · Cited by 15 · cited 2×
“[14] is a 2015 MLR on iOS applications testing.”
MethodElmore, R. F. (1991) · Review of Educational Research · doi:10.3102/00346543061003293 · Cited by 14 · cited 1×
“Researchers also have reported that: “another potential use of multivocal literature reviews is in closing the gap between academic research and professional practice ” [8].”
BackgroundYasin, A., & Hasnain, M. (2012) · Cited by 11
Nicolette, D. (2015) · International Linear Collider · Cited by 7
Test Automation using Selenium WebDriver with Java: Step by Step Guide
Garg, N. (2014) · Cited by 1
Research works printed but not linked: 3
- 10.3102/00346543061003287 · doi:10.3102/00346543061003287 · Paper page
- Yasin A., 2012, Sweden
- D. J. Anderson Kanban: Blue Hole Press 2010. D. J. Anderson Kanban : Blue Hole Press 2010.
Cited by
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831
Self-citationSmart contract applications within blockchain technology: a systematic mapping study
Macrinici, D., Cartofeanu, C., & Gao, S. (2018) · Telematics and Informatics · doi:10.1016/j.tele.2018.10.004 · Cited by 458
“Garousi et al. (2016) underlined the importance of such literature in their work.”
BackgroundThe pains and gains of microservices: A Systematic grey literature review
Soldani, J., Tamburri, D. A., & Van Den Heuvel, W.-J. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.09.082 · Cited by 435
Garousi, V., Petersen, K., & Ozkan, B. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.07.006 · Cited by 241
Self-citationLwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241
Metadata
- Authors
- Garousi, V., Felderer, M., & Mäntylä, M. V.
- Affiliations
- Hacettepe University (TR)
- Universität Innsbruck (AT)
- University of Oulu (FI)
- Venue
- Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2016-06
- Pages
- 1-6
- OpenAlex
- W2407772516
- Cited by
- 210
- References
- 37
- Citations by year
- 2026: 5 · 2025: 20 · 2024: 17 · 2023: 16 · 2022: 25 · 2021: 28 · 2020: 25 · 2019: 21 · 2018: 34 · 2017: 13 · 2016: 6
- FWCI
- 20.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Optics and Image Analysis
- Keywords
- systematic reviews, grey literature, software engineering, state of practice, research guidelines, knowledge gaps
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.