Towards Rigor in Reviews of Multivocal Literatures: Applying the Exploratory Case Study Method
Abstract
This article aims to stimulate discussion about the issue of rigor in conducting reviews of multivocal literatures. Multivocal literatures, which abound in the field of education, are comprised of all accessible writings on a common, often contemporary topic. The exploratory case study method is proposed as a means to engender rigor in reviews of such literatures. It is argued that it is appropriate to apply the concept of rigor to reviews of multivocal literatures and to use the exploratory case study method as a tool for thinking about procedures that could enhance rigor in such reviews. The article draws upon the authors’ experiences in conducting a review of the literature on school-based management to illustrate how the proposed procedures might be employed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
GL poses challenges for researchers: accessibility, discoverability, bias, low quality, and lack of standardization.
“Some of the challenges for researchers while selecting GL sources are accessibility, discoverability, bias, low quality, and lack of standardization [49] [41].”
From DevSecOps practices and tools (2025) · cites this paper as [49]
Cites
(1984) · Cited by 81,983 · see citing works
Wagner, H. R., Glaser, B. G., & Strauss, A. L. (1968) · Social Forces · doi:10.2307/2575405 · Cited by 46,470 · see citing works
The Discovery of Grounded Theory; Strategies for Qualitative Research
Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968) · Nursing Research · doi:10.1097/00006199-196807000-00014 · Cited by 39,449 · see citing works
DiMaggio, P. J., & Powell, W. W. (1983) · American Sociological Review · doi:10.2307/2095101 · Cited by 38,435 · see citing works
Basics of Qualitative Research: Grounded Theory Procedures and Techniques.
Ellis, C., Strauss, A., & Corbin, J. M. (1992) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2074814 · Cited by 29,279 · see citing works
DiMaggio, P., & Powell, W. W. (2004) · Advances in strategic management · doi:10.1016/s0742-3322(00)17011-1 · Cited by 25,866 · see citing works
(1990) · Cited by 24,217 · see citing works
Qualitative evaluation and research methods
Ellis, D. J. (1990) · International Journal of Information Management · doi:10.1016/0268-4012(90)90041-p · Cited by 17,942 · see citing works
Basics of Qualitative Research: Grounded Theory Procedures and Techniques
Long, D. R., Strauss, A., & Corbin, J. M. (1993) · Modern Language Journal · doi:10.2307/328955 · Cited by 17,612 · see citing works
(1990) · Cited by 8,425 · see citing works
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · see citing works
Book Reviews: Qualitative Evaluation Methods
Cordray, D. S., & Patton, M. Q. (1981) · Educational Evaluation and Policy Analysis · doi:10.2307/1164029 · Cited by 4,163 · see citing works
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,896 · see citing works
Introduction to Qualitative Research Methods
Taylor, S. J., Bogdan, R., & DeVault, M. L. (2015) · Wiley eBooks · doi:10.1002/9781394260485 · Cited by 3,070 · see citing works
Institutional Theories Of Organization
Zucker, L. G. (1987) · Annual Review of Sociology · doi:10.1146/annurev.so.13.080187.002303 · Cited by 2,720 · see citing works
Beyond the two disciplines of scientific psychology.
Cronbach, L. J. (1975) · American Psychologist · doi:10.1037/h0076829 · Cited by 2,097 · see citing works
Problems of Reliability and Validity in Ethnographic Research
LeCompte, M. D., & Goetz, J. P. (1982) · Review of Educational Research · doi:10.3102/00346543052001031 · Cited by 1,740 · see citing works
Levy, T. S., & Lofland, J. (1974) · Journal of Architectural Education (1947-1974) · doi:10.2307/1423857 · Cited by 1,573 · see citing works
A Framework for Political Analysis.
Rogow, A. A., & Easton, D. (1966) · Midwest Journal of Political Science · doi:10.2307/2108792 · Cited by 1,468 · see citing works
Drawing Valid Meaning from Qualitative Data: Toward a Shared Craft
Miles, M. B., & Huberman, A. M. (1984) · Educational Researcher · doi:10.3102/0013189x013005020 · Cited by 1,173 · see citing works
III. “Degrees of Freedom” and the Case Study
Campbell, D. T. (1975) · Comparative Political Studies · doi:10.1177/001041407500800204 · Cited by 1,059 · see citing works
A framework for political analysis
Easton, D. (1965) · Medical Entomology and Zoology · Cited by 937 · see citing works
Best-Evidence Synthesis: An Alternative to Meta-Analytic and Traditional Reviews
Slavin, R. E. (1986) · Educational Researcher · doi:10.3102/0013189x015009005 · Cited by 699 · see citing works
SMITH, J. K., & HESHUSIUS, L. (1986) · Educational Researcher · doi:10.3102/0013189x015001004 · Cited by 675 · see citing works
Environments and Organizations
Pennings, J. M., & Meyer, M. W. (1979) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2064601 · Cited by 587 · see citing works
Complementary methods for research in education
Winne, P. H. (1989) · Teaching and Teacher Education · doi:10.1016/0742-051x(89)90032-2 · Cited by 547 · see citing works
Wolfle, D. L. (1964) · Science · doi:10.1126/science.143.3608.767 · Cited by 406 · see citing works
Standards for Qualitative (And Quantitative) Research: A Prolegomenon
Howe, K. R., & EISENHART, M. A. (1990) · Educational Researcher · doi:10.3102/0013189x019004002 · Cited by 367 · see citing works
Introduction to Qualitative Research Methods
Bulmer, M., Bogdan, R., Taylor, S. J., Johnson, J. M., & Costner, H. L. (1976) · British Journal of Sociology · doi:10.2307/590191 · Cited by 359 · see citing works
Using the Case Survey Method to Analyze Policy Studies
Yin, R. K., & Heald, K. A. (1975) · Administrative Science Quarterly · doi:10.2307/2391997 · Cited by 314 · see citing works
Reviewing the literature: A comparison of traditional methods with meta-analysis
Cook, T. D., & Leviton, L. C. (1980) · Journal of Personality · doi:10.1111/j.1467-6494.1980.tb02379.x · Cited by 270 · see citing works
Disciplines of Inquiry in Education: An Overview
Shulman, L. S. (1981) · Educational Researcher · doi:10.3102/0013189x010006005 · Cited by 239 · see citing works
Choice and Control in American Education
Hawley, W. D., Clune, W. H., & Witte, J. F. (1992) · Educational Evaluation and Policy Analysis · doi:10.2307/1164502 · Cited by 228 · see citing works
Kara, H. (2023) · Policy Press eBooks · doi:10.56687/9781447366263-015 · Cited by 211 · see citing works
Synthesis of Research on School-Based Management
David, J. L. (1989) · Educational leadership · Cited by 206 · see citing works
Knowledge and Values in Social and Educational Research
Bredo, E., & Feinberg, W. (1983) · Medical Entomology and Zoology · Cited by 205 · see citing works
Meta-Analysis in Education: How Has It Been Used?
Slavin, R. E. (1984) · Educational Researcher · doi:10.3102/0013189x013008006 · Cited by 191 · see citing works
Dimensions of Organizations in Complex Societies: The Educational Sector
Holdaway, E. A., Newberry, J. F., Hickson, D. J., & Heron, R. (1975) · Administrative Science Quarterly · doi:10.2307/2392122 · Cited by 90 · see citing works
The design of social policy research
Mayer, R. R., & Greenwood, E. (1980) · Medical Entomology and Zoology · Cited by 65 · see citing works
Best-Evidence Synthesis: Why Less Is More
Slavin, R. E. (1987) · Educational Researcher · doi:10.3102/0013189x016004015 · Cited by 42 · see citing works
Pechman, J. A. (1983) · The Brookings Review · doi:10.2307/20079809 · Cited by 42 · see citing works
Integrating research evidence: A commentary on meta-analysis.
Mintz, J. (1983) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.51.1.71 · Cited by 41 · see citing works
Cole, R. L. (1976) · Political Science Quarterly · doi:10.2307/2148428 · Cited by 33 · see citing works
Teachers, power and school change
Louis, K. S. (1990) · Cited by 22 · see citing works
Frost, S. B. (1969) · McGill-Queen's University Press eBooks · doi:10.1515/9780773592827-006 · Cited by 17 · see citing works
A Rigorous Yet Delicate Touch: A Response to Slavin's Proposal for "Best-Evidence" Reviews
Joyce, B. R. (1987) · Educational Researcher · doi:10.3102/0013189x016004012 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Purposeful Sampling in Qualitative Research Synthesis
Suri, H. (2011) · Qualitative Research Journal · doi:10.3316/qrj1102063 · Cited by 2,284 · see citing works
“A rigorous research synthesis makes much more demand on time and resources when compared with ad hoc reviews (Elmore, 1991; Ogawa & Malen, 1991; Stock, Benito, and Lasa, 1996).”
MethodConceptualizing Teacher Professional Learning
Opfer, V. D., & Pedder, D. (2011) · Review of Educational Research · doi:10.3102/0034654311413609 · Cited by 1,814 · see citing works
“To help develop and validate this understanding, we relied on “informant confirmation” (Boote & Beile, 2005; Ogawa & Malen, 1991).”
MethodGarousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831 · see citing works
A Framework for Guiding and Evaluating Literature Reviews
Templier, M., & Paré, G. (2015) · Communications of the Association for Information Systems · doi:10.17705/1cais.03706 · Cited by 553 · see citing works
“According to Oxman and Guyatt (1988), “important aspects of the design and conduct of each primary study should be critiqued and the standard used in these critiques made explicit” (p.”
InfluentialAn exploration of technical debt
Tom, E., Aurum, A., & Vidgen, R. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2012.12.052 · Cited by 438 · see citing works
Advances in Learning and Behavioural Disabilities
Scruggs, T. E. (1999) · Cited by 378 · see citing works
Research synthesis in software engineering: A tertiary study
Cruzes, D. S., & Dybå, T. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.01.004 · Cited by 339 · see citing works
“As SRs evolve and continue to gain popularity, awareness is needed to ensure greater transparency and methodological rigor, which will increase the legitimacy of findings and relevance for practice [40].”
BackgroundWhat We Know about Effective Instructional Practices for English-Language Learners
Gersten, R. M., & Baker, S. (2000) · Exceptional Children · doi:10.1177/001440290006600402 · Cited by 322 · see citing works
Advancements in Research Synthesis Methods: From a Methodologically Inclusive Perspective
Suri, H., & Clarke, D. J. (2009) · Review of Educational Research · doi:10.3102/0034654308326349 · Cited by 320 · see citing works
“…(e.g., Engberg, 2004; Kasworm, 1990); statistical techniques (Cooper & Hedges, 1994a); interpretive techniques (Bair, 1999; Noblit & Hare, 1988; Ogawa & Malen, 1991b); historical methods (Donato & Lazerson, 2000); visual displays (Pope et al., 2007); “narrative juxtaposition” (Dixon-Woods,…”
BackgroundInfluentialFatmi, M., Hartling, L., Hillier, T., Campbell, S., & Oswald, A. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.849802 · Cited by 306 · see citing works
“The data were synthesized with guidance from methods described by Ogawa and Malen (1991) who have developed a method for synthesis based on the exploratory case study method.”
MethodLearning in Communities of Inquiry: A Review of the Literature
Rourke, L., & Kanuka, H. (2009) · Cited by 295 · see citing works
Garousi, V., Felderer, M., & Mäntylä, M. V. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2916008 · Cited by 210 · see citing works
“MLRs have become popular in several other fields, e.g., education, e.g., [5, 6].”
BackgroundMethodInfluentialDevSecOps: A Multivocal Literature Review
Myrbakken, H., & Colomo-Palacios, R. (2017) · Communications in Computer and Information Science · doi:10.1007/978-3-319-67383-7_2 · Cited by 205 · see citing works
“By using this variety of literature the results will give a more nuanced look at the topic, since it includes the voices and opinions of academics, practitioners, independent researchers, development firms and others with experience [16].”
BackgroundSmells in software test code: A survey of knowledge in industry and academia
Garousi, V., & Küçük, B. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.12.013 · Cited by 189 · see citing works
The financial aspect of managing technical debt: A systematic literature review
Ampatzoglou, A., Ampatzoglou, A., Chatzigeorgiou, A., & Avgeriou, P. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.04.001 · Cited by 186 · see citing works
“Consequently, concerning technical debt, the multi-vocal results [35] are expected to provide indications on how the concept is exploited by both researchers and practitioners; although it goes without saying that using such not peer-reviewed experience reports or position articles raises significant reliability and v…”
BackgroundWhen and what to automate in software testing? A multi-vocal literature review
Garousi, V., & Mäntylä, M. V. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.04.015 · Cited by 182 · see citing works
“A 1991 paper [29] discussed rigor in MLRs and proposed a method based on exploratory case study to conduct MLRs rigorously.”
BackgroundMethodThe Blockchain Oracle Problem in Decentralized Finance—A Multivocal Approach
Caldarelli, G., & Ellul, J. (2021) · Applied Sciences · doi:10.3390/app11167572 · Cited by 181 · see citing works
Volunteer Tutoring Programs in Reading: A Review
Wasik, B. A. (1998) · Reading Research Quarterly · doi:10.1598/rrq.33.3.2 · Cited by 150 · see citing works
Digital Leadership ‒ Mountain or Molehill? A Literature Review
Eberl, J. K., & Drews, P. (2021) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-86800-0_17 · Cited by 134 · see citing works
“To avoid a lack of documentation and ensure reliability, an additional step from Ogawa & Malen’s [15] framework was added as first step resulting in the following steps for the literature review: (1) create an audit trail documenting the reviewers’ steps, (2) define the research scope, (3) search for literature, (4) s…”
MethodReading Interventions for Students with Emotional/Behavioral Disorders
Coleman, M., & Vaughn, S. R. (2000) · Behavioral Disorders · doi:10.1177/019874290002500201 · Cited by 133 · see citing works
Insights into startup ecosystems through exploration of multi-vocal literature
Tripathi, N., Seppänen, P., Boominathan, G., Oivo, M., & Liukkunen, K. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.08.005 · Cited by 127 · see citing works
The Effectiveness of Reading Interventions for English Learners
Richards‐Tutor, C., Baker, D. L., Gersten, R. M., Baker, S. K., & Smith, J. M. (2015) · Exceptional Children · doi:10.1177/0014402915585483 · Cited by 123 · see citing works
“…few experimental studies and consequently expanded their synthesis to include descriptive and qualitative studies and also incorporated the professional opinions of practitioners with specific expertise in working effectively with English learners (i.e., Noblit & Hare, 1988; Ogawa & Malen, 1991).”
BackgroundAn exploration of IoT platform development
Fahmideh, M., & Zowghi, D. (2019) · Information Systems · doi:10.1016/j.is.2019.06.005 · Cited by 119 · see citing works
Behutiye, W. N., Rodríguez, P., Oivo, M., & Tosun, A. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2016.10.004 · Cited by 116 · see citing works
“(multivocal literature reviews are comprised of all accessible writings, including non-academic writings on a topic [22].)”
BackgroundThe Subject Matter Knowledge of Preservice Science Teachers
Cochran, K. F., & Jones, L. L. (1998) · Springer eBooks · doi:10.1007/978-94-011-4940-2_41 · Cited by 112 · see citing works
Characterizing DevOps by Hearing Multiple Voices
de França, B. B. N., Jeronimo, H., & Travassos, G. H. (2016) · doi:10.1145/2973839.2973845 · Cited by 109 · see citing works
“The diversity of sources appears in a variety of forms (e.g., academic literature, industry reports , tool vendors’ websites, and blogs), reflecting different purposes and perspectives [8].”
BackgroundMethodInfluentialSoftware engineering in start-up companies: An analysis of 88 experience reports
Klotins, E., Unterkalmsteiner, M., & Gorschek, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9620-y · Cited by 106 · see citing works
Software test maturity assessment and test process improvement: A multivocal literature review
Garousi, V., Felderer, M., & Hacaloğlu, T. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.01.001 · Cited by 105 · see citing works
An integrated framework to prioritize blockchain-based supply chain success factors
Shoaib, M., Lim, M. K., & Wang, C. (2020) · Industrial Management & Data Systems · doi:10.1108/imds-04-2020-0194 · Cited by 108 · see citing works
Sauerwein, C., Sillaber, C., Mussmann, A., & Breu, R. (2017) · Journal of the Association for Information Systems · Cited by 100 · see citing works
Do We Perform Systematic Literature Review Right? A Scientific Mapping and Methodological Assessment
Azarian, M., Yu, H., Shiferaw, A. T., & Stevik, T. K. (2023) · Logistics · doi:10.3390/logistics7040089 · Cited by 140 · see citing works
Maturity assessment and maturity models in health care: A multivocal literature review
Tarhan, A., Garousi, V., Türetken, O., Söylemez, M., & Garossi, S. (2020) · Digital Health · doi:10.1177/2055207620914772 · Cited by 11 · see citing works
Anderson, G. L., & Jones, F. B. (2000) · Educational Administration Quarterly · doi:10.1177/00131610021969056 · Cited by 92 · see citing works
Global women political leaders: An invisible history, an increasingly important future
Adler, N. J. (1996) · The Leadership Quarterly · doi:10.1016/s1048-9843(96)90038-7 · Cited by 91 · see citing works
A Systematic Investigation of the Integration of Machine Learning into Supply Chain Risk Management
Schroeder, M., & Lodemann, S. (2021) · Logistics · doi:10.3390/logistics5030062 · Cited by 86 · see citing works
“] are comprised of all accessible writings on a common, often contemporary topic” [43].”
BackgroundMethodThe changing leadership role of the principal: Implications for principal assessment
Glasman, N. S., & Heck, R. H. (1990) · Peabody Journal of Education · doi:10.1080/01619569209538708 · Cited by 82 · see citing works
Rethinking the Politics of Fit and Educational Leadership
Tooms, A. K., Lugg, C. A., & Bogotch, I. E. (2009) · Educational Administration Quarterly · doi:10.1177/1094670509353044 · Cited by 73 · see citing works
Trend spotting: fashions in medical education
Campbell, J. K., & Johnson, C. A. (1999) · BMJ · doi:10.1136/bmj.318.7193.1272 · Cited by 72 · see citing works
Relationship of DevOps to Agile, Lean and Continuous Deployment
Lwakatare, L. E., Kuvaja, P., & Oivo, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49094-6_27 · Cited by 71 · see citing works
Synthesizing evidence in software engineering research
Cruzes, D. S., & Dybå, T. (2010) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1852786.1852788 · Cited by 70 · see citing works
“As SRs evolve and continue to gain popularity, awareness is needed to ensure greater transparency and methodological rigor that will increase the legitimacy of findings and relevance for practice [20].”
BackgroundGlobal Trends and Research Aims for English Academic Oral Presentations
Barrett, N. E., & Liu, G. (2016) · Review of Educational Research · doi:10.3102/0034654316628296 · Cited by 69 · see citing works
“We adapted the methodology of Ogawa and Malen (1991) and followed the guidelines set down by Randolph: (a) create an audit trail of each step in the review, (b) define the focus of the review to explain what to include and what to omit, (c) search for relevant literature, (d) classify the…”
MethodOpportunities and risks involved in using ChatGPT to create first grade science lesson plans
Powell, W. A., & Courchesne, S. (2024) · PLoS ONE · doi:10.1371/journal.pone.0305337 · Cited by 91 · see citing works
The Politics of Advocacy in Education
Scott, J. T., Lubienski, C., & DeBray, E. H. (2008) · Educational Policy · doi:10.1177/0895904808328530 · Cited by 67 · see citing works
“Finally, Gary Anderson provides a polyvocal (Ogawa & Malen, 1985) overview of the context of military presence in American schooling.”
BackgroundResponsible slum tourism: Egyptian experience
Mekawy, M. A. (2012) · Annals of Tourism Research · doi:10.1016/j.annals.2012.07.006 · Cited by 67 · see citing works
Special Purpose Entities in Megaprojects: Empty Boxes or Real Companies?
Sainati, T., Brookes, N. J., & Locatelli, G. (2017) · Project Management Journal · doi:10.1177/875697281704800205 · Cited by 63 · see citing works
“58 April/May 2017 ■ Project Management Journal P A P E R S and project management domains, the research adopts the review criteria introduced by Ogawa and Malen (1991) and Randolph (2009).”
MethodTowards a Definition of Environmental Sustainability Evaluation in Higher Education
Alba, D., del Álamo, J. B., & Pérez, J. G. (2018) · Higher Education Policy · doi:10.1057/s41307-018-0106-8 · Cited by 58 · see citing works
Migrating to GraphQL: A Practical Assessment
Brito, G., Mombach, T., & Valente, M. T. (2019) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8667986 · Cited by 58 · see citing works
“Since the query language has just two years, we focus on grey literature, instead of analysing scientific papers, as usually recommended for emerging technologies [6]–[8].”
BackgroundEpistemological pluralism in research synthesis methods
Suri, H. (2012) · International Journal of Qualitative Studies in Education · doi:10.1080/09518398.2012.691565 · Cited by 58 · see citing works
A literature review of tourism management (1990–2013): a content analysis perspective
Chang, W., & Katrichis, J. M. (2016) · Current Issues in Tourism · doi:10.1080/13683500.2016.1167177 · Cited by 53 · see citing works
Practices and Processes in BIM Projects: An Exploratory Case Study
Boton, C., & Forgues, D. (2018) · Advances in Civil Engineering · doi:10.1155/2018/7259659 · Cited by 53 · see citing works
“It is used as “a sound and sensible rst step” when extensive empirical research has not been yet dedicated to the topic of interest [60].”
BackgroundMethodTransformational Leadership: An Integrative Review of the Literature
Gasper, J. M. (1992) · ScholarWorks - WMU (Western Michigan University) · Cited by 53 · see citing works
Impact of Armed Conflicts on Education and Educational Agents: A Multivocal Review
Cervantes-Duarte, L., & Fernández-Cano, A. (2016) · Revista Electrónica Educare · doi:10.15359/ree.20-3.12 · Cited by 22 · see citing works
Leoto, R., & Lizarralde, G. (2019) · Building and Environment · doi:10.1016/j.buildenv.2019.03.041 · Cited by 51 · see citing works
Secondary School Curricula Issues: Impact on Postsecondary Students with Disabilities
Stodden, R. A., Galloway, L. M., & Stodden, N. J. (2003) · Exceptional Children · doi:10.1177/001440290307000101 · Cited by 51 · see citing works
“…project reports, and other documents deemed pertinent to the three areas of question were systematically analyzed applying multivocal literature (Ogawa & Malen, 1991) and participatory action research (PAR) methods (Brydon-Miller, 1997; White, Nary, & Froehlich, in press; Whyte, 1991),…”
MethodMeltzer, J., & Hamann, E. ‘. (2004) · Lincoln (University of Nebraska) · Cited by 50 · see citing works
Meltzer, J., & Hamann, E. ‘. (2005) · Lincoln (University of Nebraska) · Cited by 50 · see citing works
Practical relevance of software engineering research: synthesizing the community’s voice
Garousi, V., Borg, M., & Oivo, M. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09803-0 · Cited by 48 · see citing works
“…conducting MLRs in SE has been published (Garousi et al. 2019), which is based on the SLR guidelines proposed by Kitchenham and Charters (Kitchenham and Charters 2007b), and MLR guidelines in other disciplines, e.g., in medicine (Hopewell et al. 2007) and education sciences (Ogawa and Malen 1991).”
MethodPrates, L., & Pereira, R. (2025) · International Journal of Information Security · doi:10.1007/s10207-024-00914-z · Cited by 49 · see citing works
“Some of the challenges for researchers while selecting GL sources are accessibility, dis-coverability, bias, low quality, and lack of standardization [49] [41].”
Ahmed, R. A., Farooq, A. O., Storie, D., Hartling, L., & Oswald, A. (2015) · Medical Teacher · doi:10.3109/0142159x.2015.1112893 · Cited by 47 · see citing works
A Review on Tools, Mechanics, Benefits, and Challenges of Gamified Software Testing
Fulcini, T., Coppola, R., Ardito, L., & Torchiano, M. (2023) · ACM Computing Surveys · doi:10.1145/3582273 · Cited by 45 · see citing works
“This result suggests that the Grey Literature sources about gamiication in software testing are less technical than White Literature ones, and that no actual validation of the beneits of gamiication is carried out outside peer-reviewed academic research eforts.”
MethodInfluentialCapabilities and Practices in DevOps: A Multivocal Literature Review
Amaro, R., Pereira, R., & da Silva, M. M. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3166626 · Cited by 43 · see citing works
“…DevOps research, in the same area, using MLR already exist [22], [23], [24], thus corroborating the practical usefulness of this method for the proposed research, expanding the diversity of sources that are available in a variety of forms, reflecting different purposes and perspectives [53].”
BackgroundMethodSoftware Engineering for Internet of Things: The Practitioners' Perspective
Fahmideh, M., Ahmad, A., Behnaz, A., Grundy, J., & Susilo, W. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3070692 · Cited by 1 · see citing works
Towers, N. S., & Xu, K. (2015) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2015.09.003 · Cited by 37 · see citing works
“However in our research a trade-off was essential between the depth and completion of data and the number of cases to be studied, given the sampling method was non-random (Ogawa and Malen, 1991).”
Casprini, E., & Palumbo, R. (2022) · Global Public Policy and Governance · doi:10.1007/s43508-022-00056-9 · Cited by 36 · see citing works
“…digital transformation in healthcare and set the ground for the establishment of a service ecosystem, we adopted 1 3 an exploratory case study, which enabled us “…to inductively generate, rather than deductively confirm, insights regarding the phenomenon of interest” (Ogawa & Malen, 1991: p. 271).”
MethodWhat We Know about Software Test Maturity and Test Process Improvement
Garousi, V., Felderer, M., & Hacaloğlu, T. (2017) · IEEE Software · doi:10.1109/ms.2017.4541043 · Cited by 33 · see citing works
“5,6 MLRs related to software engineering have recently started appearing; one example is an MLR on technical debt.”
BackgroundWrapping Trust for Interoperability: A Preliminary Study of Wrapped Tokens
Caldarelli, G. (2021) · Information · doi:10.3390/info13010006 · Cited by 35 · see citing works
Innovation in tradition-based firms: dynamic knowledge for international competitiveness
Corte, V. D., Zamparelli, G., & Micera, R. (2013) · European Journal of Innovation Management · doi:10.1108/ejim-06-2012-0065 · Cited by 34 · see citing works
Simone, K., Ahmed, R. A., Konkin, J., Campbell, S., Hartling, L., & Oswald, A. (2018) · Medical Teacher · doi:10.1080/0142159x.2018.1473562 · Cited by 34 · see citing works
“Due to heterogeneity in intervention types, outcomes and study designs, we conducted a qualitative synthesis of the evidence using procedures such as those outlined by Ogawa and Malen (1991) for synthesizing multi-vocal bodies of literature.”
MethodWhat We Know About Smells in Software Test Code
Garousi, V., Kucuk, B., & Felderer, M. (2018) · IEEE Software · doi:10.1109/ms.2018.2875843 · Cited by 34 · see citing works
Procedural methodology for a grounded meta‐analysis of qualitative case studies
Stall‐Meadows, C., & Hyle, A. E. (2010) · International Journal of Consumer Studies · doi:10.1111/j.1470-6431.2010.00882.x · Cited by 34 · see citing works
Towards Utility in Reviews of Multivocal Literatures
Patton, M. Q. (1991) · Review of Educational Research · doi:10.3102/00346543061003287 · Cited by 33 · see citing works
“However, I would be inclined to use the broader methodological framework of qualitative methods (cf., Patton, 1990) rather than the narrower descriptor of "exploratory case study method" (Ogawa & Malen, 1991).”
MethodInfluentialBui, T. T. P., Wilkinson, S. J., MacGregor, C., & Domingo, N. (2023) · Building and Environment · doi:10.1016/j.buildenv.2023.110557 · Cited by 32 · see citing works
Wen, R., Zeng, D., Yang, Z., Jiang, L., Ma, M., Chen, B., & van Beek, T. A. (2018) · Journal of Agricultural and Food Chemistry · doi:10.1021/acs.jafc.8b02388 · Cited by 31 · see citing works
Multivocal literature reviews in software engineering: Preliminary findings from a tertiary study
Neto, G. T. G., Santos, W. B., Endo, P. T., & Fagundes, R. A. A. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870142 · Cited by 29 · see citing works
“[20], however this study did not offer any recommendations on how to conduct MLR studies.”
BackgroundAn evidence-based inquiry into the use of grey literature in software engineering
Zhang, H., Zhou, X., Huang, X., Huang, H., & Babar, M. A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380336 · Cited by 29 · see citing works
“Different from white literature (or academic publications), GL covers a wide range of non-conventional documents, for instance, technical reports, theses and official documents [47, 50, 61] that report recent research or practice results [24, 48].”
BackgroundReusing empirical knowledge during cloud computing adoption
Fahmideh, M., & Beydoun, G. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.12.011 · Cited by 27 · see citing works
A Financial Approach for Managing Interest in Technical Debt
Ampatzoglou, A., Ampatzoglou, A., Avgeriou, P., & Chatzigeorgiou, A. (2016) · Lecture notes in business information processing · doi:10.1007/978-3-319-40512-4_7 · Cited by 26 · see citing works
“However, the use of non-peer-reviewed reports or articles raises significant reliability and validity issues [21].”
BackgroundАнатольевич, С. А., & Smolander, K. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106510 · Cited by 27 · see citing works
DevOps Metrics and KPIs: A Multivocal Literature Review
Amaro, R., Pereira, R., & da Silva, M. M. (2024) · ACM Computing Surveys · doi:10.1145/3652508 · Cited by 37 · see citing works
Serious games for teaching agile methods: A review of multivocal literature
Rodríguez, G., Caino, P. C. G., & Resett, S. A. (2021) · Computer Applications in Engineering Education · doi:10.1002/cae.22430 · Cited by 24 · see citing works
“To meet general, minimum requirements, improving rigor in studies of this literature must address issues, such as (a) data exclusion, (b) the selective use of data in documents, and (c) the complexity of data collection and analysis procedures [50].”
BackgroundGunn, T. M., Pomahac, G. A., Striker, E. G., & Tailfeathers, J. (2010) · Journal of Educational Change · doi:10.1007/s10833-010-9148-4 · Cited by 23 · see citing works
“An exploratory approach ( Ogawa and Malen 1991 ) was employed for the thematic analysis.”
MethodThe maturity of open government data maturity: a multivocal literature review
Çaldağ, M. T., & Gökalp, E. (2022) · Aslib Journal of Information Management · doi:10.1108/ajim-11-2021-0354 · Cited by 22 · see citing works
Yin, R. K. (1991) · Review of Educational Research · doi:10.3102/00346543061003299 · Cited by 21 · see citing works
The pedagogical enhancement of open education: An examination of problem-based learning
Gurell, S., Kuo, Y.-C., & Walker, A. E. (2010) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v11i3.886 · Cited by 21 · see citing works
“Given the emergent nature of the work, and the combination of quantitative and qualitative studies included, a qualitative literature review (Ogawa & Malen, 1991) was conducted.”
MethodWhat is Research Rigor? Lessons for a Transdiscipline
Gill, G., & Gill, T. R. (2020) · Informing Science The International Journal of an Emerging Transdiscipline · doi:10.28945/4528 · Cited by 20 · see citing works
“Ivarsson & Gorschek (2011), Ogawa & Malen (1991), Poland (1995), Rand & Rust (2011), Stewart et al. (2017), Tushman et al.”
BackgroundInfluentialNavigating the DevOps landscape
Zhang, X., Zhao, P., & Jaskolka, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112331 · Cited by 21 · see citing works
Choice Architecture as a Strategy to Encourage Elegant Infrastructure Outcomes
Shealy, T., & Klotz, L. E. (2017) · Journal of Infrastructure Systems · doi:10.1061/(asce)is.1943-555x.0000311 · Cited by 18 · see citing works
Artificial Intelligence in MOBA Games: A Multivocal Literature Mapping
Costa, L. M., Drachen, A., Souza, F. C. M., & Xexeo, G. B. (2023) · IEEE Transactions on Games · doi:10.1109/tg.2023.3282157 · Cited by 19 · see citing works
“We consider publications that appear in various forms (e.g., academic literature, industry reports, tool vendors’ websites, and blogs), reflecting different purposes and perspectives [21].”
BackgroundTowards a Taxonomy of Industrial Challenges and Enabling Technologies in Industry 4.0
Figliè, R., Amadio, R., Tyrovolas, M., Stylios, C. D., Paśko, Ł., Stadnicka, D., Carreras-Coch, A., Zaballos, A., Navarro, J., & Mazzei, D. (2024) · IEEE Access · doi:10.1109/access.2024.3356349 · Cited by 19 · see citing works
“To ensure this holistic understanding, a Multivocal Literature Review (MLR) was conducted to analyze both state-of-the-art and state-of-practice in the industrial sector [42].”
MethodFreire, D. L., Frantz, R. Z., Frantz, F. C. R., & Sawicki, S. (2018) · Software Practice and Experience · doi:10.1002/spe.2670 · Cited by 19 · see citing works
“Ogawa and Malen 37 states that “multivocal literature is comprised of all accessible writings on a common, often contemporary topic.”
BackgroundChallenges for integrated design (ID) in sustainable buildings
Leoto, R., & Lizarralde, G. (2019) · Construction Management and Economics · doi:10.1080/01446193.2019.1569249 · Cited by 19 · see citing works
“This strategy is employed to inductively generate insights regarding the phenomenon to be studied (Ogawa and Malen 1991).”
MethodBevilacqua, C., Pizzimenti, P. F., & Ou, Y. (2023) · Sustainability · doi:10.3390/su151813346 · Cited by 18 · see citing works
Advances in auditing and business continuity: A study in financial companies
Brás, J. C., Pereira, R., Fonseca, M., Ribeiro, R., & Bianchi, I. S. (2024) · Journal of Open Innovation Technology Market and Complexity · doi:10.1016/j.joitmc.2024.100304 · Cited by 18 · see citing works
Kitano, M. K., & Lewis, R. B. (2007) · journal for the education of the gifted · doi:10.1177/016235320703000302 · Cited by 18 · see citing works
Frasier, M. M., Hunsaker, S. L., Lee, J.-J., Mitchell, S., Cramond, B. L., Krisel, S. C., García, J. H., Martı́n, D., Frank, E., & Finley, V. (1995) · Digital Commons - USU (Utah State University) · Cited by 18 · see citing works
Beyond Watching Over Established Ways: A Review as Recasting the Literature, Recasting the Lived
Livingston, G. (1999) · Review of Educational Research · doi:10.2307/1170642 · Cited by 18 · see citing works
Capabilities and metrics in DevOps: A design science study
Amaro, R., Pereira, R., & da Silva, M. M. (2023) · Information & Management · doi:10.1016/j.im.2023.103809 · Cited by 21 · see citing works
Jamil, A. H. A., & Fathi, M. S. (2019) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/513/1/012035 · Cited by 17 · see citing works
Strategy Process Research: What do we know, what should we know?
Lechner, C. F., & Müller‐Stewens, G. (2000) · Alexandria (UniSG) (University of St.Gallen) · Cited by 17 · see citing works
A Multivocal Literature Review on the use of DevOps for e-Learning systems
Sánchez‐Gordón, M., & Colomo‐Palacios, R. (2018) · Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM'21) · doi:10.1145/3284179.3284328 · Cited by 16 · see citing works
- Authors
- Ogawa, R. T., & Malen, B.
- Affiliations
- University of Utah (US)
- University of Washington (US)
- Venue
- Review of Educational Research
- Published
- 1991-09
- Volume
- 61
- Issue
- 3
- Pages
- 265-286
- ISSN
- 0034-6543 (ISSN-L)
- 1935-1046
- OpenAlex
- W2138571267
- Cited by
- 247
- References
- 46
- Citations by year
- 2026: 8 · 2025: 14 · 2024: 20 · 2023: 11 · 2022: 17 · 2021: 18 · 2020: 12 · 2019: 8 · 2018: 21 · 2017: 17 · 2016: 12 · 2015: 9 · 2014: 1 · 2013: 9 · 2012: 3
- FWCI
- 26.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Review of Educational Research (publishedVersion)
- Topics
- Social Sciences › Education › Educational Research and Methods
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- research rigor, school-based management, review procedures, educational research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
