Cited by 247Educational Research and MethodsMeta-analysis and systematic reviewsQualitative Research Methods and Applications

The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.

Towards Rigor in Reviews of Multivocal Literatures: Applying the Exploratory Case Study Method

Ogawa, Rodney T., Malen, Betty·1991·Review of Educational Research

DOI 10.3102/00346543061003265

Open this paper in CoChat — free account

Abstract

This article aims to stimulate discussion about the issue of rigor in conducting reviews of multivocal literatures. Multivocal literatures, which abound in the field of education, are comprised of all accessible writings on a common, often contemporary topic. The exploratory case study method is proposed as a means to engender rigor in reviews of such literatures. It is argued that it is appropriate to apply the concept of rigor to reviews of multivocal literatures and to use the exploratory case study method as a tool for thinking about procedures that could enhance rigor in such reviews. The article draws upon the authors’ experiences in conducting a review of the literature on school-based management to illustrate how the proposed procedures might be employed.

Integrity signals

Registry

Crossref · 10.3102/00346543061003265

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Untitled work
(1984) · Cited by 81,983
THE DISCOVERY OF GROUNDED THEORY: STRATEGIES FOR QUALITATIVE RESEARCH. By Barney G. Glaser and Anselm L. Strauss. Chicago: Aldine Publishing Company, 1967. 271 pp. $6.75
Wagner, H. R., Glaser, B. G., & Strauss, A. L. (1968) · Social Forces · doi:10.2307/2575405 · Cited by 46,525
The Discovery of Grounded Theory; Strategies for Qualitative Research
Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968) · Nursing Research · doi:10.1097/00006199-196807000-00014 · Cited by 39,449
The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields
DiMaggio, P. J., & Powell, W. W. (1983) · American Sociological Review · doi:10.2307/2095101 · Cited by 38,435
Basics of Qualitative Research: Grounded Theory Procedures and Techniques.
Ellis, C., Strauss, A., & Corbin, J. M. (1992) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2074814 · Cited by 29,296
The iron cage revisited institutional isomorphism and collective rationality in organizational fields
DiMaggio, P., & Powell, W. W. (2004) · Advances in strategic management · doi:10.1016/s0742-3322(00)17011-1 · Cited by 25,973
Untitled work
(1990) · Cited by 24,217
Qualitative evaluation and research methods
Ellis, D. J. (1990) · International Journal of Information Management · doi:10.1016/0268-4012(90)90041-p · Cited by 17,943
Basics of Qualitative Research: Grounded Theory Procedures and Techniques
Long, D. R., Strauss, A., & Corbin, J. M. (1993) · Modern Language Journal · doi:10.2307/328955 · Cited by 17,625
Untitled work
(1990) · Cited by 8,425

Show all 46 references

Research works printed but not linked: 15

Argyris C, 1982, Theory in practice
Berkhofer R, 1976, The difficulty of studying man in past time
Gottschalk L, 1969, Understanding history
Guba E, 1981, Effective evaluation
Johnson SM, 1990, Choice and control in American education, 2, 343
Kingdon J, 1984, Agendas, alternative, and public policies
Lindblom CE, 1979, Usable knowledge
Malen B, 1990, Choice and control in American education, 2, 289
10.2307/1174243 · DOI
Mosher E, 1972, A paper presented at the annual meeting of the American Educational Research Association
Murphy JT, 1980, Getting the facts
Patton MQ, 1980, Qualitative evaluation methods
Raywid MA, 1988, Paper presented at the meeting of the Center for Policy Research in Education
Schatzman L, 1973, Field research
Yin RK, 1974, Street-level governments: Assessing decentralization and urban services

Sorted by citations

Purposeful Sampling in Qualitative Research Synthesis
Suri, H. (2011) · Qualitative Research Journal · doi:10.3316/qrj1102063 · Cited by 2,284
“A rigorous research synthesis makes much more demand on time and resources when compared with ad hoc reviews (Elmore, 1991; Ogawa & Malen, 1991; Stock, Benito, and Lasa, 1996).”
Method
Conceptualizing Teacher Professional Learning
Opfer, V. D., & Pedder, D. (2011) · Review of Educational Research · doi:10.3102/0034654311413609 · Cited by 1,814
“To help develop and validate this understanding, we relied on “informant confirmation” (Boote & Beile, 2005; Ogawa & Malen, 1991).”
Method
Guidelines for including grey literature and conducting multivocal literature reviews in software engineering
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831
A Framework for Guiding and Evaluating Literature Reviews
Templier, M., & Paré, G. (2015) · Communications of the Association for Information Systems · doi:10.17705/1cais.03706 · Cited by 553
“According to Oxman and Guyatt (1988), “important aspects of the design and conduct of each primary study should be critiqued and the standard used in these critiques made explicit” (p.”
Influential
An exploration of technical debt
Tom, E., Aurum, A., & Vidgen, R. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2012.12.052 · Cited by 438
Advances in Learning and Behavioural Disabilities
Scruggs, T. E. (1999) · Cited by 378
Research synthesis in software engineering: A tertiary study
Cruzes, D. S., & Dybå, T. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.01.004 · Cited by 339
“As SRs evolve and continue to gain popularity, awareness is needed to ensure greater transparency and methodological rigor, which will increase the legitimacy of findings and relevance for practice [40].”
Background
What We Know about Effective Instructional Practices for English-Language Learners
Gersten, R. M., & Baker, S. (2000) · Exceptional Children · doi:10.1177/001440290006600402 · Cited by 322
Advancements in Research Synthesis Methods: From a Methodologically Inclusive Perspective
Suri, H., & Clarke, D. J. (2009) · Review of Educational Research · doi:10.3102/0034654308326349 · Cited by 346
“…(e.g., Engberg, 2004; Kasworm, 1990); statistical techniques (Cooper & Hedges, 1994a); interpretive techniques (Bair, 1999; Noblit & Hare, 1988; Ogawa & Malen, 1991b); historical methods (Donato & Lazerson, 2000); visual displays (Pope et al., 2007); “narrative juxtaposition” (Dixon-Woods,…”
BackgroundInfluential
The effectiveness of team-based learning on learning outcomes in health professions education: BEME Guide No. 30
Fatmi, M., Hartling, L., Hillier, T., Campbell, S., & Oswald, A. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.849802 · Cited by 306
“The data were synthesized with guidance from methods described by Ogawa and Malen (1991) who have developed a method for synthesis based on the exploratory case study method.”
Method
Learning in Communities of Inquiry: A Review of the Literature
Rourke, L., & Kanuka, H. (2009) · Cited by 324
The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature
Garousi, V., Felderer, M., & Mäntylä, M. V. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2916008 · Cited by 210
“MLRs have become popular in several other fields, e.g., education, e.g., [5, 6].”
BackgroundMethodInfluential
DevSecOps: A Multivocal Literature Review
Myrbakken, H., & Colomo-Palacios, R. (2017) · Communications in Computer and Information Science · doi:10.1007/978-3-319-67383-7_2 · Cited by 205
“By using this variety of literature the results will give a more nuanced look at the topic, since it includes the voices and opinions of academics, practitioners, independent researchers, development firms and others with experience [16].”
Background
Smells in software test code: A survey of knowledge in industry and academia
Garousi, V., & Küçük, B. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.12.013 · Cited by 189
The financial aspect of managing technical debt: A systematic literature review
Ampatzoglou, A., Ampatzoglou, A., Chatzigeorgiou, A., & Avgeriou, P. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.04.001 · Cited by 186
“Consequently, concerning technical debt, the multi-vocal results [35] are expected to provide indications on how the concept is exploited by both researchers and practitioners; although it goes without saying that using such not peer-reviewed experience reports or position articles raises significant reliability and v…”
Background
When and what to automate in software testing? A multi-vocal literature review
Garousi, V., & Mäntylä, M. V. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.04.015 · Cited by 182
“A 1991 paper [29] discussed rigor in MLRs and proposed a method based on exploratory case study to conduct MLRs rigorously.”
BackgroundMethod
The Blockchain Oracle Problem in Decentralized Finance—A Multivocal Approach
Caldarelli, G., & Ellul, J. (2021) · Applied Sciences · doi:10.3390/app11167572 · Cited by 181
Volunteer Tutoring Programs in Reading: A Review
Wasik, B. A. (1998) · Reading Research Quarterly · doi:10.1598/rrq.33.3.2 · Cited by 150
Digital Leadership ‒ Mountain or Molehill? A Literature Review
Eberl, J. K., & Drews, P. (2021) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-86800-0_17 · Cited by 134
“To avoid a lack of documentation and ensure reliability, an additional step from Ogawa & Malen’s [15] framework was added as first step resulting in the following steps for the literature review: (1) create an audit trail documenting the reviewers’ steps, (2) define the research scope, (3) search for literature, (4) s…”
Method
Reading Interventions for Students with Emotional/Behavioral Disorders
Coleman, M., & Vaughn, S. R. (2000) · Behavioral Disorders · doi:10.1177/019874290002500201 · Cited by 133
Insights into startup ecosystems through exploration of multi-vocal literature
Tripathi, N., Seppänen, P., Boominathan, G., Oivo, M., & Liukkunen, K. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.08.005 · Cited by 127
The Effectiveness of Reading Interventions for English Learners
Richards‐Tutor, C., Baker, D. L., Gersten, R. M., Baker, S. K., & Smith, J. M. (2015) · Exceptional Children · doi:10.1177/0014402915585483 · Cited by 123
“…few experimental studies and consequently expanded their synthesis to include descriptive and qualitative studies and also incorporated the professional opinions of practitioners with specific expertise in working effectively with English learners (i.e., Noblit & Hare, 1988; Ogawa & Malen, 1991).”
Background
An exploration of IoT platform development
Fahmideh, M., & Zowghi, D. (2019) · Information Systems · doi:10.1016/j.is.2019.06.005 · Cited by 119
Analyzing the concept of technical debt in the context of agile software development: A systematic literature review
Behutiye, W. N., Rodríguez, P., Oivo, M., & Tosun, A. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2016.10.004 · Cited by 116
“(multivocal literature reviews are comprised of all accessible writings, including non-academic writings on a topic [22].)”
Background
The Subject Matter Knowledge of Preservice Science Teachers
Cochran, K. F., & Jones, L. L. (1998) · Springer eBooks · doi:10.1007/978-94-011-4940-2_41 · Cited by 110
Characterizing DevOps by Hearing Multiple Voices
de França, B. B. N., Jeronimo, H., & Travassos, G. H. (2016) · doi:10.1145/2973839.2973845 · Cited by 109
“The diversity of sources appears in a variety of forms (e.g., academic literature, industry reports , tool vendors’ websites, and blogs), reflecting different purposes and perspectives [8].”
BackgroundMethodInfluential
Software engineering in start-up companies: An analysis of 88 experience reports
Klotins, E., Unterkalmsteiner, M., & Gorschek, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9620-y · Cited by 106
Software test maturity assessment and test process improvement: A multivocal literature review
Garousi, V., Felderer, M., & Hacaloğlu, T. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.01.001 · Cited by 105
An integrated framework to prioritize blockchain-based supply chain success factors
Shoaib, M., Lim, M., & Wang, C. (2020) · Industrial management & data systems · doi:10.1108/imds-04-2020-0194 · Cited by 84
Threat Intelligence Sharing Platforms: An Exploratory Study of Software Vendors and Research Perspectives
Sauerwein, C., Sillaber, C., Mussmann, A., & Breu, R. (2017) · Journal of the Association for Information Systems · Cited by 100
Do We Perform Systematic Literature Review Right? A Scientific Mapping and Methodological Assessment
Azarian, M., Yu, H., Shiferaw, A. T., & Stevik, T. K. (2023) · Logistics · doi:10.3390/logistics7040089 · Cited by 164
Maturity assessment and maturity models in health care: A multivocal literature review
Tarhan, A., Garousi, V., Türetken, O., Söylemez, M., & Garossi, S. (2020) · Digital Health · doi:10.1177/2055207620914772 · Cited by 11
Knowledge Generation in Educational Administration from the Inside Out: The Promise and Perils of Site-Based, Administrator Research
Anderson, G. L., & Jones, F. B. (2000) · Educational Administration Quarterly · doi:10.1177/00131610021969056 · Cited by 92
Global women political leaders: An invisible history, an increasingly important future
Adler, N. J. (1996) · The Leadership Quarterly · doi:10.1016/s1048-9843(96)90038-7 · Cited by 91
A Systematic Investigation of the Integration of Machine Learning into Supply Chain Risk Management
Schroeder, M., & Lodemann, S. (2021) · Logistics · doi:10.3390/logistics5030062 · Cited by 86
“] are comprised of all accessible writings on a common, often contemporary topic” [43].”
BackgroundMethod
The changing leadership role of the principal: Implications for principal assessment
Glasman, N. S., & Heck, R. H. (1990) · Peabody Journal of Education · doi:10.1080/01619569209538708 · Cited by 82
Rethinking the Politics of Fit and Educational Leadership
Tooms, A. K., Lugg, C. A., & Bogotch, I. E. (2009) · Educational Administration Quarterly · doi:10.1177/1094670509353044 · Cited by 73
Trend spotting: fashions in medical education
Campbell, J. K., & Johnson, C. A. (1999) · BMJ · doi:10.1136/bmj.318.7193.1272 · Cited by 72
Relationship of DevOps to Agile, Lean and Continuous Deployment
Lwakatare, L. E., Kuvaja, P., & Oivo, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49094-6_27 · Cited by 71
Synthesizing evidence in software engineering research
Cruzes, D. S., & Dybå, T. (2010) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1852786.1852788 · Cited by 70
“As SRs evolve and continue to gain popularity, awareness is needed to ensure greater transparency and methodological rigor that will increase the legitimacy of findings and relevance for practice [20].”
Background
Global Trends and Research Aims for English Academic Oral Presentations
Barrett, N. E., & Liu, G. (2016) · Review of Educational Research · doi:10.3102/0034654316628296 · Cited by 69
“We adapted the methodology of Ogawa and Malen (1991) and followed the guidelines set down by Randolph: (a) create an audit trail of each step in the review, (b) define the focus of the review to explain what to include and what to omit, (c) search for relevant literature, (d) classify the…”
Method
Opportunities and risks involved in using ChatGPT to create first grade science lesson plans
Powell, W. A., & Courchesne, S. (2024) · PLoS ONE · doi:10.1371/journal.pone.0305337 · Cited by 102
The Politics of Advocacy in Education
Scott, J. T., Lubienski, C., & DeBray, E. H. (2008) · Educational Policy · doi:10.1177/0895904808328530 · Cited by 67
“Finally, Gary Anderson provides a polyvocal (Ogawa & Malen, 1985) overview of the context of military presence in American schooling.”
Background
Responsible slum tourism: Egyptian experience
Mekawy, M. A. (2012) · Annals of Tourism Research · doi:10.1016/j.annals.2012.07.006 · Cited by 67
Special Purpose Entities in Megaprojects: Empty Boxes or Real Companies?
Sainati, T., Brookes, N. J., & Locatelli, G. (2017) · Project Management Journal · doi:10.1177/875697281704800205 · Cited by 63
“58 April/May 2017 ■ Project Management Journal P A P E R S and project management domains, the research adopts the review criteria introduced by Ogawa and Malen (1991) and Randolph (2009).”
Method
Towards a Definition of Environmental Sustainability Evaluation in Higher Education
Alba, D., del Álamo, J. B., & Pérez, J. G. (2018) · Higher Education Policy · doi:10.1057/s41307-018-0106-8 · Cited by 58
Migrating to GraphQL: A Practical Assessment
Brito, G., Mombach, T., & Valente, M. T. (2019) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8667986 · Cited by 58
“Since the query language has just two years, we focus on grey literature, instead of analysing scientific papers, as usually recommended for emerging technologies [6]–[8].”
Background
Epistemological pluralism in research synthesis methods
Suri, H. (2012) · International Journal of Qualitative Studies in Education · doi:10.1080/09518398.2012.691565 · Cited by 68
A literature review of tourism management (1990–2013): a content analysis perspective
Chang, W., & Katrichis, J. M. (2016) · Current Issues in Tourism · doi:10.1080/13683500.2016.1167177 · Cited by 53
Practices and Processes in BIM Projects: An Exploratory Case Study
Boton, C., & Forgues, D. (2018) · Advances in Civil Engineering · doi:10.1155/2018/7259659 · Cited by 60
“It is used as “a sound and sensible rst step” when extensive empirical research has not been yet dedicated to the topic of interest [60].”
BackgroundMethod
Transformational Leadership: An Integrative Review of the Literature
Gasper, J. M. (1992) · ScholarWorks - WMU (Western Michigan University) · Cited by 53
Impact of Armed Conflicts on Education and Educational Agents: A Multivocal Review
Cervantes-Duarte, L., & Fernández-Cano, A. (2016) · Revista Electrónica Educare · doi:10.15359/ree.20-3.12 · Cited by 22
Challenges in evaluating strategies for reducing a building's environmental impact through Integrated Design
Leoto, R., & Lizarralde, G. (2019) · Building and Environment · doi:10.1016/j.buildenv.2019.03.041 · Cited by 51
Secondary School Curricula Issues: Impact on Postsecondary Students with Disabilities
Stodden, R. A., Galloway, L. M., & Stodden, N. J. (2003) · Exceptional Children · doi:10.1177/001440290307000101 · Cited by 51
“…project reports, and other documents deemed pertinent to the three areas of question were systematically analyzed applying multivocal literature (Ogawa & Malen, 1991) and participatory action research (PAR) methods (Brydon-Miller, 1997; White, Nary, & Froehlich, in press; Whyte, 1991),…”
Method
Meeting the Literacy Development Needs of Adolescent English Language Learners through Content Area Learning. Part One: Focus on Motivation and Engagement.
Meltzer, J., & Hamann, E. ‘. (2004) · Lincoln (University of Nebraska) · Cited by 50
Meeting the Literacy Development Needs of Adolescent English Language Learners Through Content-Area Learning - PART TWO: Focus on Classroom Teaching and Learning Strategies
Meltzer, J., & Hamann, E. ‘. (2005) · Lincoln (University of Nebraska) · Cited by 50
Practical relevance of software engineering research: synthesizing the community’s voice
Garousi, V., Borg, M., & Oivo, M. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09803-0 · Cited by 48
“…conducting MLRs in SE has been published (Garousi et al. 2019), which is based on the SLR guidelines proposed by Kitchenham and Charters (Kitchenham and Charters 2007b), and MLR guidelines in other disciplines, e.g., in medicine (Hopewell et al. 2007) and education sciences (Ogawa and Malen 1991).”
Method
DevSecOps practices and tools
Prates, L., & Pereira, R. (2025) · International Journal of Information Security · doi:10.1007/s10207-024-00914-z · Cited by 49
“Some of the challenges for researchers while selecting GL sources are accessibility, dis-coverability, bias, low quality, and lack of standardization [49] [41].”
Building capacity for education research among clinical educators in the health professions: A BEME (Best Evidence Medical Education) Systematic Review of the outcomes of interventions: BEME Guide No. 34
Ahmed, R. A., Farooq, A. O., Storie, D., Hartling, L., & Oswald, A. (2015) · Medical Teacher · doi:10.3109/0142159x.2015.1112893 · Cited by 47
A Review on Tools, Mechanics, Benefits, and Challenges of Gamified Software Testing
Fulcini, T., Coppola, R., Ardito, L., & Torchiano, M. (2023) · ACM Computing Surveys · doi:10.1145/3582273 · Cited by 50
“This result suggests that the Grey Literature sources about gamiication in software testing are less technical than White Literature ones, and that no actual validation of the beneits of gamiication is carried out outside peer-reviewed academic research eforts.”
MethodInfluential
Capabilities and Practices in DevOps: A Multivocal Literature Review
Amaro, R., Pereira, R., & da Silva, M. M. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3166626 · Cited by 43
“…DevOps research, in the same area, using MLR already exist [22], [23], [24], thus corroborating the practical usefulness of this method for the proposed research, expanding the diversity of sources that are available in a variety of forms, reﬂecting different purposes and perspectives [53].”
BackgroundMethod
Software Engineering for Internet of Things: The Practitioners' Perspective
Fahmideh, M., Ahmad, A., Behnaz, A., Grundy, J., & Susilo, W. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3070692 · Cited by 30
The influence of guanxi on physical distribution service quality availability in e-commerce sourcing fashion garments from China
Towers, N., & Xu, K. (2015) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2015.09.003 · Cited by 39
“However in our research a trade-off was essential between the depth and completion of data and the number of cases to be studied, given the sampling method was non-random (Ogawa and Malen, 1991).”
Reaping the benefits of digital transformation through Public-Private Partnership: A service ecosystem view applied to healthcare
Casprini, E., & Palumbo, R. (2022) · Global Public Policy and Governance · doi:10.1007/s43508-022-00056-9 · Cited by 36
“…digital transformation in healthcare and set the ground for the establishment of a service ecosystem, we adopted 1 3 an exploratory case study, which enabled us “…to inductively generate, rather than deductively confirm, insights regarding the phenomenon of interest” (Ogawa & Malen, 1991: p. 271).”
Method
What We Know about Software Test Maturity and Test Process Improvement
Garousi, V., Felderer, M., & Hacaloğlu, T. (2017) · IEEE Software · doi:10.1109/ms.2017.4541043 · Cited by 33
“5,6 MLRs related to software engineering have recently started appearing; one example is an MLR on technical debt.”
Background
Wrapping Trust for Interoperability: A Preliminary Study of Wrapped Tokens
Caldarelli, G. (2021) · Information · doi:10.3390/info13010006 · Cited by 35
Innovation in tradition-based firms: dynamic knowledge for international competitiveness
Corte, V. D., Zamparelli, G., & Micera, R. (2013) · European Journal of Innovation Management · doi:10.1108/ejim-06-2012-0065 · Cited by 34
What are the features of targeted or system-wide initiatives that affect diversity in health professions trainees? A BEME systematic review: BEME Guide No. 50
Simone, K., Ahmed, R. A., Konkin, J., Campbell, S., Hartling, L., & Oswald, A. (2018) · Medical Teacher · doi:10.1080/0142159x.2018.1473562 · Cited by 34
“Due to heterogeneity in intervention types, outcomes and study designs, we conducted a qualitative synthesis of the evidence using procedures such as those outlined by Ogawa and Malen (1991) for synthesizing multi-vocal bodies of literature.”
Method
What We Know About Smells in Software Test Code
Garousi, V., Kucuk, B., & Felderer, M. (2018) · IEEE Software · doi:10.1109/ms.2018.2875843 · Cited by 34
Procedural methodology for a grounded meta‐analysis of qualitative case studies
Stall‐Meadows, C., & Hyle, A. E. (2010) · International Journal of Consumer Studies · doi:10.1111/j.1470-6431.2010.00882.x · Cited by 34
Towards Utility in Reviews of Multivocal Literatures
Patton, M. Q. (1991) · Review of Educational Research · doi:10.3102/00346543061003287 · Cited by 33
“However, I would be inclined to use the broader methodological framework of qualitative methods (cf., Patton, 1990) rather than the narrower descriptor of "exploratory case study method" (Ogawa & Malen, 1991).”
MethodInfluential
Decision making in reducing carbon emissions for building refurbishment: Case studies of university buildings in New Zealand
Bui, T. T. P., Wilkinson, S. J., MacGregor, C., & Domingo, N. (2023) · Building and Environment · doi:10.1016/j.buildenv.2023.110557 · Cited by 32
Rapid Analysis of Illegal Cationic Dyes in Foods and Surface Waters Using High Temperature Direct Analysis in Real Time High-Resolution Mass Spectrometry
Wen, R., Zeng, D., Yang, Z., Jiang, L., Ma, M., Chen, B., & van Beek, T. A. (2018) · Journal of Agricultural and Food Chemistry · doi:10.1021/acs.jafc.8b02388 · Cited by 31
Multivocal literature reviews in software engineering: Preliminary findings from a tertiary study
Neto, G. T. G., Santos, W. B., Endo, P. T., & Fagundes, R. A. A. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870142 · Cited by 29
“[20], however this study did not offer any recommendations on how to conduct MLR studies.”
Background
An evidence-based inquiry into the use of grey literature in software engineering
Zhang, H., Zhou, X., Huang, X., Huang, H., & Babar, M. A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380336 · Cited by 30
“Different from white literature (or academic publications), GL covers a wide range of non-conventional documents, for instance, technical reports, theses and official documents [47, 50, 61] that report recent research or practice results [24, 48].”
Background
Reusing empirical knowledge during cloud computing adoption
Fahmideh, M., & Beydoun, G. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.12.011 · Cited by 27
A Financial Approach for Managing Interest in Technical Debt
Ampatzoglou, A., Ampatzoglou, A., Avgeriou, P., & Chatzigeorgiou, A. (2016) · Lecture notes in business information processing · doi:10.1007/978-3-319-40512-4_7 · Cited by 26
“However, the use of non-peer-reviewed reports or articles raises significant reliability and validity issues [21].”
Background
Bridging the state-of-the-art and the state-of-the-practice of SaaS pricing: A multivocal literature review
Анатольевич, С. А., & Smolander, K. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106510 · Cited by 27
DevOps Metrics and KPIs: A Multivocal Literature Review
Amaro, R., Pereira, R., & da Silva, M. M. (2024) · ACM Computing Surveys · doi:10.1145/3652508 · Cited by 37
Serious games for teaching agile methods: A review of multivocal literature
Rodríguez, G., Caino, P. C. G., & Resett, S. A. (2021) · Computer Applications in Engineering Education · doi:10.1002/cae.22430 · Cited by 24
“To meet general, minimum requirements, improving rigor in studies of this literature must address issues, such as (a) data exclusion, (b) the selective use of data in documents, and (c) the complexity of data collection and analysis procedures [50].”
Background
First Nations, Métis, and Inuit education: The Alberta initiative for school improvement approach to improve indigenous education in Alberta
Gunn, T. M., Pomahac, G. A., Striker, E. G., & Tailfeathers, J. (2010) · Journal of Educational Change · doi:10.1007/s10833-010-9148-4 · Cited by 23
“An exploratory approach ( Ogawa and Malen 1991 ) was employed for the thematic analysis.”
Method
The maturity of open government data maturity: a multivocal literature review
Çaldağ, M. T., & Gökalp, E. (2022) · Aslib Journal of Information Management · doi:10.1108/ajim-11-2021-0354 · Cited by 22
Advancing Rigorous Methodologies: A Review of "Towards Rigor in Reviews of Multivocal Literatures... "
Yin, R. K. (1991) · Review of Educational Research · doi:10.3102/00346543061003299 · Cited by 21
The pedagogical enhancement of open education: An examination of problem-based learning
Gurell, S., Kuo, Y.-C., & Walker, A. E. (2010) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v11i3.886 · Cited by 21
“Given the emergent nature of the work, and the combination of quantitative and qualitative studies included, a qualitative literature review (Ogawa & Malen, 1991) was conducted.”
Method
What is Research Rigor? Lessons for a Transdiscipline
Gill, G., & Gill, T. R. (2020) · Informing Science The International Journal of an Emerging Transdiscipline · doi:10.28945/4528 · Cited by 20
“Ivarsson & Gorschek (2011), Ogawa & Malen (1991), Poland (1995), Rand & Rust (2011), Stewart et al. (2017), Tushman et al.”
BackgroundInfluential
Navigating the DevOps landscape
Zhang, X., Zhao, P., & Jaskolka, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112331 · Cited by 21
Choice Architecture as a Strategy to Encourage Elegant Infrastructure Outcomes
Shealy, T., & Klotz, L. E. (2016) · Journal of Infrastructure Systems · doi:10.1061/(asce)is.1943-555x.0000311 · Cited by 26
Artificial Intelligence in MOBA Games: A Multivocal Literature Mapping
Costa, L. M., Drachen, A., Souza, F. C. M., & Xexeo, G. B. (2023) · IEEE Transactions on Games · doi:10.1109/tg.2023.3282157 · Cited by 19
“We consider publications that appear in various forms (e.g., academic literature, industry reports, tool vendors’ websites, and blogs), reﬂecting different purposes and perspectives [21].”
Background
Towards a Taxonomy of Industrial Challenges and Enabling Technologies in Industry 4.0
Figliè, R., Amadio, R., Tyrovolas, M., Stylios, C. D., Paśko, Ł., Stadnicka, D., Carreras-Coch, A., Zaballos, A., Navarro, J., & Mazzei, D. (2024) · IEEE Access · doi:10.1109/access.2024.3356349 · Cited by 19
“To ensure this holistic understanding, a Multivocal Literature Review (MLR) was conducted to analyze both state-of-the-art and state-of-practice in the industrial sector [42].”
Method
Survey on the run‐time systems of enterprise application integration platforms focusing on performance
Freire, D. L., Frantz, R. Z., Frantz, F. C. R., & Sawicki, S. (2018) · Software Practice and Experience · doi:10.1002/spe.2670 · Cited by 19
“Ogawa and Malen 37 states that “multivocal literature is comprised of all accessible writings on a common, often contemporary topic.”
Background
Challenges for integrated design (ID) in sustainable buildings
Leoto, R., & Lizarralde, G. (2019) · Construction Management and Economics · doi:10.1080/01446193.2019.1569249 · Cited by 19
“This strategy is employed to inductively generate insights regarding the phenomenon to be studied (Ogawa and Malen 1991).”
Method
Cities in Transition and Urban Innovation Ecosystems: Place and Innovation Dynamics in the Case of Boston and Cambridge (USA)
Bevilacqua, C., Pizzimenti, P. F., & Ou, Y. (2023) · Sustainability · doi:10.3390/su151813346 · Cited by 18
Advances in auditing and business continuity: A study in financial companies
Brás, J. C., Pereira, R., Fonseca, M., Ribeiro, R., & Bianchi, I. S. (2024) · Journal of Open Innovation Technology Market and Complexity · doi:10.1016/j.joitmc.2024.100304 · Cited by 18
Examining the Relationships between Reading Achievement and Tutoring Duration and Content for Gifted Culturally and Linguistically Diverse Students from Low-Income Backgrounds
Kitano, M. K., & Lewis, R. B. (2007) · journal for the education of the gifted · doi:10.1177/016235320703000302 · Cited by 18
Core Attributes of Giftedness: A Foundation for Recognizing the Gifted Potential of Minority and Economically Disadvantaged Students
Frasier, M. M., Hunsaker, S. L., Lee, J.-J., Mitchell, S., Cramond, B. L., Krisel, S. C., García, J. H., Martı́n, D., Frank, E., & Finley, V. (1995) · Digital Commons - USU (Utah State University) · Cited by 18
Beyond Watching Over Established Ways: A Review as Recasting the Literature, Recasting the Lived
Livingston, G. (1999) · Review of Educational Research · doi:10.2307/1170642 · Cited by 18
Capabilities and metrics in DevOps: A design science study
Amaro, R., Pereira, R., & da Silva, M. M. (2023) · Information & Management · doi:10.1016/j.im.2023.103809 · Cited by 21
Contractual issues for Building Information Modelling (BIM)-based construction projects: An exploratory case study
Jamil, A. H. A., & Fathi, M. S. (2019) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/513/1/012035 · Cited by 17
Strategy Process Research: What do we know, what should we know?
Lechner, C. F., & Müller‐Stewens, G. (2000) · Alexandria (UniSG) (University of St.Gallen) · Cited by 17
A Multivocal Literature Review on the use of DevOps for e-Learning systems
Sánchez‐Gordón, M., & Colomo‐Palacios, R. (2018) · Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM'21) · doi:10.1145/3284179.3284328 · Cited by 16

Authors: Ogawa, R. T., & Malen, B.
Affiliations: University of Utah (US)
University of Washington (US)
Venue: Review of Educational Research
Publisher: American Educational Research Association (AERA)
Published: 1991-09
Volume: 61
Issue: 3
Pages: 265-286
ISSN: 0034-6543 (ISSN-L)
1935-1046
DOI: 10.3102/00346543061003265
OpenAlex: W2138571267
Cited by: 247
References: 46
Citations by year: 2026: 8 · 2025: 14 · 2024: 20 · 2023: 11 · 2022: 17 · 2021: 18 · 2020: 12 · 2019: 8 · 2018: 21 · 2017: 17 · 2016: 12 · 2015: 9 · 2014: 1 · 2013: 9 · 2012: 3
FWCI: 26.09
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Review of Educational Research (publishedVersion)
Topics: Social Sciences › Education › Educational Research and Methods
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Education › Qualitative Research Methods and Applications
Keywords: research rigor, school-based management, review procedures, educational research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Towards Rigor in Reviews of Multivocal Literatures: Applying the Exploratory Case Study Method

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks