Reviewing Reviews of Research in Educational Leadership: An Empirical Assessment
The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Abstract
Purpose: Reviews of research play a critical but underappreciated role in knowledge production and accumulation. Yet, until relatively recently, limited attention has been given to the “methodology” of conducting reviews of research. This observation also applies in educational leadership and management where reviews of research have charted intellectual progress since the field’s inception in the 1950s and 1960s. This paper was framed as a “methodological review of reviews of research” in educational leadership and management. Method: The author analyzed 38 reviews of research in educational leadership published in nine international refereed journals over the past 52 years. The author applies a conceptual framework and analytical rubric for conducting systematic reviews of research in analyzing this sample of research reviews. Data analysis focuses on describing methodological characteristics, illuminating patterns of strength and weakness in review methods, and identifying a set of exemplary reviews. Findings: Despite publication of a number of “exemplary reviews,” there remains considerable room for improvement in the methodology of conducting systematic reviews of research in educational leadership and management. The study identified a tendency for research reviews in this field to omit key information concerning the rationale and nature of the studies included in the reviews, methods of data collection, extraction, evaluation and analysis, and how these choices impacted interpretation of the findings. Implications: This comprehensive set of 38 published review articles tracks the historical development of the field and, by itself, represents a rich harvest from the study. Within this historical corpus of reviews, the study identified a subset of “exemplary reviews” that can serve as useful models for future scholarship. Finally, by identifying patterns of methodological strength and weakness among the reviews as a group, the report offers empirically grounded recommendations for strengthening future reviews of research in educational leadership and management.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312
(2000) · Cited by 9,198
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
International Journal of Educational Management
(2023) · International Journal of Educational Management · doi:10.1108/ijem · Cited by 2,899
Robinson, V. M. J., Lloyd, C. A., & Rowe, K. J. (2008) · Educational Administration Quarterly · doi:10.1177/0013161x08321509 · Cited by 2,549 · cited 7×
“This category of research review tends to attract the attention of practitioners and policy makers, as well as scholars, resulting in higher citation rates (e.g., see reviews by Bossert et al., 1982; Hallinger & Heck, 1998; Leithwood et al., 2008; Robinson et al., 2008).”
BackgroundInfluentialSeven strong claims about successful school leadership
Leithwood, K., Harris, A., & Hopkins, D. (2008) · School Leadership & Management · doi:10.1080/13632430701800060 · Cited by 2,449 · cited 5×
“This category of research review tends to attract the attention of practitioners and policy makers, as well as scholars, resulting in higher citation rates (e.g., see reviews by Bossert et al., 1982; Hallinger & Heck, 1998; Leithwood et al., 2008; Robinson et al., 2008).”
BackgroundMethodInfluentialMethods for the synthesis of qualitative research: a critical review
Barnett-Page, E., & Thomas, J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-59 · Cited by 2,042
Handbook for Synthesizing Qualitative Research
Sandelowski, M., & Barroso, J. V. (2006) · Medical Entomology and Zoology · Cited by 1,938
Reassessing the Principal's Role in School Effectiveness: A Review of Empirical Research, 1980-1995
Hallinger, P., & Heck, R. H. (1996) · Educational Administration Quarterly · doi:10.1177/0013161x96032001002 · Cited by 1,751
Self-citationExploring the Principal's Contribution to School Effectiveness: 1980‐1995∗
Hallinger, P., & Heck, R. H. (1998) · School Effectiveness and School Improvement · doi:10.1080/0924345980090203 · Cited by 1,569
Self-citationThe review of educational research.
(1931) · Journal of Educational Psychology · doi:10.1037/h0066448 · Cited by 1,562
Summing up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
Leadership for learning: lessons from 40 years of empirical research
Hallinger, P. (2011) · Journal of Educational Administration · doi:10.1108/09578231111116699 · Cited by 1,091
Self-citationInstructional Leadership and the School Principal: A Passing Fancy that Refuses to Fade Away
Hallinger, P. (2005) · Leadership and Policy in Schools · doi:10.1080/15700760500244793 · Cited by 1,016 · cited 3×
“Hallinger and Heck (1996, 1998) applied a framework comprising competing conceptual models for organizing studies of school leadership effects. Riehl’s (2000) review employed a lens from critical theory in the analysis of leadership for student diversity and inclusive education.”
BackgroundMethodSelf-citationHaussler, S. C., & Cloutterbuck, J. (2008) · The Journal of Continuing Education in Nursing · doi:10.3928/00220124-20080101-07 · Cited by 940
Educative assessment: designing assessments to inform and improve student performance
(1999) · Choice Reviews Online · doi:10.5860/choice.36-2887 · Cited by 931 · cited 1×
“” Analytical rubrics enable higher levels of interrater agreement by providing explicit conditional statements for different levels of criterion attainment (Wiggins, 1998).”
BackgroundThe Instructional Management Role of the Principal
Bossert, S. T., Dwyer, D. C., Rowan, B., & Lee, G. V. (1982) · Educational Administration Quarterly · doi:10.1177/0013161x82018003004 · Cited by 899 · cited 3×
“Reviews conducted by Bridges (1982), Bossert et al.”
BackgroundMethodHow can systematic reviews incorporate qualitative research? A critical perspective
Dixon-Woods, M., Bonas, S., Booth, A., Jones, D. R., Miller, T., Sutton, A. J., Shaw, R. L., Smith, J. A., & Young, B. (2006) · Qualitative Research · doi:10.1177/1468794106058867 · Cited by 860 · cited 3×
“The procedures used to synthesize findings from both qualitative (Barnett-Page & Thomas, 2009; Dixon-Woods et al., 2006; Lorenc et al., 2012; Paterson, Thorne, Canam, & Jillings, 2001; Sandelowski & Barroso, 2007; Thomas & Harden, 2008; Weed, 2005) and quantitative studies (Lipsey & Wilson, 2001; Lucas et al.”
MethodWeight of Evidence: a framework for the appraisal of the quality and relevance of evidence
Gough, D. (2007) · Research Papers in Education · doi:10.1080/02671520701296189 · Cited by 830 · cited 10×
“Thus, I agree with others (e.g., Cooper & Hedges, 2009; Gough, 2007; Jackson, 1980; Light & Pillemer, 1984; Lorenc et al., 2012; Lucas et al., 2012; Valentine et al., 2010) who assert that even reviews which rely primarily on the synthesis of ideas benefit from being more systematic and explicit in the execution and r…”
BackgroundMethodInfluentialEducational Leadership and Student Achievement: The Elusive Search for an Association
Witziers, B., Bosker, R. J., & Krüger, M. L. (2003) · Educational Administration Quarterly · doi:10.1177/0013161x03253411 · Cited by 809 · cited 5×
“Less than 25% of the 38 articles (i.e., Bridges, 1982; Eagly et al., 1992; Leithwood & Jantzi, 2005; Leithwood & Montgomery, 1982; Leithwood & Sun, 2012; Murphy, 2004, 2008; Robinson et al., 2008; Witziers et al., 2003) explicitly referenced published sources on the methodology of conducting reviews of research.”
BackgroundInfluentialA Review of Transformational School Leadership Research 1996–2005
Leithwood, K., & Jantzi, D. (2005) · Leadership and Policy in Schools · doi:10.1080/15700760500244769 · Cited by 776
Distributed leadership: according to the evidence
Harris, A. (2008) · Journal of Educational Administration · doi:10.1108/09578230810863253 · Cited by 671 · cited 3×
“Leithwood, Begley, and Cousins (1990) employed a framework focusing on the nature, causes and consequences of principal leadership.”
BackgroundResultMeta-Study of Qualitative Health Research
Paterson, B., Thorne, S., Canam, C., & Jillings, C. (2001) · doi:10.4135/9781412985017 · Cited by 664
Riehl, C. J. (2000) · Review of Educational Research · doi:10.3102/00346543070001055 · Cited by 648 · cited 4×
“Judging by their citation impact (see Table 1), several exploratory reviews have had a substantial and lasting impact on the field (e.g., Adkinson, 1981; Bossert et al.,1979; Bridges, 1982; Leithwood & Jantzi, 2005; Leithwood & Montgomery, 1982; Riehl, 2000; Southworth, 2002).”
BackgroundInfluentialLeithwood, K., & Sun, J. (2012) · Educational Administration Quarterly · doi:10.1177/0013161x11436268 · Cited by 601
Lucas, P. J., Baird, J., Arai, L., Law, C., & Roberts, H. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-4 · Cited by 553
Instructional Leadership in Schools: Reflections and empirical evidence
Southworth, G. (2002) · School Leadership & Management · doi:10.1080/13632430220143042 · Cited by 498 · cited 5×
“Thus, several widely cited reviews that did not attain high scores on the rubric (e.g., Bossert et al., 1982; Harris, 2008; Heck & Hallinger, 2005; Leithwood et al., 2008; Riehl, 2000; Southworth, 2002) could perhaps have made even stronger contributions to knowledge had they incorporated more of the structural elemen…”
BackgroundInfluentialMethods for Integrative Reviews
Jackson, G. B. (1980) · Review of Educational Research · doi:10.2307/1170440 · Cited by 484
Meta-study of Qualitative Health Research: A Practical Guide to Meta-analysis and Meta-synthesis
(2001) · International Journal of Health Care Quality Assurance · doi:10.1108/ijhcqa.2001.06214fae.007 · Cited by 455 · cited 1×
“, Murphy, 2008) took advantage of recently developed tools for synthesizing nonquantitative data (see Barnett-Page & Thomas, 2009; Dixon-Woods et al., 2006; Light & Pillemer, 1984; Lorenc et al., 2012; Paterson et al., 2001; Sandelowski & Barroso, 2007; Thomas & Harden, 2008; Weed, 2005).”
The Role of the Elementary School Principal in Program Improvement
Leithwood, K. A., & Montgomery, D. J. (1982) · Review of Educational Research · doi:10.2307/1170421 · Cited by 398
Hallinger, P. (2011) · Educational Administration Quarterly · doi:10.1177/0013161x10383412 · Cited by 337 · cited 4×
“One of the reviews focused exclusively on methodological features of the literature (Hallinger, 2011a).”
BackgroundInfluentialSelf-citationA conceptual framework for systematic reviews of research in educational leadership and management
Hallinger, P. (2013) · Journal of Educational Administration · doi:10.1108/09578231311304670 · Cited by 332
Self-citationClarke, J. (2011) · Evidence Based Nursing · doi:10.1136/ebn.2011.0049 · Cited by 281
The Study of Educational Leadership and Management: Where Does the Field Stand Today?
Heck, R. H., & Hallinger, P. (2005) · Educational Management Administration & Leadership · doi:10.1177/1741143205051055 · Cited by 278
Self-citationResearch on the School Administrator: The State of the Art, 1967-19801
Bridges, E. M. (1982) · Educational Administration Quarterly · doi:10.1177/0013161x82018003003 · Cited by 249
Leadership effects on student achievement and sustained school success
Jacobson, S. (2011) · International Journal of Educational Management · doi:10.1108/09513541111100107 · Cited by 244 · cited 1×
“More specifically, only 3 of the 24 studies that relied exclusively on critical synthesis (Leithwood et al., 1990; Murphy et al., 2007; Walker et al., 2012) were explicit in clarifying data collection and analysis procedures. Murphy’s (2008) description of procedures for data extraction and transformation stands out a…”
BackgroundSchool leadership in the context of accountability policies
Leithwood, K. (2001) · International Journal of Leadership in Education · doi:10.1080/13603120110057082 · Cited by 218 · cited 2×
“However, the author’s review methods, as presented in the article, appeared ad hoc (e.g., Harris, 2008; Harvey, 1994; Heck & Hallinger, 2005; Leithwood, 2001; Leithwood et al., 2008).”
BackgroundMethodThe Nature, Causes and Consequences of Principals′ Practices: An Agenda for Future Research
Leithwood, K. A., Begley, P. T., & Bradley Cousins, J. (1990) · Journal of Educational Administration · doi:10.1108/09578239010001014 · Cited by 198 · cited 2×
“More specifically, only 3 of the 24 studies that relied exclusively on critical synthesis (Leithwood et al., 1990; Murphy et al., 2007; Walker et al., 2012) were explicit in clarifying data collection and analysis procedures.”
Background"Meta Interpretation": A Method for the Interpretive Synthesis of Qualitative Research
Weed, M. (2008) · Forum qualitative Sozialforschung · doi:10.17169/fqs-6.1.508 · Cited by 192 · cited 3×
“, 2010), scholars have sought to identify a clear set of methods, criteria, and standards for conducting and assessing reviews of research (e.g., Cooper & Hedges, 2009; Gough, 2007; Lipsey & Wilson, 2001; Lucas et al., 2007; Sandelowski & Barroso, 2007; Thomas & Harden, 2008; Weed, 2005).”
BackgroundMethodGender and Leadership Style among School Principals: A Meta-Analysis
Eagly, A. H., Karau, S. J., & Johnson, B. T. (1992) · Educational Administration Quarterly · doi:10.1177/0013161x92028001004 · Cited by 178 · cited 1×
“Less than 25% of the 38 articles (i.e., Bridges, 1982; Eagly et al., 1992; Leithwood & Jantzi, 2005; Leithwood & Montgomery, 1982; Leithwood & Sun, 2012; Murphy, 2004, 2008; Robinson et al., 2008; Witziers et al., 2003) explicitly referenced published sources on the methodology of conducting reviews of research.”
BackgroundMapping Leadership Studies in Education: Towards a Typology of Knowledge Domains
Ribbins, P., & Gunter, H. (2002) · Educational Management & Administration · doi:10.1177/0263211x020304002 · Cited by 160
Mapping the terrain of educational leadership and management in East Asia
Hallinger, P., & Bryant, D. A. (2013) · Journal of Educational Administration · doi:10.1108/jea-05-2012-0066 · Cited by 159
Self-citationPrincipal leadership in China: an initial review
Walker, A. D., Hu, R., & Qian, H. (2012) · School Effectiveness and School Improvement · doi:10.1080/09243453.2012.678863 · Cited by 155 · cited 3×
“More specifically, only 3 of the 24 studies that relied exclusively on critical synthesis (Leithwood et al., 1990; Murphy et al., 2007; Walker et al., 2012) were explicit in clarifying data collection and analysis procedures.”
BackgroundMethodThe development of assistant principals: a literature review
Oleszewski, A., Shoho, A. R., & Barnett, B. G. (2012) · Journal of Educational Administration · doi:10.1108/09578231211223301 · Cited by 152
Erickson, D. A. (1967) · Review of Educational Research · doi:10.2307/1169816 · Cited by 123 · cited 1×
“Exploratory reviews are most suitable when a problem is poorly understood (e.g., Bossert, Dwyer, Rowan, & Lee, 1982; Erickson, 1967) and/or when relevant empirical research on the topic is limited (e.”
BackgroundLeadership and organisational performance: from research to prescription?
Muijs, D. (2011) · International Journal of Educational Management · doi:10.1108/09513541111100116 · Cited by 101 · cited 1×
“During the past decade, reviews have been published by scholars from not only North America but also the United Kingdom (e.g., Harris, 2008; Muijs, 2011; Southworth, 2002; Turner, 2003), Europe (Witziers et al.”
BackgroundResearch on Educational Administration: The State-of-the-Art
Erickson, D. A. (1979) · Educational Researcher · doi:10.2307/1174358 · Cited by 93
Murphy, J., Vriesenga, M., & Storey, V. A. (2007) · Educational Administration Quarterly · doi:10.1177/0013161x07307796 · Cited by 87 · cited 2×
“More specifically, only 3 of the 24 studies that relied exclusively on critical synthesis (Leithwood et al., 1990; Murphy et al., 2007; Walker et al., 2012) were explicit in clarifying data collection and analysis procedures.”
BackgroundThe place of leadership in turnaround schools
Murphy, J. (2008) · Journal of Educational Administration · doi:10.1108/09578230810849826 · Cited by 75 · cited 1×
“” For example, Ribbins and Gunter (2002) differentiate between five different types of knowledge domains: conceptual, humanistic, critical, evaluative, and instrumental.”
BackgroundThe role of systematic reviews of qualitative evidence in evaluating interventions: a case study
Lorenc, T., Pearson, M., Jamal, F., Cooper, C., & Garside, R. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.1036 · Cited by 61 · cited 5×
“Thus, I agree with others (e.g., Cooper & Hedges, 2009; Gough, 2007; Jackson, 1980; Light & Pillemer, 1984; Lorenc et al., 2012; Lucas et al., 2012; Valentine et al., 2010) who assert that even reviews which rely primarily on the synthesis of ideas benefit from being more systematic and explicit in the execution and r…”
BackgroundMethodInfluentialVision Effects: A Critical Gap in Educational Leadership Research
Kantabutra, S. (2010) · International Journal of Educational Management · doi:10.1108/09513541011055956 · Cited by 58 · cited 1×
“, 2003), Asia (e.g., Hallinger, 2005, 2011a, 2011b; Kantabutra, 2010; Walker et al., 2012), and New Zealand (e.”
Chapter IV: Organizational Character of Education: Administrative Behavior
Lipham, J. M. (1964) · Review of Educational Research · doi:10.2307/1169697 · Cited by 46 · cited 2×
“Thus, the empirical knowledge base was sparse, fragmented, and to a large degree ad hoc (Bridges, 1982; Campbell & Faber, 1961; Erickson, 1979; Haller, 1979; Lipham, 1964).”
BackgroundChapter VII: The Science of Administration
Briner, C., & Campbell, R. F. (1964) · Review of Educational Research · doi:10.3102/00346543034004485 · Cited by 43
Valentine, J. C., Cooper, H., Patall, E. A., Tyson, D., & Robinson, J. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.3 · Cited by 37
Evidence synthesis, economics and public policy
Shemilt, I., Mugford, M., Vale, L. D., Marsh, K., Donaldson, C., & Drummond, M. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.14 · Cited by 34 · cited 2×
“, 2012; Paterson, Thorne, Canam, & Jillings, 2001; Sandelowski & Barroso, 2007; Thomas & Harden, 2008; Weed, 2005) and quantitative studies (Lipsey & Wilson, 2001; Lucas et al., 2007; Shemilt et al., 2010; Valentine et al., 2010) have undergone increased scrutiny and development in recent years.”
BackgroundFehrmann, P., & Thomas, J. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.31 · Cited by 27
DeGeest, D. S., & Schmidt, F. L. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.22 · Cited by 24
Cited by
Bringing context out of the shadows of leadership
Hallinger, P. (2018) · Educational Management Administration & Leadership · doi:10.1177/1741143216670652 · Cited by 755
“While the knowledge base is not ready for meta-analytic synthesis of findings, exploratory reviews will move the field forward by consolidating what we know (Hallinger, 2013, 2014).”
BackgroundSelf-citationA systematic review of studies on leadership models in educational research from 1980 to 2014
Gumus, S., Bellibas, M. S., Esen, M., & Gumus, E. (2018) · Educational Management Administration & Leadership · doi:10.1177/1741143216659296 · Cited by 539
Hallinger, P., & Kovačević, J. (2019) · Review of Educational Research · doi:10.3102/0034654319830380 · Cited by 490
Self-citationHitt, D. H., & Tucker, P. D. (2016) · Review of Educational Research · doi:10.3102/0034654315614911 · Cited by 404
Hallinger, P. (2020) · Educational Management Administration & Leadership · doi:10.1177/1741143218822772 · Cited by 202
“This pattern is interesting in that reviews of research often dominate lists of highly cited articles (Hallinger, 2014, 2018b).”
BackgroundSelf-citation
Metadata
- Authors
- Hallinger, P.
- Venue
- Educational Administration Quarterly
- Publisher
- SAGE Publications
- Published
- 2013-10-23
- Volume
- 50
- Issue
- 4
- Pages
- 539-576
- ISSN
- 0013-161X (ISSN-L)
- 1552-3519
- OpenAlex
- W2113394919
- Cited by
- 161
- References
- 58
- FWCI
- 7.53
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.