Testing Multilevel Mediation Using Hierarchical Linear Models: Problems and Solutions
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
Testing multilevel mediation using hierarchical linear modeling (HLM) has gained tremendous popularity in recent years. However, potential confounding in multilevel mediation effect estimates can arise in these models when within-group effects differ from between-group effects. This study summarizes three types of HLM-based multilevel mediation models, and then explains that in two types of these models confounding can be produced and erroneous conclusions may be derived when using popularly recommended procedures. A Monte Carlo simulation study illustrates that these procedures can underestimate or overestimate true mediation effects. Recommendations are provided for appropriately testing multilevel mediation and for differentiating within-group versus between-group effects in multilevel settings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology · doi:10.2307/2075823 · Cited by 18,724
Multilevel analysis : an introduction to basic and advanced multilevel modeling
Snijders, T. A. B., & Bosker, R. (1999) · Cited by 13,775
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
Asymptotic Confidence Intervals for Indirect Effects in Structural Equation Models
Sobel, M. E. (1982) · Sociological Methodology · doi:10.2307/270723 · Cited by 12,810
A comparison of methods to test mediation and other intervening variable effects
MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.83 · Cited by 9,406 · cited 1×
“Empirical power is calculated in the gð4Þ02 ¼ .14 situation, because we are interested in the empirical power to detect a small effect size and .14 represents a small effect size (MacKinnon et al., 2002).”
MethodConfidence Limits for the Indirect Effect: Distribution of the Product and Resampling Methods
MacKinnon, D. P., Lockwood, C. M., & Williams, J. (2004) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3901_4 · Cited by 7,806
Introduction to Statistical Mediation Analysis
MacKinnon, D. P. (2012) · doi:10.4324/9780203809556 · Cited by 5,618
Edwards, J. R., & Lambert, L. S. (2007) · Psychological Methods · doi:10.1037/1082-989x.12.1.1 · Cited by 4,711
Equivalence of the Mediation, Confounding and Suppression Effect
MacKinnon, D. P., Krull, J. L., & Lockwood, C. M. (2000) · Prevention Science · doi:10.1023/a:1026595011371 · Cited by 4,188
Introducing Multilevel Modeling
Kreft, I., & de Leeuw, J. (1998) · doi:10.4135/9781849209366 · Cited by 2,997
Centering Decisions in Hierarchical Linear Models: Implications for Research in Organizations
Hofmann, D. (1998) · Journal of Management · doi:10.1016/s0149-2063(99)80077-4 · Cited by 2,109
Cheung, G. W., & Lau, R. S. (2008) · Organizational Research Methods · doi:10.1177/1094428107300343 · Cited by 1,965 · cited 2×
“were chosen to correspond to zero, small, medium, and large effect sizes (see Cohen, 1988, pp. 412-414) and to correspond as closely as possible with previous simulation research (e.g., Cheung & Lau, 2008; MacKinnon, Lockwood, Hoffman, West, & Sheets, 2002).”
ResultA Simulation Study of Mediated Effect Measures
MacKinnon, D. P., Warsi, G., & Dwyer, J. H. (1995) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3001_3 · Cited by 1,719
Bauer, D. J., Preacher, K. J., & Gil, K. M. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.142 · Cited by 1,704
Self-citationFrom Micro to Meso: Critical Steps in Conceptualizing and Conducting Multilevel Research
Klein, K. J., & Kozlowski, S. W. J. (2000) · Organizational Research Methods · doi:10.1177/109442810033001 · Cited by 1,471
Multilevel Modeling of Individual and Group Level Mediated Effects
Krull, J. L., & MacKinnon, D. P. (2001) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3602_06 · Cited by 1,330
Tests of the three-path mediated effect
Taylor, A. B., MacKinnon, D. P., & Tein, J.-Y. (2008) · Organizational Research Methods · doi:10.1177/1094428107300344 · Cited by 1,209
A multilevel study of leadership, empowerment, and performance in teams.
Chen, G., Kirkman, B. L., Kanfer, R., Allen, D., & Rosen, B. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.2.331 · Cited by 759
Chen, G., & Bliese, P. D. (2002) · Journal of Applied Psychology · doi:10.1037//0021-9010.87.3.549 · Cited by 710
Lower level mediation in multilevel models.
Kenny, D. A., Korchmaros, J. D., & Bolger, N. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.2.115 · Cited by 687
The application of hierarchical linear modeling to organizational research.
Hofmann, D. A., Griffin, M. A., & Gavin, M. B. (2000) · Cited by 673
Seibert, S. E., Silver, S. R., & Randolph, W. A. (2004) · Academy of Management Journal · doi:10.5465/20159585 · Cited by 651
Multilevel Mediation Modeling in Group-Based Intervention Studies
Krull, J. L., & Mackinnon, D. P. (1999) · Evaluation Review · doi:10.1177/0193841x9902300404 · Cited by 419
Theory Testing and Organizational Behavior: The Varient Approach.
House, R. J., Dansereau, F., Alutto, J. A., & Yammarino, F. J. (1987) · Administrative Science Quarterly · doi:10.2307/2392921 · Cited by 381
A framework for testing meso‐mediational relationships in Organizational Behavior
Mathieu, J. E., & Taylor, S. R. (2007) · Journal of Organizational Behavior · doi:10.1002/job.436 · Cited by 374
The Relative Validity of Inferences About Mediation as a Function of Research Design Characteristics
Stone-Romero, E. F., & Rosopa, P. J. (2008) · Organizational Research Methods · doi:10.1177/1094428107300342 · Cited by 346
Comparing correlations based on individual-level and aggregated data.
Ostroff, C. L. (1993) · Journal of Applied Psychology · doi:10.1037//0021-9010.78.4.569 · Cited by 320
The Journal of Experimental Education: A Quarterly
(1972) · The Journal of Experimental Education · doi:10.1080/00220973.1972.11011361 · Cited by 283
Sample Size for Studying Intermediate Endpoints within Intervention Trials or Observational Studies
Freedman, L. S., & Schatzkin, A. (1992) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116581 · Cited by 251
A Technique for Analyzing the Effects of Group Composition
Davis, J. A., Spaeth, J. L., & Huson, C. (1961) · American Sociological Review · doi:10.2307/2089857 · Cited by 155
Mediational Inferences in Organizational Research: Then, Now, and Beyond
Mathieu, J. E., DeShon, R. P., & Bergh, D. D. (2008) · Organizational Research Methods · doi:10.1177/1094428107310089 · Cited by 151
The Performance of Methods to Test Upper-Level Mediation in the Presence of Nonnormal Data
Pituch, K. A., & Stapleton, L. M. (2008) · Multivariate Behavioral Research · doi:10.1080/00273170802034844 · Cited by 117
Three-Level Models for Indirect Effects in School- and Class-Randomized Experiments in Education
Pituch, K. A., Murphy, D. L., & Tate, R. L. (2009) · The Journal of Experimental Education · doi:10.1080/00220970903224685 · Cited by 58
Adjusting for a Mediator in Models With Two Crossed Treatment Variables
Muller, D., Yzerbyt, V. Y., & Judd, C. M. (2008) · Organizational Research Methods · doi:10.1177/1094428106296639 · Cited by 46
Cited by
mediation:R Package for Causal Mediation Analysis
Tingley, D., Yamamoto, T., Hirose, K., Keele, L., & Imai, K. (2014) · Journal of Statistical Software · doi:10.18637/jss.v059.i05 · Cited by 4,155
A general multilevel SEM framework for assessing multilevel mediation
Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010) · Psychological Methods · doi:10.1037/a0020141 · Cited by 3,796
Self-citationThe Impact of High-Performance Human Resource Practices on Employees’ Attitudes and Behaviors
Kehoe, R. R., & Wright, P. M. (2013) · Journal of Management · doi:10.1177/0149206310365901 · Cited by 1,289
Two-condition within-participant statistical mediation analysis: A path-analytic framework.
Montoya, A. K., & Hayes, A. F. (2017) · Psychological Methods · doi:10.1037/met0000086 · Cited by 931
Expressed humility in organizations: Implications for performance, teams, and leadership
Owens, B. P., Johnson, M. D., & Mitchell, T. R. (2013) · Organization Science · doi:10.1287/orsc.1120.0795 · Cited by 927
Metadata
- Authors
- Zhang, Z., Zyphur, M. J., & Preacher, K. J.
- Venue
- Organizational Research Methods
- Publisher
- SAGE Publications
- Published
- 2008-12-17
- Volume
- 12
- Issue
- 4
- Pages
- 695-719
- ISSN
- 1094-4281 (ISSN-L)
- 1552-7425
- OpenAlex
- W2116952601
- Cited by
- 1,093
- References
- 38
- FWCI
- 10.24
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.