The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Reliability estimation in a multilevel confirmatory factor analysis framework.
Abstract
Scales with varying degrees of measurement reliability are often used in the context of multistage sampling, where variance exists at multiple levels of analysis (e.g., individual and group). Because methodological guidance on assessing and reporting reliability at multiple levels of analysis is currently lacking, we discuss the importance of examining level-specific reliability. We present a simulation study and an applied example showing different methods for estimating multilevel reliability using multilevel confirmatory factor analysis and provide supporting Mplus program code. We conclude that (a) single-level estimates will not reflect a scale's actual reliability unless reliability is identical at each level of analysis, (b) 2-level alpha and composite reliability (omega) perform relatively well in most settings, (c) estimates of maximal reliability (H) were more biased when estimated using multilevel data than either alpha or omega, and (d) small cluster size can lead to overestimates of reliability at the between level of analysis. We also show that Monte Carlo confidence intervals and Bayesian credible intervals closely reflect the sampling distribution of reliability estimates under most conditions. We discuss the estimation of credible intervals using Mplus and provide R code for computing Monte Carlo confidence intervals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(2001) · RePEc: Research Papers in Economics · Cited by 27,577
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology · doi:10.2307/2075823 · Cited by 18,724
Confirmatory factor analysis for applied research
(2007) · Choice Reviews Online · doi:10.5860/choice.44-2769 · Cited by 15,614
Statistical Theories of Mental Test Scores
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,022
Confidence Limits for the Indirect Effect: Distribution of the Product and Resampling Methods
MacKinnon, D. P., Lockwood, C. M., & Williams, J. (2004) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3901_4 · Cited by 7,806
Sample size in factor analysis.
MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999) · Psychological Methods · doi:10.1037/1082-989x.4.1.84 · Cited by 5,497
Chan, D. K. (1998) · Journal of Applied Psychology · doi:10.1037//0021-9010.83.2.234 · Cited by 2,972
On the use, the misuse, and the very limited usefulness of Cronbach’s alpha
Sijtsma, K. (2009) · Psychometrika · doi:10.1007/s11336-008-9101-0 · Cited by 2,814
(1999) · Cited by 2,653
(2000) · Cited by 2,531
Bootstrap confidence intervals
DiCiccio, T. J., & Efron, B. (1996) · Statistical Science · doi:10.1214/ss/1032280214 · Cited by 2,286
The Theory of the Estimation of Test Reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 2,002
Advantages of Monte Carlo confidence intervals for indirect effects
Preacher, K. J., & Selig, J. P. (2012) · Communication Methods and Measures · doi:10.1080/19312458.2012.679848 · Cited by 1,908
Self-citationEstimation of composite reliability for congeneric measures
Raykov, T. (1997) · Applied Psychological Measurement · doi:10.1177/01466216970212006 · Cited by 1,775
Intraclass Reliability Estimates: Testing Structural Assumptions
Werts, C. E., Linn, R. L., & Jöreskog, K. G. (1974) · Educational and Psychological Measurement · doi:10.1177/001316447403400104 · Cited by 1,655
Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005) · Psychometrika · doi:10.1007/s11336-003-0974-7 · Cited by 1,439
Multilevel Covariance Structure Analysis
MUTHÉN, B. O. (1994) · Sociological Methods & Research · doi:10.1177/0049124194022003006 · Cited by 1,171
Sample Size in Factor Analysis: The Role of Model Error
MacCallum, R. C., Widaman, K. F., Preacher, K. J., & Hong, S. (2001) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3604_06 · Cited by 1,124
Self-citationA Basis for Analyzing Test-Retest Reliability
Guttman, L. (1945) · Psychometrika · doi:10.1007/bf02288892 · Cited by 1,122
Introduction to Psychometric Theory
Raykov, T., & Marcoulides, G. A. (2011) · Routledge eBooks · doi:10.4324/9780203841624 · Cited by 1,095
Panel Models in Sociological Research: Theory into Practice
Halaby, C. N. (2004) · Annual Review of Sociology · doi:10.1146/annurev.soc.30.012703.110629 · Cited by 946
On structural equation modeling with data that are not missing completely at random
Muthén, B., Kaplan, D., & Hollis, M. (1987) · Psychometrika · doi:10.1007/bf02294365 · Cited by 883
Cranford, J. A., Shrout, P. E., Iida, M., Rafaeli, E., Yip, T., & Bolger, N. (2006) · Personality and Social Psychology Bulletin · doi:10.1177/0146167206287721 · Cited by 856
McDonald, R. P. (1970) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1970.tb00432.x · Cited by 781
Handbook of Advanced Multilevel Analysis
(2011) · Routledge eBooks · doi:10.4324/9780203848852 · Cited by 743
Factor Analysis with Ordinal Indicators: A Monte Carlo Study Comparing DWLS and ULS Estimation
Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705510903203573 · Cited by 685
Encyclopedia of social measurement
(2005) · Choice Reviews Online · doi:10.5860/choice.42-5629 · Cited by 616
Wilhelm, P., & Schoebi, D. (2007) · European Journal of Psychological Assessment · doi:10.1027/1015-5759.23.4.258 · Cited by 472
The statistical analysis of data from small groups.
Kenny, D. A., Mannetti, L., Pierro, A., Livi, S., & Kashy, D. A. (2002) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.83.1.126 · Cited by 469
COEFFICIENT ALPHA AND THE RELIABILITY OF COMPOSITE MEASUREMENTS
Novick, M. R., & Lewis, C. (1966) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1966.tb00356.x · Cited by 446
Raykov, T., & Shrout, P. E. (2002) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0902_3 · Cited by 283
A General Model for the Analysis of Multilevel Data
Goldstein, H., & McDonald, R. P. (1988) · Psychometrika · doi:10.1007/bf02294400 · Cited by 240
How few countries will do? Comparative survey analysis from a Bayesian perspective
Hox, J. J., van de Schoot, R., & Matthijsse, S. M. (2011) · Survey research methods · doi:10.18148/srm/2012.v6i2.5033 · Cited by 225
Growth mixture modeling: Analysis with non-Gaussian random effects
Asparouhov, T., & Muthen, B. (2008) · Chapman & Hall/CRC Handbooks of Modern Statistical Methods · doi:10.1201/9781420011579.ch6 · Cited by 207
On Multilevel Model Reliability Estimation From the Perspective of Structural Equation Modeling
Raykov, T., & Marcoulides, G. A. (2006) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem1301_7 · Cited by 181
A Monte Carlo sample size study: how many countries are needed for accurate multilevel SEM?
Meuleman, B., & Billiet, J. (2008) · Survey research methods · doi:10.18148/srm/2009.v3i1.666 · Cited by 177
First graders' literacy and self-regulation gains: The effect of individualizing student instruction
Connor, C. M., Ponitz, C. C., Phillips, B. M., Travis, Q. M., Glasney, S., & Morrison, F. J. (2010) · Journal of School Psychology · doi:10.1016/j.jsp.2010.06.003 · Cited by 152
Raykov, T. (1998) · Applied Psychological Measurement · doi:10.1177/014662169802200406 · Cited by 96
Analytic Estimation of Standard Error and Confidence Interval for Scale Reliability
Raykov, T. (2002) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3701_04 · Cited by 94
Estimation of maximal reliability: A note on a covariance structure modelling approach
Raykov, T. (2004) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711004849295 · Cited by 83
Mean and Covariance Structure Analysis of Hierarchical Data
Muthén, B. (2011) · eScholarship (California Digital Library) · Cited by 72
The Bootstrap and Markov-Chain Monte Carlo
Efron, B. (2011) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543406.2011.607736 · Cited by 64
A Unifying Expression for the Maximal Reliability of a Linear Composite
Li, H. (1997) · Psychometrika · doi:10.1007/bf02295278 · Cited by 63
Using Generalizability Theory in Counseling and Development
Webb, N. M., ROWLEY, G. L., & Shavelson, R. J. (1988) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.1988.12022886 · Cited by 56
Franić, S., Dolan, C. V., Borsboom, D., Hudziak, J. J., van Beijsterveldt, C. E. M., & Boomsma, D. I. (2013) · Psychological Methods · doi:10.1037/a0032755 · Cited by 47
Kistner, E. O., & Muller, K. E. (2004) · Psychometrika · doi:10.1007/bf02295646 · Cited by 43
Covariance Structure Models for Maximal Reliability of Unit-Weighted Composites
Bentler, P. M. (2007) · Elsevier eBooks · doi:10.1016/b978-044452044-9/50004-5 · Cited by 36
Gottfredson, N. C., Panter, A. T., Daye, C. E., Allen, W. F., & Wightman, L. F. (2009) · Multivariate Behavioral Research · doi:10.1080/00273170902949719 · Cited by 35
Maximally Reliable Composites for Unidimensional Measures
Conger, A. J. (1980) · Educational and Psychological Measurement · doi:10.1177/001316448004000213 · Cited by 18
Bootstrapping in Multilevel Models
Goldstein, H. (2015) · Routledge eBooks · doi:10.4324/9780203848852.ch9 · Cited by 18
Estimation of Reliability for Multiple-Component Measuring Instruments in Hierarchical Designs
Raykov, T., & du Toit, S. H. C. (2005) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem1204_2 · Cited by 15
Evaluation of Reliability Coefficients for Two-Level Models via Latent Variable Analysis
Raykov, T., & Penev, S. I. (2010) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2010.510052 · Cited by 10
Huang, P., & Weng, L. (2012) · Multivariate Behavioral Research · doi:10.1080/00273171.2012.673924 · Cited by 9
Cited by
Thanks coefficient alpha, we’ll take it from here.
McNeish, D. (2018) · Psychological Methods · doi:10.1037/met0000144 · Cited by 2,533
Gabriel, A. S., Podsakoff, N. P., Beal, D. J., Scott, B. A., Sonnentag, S., Trougakos, J. P., & Butts, M. M. (2019) · Organizational Research Methods · doi:10.1177/1094428118802626 · Cited by 755
The Concept of Mental Toughness: Tests of Dimensionality, Nomological Network, and Traitness
Gucciardi, D. F., Hanton, S., Gordon, S., Mallett, C. J., & Temby, P. (2015) · Journal of Personality · doi:10.1111/jopy.12079 · Cited by 483
Multilevel structural equation models for assessing moderation within and across levels of analysis.
Preacher, K. J., Zhang, Z., & Zyphur, M. J. (2016) · Psychological Methods · doi:10.1037/met0000052 · Cited by 466
Self-citationKhajavy, G. H., MacIntyre, P. D., & Barabadi, E. (2018) · Studies in Second Language Acquisition · doi:10.1017/s0272263117000304 · Cited by 451
Metadata
- Authors
- Geldhof, G. J., Preacher, K. J., & Zyphur, M. J.
- Venue
- Psychological Methods
- Published
- 2014-03
- Volume
- 19
- Issue
- 1
- Pages
- 72-91
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2114473532
- Cited by
- 1,496
- References
- 55
- FWCI
- 55.31
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.