The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Confidence intervals for population reliability coefficients: Evaluation of methods, recommendations, and software for composite measures.
Abstract
A composite score is the sum of a set of components. For example, a total test score can be defined as the sum of the individual items. The reliability of composite scores is of interest in a wide variety of contexts due to their widespread use and applicability to many disciplines. The psychometric literature has devoted considerable time to discussing how to best estimate the population reliability value. However, all point estimates of a reliability coefficient fail to convey the uncertainty associated with the estimate as it estimates the population value. Correspondingly, a confidence interval is recommended to convey the uncertainty with which the population value of the reliability coefficient has been estimated. However, many confidence interval methods for bracketing the population reliability coefficient exist and it is not clear which method is most appropriate in general or in a variety of specific circumstances. We evaluate these confidence interval methods for 4 reliability coefficients (coefficient alpha, coefficient omega, hierarchical omega, and categorical omega) under a variety of conditions with 3 large-scale Monte Carlo simulation studies. Our findings lead us to generally recommend bootstrap confidence intervals for hierarchical omega for continuous items and categorical omega for categorical items. All of the methods we discuss are implemented in the freely available R language and environment via the MBESS package.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we agree with those viewing h as being a more appropriate measure of second-order reliability (Kelley & Pornprasertmanit, 2016;Zinbarg et al., 2005) because it considers both 2 and group-specific error as comprising the error term.”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley & Pornprasertmanit, 2016
In the citing paper
“Omega total ( T ) represents the reliability of the combined first-order factors, whereas omega hierarchical ( h ) is the reliability of the second-order factor (Kelley & Pornprasertmanit, 2016;Zinbarg et al., 2005)”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley & Pornprasertmanit, 2016
In the citing paper
“We prefer PCIs over bias-corrected and accelerated CIs (BC a CI) because PCIs have better coverage for both normal and nonnormal item distributions (Kelley & Pornprasertmanit, 2016).”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley & Pornprasertmanit, 2016
In the citing paper
“While the appropriate sample size depends on the complexity of the model and the estimation method, Kelley and Pornprasertmanit (2016) found, using a simulation, that the coverage rates of CIs are good across reliability levels even at N = 50.”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley and Pornprasertmanit (2016)
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1998) · Cited by 36,049
lavaan : An R Package for Structural Equation Modeling
Rosseel, Y. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i02 · Cited by 27,955
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Statistical Theories of Mental Test Scores
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,022
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,014
(2005) · Cited by 6,097
A scaled difference chi-square test statistic for moment structure analysis
Satorra, A., & Bentler, P. M. (2001) · Psychometrika · doi:10.1007/bf02296192 · Cited by 5,282
Introduction to Classical and Modern Test Theory
Crocker, L. M., & Algina, J. J. (1986) · Cited by 5,090
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
Flora, D. B., & Curran, P. J. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.466 · Cited by 3,019
Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012) · Psychological Methods · doi:10.1037/a0029315 · Cited by 2,816
On the use, the misuse, and the very limited usefulness of Cronbach’s alpha
Sijtsma, K. (2009) · Psychometrika · doi:10.1007/s11336-008-9101-0 · Cited by 2,814
(1999) · Cited by 2,653
Applications of Structural Equation Modeling in Psychological Research
MacCallum, R. C., & Austin, J. T. (2000) · Annual Review of Psychology · doi:10.1146/annurev.psych.51.1.201 · Cited by 2,623
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
The Theory of the Estimation of Test Reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 2,002
Model fit and model selection in structural equation modeling.
West, S. G., Taylor, A. B., & Wu, W. (2012) · Cited by 1,812
Estimation of composite reliability for congeneric measures
Raykov, T. (1997) · Applied Psychological Measurement · doi:10.1177/01466216970212006 · Cited by 1,775
Coefficients Alpha, Beta, Omega, and the glb: Comments on Sijtsma
Revelle, W., & Zinbarg, R. E. (2009) · Psychometrika · doi:10.1007/s11336-008-9102-z · Cited by 1,628
Muthén, B. (1984) · Psychometrika · doi:10.1007/bf02294210 · Cited by 1,544
Reliability estimation in a multilevel confirmatory factor analysis framework.
Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014) · Psychological Methods · doi:10.1037/a0032138 · Cited by 1,496
Oehlert, G. W. (1992) · The American Statistician · doi:10.1080/00031305.1992.10475842 · Cited by 1,379
STATISTICAL ANALYSIS OF SETS OF CONGENERIC TESTS*
Jo̎reskog, K. G. (1969) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1969.tb00777.x · Cited by 1,203
OpenMx: An Open Source Extended Structural Equation Modeling Framework
Boker, S., Neale, M., Maes, H., Wilde, M., Spiegel, M., Brick, T., Spies, J., Estabrook, R., Kenny, S., Bates, T., et al. (2011) · Psychometrika · doi:10.1007/s11336-010-9200-6 · Cited by 1,163
A Basis for Analyzing Test-Retest Reliability
Guttman, L. (1945) · Psychometrika · doi:10.1007/bf02288892 · Cited by 1,122
Introduction to Psychometric Theory
Raykov, T., & Marcoulides, G. A. (2011) · Routledge eBooks · doi:10.4324/9780203841624 · Cited by 1,095
Kelley, K., & Preacher, K. J. (2012) · Psychological Methods · doi:10.1037/a0028086 · Cited by 994
Self-citationStatistical Methods for Research Workers
Tukey, J., & Fisher, R. A. (1952) · Econometrica · doi:10.2307/1907425 · Cited by 949
Enders, C. K. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.352 · Cited by 932
Assessing Factorial Invariance in Ordered-Categorical Measures
Millsap, R. E., & Yun-Tein, J. (2004) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3903_4 · Cited by 862
Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008) · Psychological Methods · doi:10.1037/a0012869 · Cited by 823
Olsson, U. H., Foss, T., Troye, S. V., & Howell, R. D. (2000) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0704_3 · Cited by 648
Sample Size Requirements for Testing and Estimating Coefficient Alpha
Bonett, D. G. (2002) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986027004335 · Cited by 577
Marcoulides, G. A., & Saunders, C. (2006) · MIS Quarterly · doi:10.2307/25148727 · Cited by 571
Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.59.103006.093735 · Cited by 542
Self-citationStatistical Inference for Coefficient Alpha
Feldt, L. S., Woodruff, D. J., & Salih, F. A. (1987) · Applied Psychological Measurement · doi:10.1177/014662168701100107 · Cited by 517
Test Reliability Estimated by Analysis of Variance
Hoyt, C. (1941) · Psychometrika · doi:10.1007/bf02289270 · Cited by 513
COEFFICIENT ALPHA AND THE RELIABILITY OF COMPOSITE MEASUREMENTS
Novick, M. R., & Lewis, C. (1966) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1966.tb00356.x · Cited by 446
Simulating Multivariate Nonnormal Distributions
Vale, C. D., & Maurelli, V. A. (1983) · Psychometrika · doi:10.1007/bf02293687 · Cited by 446
Green, S. B., & Yang, Y. (2009) · Psychometrika · doi:10.1007/s11336-008-9099-3 · Cited by 428
Zinbarg, R. E., Yovel, I., Revelle, W., & McDonald, R. P. (2006) · Applied Psychological Measurement · doi:10.1177/0146621605278814 · Cited by 422
Alpha, Dimension-Free, and Model-Based Internal Consistency Reliability
Bentler, P. M. (2009) · Psychometrika · doi:10.1007/s11336-008-9100-1 · Cited by 405
Miller, M. B. (1995) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705519509540013 · Cited by 377
Commentary on Coefficient Alpha: A Cautionary Tale
Green, S. B., & Yang, Y. (2009) · Psychometrika · doi:10.1007/s11336-008-9098-4 · Cited by 312
Kelley, K., & Maxwell, S. E. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.3.305 · Cited by 310
Self-citationThe Approximate Sampling Distribution of Kuder-Richardson Reliability Coefficient Twenty
Feldt, L. S. (1965) · Psychometrika · doi:10.1007/bf02289499 · Cited by 288
Raykov, T., & Shrout, P. E. (2002) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0902_3 · Cited by 283
Advancing Alpha: Measuring Reliability with ConfidencePreprint
Iacobucci, D., & Duhachek, A. (2003) · SSRN Electronic Journal · Cited by 282
The two-sample t test: pre-testing its assumptions does not pay off
Rasch, D., Kubinger, K. D., & Moder, K. (2011) · Statistical Papers · doi:10.1007/s00362-009-0224-x · Cited by 269
A K-Sample Significance Test for Independent Alpha Coefficients
Hakstian, A. R., & Whalen, T. E. (1976) · Psychometrika · doi:10.1007/bf02291840 · Cited by 257
Methods for the Behavioral, Educational, and Social Sciences: An R package
Kelley, K. (2007) · Behavior Research Methods · doi:10.3758/bf03192993 · Cited by 255
Self-citationConfidence Intervals for Standardized Effect Sizes: Theory, Application, and Implementation
Kelley, K. (2007) · Journal of Statistical Software · doi:10.18637/jss.v020.i08 · Cited by 236
Self-citationAlpha's Standard Error (ASE): An Accurate and Precise Confidence Interval Estimate.
Duhachek, A., & Iacobucci, D. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.5.792 · Cited by 171
Woodhouse, B., & Jackson, P. H. (1977) · Psychometrika · doi:10.1007/bf02295980 · Cited by 156
Schneider, B. (2006) · Cited by 149
Cudeck, R. A., & Browne, M. W. (1992) · Psychometrika · doi:10.1007/bf02295424 · Cited by 148
Scale construction and development using structural equation modeling.
Raykov, T. (2012) · Cited by 148
Coefficient Alpha as an Estimate of Test Reliability Under Violation of Two Assumptions
Zimmerman, D. W., Zumbo, B. D., & Lalonde, C. (1993) · Educational and Psychological Measurement · doi:10.1177/0013164493053001003 · Cited by 142
Correlated Errors in True Score Models and Their Effect on Coefficient Alpha
Green, S. B., & Hershberger, S. L. (2000) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem0702_6 · Cited by 141
On the Distribution of the Maximum Likelihood Estimator of Cronbach's Alpha
Zyl, V., Neudecker, H., & Nel, D. G. (2000) · Psychometrika · doi:10.1007/bf02296146 · Cited by 140
Testing Model Nesting and Equivalence
Bentler, P., & Satorra, A. (2008) · Psychological methods · doi:10.1037/a0019625 · Cited by 133
Asymptotically distribution-free (ADF) interval estimation of coefficient alpha.
Maydeu‐Olivares, A., Coffman, D. L., & Hartmann, W. M. (2007) · Psychological Methods · doi:10.1037/1082-989x.12.2.157 · Cited by 130
Evaluation of Scale Reliability With Binary Measures Using Latent Variable Modeling
Raykov, T., Dimitrov, D. M., & Asparouhov, T. (2010) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511003659417 · Cited by 127
Probability Spaces, Hilbert Spaces, and the Axioms of Test Theory
Zimmerman, D. W. (1975) · Psychometrika · doi:10.1007/bf02291765 · Cited by 111
Varying coefficient meta-analytic methods for alpha reliability.
Bonett, D. G. (2010) · Psychological Methods · doi:10.1037/a0020142 · Cited by 97
Analytic Estimation of Standard Error and Confidence Interval for Scale Reliability
Raykov, T. (2002) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3701_04 · Cited by 94
Komaroff, E. (1997) · Applied Psychological Measurement · doi:10.1177/01466216970214004 · Cited by 94
Constructing Approximate Confidence Intervals for Parameters With Structural Equation Models
Cheung, M. W. ‐. (2009) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705510902751291 · Cited by 84
A Monte Carlo Study of Eight Confidence Interval Methods for Coefficient Alpha
Romano, J. L., Kromrey, J. D., & Hibbard, S. T. (2010) · Educational and Psychological Measurement · doi:10.1177/0013164409355690 · Cited by 55
Terry, L. J., & Kelley, K. (2011) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.2011.02030.x · Cited by 47
Self-citationYuan, K., Guarnaccia, C. A., & Hayslip, B. (2003) · Educational and Psychological Measurement · doi:10.1177/0013164402239314 · Cited by 44
Coefficient Alpha Bootstrap Confidence Interval Under Nonnormality
Padilla, M. A., Divers, J., & Newton, M. (2012) · Applied Psychological Measurement · doi:10.1177/0146621612445470 · Cited by 40
Kelley, K., & Cheng, Y. (2011) · Methodology · doi:10.1027/1614-2241/a000036 · Cited by 39
Self-citationCoefficient Omega Bootstrap Confidence Intervals
Padilla, M. A., & Divers, J. (2013) · Educational and Psychological Measurement · doi:10.1177/0013164413492765 · Cited by 21
Bootstrap Interval Estimation of Reliability via Coefficient Omega
Padilla, M. A., & Divers, J. (2013) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1367381520 · Cited by 21
Meta-Analysis of Scale Reliability Using Latent Variable Modeling
Raykov, T., & Marcoulides, G. A. (2013) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2013.769396 · Cited by 17
Cui, Y. D., & Li, J. (2012) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.2012.02038.x · Cited by 16
Automated Procedure for Obtaining Standard Error and Confidence Interval for Scale Reliability
Raykov, T. (2002) · Understanding Statistics · doi:10.1207/s15328031us0102_01 · Cited by 7
Cited by
Thanks coefficient alpha, we’ll take it from here.
McNeish, D. (2018) · Psychological Methods · doi:10.1037/met0000144 · Cited by 2,533
Cheung, G. W., Cooper-Thomas, H. D., Lau, R. S., & Wang, L. C. (2024) · Asia Pacific Journal of Management · doi:10.1007/s10490-023-09871-y · Cited by 2,262
“…factor reliability because they treat 2 as the only error term, we agree with those viewing h as being a more appropriate measure of second-order reliability (Kelley & Pornprasertmanit, 2016; Zinbarg et al., 2005) because it considers both 2 and group-specific error as comprising the error term.”
InfluentialA Review of Key Likert Scale Development Advances: 1995–2019
Jebb, A. T., Ng, V., & Tay, L. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.637547 · Cited by 770
Un viaje alrededor de alfa y omega para estimar la fiabilidad de consistencia interna
Viladrich, C., Angulo-Brunet, A., & Doval, E. (2017) · Anales de Psicología · doi:10.6018/analesps.33.3.268401 · Cited by 699
Flora, D. B. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920951747 · Cited by 561
Metadata
- Authors
- Kelley, K., & Pornprasertmanit, S.
- Affiliations
- University of Notre Dame (US)
- Texas Tech University (US)
- Venue
- Psychological Methods
- Published
- 2016
- Volume
- 21
- Issue
- 1
- Pages
- 69-92
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2295825737
- Cited by
- 259
- References
- 80
- Citations by year
- 2026: 20 · 2025: 17 · 2024: 27 · 2023: 31 · 2022: 35 · 2021: 38 · 2020: 29 · 2019: 22 · 2018: 22 · 2017: 13 · 2016: 4
- FWCI
- 22.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Notre Dame Deloitte Center for Ethical Leadership
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- confidence intervals, composite score, Cronbach's alpha, bootstrap confidence intervals, reliability estimation, Monte Carlo simulation, R software
- MeSH terms
- Humans, Psychometrics, Reproducibility of Results, Confidence Intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.