Confidence intervals for population reliability coefficients: Evaluation of methods, recommendations, and software for composite measures.
The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Abstract
A composite score is the sum of a set of components. For example, a total test score can be defined as the sum of the individual items. The reliability of composite scores is of interest in a wide variety of contexts due to their widespread use and applicability to many disciplines. The psychometric literature has devoted considerable time to discussing how to best estimate the population reliability value. However, all point estimates of a reliability coefficient fail to convey the uncertainty associated with the estimate as it estimates the population value. Correspondingly, a confidence interval is recommended to convey the uncertainty with which the population value of the reliability coefficient has been estimated. However, many confidence interval methods for bracketing the population reliability coefficient exist and it is not clear which method is most appropriate in general or in a variety of specific circumstances. We evaluate these confidence interval methods for 4 reliability coefficients (coefficient alpha, coefficient omega, hierarchical omega, and categorical omega) under a variety of conditions with 3 large-scale Monte Carlo simulation studies. Our findings lead us to generally recommend bootstrap confidence intervals for hierarchical omega for continuous items and categorical omega for categorical items. All of the methods we discuss are implemented in the freely available R language and environment via the MBESS package.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we agree with those viewing h as being a more appropriate measure of second-order reliability (Kelley & Pornprasertmanit, 2016;Zinbarg et al., 2005) because it considers both 2 and group-specific error as comprising the error term.”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley & Pornprasertmanit, 2016
In the citing paper
“Omega total ( T ) represents the reliability of the combined first-order factors, whereas omega hierarchical ( h ) is the reliability of the second-order factor (Kelley & Pornprasertmanit, 2016;Zinbarg et al., 2005)”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley & Pornprasertmanit, 2016
In the citing paper
“We prefer PCIs over bias-corrected and accelerated CIs (BC a CI) because PCIs have better coverage for both normal and nonnormal item distributions (Kelley & Pornprasertmanit, 2016).”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley & Pornprasertmanit, 2016
In the citing paper
“While the appropriate sample size depends on the complexity of the model and the estimation method, Kelley and Pornprasertmanit (2016) found, using a simulation, that the coverage rates of CIs are good across reliability levels even at N = 50.”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley and Pornprasertmanit (2016)
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931 · cited 1×
“The most widely used reliability coefficient used to estimate $(Y) is coefficient alpha (Cronbach, 1951).”
MethodAn Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1998) · Cited by 36,049
lavaan : An R Package for Structural Equation Modeling
Rosseel, Y. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i02 · Cited by 27,955
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Statistical Theories of Mental Test Scores
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,022
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,014
(2005) · Cited by 6,097
A scaled difference chi-square test statistic for moment structure analysis
Satorra, A., & Bentler, P. M. (2001) · Psychometrika · doi:10.1007/bf02296192 · Cited by 5,282
Introduction to Classical and Modern Test Theory
Crocker, L. M., & Algina, J. J. (1986) · Cited by 5,090
Cited by
Thanks coefficient alpha, we’ll take it from here.
McNeish, D. (2018) · Psychological Methods · doi:10.1037/met0000144 · Cited by 2,533
Cheung, G. W., Cooper-Thomas, H. D., Lau, R. S., & Wang, L. C. (2024) · Asia Pacific Journal of Management · doi:10.1007/s10490-023-09871-y · Cited by 2,262
“…factor reliability because they treat 2 as the only error term, we agree with those viewing h as being a more appropriate measure of second-order reliability (Kelley & Pornprasertmanit, 2016; Zinbarg et al., 2005) because it considers both 2 and group-specific error as comprising the error term.”
InfluentialA Review of Key Likert Scale Development Advances: 1995–2019
Jebb, A. T., Ng, V., & Tay, L. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.637547 · Cited by 770
Un viaje alrededor de alfa y omega para estimar la fiabilidad de consistencia interna
Viladrich, C., Angulo-Brunet, A., & Doval, E. (2017) · Anales de Psicología · doi:10.6018/analesps.33.3.268401 · Cited by 699
Flora, D. B. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920951747 · Cited by 561
Metadata
- Authors
- Kelley, K., & Pornprasertmanit, S.
- Affiliations
- University of Notre Dame (US)
- Texas Tech University (US)
- Venue
- Psychological Methods
- Published
- 2016
- Volume
- 21
- Issue
- 1
- Pages
- 69-92
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2295825737
- Cited by
- 259
- References
- 80
- Citations by year
- 2026: 20 · 2025: 17 · 2024: 27 · 2023: 31 · 2022: 35 · 2021: 38 · 2020: 29 · 2019: 22 · 2018: 22 · 2017: 13 · 2016: 4
- FWCI
- 22.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Notre Dame Deloitte Center for Ethical Leadership
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- confidence intervals, composite score, Cronbach's alpha, bootstrap confidence intervals, reliability estimation, Monte Carlo simulation, R software
- MeSH terms
- Humans, Psychometrics, Reproducibility of Results, Confidence Intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.