Confidence intervals for population reliability coefficients: Evaluation of methods, recommendations, and software for composite measures.
The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Abstract
A composite score is the sum of a set of components. For example, a total test score can be defined as the sum of the individual items. The reliability of composite scores is of interest in a wide variety of contexts due to their widespread use and applicability to many disciplines. The psychometric literature has devoted considerable time to discussing how to best estimate the population reliability value. However, all point estimates of a reliability coefficient fail to convey the uncertainty associated with the estimate as it estimates the population value. Correspondingly, a confidence interval is recommended to convey the uncertainty with which the population value of the reliability coefficient has been estimated. However, many confidence interval methods for bracketing the population reliability coefficient exist and it is not clear which method is most appropriate in general or in a variety of specific circumstances. We evaluate these confidence interval methods for 4 reliability coefficients (coefficient alpha, coefficient omega, hierarchical omega, and categorical omega) under a variety of conditions with 3 large-scale Monte Carlo simulation studies. Our findings lead us to generally recommend bootstrap confidence intervals for hierarchical omega for continuous items and categorical omega for categorical items. All of the methods we discuss are implemented in the freely available R language and environment via the MBESS package.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we agree with those viewing h as being a more appropriate measure of second-order reliability (Kelley & Pornprasertmanit, 2016;Zinbarg et al., 2005) because it considers both 2 and group-specific error as comprising the error term.”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley & Pornprasertmanit, 2016
In the citing paper
“Omega total ( T ) represents the reliability of the combined first-order factors, whereas omega hierarchical ( h ) is the reliability of the second-order factor (Kelley & Pornprasertmanit, 2016;Zinbarg et al., 2005)”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley & Pornprasertmanit, 2016
In the citing paper
“We prefer PCIs over bias-corrected and accelerated CIs (BC a CI) because PCIs have better coverage for both normal and nonnormal item distributions (Kelley & Pornprasertmanit, 2016).”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley & Pornprasertmanit, 2016
In the citing paper
“While the appropriate sample size depends on the complexity of the model and the estimation method, Kelley and Pornprasertmanit (2016) found, using a simulation, that the coverage rates of CIs are good across reliability levels even at N = 50.”
From Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations (2024) · cites this paper as Kelley and Pornprasertmanit (2016)
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931 · cited 1×
“The most widely used reliability coefficient used to estimate $(Y) is coefficient alpha (Cronbach, 1951).”
MethodAn Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1998) · Cited by 36,049
lavaan : An R Package for Structural Equation Modeling
Rosseel, Y. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i02 · Cited by 27,955
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Statistical Theories of Mental Test Scores
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,022
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,014
(2005) · Cited by 6,097
A scaled difference chi-square test statistic for moment structure analysis
Satorra, A., & Bentler, P. M. (2001) · Psychometrika · doi:10.1007/bf02296192 · Cited by 5,282
Introduction to Classical and Modern Test Theory
Crocker, L. M., & Algina, J. J. (1986) · Cited by 5,090
Cited by
Items 201–280 of 280
Ventura‐León, J. (2025) · Revista de Psicología Clínica con Niños y Adolescentes · doi:10.21134/rpcna.2025.12.1.7 · Cited by 2
Reinke, W. M., Herman, K. C., Thompson, A. M., Shaoli, S. S., & Bhuiyan, P. (2026) · Journal of School Psychology · doi:10.1016/j.jsp.2025.101545 · Cited by 3
Validität und Reliabilität virtualisierter RTR-Messungen
Waldvogel, T., Wagschal, U., Metz, T., Becker, B., Feiten, L., & Weishaupt, S. (2021) · Publizistik · doi:10.1007/s11616-021-00680-1 · Cited by 2
An In-Depth Analysis of the Perceived Stress Scale Among College Freshmen and Transfer Students
Lo, M., Perry, A. H., & Berenbon, R. (2022) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.2022.2034479 · Cited by 2
Independent Validation of the Video Game Dispositional Flow Scale With League of Legends Players
von Felten, N., Brühlmann, F., & Perrig, S. A. C. (2022) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3505270.3558351 · Cited by 2
Xiao, L., & Hau, K. (2023) · Applied Measurement in Education · doi:10.1080/08957347.2023.2172016 · Cited by 2
Roberson, A. J. (2019) · Louisiana State University · doi:10.31390/gradschool_dissertations.5051 · Cited by 4
Need Frustration and Short-Term Wellbeing: Restorative Experiences in Videogame Play
Tyack, A. (2019) · Queensland University of Technology · doi:10.5204/thesis.eprints.134476 · Cited by 4
Gwyn, W. G. (2020) · Research Bank (Australian Catholic University) · doi:10.26199/ekw9-2b57 · Cited by 2
Sharma, S. K., Sharma, E., & Mirosnik, K. (2021) · Research Square · doi:10.21203/rs.3.rs-433255/v1 · Cited by 2
Zhao, S., & Plonsky, L. (2024) · Elia · doi:10.12795/elia.2024.i24.8 · Cited by 1
Riaño-Hernández, D., Leenen, I., Ramírez-Conde, A., Atehortúa, P., & Piqueras, J. A. (2025) · PLoS ONE · doi:10.1371/journal.pone.0314488 · Cited by 1
Ventura‐León, J., Tocto‐Muñoz, S., Lino‐Cruz, C., Carbajal‐León, C., García-Godos, F., & Ortiz-Saenz, C. (2025) · Journal of Family Trauma Child Custody & Child Development · doi:10.1080/26904586.2025.2478079 · Cited by 1
Alfaro-Urquiola, A. L., Burgos‐Benavides, L., Ramírez, A., Díez, J. H., & Rodríguez‐Díaz, F. J. (2024) · Revista Latinoamericana de Psicología · doi:10.14349/rlp.2024.v56.28 · Cited by 2
Shi, B., Mao, Z., & Wang, S. (2025) · Journal of Quality Assurance in Hospitality & Tourism · doi:10.1080/1528008x.2025.2585329 · Cited by 1
Vera-Ruiz, G., Ballena-Descalzo, R. O., Bardales-Andrade, M. E., Villarreal-Huertas, D., Lázaro-Illatopa, W. I., Quispe-Callo, G., Dámaso-Román, A., Portugal, A. P. M., Relvas, A. P., Jiménez, J. S. F. G., et al. (2026) · Acta Psychologica · doi:10.1016/j.actpsy.2026.106304 · Cited by 1
Rojas‐Valverde, D., Becerra-Patiño, B. A., Sepulveda, R. A. Y., Calleja-Núñez, J. J., Uribe, J. D. P., & Bonilla, D. A. (2026) · Dermato · doi:10.3390/dermato6040043 · Cited by 1
Dimensionality of the Penn State Worry Questionnaire and its brief versions
Valencia, P. D., & Paredes-Ángeles, R. (2021) · Revista Argentina de Ciencias del Comportamiento ( RACC ) · doi:10.32348/1852.4206.v13.n2.27395 · Cited by 1
Zehtner, R. I., Baeurle, C. L., Walter, B., Stark, R. M., & Hermann, A. (2021) · Psychological Test Adaptation and Development · doi:10.1027/2698-1866/a000015 · Cited by 1
Research on test reliability in China’s mainland from 2001 to 2020
Zhonglin, W. E. N., Hongxi, C. H. E. N., Jie, F. A. N. G., Baojuan, Y. E., & Baozhen, C. A. I. (2022) · Advances in Psychological Science · doi:10.3724/sp.j.1042.2022.01682 · Cited by 1
Capuia, D., da Cruz, M., Masseca, A., Marques, E., Leite, P., Mangus, A. R., Webb, E. K., Ravichandran, C. T., Ressler, K. J., & Moreland-Capuia, A. R. (2024) · Journal of Traumatic Stress · doi:10.1002/jts.23062 · Cited by 1
Salas-Blas, E., Vallejos-Flores, M. A., la Cruz, G. C.-D., Medina, E., Fonseca‐Pedrero, E., Mayo‐Puchoc, N., & Copez-Lonzoy, A. J. E. (2025) · BMC Psychology · doi:10.1186/s40359-025-02437-1 · Cited by 1
On being nurtured while depressed
Moyle, W. (1997) · Cited by 1
Everyday Diplomacy: dealing with controversy online and face-to-face
Roos, C. A. (2022) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.230455324 · Cited by 2
Aubry, A. (2018) · theses.fr (ABES) · doi:10.70675/38a712a9ze5bfz4adez87b3z86fca53ec446 · Cited by 4
Intrinsic equivalence between developmental prosopagnosia questionnairesPreprint
Matsuyoshi, D., & Watanabe, K. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/267351
Stigma und subjektive Erfahrungen in der Früherkennung von Psychosen
Uttinger, M. (2017) · edoc (University of Basel) · doi:10.5451/unibas-006785248
Mental health outcomes and change processes in compassion focused therapy
Sommers‐Spijkerman, M. (2018) · doi:10.3990/1.9789036546652
Kennedy, S. J., Millin, P. M., & Kennedy, G. J. (2021) · Journal of Drug Issues · doi:10.1177/00220426211037267
The characteristics of occupational Tuberculosis risk in health care workersPreprint
Zein, R. A., Wulandari, R. D., Ridlo, I. A., Hendriani, W., Suhariadi, F., & Rianto, A. (2019) · INA-Rxiv (OSF Preprints) · doi:10.31227/osf.io/4qke5
Moscoso‐Porras, M., Villarreal‐Zegarra, D., Castillo, D. D., Zavaleta-Lopez, E., & Miranda, J. J. (2022) · Wellcome Open Research · doi:10.12688/wellcomeopenres.18397.1
A rede de crenças disfuncionais sobre o sono: uma reanálise estrutural da DBAS-16
do Carmo, M. M. I. D. B. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.5.2023.tde-27112023-155347
Insights in: Psychology for Clinical Settings 2022
Castelnuovo, G., Ginsberg, J. P., & Rocha, N. (2023) · Frontiers research topics · doi:10.3389/978-2-8325-4081-7
School‐based motivational interviewing: Change and sustain talks during goal setting
Qi, T., Strait, G. G., Roberson, A. J., & Terry, J. D. (2024) · Psychology in the Schools · doi:10.1002/pits.23313
Adaptation and validation of the stress control mindset measure in Italian university students
Pisanti, R., Soraci, P., Hagger, M. S., Hamilton, K., & Keech, J. J. (2024) · Current Psychology · doi:10.1007/s12144-024-06724-9
Correcting for Differences in Measurement Unreliability in Meta-Analysis of Variances
Jansen, K., & Nestler, S. (2025) · Multivariate Behavioral Research · doi:10.1080/00273171.2025.2469789
Fundamentalism and Hope in a Religiously Diverse Sample in the United States
Warlick, C. A., Alsayer, A. A., Shumate, C., & Templin, J. L. (2025) · Pastoral Psychology · doi:10.1007/s11089-025-01212-8 · Cited by 1
The Analytic Fluency Scale: Measuring the Skill of a Data Analyst
Vanaman, M., & Peng, R. D. (2026) · Collabra Psychology · doi:10.1525/collabra.163031
Zheng, M., Miroshnik, K. G., & Kaufman, J. C. (2025) · The Journal of Creative Behavior · doi:10.1002/jocb.70050
Maternal Ambivalence Scale (MAS): the russian adaptation and validization
Korgozha, M. A., & Швець, Д. В. (2025) · Counseling Psychology and Psychotherapy · doi:10.17759/cpp.2025330206
Mendoza, J., Gómez, D., Rivera, J. D. P., Valle, M., Pantoja, M. C. B., Coneo, A. M. C., & Mebarak, M. (2025) · European Journal of Psychological Assessment · doi:10.1027/1015-5759/a000934
Blahuš, P., Brown, B. L., Bunderson, C. V., & Olsen, J. A. (2025) · Wiley eBooks · doi:10.1002/9781119312246.oth1
Villarreal-Zegarra, D., Torres-Puente, R., Castillo-Blanco, R., Cabieses, B., Bellido-Boza, L., & Mezones-Holguin, E. (2021) · Figshare · doi:10.6084/m9.figshare.16864913
Suzman, E., Williams, Z. J., Feldman, J. I., Failla, M. D., Cascio, C. J., Wallace, M., Niarchou, M., Sutcliffe, J. S., Wodka, E. L., & Woynaroski, T. G. (2021) · Figshare · doi:10.6084/m9.figshare.14746765
Zamora, Á., Desdentado, L., Herrero, R., Miragall, M., & Baños, R. (2024) · Figshare · doi:10.6084/m9.figshare.25698296
Attachment in Mentoring RelationshipsPreprint
Zhang, F., & Robertson, M. M. (2021) · OSF Preprints (OSF Preprints) · doi:10.17605/osf.io/v86wq
Villarreal-Zegarra, D., Torres-Puente, R., Castillo-Blanco, R., Cabieses, B., Bellido-Boza, L., & Mezones-Holguin, E. (2021) · Figshare · doi:10.6084/m9.figshare.16864910
Villarreal-Zegarra, D., Torres-Puente, R., Castillo-Blanco, R., Cabieses, B., Bellido-Boza, L., & Mezones-Holguin, E. (2021) · Figshare · doi:10.6084/m9.figshare.16864907
Context and uncertainty in decisions from experience
Holwerda, J. (2023) · UNSWorks (University of New South Wales, Sydney, Australia) · doi:10.26190/unsworks/24747
Anàlisi de les reaccions adaptatives del sistema nerviós autònom
Cabrera Castro, A. (2023) · Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona)
Salisu, I. (2024) · UUM Electronic Theses and Dissertation [eTheses] (Northern University of Malaysia)
Gestión de representación política y el desarrollo sostenible, región San Martín - 2023
Tomanguillo, N. D. P. P. (2023) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas)
On Monte Carlo simulation algorithms for research in psychometrics
Astivia, O. L. O. (2017) · cIRcle (University of British Columbia)
Kwan, H. M. (2024) · University Of Wales Trinity Saint David Research Repository (University of Wales Trinity Saint David) · doi:10.82227/repository.uwtsd.ac.uk.00003194
Chalupny, U., Meier, P. V., Schärer, C., Silva, D. V., & Christ, O. (2026) · Frontiers in Virtual Reality · doi:10.3389/frvir.2026.1958188
Siitonen, P., Honkapuro, S., Annala, S., & Wolff, A. (2026) · Environment and Behavior · doi:10.1177/00139165261419601
Chen, D., Feng, S., Leng, Y., Zhou, G., Hofmann, S. G., & Liu, X. (2026) · BMC Psychology · doi:10.1186/s40359-026-04176-3
Measuring Teacher Self-Efficacy in the Moral Work of Teaching
Osguthorpe, R. D., Clark, S., & Sudweeks, R. R. (2023) · Journal of Character Education · doi:10.1108/jced-11-2023-0006
Silva-Dominguez, D. S., Salazar, J. O., Ochoa, P. C., & Juárez, C. P. (2026) · Revista de la Facultad de Medicina Humana · doi:10.25176/rfmh.v26i1.8039
Hoffmann, M. (2025) · theses.fr (ABES) · doi:10.70675/1b9ca3d9z3352z4d23za977zff9e4885f911
Wijk, C. H. V., Muller, C. J. B., & Andrews, B. N. (2026) · South African Journal of Psychiatry · doi:10.4102/sajpsychiatry.v32i0.2578
Marchetti, D., Fontanesi, L., Giandomenico, S. D., Facchino, A. P., & Verrocchio, M. C. (2026) · Journal of Marital and Family Therapy · doi:10.1111/jmft.70139
Argyrides, M., Todd, J., Reppa, G. P., & Swami, V. (2026) · Body Image · doi:10.1016/j.bodyim.2026.102110
Wijk, C. H. V. (2026) · Psychological Reports · doi:10.1177/00332941261464049
Wijk, C. H. V., & Muller, C. J. B. (2026) · African Journal of Psychological Assessment · doi:10.4102/ajopa.v8i0.195
Trotta, F., Petrosino, F., Napolitano, D., Cascio, A. L., Cilluffo, S., Bozzetti, M., Pucciarelli, G., Vellone, E., & Bartoli, D. (2026) · Gastroenterology Nursing · doi:10.1097/sga.0000000000000995
Erol, R., & ÖZDEMİR, A. (2026) · Eğitim Ve İnsani Bilimler Dergisi Teori Ve Uygulama · doi:10.58689/eibd.1948248
dos Santos, G. M. X., Camolez, M. M., Mauad, L. D. D. S., Secco, M., Sugano, R. M. M., Rezende, G., Lozano‐Lozano, M., & Fonseca, M. D. C. R. (2026) · Research Square · doi:10.21203/rs.3.rs-10693543/v1
DMAR: Design, Measurement, and Analysis in R (DMAR)
Kelley, K. (2026) · doi:10.32614/cran.package.dmar
Self-citationAlnofei, A. A. A. (2023) · مجلة الفتح للبحوث التربوية والنفسية
Measurement and Spatiotemporal Analysis of Urban Innovation Capacity in China
Song, P., 张进峰, & Zhao, Z. (2026) · SAGE Open · doi:10.1177/21582440261494822
PreWorkS : A Precarious Work Scale for the Cultural and Creative Industries
Piliousis, A., & Baltzis, A. (2026) · Empirical Studies of the Arts · doi:10.1177/02762374261496214
Exploring Visual Access and Proactive AI Tutoring for Handwritten MathematicsPreprint
Kotalwar, N., Borchers, C., Gandhi, S., Purkayastha, P. S., Yaron, D., & Rosé, C. P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.26411
(2026) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-80791-470-720261011
Metadata
- Authors
- Kelley, K., & Pornprasertmanit, S.
- Affiliations
- University of Notre Dame (US)
- Texas Tech University (US)
- Venue
- Psychological Methods
- Published
- 2016
- Volume
- 21
- Issue
- 1
- Pages
- 69-92
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2295825737
- Cited by
- 259
- References
- 80
- Citations by year
- 2026: 20 · 2025: 17 · 2024: 27 · 2023: 31 · 2022: 35 · 2021: 38 · 2020: 29 · 2019: 22 · 2018: 22 · 2017: 13 · 2016: 4
- FWCI
- 22.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Notre Dame Deloitte Center for Ethical Leadership
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- confidence intervals, composite score, Cronbach's alpha, bootstrap confidence intervals, reliability estimation, Monte Carlo simulation, R software
- MeSH terms
- Humans, Psychometrics, Reproducibility of Results, Confidence Intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.