Impact of the Number of Response Categories and Anchor Labels on Coefficient Alpha and Test-Retest Reliability
The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
A total of 1,247 college students participated in this study on the effect of scale format on the reliability of Likert-type rating scales. The number of response categories ranged from 3 to 9. Anchor labels on the scales were provided for each response option or for the end points only. The results indicated that the scales with fewresponse categories tended to result in lower reliability, especially lower test-retest reliability. The scales with all the response options clearly labeled were likely to yield higher test-retest reliability than those with only the end points labeled. Scale design that leads to consistent participant responses as indicated by test-retest reliability should be preferred.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
What is coefficient alpha? An examination of theory and applications
Cortina, J. M. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.1.98 · Cited by 8,810 · cited 5×
“Because high internal consistency reliability, as commonly measured by coefficient α, does not necessarily guarantee high test-retest reliability (Cortina, 1993; Crocker & Algina, 1986), the impact of scale design on test-retest reliability warrants direct investigation.”
BackgroundMethodResultInfluentialIntroduction to Classical and Modern Test Theory
Crocker, L. M., & Algina, J. J. (1986) · Cited by 5,090
The Psychology of Survey Response
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) · doi:10.1017/cbo9780511819322 · Cited by 4,734
Preston, C. C., & Colman, A. M. (2000) · Acta Psychologica · doi:10.1016/s0001-6918(99)00050-5 · Cited by 2,106
Wright, B. D., & Masters, G. N. (1982) · ACEReSearch Repository (Australian Council for Educational Research) · Cited by 1,930
Krosnick, J. A. (1999) · Annual Review of Psychology · doi:10.1146/annurev.psych.50.1.537 · Cited by 1,852
Techniques of attitude scale construction.
Edwards, A. L. (1957) · Appleton-Century-Crofts eBooks · doi:10.1037/14423-000 · Cited by 1,593
The unicorn, the normal curve, and other improbable creatures.
Micceri, T. (1989) · Psychological Bulletin · doi:10.1037//0033-2909.105.1.156 · Cited by 1,485
Educational research, methodology, and measurement: an international handbook
(1998) · Choice Reviews Online · doi:10.5860/choice.35-3089 · Cited by 716
Is There an Optimal Number of Alternatives for Likert Scale Items? Study I: Reliability and Validity
Matell, M. S., & Jacoby, J. (1971) · Educational and Psychological Measurement · doi:10.1177/001316447103100307 · Cited by 571
Effect of the number of scale points on reliability: A Monte Carlo approach.
Lissitz, R. W., & Green, S. B. (1975) · Journal of Applied Psychology · doi:10.1037/h0076268 · Cited by 488
Educational research, methodology, and measurement: an international handbook
(1989) · Choice Reviews Online · doi:10.5860/choice.26-5427 · Cited by 432
Research Design Effects on the Reliability of Rating Scales: A Meta-Analysis
Churchill, G. A., & Peter, J. P. (1984) · Journal of Marketing Research · doi:10.1177/002224378402100402 · Cited by 367
Reliability Methods: A Note on the Frequency of Use of Various Types
Hogan, T. P., Benjamin, A., & Brezinski, K. L. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970691 · Cited by 258
A K-Sample Significance Test for Independent Alpha Coefficients
Hakstian, A. R., & Whalen, T. E. (1976) · Psychometrika · doi:10.1007/bf02291840 · Cited by 257
Comparisons of Party Identification and Policy Preferences: The Impact of Survey Question Format
Krosnick, J. A., & Berent, M. K. (1993) · American Journal of Political Science · doi:10.2307/2111580 · Cited by 244
Educational and Psychological Measurement
Finch, W. H., & French, B. F. (2018) · Routledge eBooks · doi:10.4324/9781315650951 · Cited by 238
A Simple and Reliable Method of Scoring the Thurstone Attitude Scales
Likert, R., Roslow, S., & Murphy, G. (1934) · The Journal of Social Psychology · doi:10.1080/00224545.1934.9919450 · Cited by 216
On the loss of reliability in ratings due to coarseness of the scale
Symonds, P. M. (1924) · Journal of Experimental Psychology · doi:10.1037/h0074469 · Cited by 194 · cited 5×
“Why would the number of response categories be expected to affect test reliability? In considering the coarseness of categorization, Symonds (1924) described a parallel between psychological measurements and physical measurements.”
BackgroundResultInfluentialNumber of Scale Points and the Reliability of Scales
Komorita, S. S., & Graham, W. K. (1965) · Educational and Psychological Measurement · doi:10.1177/001316446502500404 · Cited by 190
A Monte Carlo study of factors affecting three indices of composite scale reliability.
Jenkins, G. D., & Taber, T. D. (1977) · Journal of Applied Psychology · doi:10.1037//0021-9010.62.4.392 · Cited by 170
Effects of Some Variations in Rating Scale Characteristics on the Means and Reliabilities of Ratings
Finn, R. H. (1972) · Educational and Psychological Measurement · doi:10.1177/001316447203200203 · Cited by 164
MASTERS, E. R. (1974) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1974.tb00970.x · Cited by 144
Johnson, S. M., Smith, P. C., & Tucker, S. M. (1982) · Journal of Applied Psychology · doi:10.1037/0021-9010.67.4.500 · Cited by 136
A comparison of the behaviorally anchored rating and mixed standard scale formats.
Dickinson, T. L., & Zellinger, P. M. (1980) · Journal of Applied Psychology · doi:10.1037//0021-9010.65.2.147 · Cited by 108
Determinants of Scale Response: Label versus Position
Wildt, A. R., & Mazis, M. B. (1978) · Journal of Marketing Research · doi:10.1177/002224377801500209 · Cited by 105
The Effects of Nonnormality and Number of Response Categories on Reliability
Bandalos, D. L., & Enders, C. K. (1996) · Applied Measurement in Education · doi:10.1207/s15324818ame0902_4 · Cited by 100
Kamphaus, R. W. (1994) · Contemporary Psychology · doi:10.1037/034494 · Cited by 98
Number of Alternatives per Choice Point and Stability of Likert-Type Scales
Oaster, T. R. F. (1989) · Perceptual and Motor Skills · doi:10.2466/pms.1989.68.2.549 · Cited by 97
ANCHOR POINT EFFECTS ON THE EQUIVALENCE OF QUESTIONNAIRE ITEMS
Lam, T. C. M., & Klockars, A. J. (1982) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1982.tb00137.x · Cited by 93
Bendig, A. W. (1953) · Journal of Applied Psychology · doi:10.1037/h0057911 · Cited by 91
Response Differences and Preferences for All-Category-Defined and End-Defined Likert Formats
Dixon, P. N., Bobo, M., & Stevick, R. A. (1984) · Educational and Psychological Measurement · doi:10.1177/0013164484441006 · Cited by 74
The Influence of Labels and Positions in Rating Scales
Klockars, A. J., & Yamagishi, M. (1988) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1988.tb00294.x · Cited by 71
Optimal refinement of the rating scale.
Champney, H., & Marshall, H. (1939) · Journal of Applied Psychology · doi:10.1037/h0054522 · Cited by 71
Number of Response Categories and Statistics on a Teacher Rating Scale
Aiken, L. R. (1983) · Educational and Psychological Measurement · doi:10.1177/001316448304300209 · Cited by 44
Effects of Verbally Labeled Anchor Points on the Distributional Parameters of Rating Measures
French‐Lazovik, G., & Gibson, C. L. (1984) · Applied Psychological Measurement · doi:10.1177/014662168400800106 · Cited by 41
Effects of Number and Type of Response Choices on Internal Consistency Reliability
Halpin, G., Halpin, G., & Arbet, S. (1994) · Perceptual and Motor Skills · doi:10.2466/pms.1994.79.2.928 · Cited by 14
Differences between odd and even number of response scale: Some empirical evidence.
Wong, C., Tam, K.-C., Fung, M.-Y., & Wan, K. K.-H. (1993) · Cited by 12
SCALING ISSUES IN FACULTY EVALUATIONS
Landrum, R. E. (1999) · Psychological Reports · doi:10.2466/pr0.1999.84.1.178 · Cited by 4
Cited by
Developing questionnaires for educational research: AMEE Guide No. 87
Artino, A. R., La Rochelle, J. S., Dezee, K. J., & Gehlbach, H. (2014) · Medical Teacher · doi:10.3109/0142159x.2014.889814 · Cited by 1,450
Varni, J. W., Limbers, C. A., & Burwinkle, T. M. (2007) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-5-1 · Cited by 1,075
Effect of the Number of Response Categories on the Reliability and Validity of Rating Scales
Lozano, L. M., García-Cueto, E., & Muñiz, J. (2008) · Methodology · doi:10.1027/1614-2241.4.2.73 · Cited by 843
Weijters, B., Cabooter, E., & Schillewaert, N. (2010) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2010.02.004 · Cited by 801
A Review of Key Likert Scale Development Advances: 1995–2019
Jebb, A. T., Ng, V., & Tay, L. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.637547 · Cited by 770
Metadata
- Authors
- Weng, L.-J.
- Venue
- Educational and Psychological Measurement
- Publisher
- SAGE Publications
- Published
- 2004-12
- Volume
- 64
- Issue
- 6
- Pages
- 956-972
- ISSN
- 0013-1644 (ISSN-L)
- 1552-3888
- OpenAlex
- W2137071805
- Cited by
- 410
- References
- 42
- FWCI
- 8.91
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.