An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales.
Abstract
This laboratory research compared the reliability, validity, and accuracy of a computerized adaptive rating scale (CARS) format and 2 relatively common and representative rating formats. The CARS is a paired-comparison rating task that uses adaptive testing principles to present pairs of scaled behavioral statements to the rater to iteratively estimate a ratee's effectiveness on 3 dimensions of contextual performance. Videotaped vignettes of 6 office workers were prepared, depicting prescripted levels of contextual performance, and 112 subjects rated these vignettes using the CARS format and one or the other competing format. Results showed 23%-37% lower standard errors of measurement for the CARS format. In addition, validity was significantly higher for the CARS format (d = .18), and Cronbach's accuracy coefficients showed significantly higher accuracy, with a median effect size of .08. The discussion focuses on possible reasons for the results.
Cites (2)
- Intraclass correlations: Uses in assessing rater reliability.Shrout, Patrick E., Fleiss, Joseph L. · 1979 · Psychological Bulletin
- The critical incident technique.Flanagan, John C. · 1954 · Psychological Bulletin
Cited by (5)
- A New Model of Work Role Performance: Positive Behavior in Uncertain and Interdependent ContextsGriffin, Mark A., Neal, Andrew, Parker, Sharon K. · 2007 · Academy of Management Journal
- The Great Eight Competencies: A Criterion-Centric Approach to Validation.Bartram, Dave · 2005 · Journal of Applied Psychology
- Citizenship and Counterproductive Behavior: Clarifying Relations Between the Two DomainsSackett, Paul R., Berry, Christopher M., Wiemann, Shelly A., Laczo, Roxanne M. · 2006 · Human Performance
- The Concept of Organizational CitizenshipBorman, Walter C. · 2004 · Current Directions in Psychological Science
- The Restriction of Variance Hypothesis and Interrater Reliability and Agreement: Are Ratings from Multiple Sources Really Dissimilar?Lebreton, James M., Burgess, Jennifer R. D., Kaiser, Robert B., Atchley, E. Kate, James, Lawrence R. · 2003 · Organizational Research Methods
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. https://app.cochat.ai/papers/10.1037//0021-9010.86.5.965
- The paper
- Borman, W. C., Buck, D. E., Hanson, M. A., Motowidlo, S. J., Stark, S., & Drasgow, F. (2001). An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. Journal of Applied Psychology. https://doi.org/10.1037//0021-9010.86.5.965
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.