An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales.

Borman, Walter C., Buck, Daren E., Hanson, Mary Ann, Motowidlo, Stephan J., Stark, Stephen, Drasgow, Fritz·2001·Journal of Applied Psychology

Abstract

This laboratory research compared the reliability, validity, and accuracy of a computerized adaptive rating scale (CARS) format and 2 relatively common and representative rating formats. The CARS is a paired-comparison rating task that uses adaptive testing principles to present pairs of scaled behavioral statements to the rater to iteratively estimate a ratee's effectiveness on 3 dimensions of contextual performance. Videotaped vignettes of 6 office workers were prepared, depicting prescripted levels of contextual performance, and 112 subjects rated these vignettes using the CARS format and one or the other competing format. Results showed 23%-37% lower standard errors of measurement for the CARS format. In addition, validity was significantly higher for the CARS format (d = .18), and Cronbach's accuracy coefficients showed significantly higher accuracy, with a median effect size of .08. The discussion focuses on possible reasons for the results.

Cites (2)

  1. Intraclass correlations: Uses in assessing rater reliability.
    Shrout, Patrick E., Fleiss, Joseph L. · 1979 · Psychological Bulletin
  2. The critical incident technique.
    Flanagan, John C. · 1954 · Psychological Bulletin

Show all 2 references

Cited by (5)

  1. A New Model of Work Role Performance: Positive Behavior in Uncertain and Interdependent Contexts
    Griffin, Mark A., Neal, Andrew, Parker, Sharon K. · 2007 · Academy of Management Journal
  2. The Great Eight Competencies: A Criterion-Centric Approach to Validation.
    Bartram, Dave · 2005 · Journal of Applied Psychology
  3. Citizenship and Counterproductive Behavior: Clarifying Relations Between the Two Domains
    Sackett, Paul R., Berry, Christopher M., Wiemann, Shelly A., Laczo, Roxanne M. · 2006 · Human Performance
  4. The Concept of Organizational Citizenship
    Borman, Walter C. · 2004 · Current Directions in Psychological Science
  5. The Restriction of Variance Hypothesis and Interrater Reliability and Agreement: Are Ratings from Multiple Sources Really Dissimilar?
    Lebreton, James M., Burgess, Jennifer R. D., Kaiser, Robert B., Atchley, E. Kate, James, Lawrence R. · 2003 · Organizational Research Methods

See all 166

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. https://app.cochat.ai/papers/10.1037//0021-9010.86.5.965
The paper
Borman, W. C., Buck, D. E., Hanson, M. A., Motowidlo, S. J., Stark, S., & Drasgow, F. (2001). An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. Journal of Applied Psychology. https://doi.org/10.1037//0021-9010.86.5.965

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.