Rating the ratings: Assessing the psychometric quality of rating data.

Abstract

Reviews research that is concerned with evaluating the psychometric qualities of data in the form of ratings (rating errors) and that has been plagued with conceptual and operational confusion and inconsistency. Following a brief historical survey, inconsistencies in definitions, quantifications, and methodologies are documented in a review of more than 20 relevant articles published in Journal of Applied Psychology, Organizational Behavior and Human Performance, and Personnel Psychology (1975-1977). Empirical implications of these inconsistencies are discussed, and a revised typology of rating criteria, combined with a multivariate analytic approach, is suggested. (65 ref) (PsycINFO Database Record (c) 2006 APA, all rights reserved)

Cites (49)

1. Statistical principles in experimental design.
   Winer, B. J. · 1962 · McGraw-Hill Book Company
2. Intraclass correlations: Uses in assessing rater reliability.
   Shrout, Patrick E., Fleiss, Joseph L. · 1979 · Psychological Bulletin
3. A constant error in psychological ratings.
   Thorndike, E.L. · 1920 · Journal of Applied Psychology
4. The halo effect: Evidence for unconscious alteration of judgments.
   Nisbett, Richard E., Wilson, Timothy D. · 1977 · Journal of Personality and Social Psychology
5. Book Reviews: Cronbach, Lee J., Gleser, Goldine C., Nanda, Harinder, and Rajaratnam, Nageswari. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. New York: John Wiley and Sons, Inc., 1972. 410 + xix pp. $12.95.
   Philip R. Merrifield, Lee J. Cronbach, Goldine C. Gleser, Harinder Nanda, Nageswari Rajaratnam · 1974 · American Educational Research Journal
6. Estimation of the reliability of ratings.
   Ebel, Robert L. · 1951 · Psychometrika
7. Development of a general solution to the problem of validity generalization.
   Schmidt, Frank L., Hunter, John E. · 1977 · Journal of Applied Psychology
8. Implicit personality theory: A review.
   Schneider, David J. · 1973 · Psychological Bulletin
9. The multitrait-multirater approach to measuring managerial job performance.
   Lawler, Edward E. · 1967 · Journal of Applied Psychology
10. Training managers to minimize rating errors in the observation of behavior.
    Gary P. Latham, Kenneth N. Wexley, Elliott D. Pursell · 1975 · Journal of Applied Psychology

Show all 49 references

Cited by (567)

1. Externalizing behavior problems and academic underachievement in childhood and adolescence: Causal relationships and underlying mechanisms.
   Stephen P. Hinshaw · 1992 · Psychological Bulletin
2. On the distinction between attentional deficits/hyperactivity and conduct problems/aggression in child psychopathology.
   Hinshaw, Stephen P. · 1987 · Psychological Bulletin
3. Community readiness: Research to practice
   Edwards, Ruth W., Jumper-Thurman, Pamela, Plested, Barbara A., Oetting, Eugene R., Swanson, Louis · 2000 · Journal of Community Psychology
4. The criterion problem: 1917-1992.
   Austin, James T., Villanova, Peter · 1992 · Journal of Applied Psychology
5. PROJECT A VALIDITY RESULTS: THE RELATIONSHIP BETWEEN PREDICTOR AND CRITERION DOMAINS
   MCHENRY, JEFFREY J., HOUGH, LEAETTA M., TOQUAM, JODY L., HANSON, MARY ANN, ASHWORTH, STEVEN · 1990 · Personnel Psychology

See all 567

Related (10)

1. Effects of rater training and diary-keeping on psychometric error in ratings.
   H. John Bernardin, C. S. Walter · 1977 · Journal of Applied Psychology
2. A constant error in psychological ratings.
   Thorndike, E.L. · 1920 · Journal of Applied Psychology
3. Evaluation von Beurteilungen:
   Thomas Eckes · 2005 · Zeitschrift für Psychologie
4. Longitudinal study of the effects of rater training on psychometric error in ratings.
   John M. Ivancevich · 1979 · Journal of Applied Psychology
5. What's wrong with rating scales? Psychology's replication and confidence crisis cannot be solved without transparency in data generation
   Jana Uher · 2023 · Social and Personality Psychology Compass

See all