Rater bias in psychological research: When is it a problem and what can we do about it?
Abstract
Rater bias is a substantial source of error in psychological research. Bias distorts observed effect sizes beyond the expected level of attenuation due to intrarater error, and the impact of bias is not accurately estimated using conventional methods of correction for attenuation. Using a model based on multivariate generalizability theory, this article illustrates how bias affects research results. The model identifies 4 types of bias that may affect findings in research using observer ratings, including the biases traditionally termed leniency and halo errors. The impact of bias depends on which of 4 classes of rating design is used, and formulas are derived for correcting observed effect sizes for attenuation (due to bias variance) and inflation (due to bias covariance) in each of these classes. The rater bias model suggests procedures for researchers seeking to minimize adverse impact of bias on study findings.
Cites (36)
- Statistical power analysis for the behavioral sciences1990 · Computers, Environment and Urban Systems
- Untitled work1998
- Intraclass correlations: Uses in assessing rater reliability.Shrout, Patrick E., Fleiss, Joseph L. · 1979 · Psychological Bulletin
- Structural Equations with Latent Variables.Clogg, Clifford C., Bollen, Kenneth A. · 1991 · Contemporary Sociology
- Convergent and discriminant validation by the multitrait-multimethod matrix.Campbell, Donald T., Fiske, Donald W. · 1959 · Psychological Bulletin
- A power primer.Cohen, Jacob · 1992 · Psychological Bulletin
- Methods of Meta-Analysis: Correcting Error and Bias in Research FindingsSchmidt, Frank L., Hunter, John E. · 2015
- The Handbook of Research Synthesis.Howel, D., Cooper, H., Hedges, L. V. · 1995 · Biometrics
- A constant error in psychological ratings.Thorndike, E.L. · 1920 · Journal of Applied Psychology
- The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and ProfilesMerrifield, Philip R., Cronbach, Lee J., Gleser, Goldine C., Nanda, Harinder, Rajaratnam, Nageswari · 1974 · American Educational Research Journal
Cited by (22)
- A meta-analysis of the five-factor model of personality and academic performance.Poropat, Arthur E. · 2009 · Psychological Bulletin
- A Meta-analysis of the Effectiveness of Intervention Programs Designed to Support AutonomySu, Yu-Lan, Reeve, Johnmarshall · 2011 · Educational Psychology Review
- Charisma, positive emotions and mood contagionBono, Joyce E., Ilies, Remus · 2006 · The Leadership Quarterly
- Classroom Climate and Contextual Effects: Conceptual and Methodological Issues in the Evaluation of Group-Level EffectsMarsh, Herbert W., Lüdtke, Oliver, Nagengast, Benjamin, Trautwein, Ulrich, Morin, Alexandre J. S., Abduljabbar, Adel S., Köller, Olaf · 2012 · Educational Psychologist
- Planned missing data designs in psychological research.Graham, John W., Taylor, Bonnie J., Olchowski, Allison E., Cumsille, Patricio E. · 2006 · Psychological Methods
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Rater bias in psychological research: When is it a problem and what can we do about it? https://app.cochat.ai/papers/10.1037//1082-989x.5.1.64
- The paper
- Hoyt, W. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods. https://doi.org/10.1037//1082-989x.5.1.64
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.