Answers to 20 Questions About Interrater Reliability and Interrater Agreement
The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
The use of interrater reliability (IRR) and interrater agreement (IRA) indices has increased dramatically during the past 20 years. This popularity is, at least in part, because of the increased role of multilevel modeling techniques (e.g., hierarchical linear modeling and multilevel structural equation modeling) in organizational research. IRR and IRA indices are often used to justify aggregating lower-level data used in composition models. The purpose of the current article is to expose researchers to the various issues surrounding the use of IRR and IRA indices often used in conjunction with multilevel models. To achieve this goal, the authors adopt a question-and-answer format and provide a tutorial in the appendices illustrating how these indices may be computed using the SPSS software.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For the 20-comment reliability subset (5 repetitions each), we computed intraclass correlations for each DV, adopting ICC ≥ 0 . 70 as the threshold for adequate reliability [59].”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [59]
In the citing paper
“This cutoff is a heuristic rather than an absolute standard; the appropriate threshold depends on the decisions being made [59].”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [59]
In the citing paper
“High-stakes decisions about specific individuals require strong agreement, whereas research estimating aggregate effects may proceed with moderate agreement [59].”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [59]
In the citing paper
“We interpret these values in light of LeBreton and Senter [59]'s guidance that reliability standards should be commensurate with the intensity of decisions being made. High-stakes decisions about specific individuals (e.g., hiring, promotion) require strong agreement (ICC > 0 . 90 ), whereas research estimating aggregate group-level effects may proceed with moderate agreement.”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [59]
In the citing paper
“The two raters' reliability [ICC (2) = .92]and agreement (average deviation = .26)were within conventional guidelines [90].”
From Absorbed in technology but digitally overloaded: Interplay effects on gig workers’ burnout and creativity (2021) · cites this paper as [90]
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology · doi:10.2307/2075823 · Cited by 18,724
Review of Psychometric theory (2nd ed.).
SITGREAVES, R. (1979) · Contemporary Psychology: A Journal of Reviews · doi:10.1037/018882 · Cited by 11,425
Statistical Theories of Mental Test Scores
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,022
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774 · cited 24×
“The rWG and rWGðJÞ indices were initially introduced by James et al. (1984).”
BackgroundMethodResultInfluential(2000) · Cited by 3,573
Chan, D. K. (1998) · Journal of Applied Psychology · doi:10.1037//0021-9010.83.2.234 · Cited by 2,972
(2000) · Cited by 2,531
rwg: an assessment of within-group interrater agreement
James, L. R., Demaree, R. G., & Wolf, G. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.2.306 · Cited by 2,221 · cited 5×
“Schmidt and Hunter (1989) critiqued the rWG and rWGðJÞ indices, largely based on semantic confusion arising from earlier writers’ labels of the rWG indices as reliability coefficients (James et al., 1984) versus agreement coefficients (James et al., 1993; Kozlowski & Hattrup, 1992).”
BackgroundMethodInfluentialGulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Aggregation bias in estimates of perceptual agreement.
James, L. R. (1982) · Journal of Applied Psychology · doi:10.1037/0021-9010.67.2.219 · Cited by 2,139 · cited 7×
“ICC(1) values may be interpreted as the level of consensus + consistency one would expect if a judge was randomly selected from the population of judges and his or her scores were compared to the mean score (i.e., estimated true score) obtained from the sample of judges (Bliese, 2000; James, 1982).”
BackgroundMethodResultInfluentialThe Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053 · cited 5×
“The commonly used cut point of .70 is by no means a definitive rule; it is simply a heuristic that is frequently (and inappropriately) invoked in research (Lance et al., 2006).”
BackgroundMethodInfluentialLeader-Member Exchange Model of Leadership: A Critique and Further Development
Dienesch, R. M., & Liden, R. C. (1986) · The Academy of Management Review · doi:10.2307/258314 · Cited by 2,006
Personality, affect, and behavior in groups
George, J. M. (1990) · Journal of Applied Psychology · doi:10.1037/0021-9010.75.2.107 · Cited by 1,372 · cited 6×
“Concerning the rWG indices, this value was first reported by George (1990), who obtained it from a personal communication with Lawrence R. James, one of the creators of rWG. LeBreton et al. (2003) recently offered an expanded explanation for the .”
BackgroundMethodInfluentialOrganizational climate: A review of theory and research.
James, L. R., & Jones, A. P. (1974) · Psychological Bulletin · doi:10.1037/h0037511 · Cited by 1,089
Understanding performance appraisal: social, organizational, and goal-based perspectives
(1995) · Choice Reviews Online · doi:10.5860/choice.32-5748 · Cited by 1,088
Beyond attribution theory: Cognitive processes in performance appraisal.
Feldman, J. M. (1981) · Journal of Applied Psychology · doi:10.1037//0021-9010.66.2.127 · Cited by 870
Errors and mistakes: Evaluating the accuracy of social judgment.
Funder, D. C. (1987) · Psychological Bulletin · doi:10.1037//0033-2909.101.1.75 · Cited by 869
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037//0033-2909.83.5.762 · Cited by 825
Statistical power analysis: a simple and general model for traditional and modern hypothesis tests
(1998) · Choice Reviews Online · doi:10.5860/choice.36-1020 · Cited by 792
Climate strength: a new direction for climate research.
Schneider, B., Salvaggio, A. N., & Subirats, M. (2002) · Journal of Applied Psychology · doi:10.1037//0021-9010.87.2.220 · Cited by 775
The application of hierarchical linear modeling to organizational research.
Hofmann, D. A., Griffin, M. A., & Gavin, M. B. (2000) · Cited by 673
Group Size, ICC Values, and Group-Level Correlations: A Simulation
Bliese, P. D. (1998) · Organizational Research Methods · doi:10.1177/109442819814001 · Cited by 636 · cited 6×
“Basically, the values obtained for the ICC(K) and ICC(A,K) asymptotically approach the values one would obtain by placing the ICC(1) and the ICC(A,1) into the SpearmanBrown prophesy formula (the correction factor being K, the number of raters; Bliese, 1998, 2000).”
BackgroundMethodInfluentialLindell, M. K., & Brandt, C. J. (2000) · Journal of Applied Psychology · doi:10.1037//0021-9010.85.3.331 · Cited by 558
A disagreement about within-group agreement: disentangling issues of consistency versus consensus
Kozlowski, S. W., & Hattrup, K. (1992) · Journal of Applied Psychology · doi:10.1037/0021-9010.77.2.161 · Cited by 514
Comparative analysis of the reliability of job performance ratings.
Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996) · Journal of Applied Psychology · doi:10.1037/0021-9010.81.5.557 · Cited by 505
Cooper, W. H. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.90.2.218 · Cited by 492
On Average Deviation Indices for Estimating Interrater Agreement
Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999) · Organizational Research Methods · doi:10.1177/109442819921004 · Cited by 462
Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide
Burke, M. J., & Dunlap, W. P. (2002) · Organizational Research Methods · doi:10.1177/1094428102005002002 · Cited by 447
Organizational Research Methods
Brewerton, P., & Millward, L. (2001) · SAGE Publications Ltd eBooks · doi:10.4135/9781849209533 · Cited by 443
Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.1.108 · Cited by 411
An examination of the antecedents and moderator influences of climate strength.
González‐Romá, V., Silla, J. M. P., & Tordera, N. (2002) · Journal of Applied Psychology · doi:10.1037//0021-9010.87.3.465 · Cited by 394
AN EXPLORATION OF CLIMATES FOR TECHNICAL UPDATING AND PERFORMANCE
KOZLOWSKI, S. W. J., & HULTS, B. M. (1987) · Personnel Psychology · doi:10.1111/j.1744-6570.1987.tb00614.x · Cited by 316
Performance Appraisal: Alternative Perspectives
Dobbins, G., & Cardy, R. L. (1993) · Medical Entomology and Zoology · Cited by 304
Interrater Agreement Reconsidered: An Alternative to the rwg Indices
Brown, R. D., & Hauenstein, N. M. A. (2005) · Organizational Research Methods · doi:10.1177/1094428105275376 · Cited by 297
A Revised Index of Interrater Agreement for Multi-Item Ratings of a Single Target
Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999) · Applied Psychological Measurement · doi:10.1177/01466219922031257 · Cited by 255
Lebreton, J. M., Burgess, J. R. D., Kaiser, R. B., Atchley, E. K., & James, L. R. (2003) · Organizational Research Methods · doi:10.1177/1094428102239427 · Cited by 254
Self-citationJob behavior, performance, and effectiveness.
Borman, W. C. (1991) · Digital Commons - University of South Florida (University of South Florida) · Cited by 241
Performance Appraisal Process Research in the 1980s: What Has It Contributed to Appraisals in Use?
Ilgen, D. R., Barnes-Farrell, J. L., & McKellin, D. B. (1993) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1993.1015 · Cited by 226
Sulsky, L. M., & Balzer, W. K. (1988) · Journal of Applied Psychology · doi:10.1037//0021-9010.73.3.497 · Cited by 224
Dunlap, W. P., Burke, M. J., & Smith-Crowe, K. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.2.356 · Cited by 219
INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS
MURPHY, K. R., & DESHON, R. (2000) · Personnel Psychology · doi:10.1111/j.1744-6570.2000.tb02421.x · Cited by 183
Rater motivation in the performance appraisal context: A theoretical framework
Harris, M. (1994) · Journal of Management · doi:10.1016/0149-2063(94)90028-0 · Cited by 174
Rater errors and rating accuracy.
Murphy, K. R., & Balzer, W. K. (1989) · Journal of Applied Psychology · doi:10.1037//0021-9010.74.4.619 · Cited by 170
Statistical properties of the rWG(J) index of agreement.
Cohen, A., Doveh, E., & Eick, U. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.3.297 · Cited by 166
Lindell, M. K., & Brandt, C. J. (1999) · Journal of Applied Psychology · doi:10.1037/0021-9010.84.4.640 · Cited by 131
INTERPERSONAL AFFECT AND PERFORMANCE APPRAISAL: A FIELD STUDY
VARMA, A., DENISI, A. S., & PETERS, L. H. (1996) · Personnel Psychology · doi:10.1111/j.1744-6570.1996.tb01803.x · Cited by 131
Interrater reliability coefficients cannot be computed when only one stimulus is rated.
Schmidt, F. L., & Hunter, J. E. (1989) · Journal of Applied Psychology · doi:10.1037//0021-9010.74.2.368 · Cited by 123
Recent Issues Regarding rWG, rWG, rWG(J), and rWG(J)
LeBreton, J. M., James, L. R., & Lindell, M. K. (2005) · Organizational Research Methods · doi:10.1177/1094428104272181 · Cited by 118
Self-citationRELIABILITY IS NOT VALIDITY AND VALIDITY IS NOT RELIABILITY
SCHMIDT, F. L., VISWESVARAN, C., & ONES, D. S. (2000) · Personnel Psychology · doi:10.1111/j.1744-6570.2000.tb02422.x · Cited by 108
Issues in Multilevel Research: Theory Development, Measurement, and Analysis
Hofmann, D. A. (2004) · Handbook of Research Methods in Industrial and Organizational Psychology · doi:10.1002/9780470756669.ch12 · Cited by 103
Implications of Three Causal Models for the Measurement of Halo Error
Fisicaro, S. A., & Lance, C. E. (1990) · Applied Psychological Measurement · doi:10.1177/014662169001400407 · Cited by 101
Measuring Interrater Agreement for Ratings of a Single Target
Lindell, M. K., & Brandt, C. J. (1997) · Applied Psychological Measurement · doi:10.1177/01466216970213006 · Cited by 86
A reexamination of the relation between halo error and accuracy.
Fisicaro, S. A. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.2.239 · Cited by 74
Reducing the Effects of Performance Expectations on Behavioral Ratings
Baltes, B. B., & Parker, C. P. (2000) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.2000.2897 · Cited by 62
PROGRESS IN PSYCHOMETRICS: CAN INDUSTRIAL AND ORGANIZATIONAL PSYCHOLOGY CATCH UP?
Murphy, K. R., & SHON, R. D. (2000) · Personnel Psychology · doi:10.1111/j.1744-6570.2000.tb02423.x · Cited by 39
Assessing and Testing Interrater Agreement on a Single Target Using Multi-Item Rating Scales
Lindell, M. K. (2001) · Applied Psychological Measurement · doi:10.1177/01466216010251007 · Cited by 33
A Simplified Guide to Statistics for Psychology and Education.
Bates, G. E., & Smith, G. M. (1964) · American Mathematical Monthly · doi:10.2307/2310938 · Cited by 27
Comments on the Measurement of Halo
Fisicaro, S. A., & Vance, R. J. (1994) · Educational and Psychological Measurement · doi:10.1177/0013164494054002010 · Cited by 25
Research works printed but not linked: 16
- 10.1037/0033-2909.83.5.762 · doi:10.1037/0033-2909.83.5.762 · Paper page
- 10.1037/0021-9010.83.2.234 · doi:10.1037/0021-9010.83.2.234 · Paper page
- Cronbach, L.J., 1972, The dependability of behavioral measurements
- 10.5465/amr.1986.4306242 · doi:10.5465/amr.1986.4306242 · Paper page
- 10.1037/0021-9010.66.2.127 · doi:10.1037/0021-9010.66.2.127 · Paper page
- 10.1037/0033-2909.101.1.75 · doi:10.1037/0033-2909.101.1.75 · Paper page
- 10.1037/0021-9010.87.3.465 · doi:10.1037/0021-9010.87.3.465 · Paper page
- 10.1037/0021-9010.85.3.331 · doi:10.1037/0021-9010.85.3.331 · Paper page
- 10.1037/0021-9010.74.4.619 · doi:10.1037/0021-9010.74.4.619 · Paper page
- Nunnally, J.C., 1978, Psychometric theory
- Nunnally, J.C., 1994, Psychometric theory
- Roberson, Q.M., Organizational Research Methods
- 10.1037/0021-9010.74.2.368 · doi:10.1037/0021-9010.74.2.368 · Paper page
- 10.1037/0021-9010.87.2.220 · doi:10.1037/0021-9010.87.2.220 · Paper page
- Schriesheim, C.A., Paper presented the annual meeting of the Academy of Management
- 10.1037/0021-9010.73.3.497 · doi:10.1037/0021-9010.73.3.497 · Paper page
Cited by
Organizational Climate and Culture
Schneider, B., Ehrhart, M. G., & Macey, W. H. (2013) · Annual Review of Psychology · doi:10.1146/annurev-psych-113011-143809 · Cited by 1,956
“It is also common for researchers to report ICC(2) [sometimes also referred to as ICC(K); LeBreton & Senter 2008].”
InfluentialGrant, A. M., & Berry, J. W. (2011) · Academy of Management Journal · doi:10.5465/amj.2011.59215085 · Cited by 1,533
“Since the two raters achieved good reliability (ICC2 .69, p .001), and agreement (average deviation .63)—well within conventional guidelines (LeBreton & Senter, 2008)— we averaged their ratings into a measure of the overall creativity of each participant’s ideas.”
BackgroundThe Microfoundations Movement in Strategy and Organization Theory
Felin, T., Foss, N. J., & Ployhart, R. E. (2015) · Academy of Management Annals · doi:10.5465/19416520.2015.1007651 · Cited by 1,253
Kirkman, B. L., Chen, G., Farh, J.-L., Chen, Z. X., & Lowe, K. B. (2009) · Academy of Management Journal · doi:10.5465/amj.2009.43669971 · Cited by 1,094
Interrater agreement and interrater reliability: Key concepts, approaches, and applications
Gisev, N., Bell, J. S., & Chen, T. F. (2013) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2012.04.004 · Cited by 924
Metadata
- Authors
- LeBreton, J. M., & Senter, J. L.
- Affiliations
- Purdue University West Lafayette (US)
- Wayne State University (US)
- Corresponding author
- LeBreton, James M.
- ORCID
- LeBreton, James M.
- Venue
- Organizational Research Methods
- Publisher
- SAGE Publications
- Published
- 2007-11-28
- Volume
- 11
- Issue
- 4
- Pages
- 815-852
- ISSN
- 1094-4281 (ISSN-L)
- 1552-7425
- OpenAlex
- W2086059573
- Cited by
- 3,608
- References
- 62
- Citations by year
- 2026: 276 · 2025: 242 · 2024: 268 · 2023: 280 · 2022: 313 · 2021: 306 · 2020: 305 · 2019: 247 · 2018: 220 · 2017: 198 · 2016: 197 · 2015: 171 · 2014: 166 · 2013: 139 · 2012: 92
- FWCI
- 33.12
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Organizational Research Methods (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- inter-rater reliability, interobserver agreement, multilevel modeling, multilevel analysis, multilevel structural equation modeling, data aggregation, composition models, measurement reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.