Answers to 20 Questions About Interrater Reliability and Interrater Agreement
The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
The use of interrater reliability (IRR) and interrater agreement (IRA) indices has increased dramatically during the past 20 years. This popularity is, at least in part, because of the increased role of multilevel modeling techniques (e.g., hierarchical linear modeling and multilevel structural equation modeling) in organizational research. IRR and IRA indices are often used to justify aggregating lower-level data used in composition models. The purpose of the current article is to expose researchers to the various issues surrounding the use of IRR and IRA indices often used in conjunction with multilevel models. To achieve this goal, the authors adopt a question-and-answer format and provide a tutorial in the appendices illustrating how these indices may be computed using the SPSS software.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For the 20-comment reliability subset (5 repetitions each), we computed intraclass correlations for each DV, adopting ICC ≥ 0 . 70 as the threshold for adequate reliability [59].”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [59]
In the citing paper
“This cutoff is a heuristic rather than an absolute standard; the appropriate threshold depends on the decisions being made [59].”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [59]
In the citing paper
“High-stakes decisions about specific individuals require strong agreement, whereas research estimating aggregate effects may proceed with moderate agreement [59].”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [59]
In the citing paper
“We interpret these values in light of LeBreton and Senter [59]'s guidance that reliability standards should be commensurate with the intensity of decisions being made. High-stakes decisions about specific individuals (e.g., hiring, promotion) require strong agreement (ICC > 0 . 90 ), whereas research estimating aggregate group-level effects may proceed with moderate agreement.”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [59]
In the citing paper
“The two raters' reliability [ICC (2) = .92]and agreement (average deviation = .26)were within conventional guidelines [90].”
From Absorbed in technology but digitally overloaded: Interplay effects on gig workers’ burnout and creativity (2021) · cites this paper as [90]
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology · doi:10.2307/2075823 · Cited by 18,724
Review of Psychometric theory (2nd ed.).
SITGREAVES, R. (1979) · Contemporary Psychology: A Journal of Reviews · doi:10.1037/018882 · Cited by 11,425
Statistical Theories of Mental Test Scores
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,022
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774 · cited 24×
“The rWG and rWGðJÞ indices were initially introduced by James et al. (1984).”
BackgroundMethodResultInfluential(2000) · Cited by 3,573
Chan, D. K. (1998) · Journal of Applied Psychology · doi:10.1037//0021-9010.83.2.234 · Cited by 2,972
Research works printed but not linked: 16
- 10.1037/0033-2909.83.5.762 · doi:10.1037/0033-2909.83.5.762 · Paper page
- 10.1037/0021-9010.83.2.234 · doi:10.1037/0021-9010.83.2.234 · Paper page
- Cronbach, L.J., 1972, The dependability of behavioral measurements
- 10.5465/amr.1986.4306242 · doi:10.5465/amr.1986.4306242 · Paper page
- 10.1037/0021-9010.66.2.127 · doi:10.1037/0021-9010.66.2.127 · Paper page
- 10.1037/0033-2909.101.1.75 · doi:10.1037/0033-2909.101.1.75 · Paper page
- 10.1037/0021-9010.87.3.465 · doi:10.1037/0021-9010.87.3.465 · Paper page
- 10.1037/0021-9010.85.3.331 · doi:10.1037/0021-9010.85.3.331 · Paper page
- 10.1037/0021-9010.74.4.619 · doi:10.1037/0021-9010.74.4.619 · Paper page
- Nunnally, J.C., 1978, Psychometric theory
- Nunnally, J.C., 1994, Psychometric theory
- Roberson, Q.M., Organizational Research Methods
- 10.1037/0021-9010.74.2.368 · doi:10.1037/0021-9010.74.2.368 · Paper page
- 10.1037/0021-9010.87.2.220 · doi:10.1037/0021-9010.87.2.220 · Paper page
- Schriesheim, C.A., Paper presented the annual meeting of the Academy of Management
- 10.1037/0021-9010.73.3.497 · doi:10.1037/0021-9010.73.3.497 · Paper page
Cited by
Organizational Climate and Culture
Schneider, B., Ehrhart, M. G., & Macey, W. H. (2013) · Annual Review of Psychology · doi:10.1146/annurev-psych-113011-143809 · Cited by 1,956
“It is also common for researchers to report ICC(2) [sometimes also referred to as ICC(K); LeBreton & Senter 2008].”
InfluentialGrant, A. M., & Berry, J. W. (2011) · Academy of Management Journal · doi:10.5465/amj.2011.59215085 · Cited by 1,533
“Since the two raters achieved good reliability (ICC2 .69, p .001), and agreement (average deviation .63)—well within conventional guidelines (LeBreton & Senter, 2008)— we averaged their ratings into a measure of the overall creativity of each participant’s ideas.”
BackgroundThe Microfoundations Movement in Strategy and Organization Theory
Felin, T., Foss, N. J., & Ployhart, R. E. (2015) · Academy of Management Annals · doi:10.5465/19416520.2015.1007651 · Cited by 1,253
Kirkman, B. L., Chen, G., Farh, J.-L., Chen, Z. X., & Lowe, K. B. (2009) · Academy of Management Journal · doi:10.5465/amj.2009.43669971 · Cited by 1,094
Interrater agreement and interrater reliability: Key concepts, approaches, and applications
Gisev, N., Bell, J. S., & Chen, T. F. (2013) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2012.04.004 · Cited by 924
Metadata
- Authors
- LeBreton, J. M., & Senter, J. L.
- Affiliations
- Purdue University West Lafayette (US)
- Wayne State University (US)
- Corresponding author
- LeBreton, James M.
- ORCID
- LeBreton, James M.
- Venue
- Organizational Research Methods
- Publisher
- SAGE Publications
- Published
- 2007-11-28
- Volume
- 11
- Issue
- 4
- Pages
- 815-852
- ISSN
- 1094-4281 (ISSN-L)
- 1552-7425
- OpenAlex
- W2086059573
- Cited by
- 3,608
- References
- 62
- Citations by year
- 2026: 276 · 2025: 242 · 2024: 268 · 2023: 280 · 2022: 313 · 2021: 306 · 2020: 305 · 2019: 247 · 2018: 220 · 2017: 198 · 2016: 197 · 2015: 171 · 2014: 166 · 2013: 139 · 2012: 92
- FWCI
- 33.12
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Organizational Research Methods (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- inter-rater reliability, interobserver agreement, multilevel modeling, multilevel analysis, multilevel structural equation modeling, data aggregation, composition models, measurement reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Answers to 20 Questions About Interrater Reliability and Interrater Agreement. https://app.cochat.ai/papers/10.1177/1094428106296642
- The paper
- LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 Questions About Interrater Reliability and Interrater Agreement. Organizational Research Methods. https://doi.org/10.1177/1094428106296642
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.