rwg: an assessment of within-group interrater agreement
The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Schmidt and Hunter (1989) critiqued the within-group interrater reliability statistic (r wg ) described by James, Demaree, and Wolf (1984). Kozlowski and Hattrup (1992) responded to the Schmidt and Hunter critique and argued that r wg is a suitable index of interrater agreement. This article focuses on the interpretation of r wg as a measure of agreement among judges' ratings of a single target. A new derivation of r wg is given that underscores this interpretation
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The MPI has been shown to have good construct and discriminant validity, internal consistency, and test-retest reliability (8, 21).”
From Empathic and nonempathic interaction in chronic pain couples. (2008) · cites this paper as [21]
Citations
Cites
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 26,755
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,007
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
Self-citationThe Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Employee and customer perceptions of service in banks: Replication and extension.
Schneider, B., & Bowen, D. E. (1985) · Journal of Applied Psychology · doi:10.1037//0021-9010.70.3.423 · Cited by 1,353
Hater, J. J., & Bass, B. M. (1988) · Journal of Applied Psychology · doi:10.1037//0021-9010.73.4.695 · Cited by 1,213
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 852
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037//0033-2909.83.5.762 · Cited by 825
Cited by
Tsai, W. (2001) · Academy of Management Journal · doi:10.2307/3069443 · Cited by 4,257
THE ANTECEDENTS, CONSEQUENCES, AND MEDIATING ROLE OF ORGANIZATIONAL AMBIDEXTERITY.
Gibson, C. B., & Birkinshaw, J. (2004) · Academy of Management Journal · doi:10.2307/20159573 · Cited by 3,919
Jansen, J. J. P., Van Den Bosch, F. A. J., & Volberda, H. W. (2006) · Management Science · doi:10.1287/mnsc.1060.0576 · Cited by 3,705
“We calculated an interrater agreement score (rwg) for data on these variables (James et al. 1993).”
MethodAnswers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608
“Schmidt and Hunter (1989) critiqued the rWG and rWGðJÞ indices, largely based on semantic confusion arising from earlier writers’ labels of the rWG indices as reliability coefficients (James et al., 1984) versus agreement coefficients (James et al., 1993; Kozlowski & Hattrup, 1992).”
BackgroundMethodInfluentialWhat's the difference? diversity constructs as separation, variety, or disparity in organizations
Harrison, D. A., & Klein, K. J. (2007) · Academy of Management Review · doi:10.5465/amr.2007.26586096 · Cited by 3,151
Metadata
- Authors
- James, L. R., Demaree, R. G., & Wolf, G.
- Affiliations
- University of Tennessee at Knoxville (US)
- Venue
- Journal of Applied Psychology
- Published
- 1993-04
- Volume
- 78
- Issue
- 2
- Pages
- 306-309
- ISSN
- 0021-9010 (ISSN-L)
- 1939-1854
- OpenAlex
- W2022950201
- Cited by
- 2,221
- References
- 21
- Citations by year
- 2026: 54 · 2025: 88 · 2024: 76 · 2023: 85 · 2022: 88 · 2021: 97 · 2020: 95 · 2019: 98 · 2018: 108 · 2017: 82 · 2016: 86 · 2015: 100 · 2014: 111 · 2013: 110 · 2012: 96
- FWCI
- 12.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Applied Psychology (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
- Keywords
- inter-rater reliability, interobserver agreement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: rwg: an assessment of within-group interrater agreement. https://app.cochat.ai/papers/10.1037/0021-9010.78.2.306
- The paper
- James, L. R., Demaree, R. G., & Wolf, G. (1993). rwg: an assessment of within-group interrater agreement. Journal of Applied Psychology. https://doi.org/10.1037/0021-9010.78.2.306
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.