The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
rwg: an assessment of within-group interrater agreement
Abstract
Schmidt and Hunter (1989) critiqued the within-group interrater reliability statistic (r wg ) described by James, Demaree, and Wolf (1984). Kozlowski and Hattrup (1992) responded to the Schmidt and Hunter critique and argued that r wg is a suitable index of interrater agreement. This article focuses on the interpretation of r wg as a measure of agreement among judges' ratings of a single target. A new derivation of r wg is given that underscores this interpretation
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The MPI has been shown to have good construct and discriminant validity, internal consistency, and test-retest reliability (8, 21).”
From Empathic and nonempathic interaction in chronic pain couples. (2008) · cites this paper as [21]
Citations
Cites
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,009
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
Self-citationThe Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Employee and customer perceptions of service in banks: Replication and extension.
Schneider, B., & Bowen, D. E. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.3.423 · Cited by 1,344
Hater, J. J., & Bass, B. M. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.4.695 · Cited by 1,136
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 883
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 830
Further Evidence on Response Sets and Test Design
Cronbach, L. J. (1950) · Educational and Psychological Measurement · doi:10.1177/001316445001000101 · Cited by 748
A disagreement about within-group agreement: disentangling issues of consistency versus consensus
Kozlowski, S. W., & Hattrup, K. (1992) · Journal of Applied Psychology · doi:10.1037/0021-9010.77.2.161 · Cited by 514
Mitchell, S. K. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.376 · Cited by 445
Item response theory : application to psychological measurement
Hulin, C. L., Drasgow, F., & Parsons, C. K. (1983) · Dow Jones-Irwin eBooks · Cited by 361
AN EXPLORATION OF CLIMATES FOR TECHNICAL UPDATING AND PERFORMANCE
Kozlowski, S. W. J., & Hults, B. M. (1987) · Personnel Psychology · doi:10.1111/j.1744-6570.1987.tb00614.x · Cited by 322
Intraclass correlations: There's more there than meets the eye.
Lahey, M. A., Downey, R. G., & Saal, F. E. (1983) · Psychological Bulletin · doi:10.1037/0033-2909.93.3.586 · Cited by 222
A Note on Estimating the Reliability of Categorical Data
Finn, R. H. (1970) · Educational and Psychological Measurement · doi:10.1177/001316447003000106 · Cited by 207
Judgment of counseling process: Reliability, agreement, and error.
Lawlis, G. F., & Lu, E. (1972) · Psychological Bulletin · doi:10.1037/h0032935 · Cited by 174
Interrater reliability coefficients cannot be computed when only one stimulus is rated.
Schmidt, F. L., & Hunter, J. E. (1989) · Journal of Applied Psychology · doi:10.1037/0021-9010.74.2.368 · Cited by 122
Interobserver relational agreement.
Stine, W. W. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.106.2.341 · Cited by 26
Bartko, J. J. (1978) · Psychological Bulletin · doi:10.1037//0033-2909.85.1.139 · Cited by 13
Flagged first, then by citations
Cited by
Sorted by citations
Tsai, W. (2001) · Academy of Management Journal · doi:10.2307/3069443 · Cited by 4,352
THE ANTECEDENTS, CONSEQUENCES, AND MEDIATING ROLE OF ORGANIZATIONAL AMBIDEXTERITY.
Gibson, C. B., & Birkinshaw, J. (2004) · Academy of Management Journal · doi:10.2307/20159573 · Cited by 3,951
Jansen, J. J. P., Van Den Bosch, F. A. J., & Volberda, H. W. (2006) · Management Science · doi:10.1287/mnsc.1060.0576 · Cited by 3,705
“We calculated an interrater agreement score (rwg) for data on these variables (James et al. 1993).”
MethodAnswers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608
“Schmidt and Hunter (1989) critiqued the rWG and rWGðJÞ indices, largely based on semantic confusion arising from earlier writers’ labels of the rWG indices as reliability coefficients (James et al., 1984) versus agreement coefficients (James et al., 1993; Kozlowski & Hattrup, 1992).”
BackgroundMethodInfluentialWhat's the difference? diversity constructs as separation, variety, or disparity in organizations
Harrison, D. A., & Klein, K. J. (2007) · Academy of Management Review · doi:10.5465/amr.2007.26586096 · Cited by 3,297
Metadata
- Authors
- James, L. R., Demaree, R. G., & Wolf, G.
- Affiliations
- University of Tennessee at Knoxville (US)
- Venue
- Journal of Applied Psychology
- Published
- 1993-04
- Volume
- 78
- Issue
- 2
- Pages
- 306-309
- ISSN
- 0021-9010 (ISSN-L)
- 1939-1854
- OpenAlex
- W2022950201
- Cited by
- 2,221
- References
- 21
- Citations by year
- 2026: 54 · 2025: 88 · 2024: 76 · 2023: 85 · 2022: 88 · 2021: 97 · 2020: 95 · 2019: 98 · 2018: 108 · 2017: 82 · 2016: 86 · 2015: 100 · 2014: 111 · 2013: 110 · 2012: 96
- FWCI
- 12.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Applied Psychology (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
- Keywords
- inter-rater reliability, interobserver agreement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
