The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Estimating within-group interrater reliability with and without response bias
Abstract
Abstract : This article presents methods for assessing agreement among the judgments made by a single group of judges on a single variable in regard to a single target. For example, the group of judges could be editorial consultants, members of an assessment center, or members of a team. The single target could be a manuscript, a lower-level manager, or a team. The variable on which the target is judged could be overall publishability in the case of the manuscript, managerial potential for the lower-level manager, or team cooperativeness for the team. The methods presented are based on new procedures for estimating interrater reliability. For situations such as the above, these procedures are shown to furnish more accurate and interpretable estimates of agreement than estimates provided by procedures commonly used to estimate agreement, consistency, or interrater reliability. In addition, the proposed methods include processes for controlling for the spurious influences of response biases (e.g., positive leniency, social desirability) on estimates of interrater reliability. (Author)
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, in couples where only one partner reported pain, spouses reported greater depressive symptoms when both partners expressed anger and contempt.”
From Empathic and nonempathic interaction in chronic pain couples. (2008) · cites this paper as [20]
In the citing paper
“Johansen and Cano 20 demonstrated that spouses rated the patients’ pain as being more severe when patients expressed anger and contempt.”
From Empathic and nonempathic interaction in chronic pain couples. (2008) · cites this paper as [20]
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1994) · Psychological Review · doi:10.1037/0033-295x.101.2.343 · Cited by 17,563
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Aggregation bias in estimates of perceptual agreement.
James, L. R. (1982) · Journal of Applied Psychology · doi:10.1037/0021-9010.67.2.219 · Cited by 2,139
Self-citationAssessing similarity between profiles.
Cronbach, L. J., & Gleser, G. C. (1953) · Psychological Bulletin · doi:10.1037/h0057173 · Cited by 1,149
Category judgment: A range-frequency model.
Parducci, A. (1965) · Psychological Review · doi:10.1037/h0022602 · Cited by 1,132
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 830
Advances in factor analysis and structural equation models
Jöreskog, K. G., Sörbom, D., & Magidson, J. (1979) · Medical Entomology and Zoology · Cited by 826
Further Evidence on Response Sets and Test Design
Cronbach, L. J. (1950) · Educational and Psychological Measurement · doi:10.1177/001316445001000101 · Cited by 748
Cited by
Sorted by citations
The job demands-resources model of burnout.
Demerouti, E., Bakker, A. B., Nachreiner, F., & Schaufeli, W. B. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.3.499 · Cited by 13,643
Ethical leadership: A social learning perspective for construct development and testing
Brown, M. E., Treviño, L. K., & Harrison, D. A. (2005) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2005.03.002 · Cited by 5,435
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608
What Makes Teams Work: Group Effectiveness Research from the Shop Floor to the Executive Suite
Cohen, S. G., & Bailey, D. E. (1997) · Journal of Management · doi:10.1177/014920639702300303 · Cited by 3,526
Differences Between Tight and Loose Cultures: A 33-Nation Study
Gelfand, M. J., Raver, J. L., Nishii, L., Leslie, L. M., Lun, J., Lim, B. C., Duan, L., Almaliach, A., Ang, S., Arnadottir, J., et al. (2011) · Science · doi:10.1126/science.1197754 · Cited by 3,252
Metadata
- Authors
- James, L. R., Demaree, R. G., & Wolf, G.
- Affiliations
- Georgia Institute of Technology (US)
- Venue
- Journal of Applied Psychology
- Published
- 1984-02
- Volume
- 69
- Issue
- 1
- Pages
- 85-98
- ISSN
- 0021-9010 (ISSN-L)
- 1939-1854
- OpenAlex
- W1971711030
- Cited by
- 4,774
- References
- 37
- Citations by year
- 2026: 127 · 2025: 164 · 2024: 145 · 2023: 162 · 2022: 173 · 2021: 210 · 2020: 204 · 2019: 231 · 2018: 202 · 2017: 222 · 2016: 207 · 2015: 216 · 2014: 229 · 2013: 227 · 2012: 232
- FWCI
- 4.17
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Applied Psychology (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- inter-rater reliability, response bias, leniency bias, social desirability bias, judgment consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
