The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Estimating within-group interrater reliability with and without response bias
Abstract
Abstract : This article presents methods for assessing agreement among the judgments made by a single group of judges on a single variable in regard to a single target. For example, the group of judges could be editorial consultants, members of an assessment center, or members of a team. The single target could be a manuscript, a lower-level manager, or a team. The variable on which the target is judged could be overall publishability in the case of the manuscript, managerial potential for the lower-level manager, or team cooperativeness for the team. The methods presented are based on new procedures for estimating interrater reliability. For situations such as the above, these procedures are shown to furnish more accurate and interpretable estimates of agreement than estimates provided by procedures commonly used to estimate agreement, consistency, or interrater reliability. In addition, the proposed methods include processes for controlling for the spurious influences of response biases (e.g., positive leniency, social desirability) on estimates of interrater reliability. (Author)
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, in couples where only one partner reported pain, spouses reported greater depressive symptoms when both partners expressed anger and contempt.”
From Empathic and nonempathic interaction in chronic pain couples. (2008) · cites this paper as [20]
In the citing paper
“Johansen and Cano 20 demonstrated that spouses rated the patients’ pain as being more severe when patients expressed anger and contempt.”
From Empathic and nonempathic interaction in chronic pain couples. (2008) · cites this paper as [20]
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1994) · Psychological Review · doi:10.1037/0033-295x.101.2.343 · Cited by 17,559
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Aggregation bias in estimates of perceptual agreement.
James, L. R. (1982) · Journal of Applied Psychology · doi:10.1037/0021-9010.67.2.219 · Cited by 2,139
Self-citationAssessing similarity between profiles.
Cronbach, L. J., & Gleser, G. C. (1953) · Psychological Bulletin · doi:10.1037/h0057173 · Cited by 1,149
Category judgment: A range-frequency model.
Parducci, A. (1965) · Psychological Review · doi:10.1037/h0022602 · Cited by 1,132
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 830
Advances in factor analysis and structural equation models
Jöreskog, K. G., Sörbom, D., & Magidson, J. (1979) · Medical Entomology and Zoology · Cited by 826
Further Evidence on Response Sets and Test Design
Cronbach, L. J. (1950) · Educational and Psychological Measurement · doi:10.1177/001316445001000101 · Cited by 748
Mitchell, S. K. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.376 · Cited by 445
The great response-style myth.
Rorer, L. G. (1965) · Psychological Bulletin · doi:10.1037/h0021888 · Cited by 438
Item response theory : application to psychological measurement
Hulin, C. L., Drasgow, F., & Parsons, C. K. (1983) · Dow Jones-Irwin eBooks · Cited by 361
Cognitive Processes and Societal Risk Taking
Slovic, P., Fischhoff, B., & Lichtenstein, S. (1976) · Springer eBooks · doi:10.1007/978-94-010-1276-8_2 · Cited by 310
Cognitive organization and change: An information processing approach
(1975) · Behavior Therapy · doi:10.1016/s0005-7894(75)80236-x · Cited by 306
Behavior in the complex environment.
Streufert, S., & Streufert, S. C. (1978) · Cited by 302
Format and training effects on rating accuracy and rater errors.
Borman, W. C. (1979) · Journal of Applied Psychology · doi:10.1037/0021-9010.64.4.410 · Cited by 245
rp and other Coefficients of Pattern Similarity
Cattell, R. B. (1949) · Psychometrika · doi:10.1007/bf02289193 · Cited by 241
Effects of rater training: Creating new response sets and decreasing accuracy.
Bernardin, H. J., & Pence, E. C. (1980) · Journal of Applied Psychology · doi:10.1037/0021-9010.65.1.60 · Cited by 222
Intraclass correlations: There's more there than meets the eye.
Lahey, M. A., Downey, R. G., & Saal, F. E. (1983) · Psychological Bulletin · doi:10.1037/0033-2909.93.3.586 · Cited by 222
THE CONTROL OF BIAS IN RATINGS: A THEORY OF RATING
Wherry, R. J., & Bartlett, C. J. (1982) · Personnel Psychology · doi:10.1111/j.1744-6570.1982.tb02208.x · Cited by 221
A Note on Estimating the Reliability of Categorical Data
Finn, R. H. (1970) · Educational and Psychological Measurement · doi:10.1177/001316447003000106 · Cited by 207
Cognitive Structure, Theory and Measurement of Individual Differences
Scott, W. A., Osgood, D. W., & Peterson, C. (1979) · Medical Entomology and Zoology · Cited by 180
Exploring upper limits of reliability and validity in job performance ratings.
Borman, W. C. (1978) · Journal of Applied Psychology · doi:10.1037/0021-9010.63.2.135 · Cited by 178
R (c): A Profile Similarity Coefficient Invariant Over Variable Reflection
Cohen, J. (1968) · PsycEXTRA Dataset · doi:10.1037/e473742008-107 · Cited by 98
Response Bias in an Unstructured Questionnaire
Berg, I. A., & Rapaport, G. M. (1954) · The Journal of Psychology · doi:10.1080/00223980.1954.9712954 · Cited by 76
The Effect of Grouping or Randomizing Items on Leniency Response Bias
Schriesheim, C. A. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100219 · Cited by 74
Group climate: An exploratory analysis of construct validity
Howe, J. G. (1977) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(77)90057-5 · Cited by 62
RESPONSE STYLES AS PERSONALITY VARIABLES: A THEORETICAL INTEGRATION OF MULTIVARIATE RESEARCH1
Damarin, F. L., & Messick, S. (1965) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1965.tb00967.x · Cited by 42
Note on multivariate methods for profile analysis.
Overall, J. E. (1964) · Psychological Bulletin · doi:10.1037/h0041666 · Cited by 34
James, L. R., & White, J. F. (1983) · Personnel Psychology · doi:10.1111/j.1744-6570.1983.tb00514.x · Cited by 26
Self-citationLatent trait theory for organizational research
Guion, R. M., & Ironson, G. H. (1983) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(83)90113-7 · Cited by 20
Response Tendency in a Questionnaire without Questions
van Heerden, J., & Hoogstraten, J. (1979) · Applied Psychological Measurement · doi:10.1177/014662167900300112 · Cited by 17
An Exact Probability Test for Use with Likert-Type Scales
Cooper, M. (1976) · Educational and Psychological Measurement · doi:10.1177/001316447603600310 · Cited by 14
Comments on the Analysis of Variance Strategy for the Computation of Intraclass Reliability
Selvage, R. (1976) · Educational and Psychological Measurement · doi:10.1177/001316447603600303 · Cited by 11
Some Cheap Tricks for Making Inferences about Distribution Shapes from Variances
Messick, D. M. (1982) · Educational and Psychological Measurement · doi:10.1177/001316448204200304 · Cited by 7
Agreement or Disagreement of a Set of Likert-Type Ratings
Hsu, L. M. (1979) · Educational and Psychological Measurement · doi:10.1177/001316447903900205 · Cited by 4
Cited by
Sorted by citations
The job demands-resources model of burnout.
Demerouti, E., Bakker, A. B., Nachreiner, F., & Schaufeli, W. B. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.3.499 · Cited by 13,643
Ethical leadership: A social learning perspective for construct development and testing
Brown, M. E., Treviño, L. K., & Harrison, D. A. (2005) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2005.03.002 · Cited by 5,435
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608
What Makes Teams Work: Group Effectiveness Research from the Shop Floor to the Executive Suite
Cohen, S. G., & Bailey, D. E. (1997) · Journal of Management · doi:10.1177/014920639702300303 · Cited by 3,526
Differences Between Tight and Loose Cultures: A 33-Nation Study
Gelfand, M. J., Raver, J. L., Nishii, L., Leslie, L. M., Lun, J., Lim, B. C., Duan, L., Almaliach, A., Ang, S., Arnadottir, J., et al. (2011) · Science · doi:10.1126/science.1197754 · Cited by 3,252
Metadata
- Authors
- James, L. R., Demaree, R. G., & Wolf, G.
- Affiliations
- Georgia Institute of Technology (US)
- Venue
- Journal of Applied Psychology
- Published
- 1984-02
- Volume
- 69
- Issue
- 1
- Pages
- 85-98
- ISSN
- 0021-9010 (ISSN-L)
- 1939-1854
- OpenAlex
- W1971711030
- Cited by
- 4,774
- References
- 37
- Citations by year
- 2026: 127 · 2025: 164 · 2024: 145 · 2023: 162 · 2022: 173 · 2021: 210 · 2020: 204 · 2019: 231 · 2018: 202 · 2017: 222 · 2016: 207 · 2015: 216 · 2014: 229 · 2013: 227 · 2012: 232
- FWCI
- 4.17
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Applied Psychology (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- inter-rater reliability, response bias, leniency bias, social desirability bias, judgment consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
