Aggregation bias in estimates of perceptual agreement.
Abstract
It is shown that estimates of agreement based on group mean scores have been incorrectly interpreted as applying to perceptual agreement among individuals. Of initial importance is a study by Drexler (1977), who concluded that a considerable proportion of the variance in climate perceptions is accounted for by organizational membership. This conclusion has been employed recently by other authors to support the assumption that individuals in the same environment tend to agree with respect to climate perceptions. The present paper demonstrates that Drexler's analysis provided inflated estimates of agreement among individuals. The logic of the approach is then extended to other studies in which inflated estimates of agreement appeared likely. (Author)
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ICC(1) = 0.21 and ICC(2) = 0.81 justified aggregating the peer ratings.
“ICC(1), indicating the ratio of between-group variance to total variance was 0.21, indicating a high score (Bliese, 2000;James, 1982).ICC(2), indicating the reliability of average peer perceptions, was 0.81, also representing a high value (Glick, 1985).Thus, our approach to aggregate the peer ratings can be justified.”
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
Development of the Job Diagnostic Survey.
Hackman, J. R., & Oldham, G. R. (1975) · Journal of Applied Psychology · doi:10.1037/h0076546 · Cited by 7,246 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Model of Distributor Firm and Manufacturer Firm Working Partnerships
Anderson, J. C., & Narus, J. A. (1990) · Journal of Marketing · doi:10.1177/002224299005400103 · Cited by 5,826 · see citing works
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774 · see citing works
Self-citationHurley, R. F., & Hult, G. T. M. (1998) · Journal of Marketing · doi:10.1177/002224299806200303 · Cited by 3,971 · see citing works
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608 · see citing works
CONDUCTING INTERORGANIZATIONAL RESEARCH USING KEY INFORMANTS.
KUMAR, N., STERN, L. W., & ANDERSON, J. C. (1993) · Academy of Management Journal · doi:10.2307/256824 · Cited by 2,467 · see citing works
Salanova, M., Agut, S., & Peiró, J. M. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.6.1217 · Cited by 2,276 · see citing works
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053 · see citing works
Levels issues in theory development, data collection and analysis
Klein, K. J., Dansereau, F., & Hall, R. J. (1994) · The Academy of Management Review · doi:10.2307/258703 · Cited by 1,607 · see citing works
NISHII, L. H., LEPAK, D. P., & SCHNEIDER, B. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00121.x · Cited by 1,410 · see citing works
EHRHART, M. G. (2004) · Personnel Psychology · doi:10.1111/j.1744-6570.2004.tb02484.x · Cited by 1,322 · see citing works
Glick, W. H. (1985) · The Academy of Management Review · doi:10.5465/amr.1985.4279045 · Cited by 1,286 · see citing works
On the validity of subjective measures of company performance
WALL, T. D., MICHIE, J., PATTERSON, M., WOOD, S. J., SHEEHAN, M., CLEGG, C. W., & WEST, M. (2004) · Personnel Psychology · doi:10.1111/j.1744-6570.2004.tb02485.x · Cited by 1,121 · see citing works
Schaubroeck, J., Lam, S. S. K., & Peng, A. C. (2011) · Journal of Applied Psychology · doi:10.1037/a0022625 · Cited by 933 · see citing works
Evolutionary Personality Psychology
Buss, D. M. (1991) · Annual Review of Psychology · doi:10.1146/annurev.ps.42.020191.002331 · Cited by 930 · see citing works
Kirkman, B. L., Rosen, B., Tesluk, P. E., & Gibson, C. B. (2004) · Academy of Management Journal · doi:10.2307/20159571 · Cited by 889 · see citing works
Shin, S. J., & Zhou, J. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.6.1709 · Cited by 845 · see citing works
The cross‐level effects of culture and climate in human service teams
Glisson, C., & James, L. R. (2002) · Journal of Organizational Behavior · doi:10.1002/job.162 · Cited by 758 · see citing works
Self-citationPeterson, R. S., Smith, D. B., Martorana, P. V., & Owens, P. D. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.5.795 · Cited by 654 · see citing works
Group Size, ICC Values, and Group-Level Correlations: A Simulation
Bliese, P. D. (1998) · Organizational Research Methods · doi:10.1177/109442819814001 · Cited by 636 · see citing works
Implementing computerized technology: An organizational analysis.
Klein, K. J., Conn, A. B., & Sorra, J. S. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.5.811 · Cited by 615 · see citing works
Dynamic nature of trust in virtual teams
Kanawattanachai, P., & Yoo, Y. (2002) · The Journal of Strategic Information Systems · doi:10.1016/s0963-8687(02)00019-7 · Cited by 594 · see citing works
Hunter, E. M., Neubert, M. J., Perry, S. J., Witt, L. A., Penney, L. M., & Weinberger, E. (2013) · The Leadership Quarterly · doi:10.1016/j.leaqua.2012.12.001 · Cited by 573 · see citing works
A disagreement about within-group agreement: disentangling issues of consistency versus consensus
Kozlowski, S. W., & Hattrup, K. (1992) · Journal of Applied Psychology · doi:10.1037/0021-9010.77.2.161 · Cited by 514 · see citing works
Hollenbeck, J. R., Ilgen, D. R., Sego, D. J., Hedlund, J., Major, D. A., & Phillips, J. (1995) · Journal of Applied Psychology · doi:10.1037/0021-9010.80.2.292 · Cited by 381 · see citing works
Authentic Leadership and Employee Voice Behavior: A Multi-Level Psychological Process
Hsiung, H.-H. (2012) · Journal of Business Ethics · doi:10.1007/s10551-011-1043-2 · Cited by 347 · see citing works
Empathy and the emergence of task and relations leaders
Kellett, J. B., Humphrey, R. H., & Sleeth, R. G. (2006) · The Leadership Quarterly · doi:10.1016/j.leaqua.2005.12.003 · Cited by 337 · see citing works
Goncalo, J. A., Flynn, F. J., & Kim, S. H. (2010) · Personality and Social Psychology Bulletin · doi:10.1177/0146167210385109 · Cited by 312 · see citing works
Reflexivity in Teams: A Measure and Correlates
Schippers, M. C., Den Hartog, D. N., & Koopman, P. L. (2007) · Applied Psychology · doi:10.1111/j.1464-0597.2006.00250.x · Cited by 287 · see citing works
Empathy and complex task performance: Two routes to leadership
Kellett, J. B., Humphrey, R. H., & Sleeth, R. G. (2002) · The Leadership Quarterly · doi:10.1016/s1048-9843(02)00142-x · Cited by 287 · see citing works
The meaning and measurement of implementation climate
Weiner, B. J., Belden, C. M., Bergmire, D. M., & Johnston, M. (2011) · Implementation Science · doi:10.1186/1748-5908-6-78 · Cited by 260 · see citing works
Location-Level Links between Diversity and Innovative Climate Depend on National Power Distance
van der Vegt, G. S., Van de Vliert, E., & Huang, X. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.19573116 · Cited by 228 · see citing works
The role of transformational leadership in enhancing team reflexivity
Schippers, M. C., Den Hartog, D. N., Koopman, P. L., & van Knippenberg, D. (2008) · Human Relations · doi:10.1177/0018726708096639 · Cited by 215 · see citing works
Hill, N. S., Seo, M.-G., Kang, J. H., & Taylor, M. S. (2012) · Organization Science · doi:10.1287/orsc.1110.0662 · Cited by 203 · see citing works
Chen, J.-X., Sharma, P., Zhan, W., & Liu, L. (2019) · Journal of Business Research · doi:10.1016/j.jbusres.2018.10.061 · Cited by 170 · see citing works
Ayoko, O. B., Callan, V. J., & Härtel, C. E. J. (2008) · Small Group Research · doi:10.1177/1046496407304921 · Cited by 159 · see citing works
Kollmann, T., Stöckmann, C., Kensbock, J. M., & Peschl, A. (2020) · Human Resource Management · doi:10.1002/hrm.21981 · Cited by 146 · see citing works
A new perspective on the social functions of emotions: Gratitude and the witnessing effect.
Algoe, S. B., Dwyer, P. C., Younge, A., & Oveis, C. (2020) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000202 · Cited by 120 · see citing works
The Impact of Cultural Value Diversity on Multicultural Team Performance☆
KIRKMAN, B., & SHAPIRO, D. (2005) · Advances in International Management · doi:10.1016/s0747-7929(05)18002-0 · Cited by 94 · see citing works
Tian, Q., Liu, Y., & Fan, J. (2015) · Journal of Management & Organization · doi:10.1017/jmo.2015.14 · Cited by 83 · see citing works
COLLECTIVE CLIMATES: A TEST OF THEIR SOCIOPSYCHOLIGICAL SIGNIFICANCE.
PATTERSON, M., PAYNE, R., & WEST, M. (1996) · Academy of Management Journal · doi:10.2307/257074 · Cited by 63 · see citing works
Liu, X., & Liu, J. (2013) · Nankai Business Review International · doi:10.1108/nbri-07-2013-0023 · Cited by 48 · see citing works
Actors conform, observers react: the effects of behavioral synchrony on conformity
Dong, P., Dai, X., & Wyer, R. S. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000001 · Cited by 46 · see citing works
Lee, A. Y.-P., Chang, P.-C., & Chang, H.-Y. (2022) · Employee Relations: The International Journal · doi:10.1108/er-06-2021-0251 · Cited by 35 · see citing works
Impact of a School‐Based Workplace Health Promotion Program on Morale of Inner‐City Teachers
Allegrante, J. P., & Michela, J. L. (1990) · Journal of School Health · doi:10.1111/j.1746-1561.1990.tb04772.x · Cited by 33 · see citing works
Bilinska, P., Wegge, J., & Kliegel, M. (2016) · Journal of Personnel Psychology · doi:10.1027/1866-5888/a000144 · Cited by 20 · see citing works
Shaw, J. D., Duffy, M. K., & Stark, E. M. (2000) · Journal of Management · doi:10.1016/s0149-2063(99)00045-8 · Cited by 151 · see citing works
- Authors
- James, L. R.
- Affiliations
- Georgia Institute of Technology (US)
- Corresponding author
- James, Lawrence R.
- Venue
- Journal of Applied Psychology
- Published
- 1982-04
- Volume
- 67
- Issue
- 2
- Pages
- 219-229
- ISSN
- 0021-9010 (ISSN-L)
- 1939-1854
- OpenAlex
- W1996339572
- Cited by
- 2,139
- References
- 2
- Citations by year
- 2026: 55 · 2025: 55 · 2024: 61 · 2023: 84 · 2022: 94 · 2021: 77 · 2020: 123 · 2019: 93 · 2018: 87 · 2017: 63 · 2016: 90 · 2015: 88 · 2014: 85 · 2013: 89 · 2012: 98
- FWCI
- 7.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Applied Psychology (publishedVersion)
- Defense Technical Information Center (DTIC) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Social and Intergroup Psychology
- Keywords
- aggregation bias, climate perception, organizational membership
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review