A disagreement about within-group agreement: disentangling issues of consistency versus consensus

Kozlowski, Steve W.; Hattrup, Keith

doi:10.1037/0021-9010.77.2.161

Cited by 515Reliability and Agreement in MeasurementPsychometric Methodologies and TestingDelphi Technique in Research

A disagreement about within-group agreement: disentangling issues of consistency versus consensus

Kozlowski, Steve W., Hattrup, Keith·1992·Journal of Applied Psychology

DOI 10.1037/0021-9010.77.2.161

Open this paper in CoChat — free account

Abstract

James, Demaree, and Wolf (1984) developed an indes, r WG , for assessing within-group agreement appropriate when only a single target is rated. The assessment of interperceiver agreement in such situations is of particular relevance to the composition model for climate. Schmidt and Hunter (1989) have criticized the conceptual foundation of r WG because it is not consistent with the classical model of reliability. They proposed an alternative approach, the use of the rating standard deviation (SD X ), the standard error of the rating mean (SE M ), and the associated confidence intervals for SE M to index interrater agreement. This comment argues that the critique of r WG dit not clearly distinguish the concepts of interrater consensus (i.e., agreement) and interrater consistency (i.e., reliability)

Integrity signals

Registry

Crossref · 10.1037/0021-9010.77.2.161

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Operant models cannot explain that negative or punishing spouse responses are associated with increased pain behaviors and pain severity.
“Newton-John 29 noted that operant models cannot explain the fact that negative or punishing spouse responses are associated with increased pain behaviors and pain severity 3, 22.”
From Empathic and nonempathic interaction in chronic pain couples. (2008) · cites this paper as [22]

A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
The Social Psychology of Organizations.
Hackman, J. R., Katz, D., & Kahn, R. L. (1979) · Administrative Science Quarterly · doi:10.2307/2989929 · Cited by 12,243 · see citing works
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · see citing works
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774 · see citing works
TRANSFER OF TRAINING: A REVIEW AND DIRECTIONS FOR FUTURE RESEARCH
Baldwin, T. T., & Ford, J. K. (1988) · Personnel Psychology · doi:10.1111/j.1744-6570.1988.tb00632.x · Cited by 3,449 · see citing works
International Review of Industrial and Organizational Psychology
Cooper, C. L., & Robertson, I. T. (2003) · doi:10.1002/0470013311.oth1 · Cited by 2,775 · see citing works
Aggregation bias in estimates of perceptual agreement.
James, L. R. (1982) · Journal of Applied Psychology · doi:10.1037/0021-9010.67.2.219 · Cited by 2,139 · see citing works
Personality, affect, and behavior in groups
George, J. M. (1990) · Journal of Applied Psychology · doi:10.1037/0021-9010.75.2.107 · Cited by 1,372 · see citing works
Employee and customer perceptions of service in banks: Replication and extension.
Schneider, B., & Bowen, D. E. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.3.423 · Cited by 1,344 · see citing works
ON THE ETIOLOGY OF CLIMATES
Schneider, B., & Reichers, A. E. (1983) · Personnel Psychology · doi:10.1111/j.1744-6570.1983.tb00500.x · Cited by 1,195 · see citing works
Superiors' evaluations and subordinates' perceptions of transformational and transactional leadership.
Hater, J. J., & Bass, B. M. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.4.695 · Cited by 1,136 · see citing works
Organizational climate: A review of theory and research.
James, L. R., & Jones, A. P. (1974) · Psychological Bulletin · doi:10.1037/h0037511 · Cited by 1,099 · see citing works
Toward an understanding of team performance and training.
Eduardo Salas, A., Dickinson, T. L., Converse, S. A., & Tannenbaum, S. I. (1992) · Cited by 909 · see citing works
Teams: Their Training and Performance
Swezey, R. W., & Salas, E. (1992) · Medical Entomology and Zoology · Cited by 891 · see citing works
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 855 · see citing works
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 818 · see citing works
Integration of climate and leadership: Examination of a neglected issue.
Kozlowski, S. W. J., & Doherty, M. L. (1989) · Journal of Applied Psychology · doi:10.1037/0021-9010.74.4.546 · Cited by 586 · see citing works
Self-citation
Interobserver agreement, reliability, and generalizability of data collected in observational studies.
Mitchell, S. K. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.376 · Cited by 445 · see citing works
Environmental Psychology: Directions and Perspectives
Feimer, N. R., & Geller, E. S. (1983) · Cited by 415 · see citing works
Theory Testing and Organizational Behavior: The Varient Approach.
House, R. J., Dansereau, F., Alutto, J. A., & Yammarino, F. J. (1987) · Administrative Science Quarterly · doi:10.2307/2392921 · Cited by 381 · see citing works
AN EXPLORATION OF CLIMATES FOR TECHNICAL UPDATING AND PERFORMANCE
Kozlowski, S. W. J., & Hults, B. M. (1987) · Personnel Psychology · doi:10.1111/j.1744-6570.1987.tb00614.x · Cited by 319 · see citing works
Self-citation
Organizational structure: a review of structural dimensions and their conceptual relationships with individual attitudes and behavior
James, L. R., & Jones, A. P. (1976) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(76)90008-8 · Cited by 250 · see citing works
Developing an Interdisciplinary Science of Organizations.
Bhagat, R. S., Roberts, K. H., Hulin, C. L., & Rousseau, D. M. (1979) · Administrative Science Quarterly · doi:10.2307/2392507 · Cited by 243 · see citing works
The construction of climate in organizational research.
Rousseau, D. M. (1988) · Cited by 240 · see citing works
A Note on Estimating the Reliability of Categorical Data
Finn, R. H. (1970) · Educational and Psychological Measurement · doi:10.1177/001316447003000106 · Cited by 207 · see citing works
Judgment of counseling process: Reliability, agreement, and error.
Lawlis, G. F., & Lu, E. (1972) · Psychological Bulletin · doi:10.1037/h0032935 · Cited by 174 · see citing works
An Integrative Model of Updating and Performance
Kozlowski, S. W. J., & Farr, J. L. (1988) · Human Performance · doi:10.1207/s15327043hup0101_1 · Cited by 138 · see citing works
Self-citation
Work Climates. An Interactionist Perspective.
Schneider, B. N. (1981) · Defense Technical Information Center (DTIC) · Cited by 126 · see citing works
Interrater reliability coefficients cannot be computed when only one stimulus is rated.
Schmidt, F. L., & Hunter, J. E. (1989) · Journal of Applied Psychology · doi:10.1037/0021-9010.74.2.368 · Cited by 122 · see citing works
Apples and Oranges: An Empirical Comparison of Commonly Used Indices of Interrater Agreement
Jones, A. P., Johnson, L. A., Butler, M. C., & Main, D. S. (1983) · Academy of Management Journal · doi:10.2307/256262 · Cited by 122 · see citing works
A Measure of Agreement Among Subjective Judgments
Lu, K. H. (1971) · Educational and Psychological Measurement · doi:10.1177/001316447103100105 · Cited by 23 · see citing works

Flagged first, then by citations

1 of the 33 references couldn’t be matched to a known paper.

Sorted by citations

PERSON‐ORGANIZATION FIT: AN INTEGRATIVE REVIEW OF ITS CONCEPTUALIZATIONS, MEASUREMENT, AND IMPLICATIONS
Kristof‐Brown, A. L. (1996) · Personnel Psychology · doi:10.1111/j.1744-6570.1996.tb01790.x · Cited by 4,849 · see citing works
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608 · see citing works
The Work Design Questionnaire (WDQ): Developing and validating a comprehensive measure for assessing job design and the nature of work.
Morgeson, F. P., & Humphrey, S. E. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.6.1321 · Cited by 2,994 · see citing works
Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models.
Chan, D. K. (1998) · Journal of Applied Psychology · doi:10.1037/0021-9010.83.2.234 · Cited by 2,981 · see citing works
Untitled work
(2000) · Cited by 2,531 · see citing works
Self-citation
HUMAN RESOURCE MANAGEMENT, MANUFACTURING STRATEGY, AND FIRM PERFORMANCE.
Youndt, M. A., Snell, S. A., J. W. DEAN, Jr., & Lepak, D. P. (1996) · Academy of Management Journal · doi:10.2307/256714 · Cited by 2,274 · see citing works
Knowledge Management Enablers, Processes, and Organizational Performance: An Integrative View and Empirical Examination
Lee, H., & Choi, B. (2003) · Journal of Management Information Systems · doi:10.1080/07421222.2003.11045756 · Cited by 2,269 · see citing works
rwg: an assessment of within-group interrater agreement
James, L. R., Demaree, R. G., & Wolf, G. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.2.306 · Cited by 2,221 · see citing works
RELATIONS BETWEEN WORK GROUP CHARACTERISTICS AND EFFECTIVENESS: IMPLICATIONS FOR DESIGNING EFFECTIVE WORK GROUPS
Campion, M. A., Medsker, G. J., & Higgs, A. C. (1993) · Personnel Psychology · doi:10.1111/j.1744-6570.1993.tb01571.x · Cited by 2,189 · see citing works
HOW MUCH DO HIGH‐PERFORMANCE WORK PRACTICES MATTER? A META‐ANALYSIS OF THEIR EFFECTS ON ORGANIZATIONAL PERFORMANCE
Combs, J. G., Liu, Y., Hall, A. T., & Ketchen, D. J. (2006) · Personnel Psychology · doi:10.1111/j.1744-6570.2006.00045.x · Cited by 2,098 · see citing works
Consensus development methods, and their use in clinical guideline development.
Murphy, M., Black, Lamping, McKee, M., Sanderson, Askham, & Marteau (1998) · Health Technology Assessment · doi:10.3310/hta2030 · Cited by 2,042 · see citing works
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053 · see citing works
Social Structure of “Coopetition” Within a Multiunit Organization: Coordination, Competition, and Intraorganizational Knowledge Sharing
Tsai, W. (2002) · Organization Science · doi:10.1287/orsc.13.2.179.536 · Cited by 1,904 · see citing works
Defining and developing measures of lean production
Shah, R., & Ward, P. T. (2007) · Journal of Operations Management · doi:10.1016/j.jom.2007.01.019 · Cited by 2,399 · see citing works
Levels issues in theory development, data collection and analysis
Klein, K. J., Dansereau, F., & Hall, R. J. (1994) · The Academy of Management Review · doi:10.2307/258703 · Cited by 1,607 · see citing works
An Overview of the Logic and Rationale of Hierarchical Linear Models
Hofmann, D. A. (1997) · Journal of Management · doi:10.1177/014920639702300602 · Cited by 1,584 · see citing works
From Micro to Meso: Critical Steps in Conceptualizing and Conducting Multilevel Research
Klein, K. J., & Kozlowski, S. W. J. (2000) · Organizational Research Methods · doi:10.1177/109442810033001 · Cited by 1,473 · see citing works
Self-citation
EMPLOYEE ATTRIBUTIONS OF THE “WHY” OF HR PRACTICES: THEIR EFFECTS ON EMPLOYEE ATTITUDES AND BEHAVIORS, AND CUSTOMER SATISFACTION
NISHII, L. H., LEPAK, D. P., & SCHNEIDER, B. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00121.x · Cited by 1,410 · see citing works
Examining the Human Resource Architecture: The Relationships Among Human Capital, Employment, and Human Resource Configurations
Lepak, D. P., & Snell, S. A. (2002) · Journal of Management · doi:10.1177/014920630202800403 · Cited by 1,370 · see citing works
Vertical versus shared leadership as predictors of the effectiveness of change management teams: An examination of aversive, directive, transactional, transformational, and empowering leader behaviors.
Pearce, C. L., & Jr. Sims Henry, P. (2002) · Group Dynamics Theory Research and Practice · doi:10.1037/1089-2699.6.2.172 · Cited by 1,311 · see citing works
The Structure and Function of Collective Constructs: Implications for Multilevel Research and Theory Development
Morgeson, F. P., & Hofmann, D. A. (1999) · Academy of Management Review · doi:10.5465/amr.1999.1893935 · Cited by 1,224 · see citing works
Measurement reliability and agreement in psychiatry
Shrout, P. E. (1998) · Statistical Methods in Medical Research · doi:10.1177/096228029800700306 · Cited by 1,211 · see citing works
Innovation in top management teams.
West, M. A., & Anderson, N. R. (1996) · Journal of Applied Psychology · doi:10.1037/0021-9010.81.6.680 · Cited by 1,148 · see citing works
On the validity of subjective measures of company performance
WALL, T. D., MICHIE, J., PATTERSON, M., WOOD, S. J., SHEEHAN, M., CLEGG, C. W., & WEST, M. (2004) · Personnel Psychology · doi:10.1111/j.1744-6570.2004.tb02485.x · Cited by 1,121 · see citing works
Individual consideration viewed at multiple levels of analysis: A multi-level framework for examining the diffusion of transformational leadership
Avolio, B. J., & Bass, B. M. (1995) · The Leadership Quarterly · doi:10.1016/1048-9843(95)90035-7 · Cited by 1,050 · see citing works
Relationships between psychological climate perceptions and work outcomes: a meta‐analytic review
Parker, C. P., Baltes, B. B., Young, S. A., Huff, J. W., Altmann, R. A., Lacost, H. A., & Roberts, J. E. (2003) · Journal of Organizational Behavior · doi:10.1002/job.198 · Cited by 1,079 · see citing works
Do they see eye to eye? Management and employee perspectives of high-performance work systems and influence processes on service quality.
Liao, H., Toya, K., Lepak, D., & Hong, Y. (2009) · Journal of Applied Psychology · doi:10.1037/a0013504 · Cited by 1,030 · see citing works
Network location and learning: the influence of network resources and firm capabilities on alliance formation
Gulati, R. (1999) · Strategic Management Journal · doi:10.1002/(sici)1097-0266(199905)20:5 · Cited by 2,482 · see citing works
MONKEY SEE, MONKEY DO: THE INFLUENCE OF WORK GROUPS ON THE ANTISOCIAL BEHAVIOR OF EMPLOYEES.
Robinson, S. L., & O’Leary-Kelly, A. M. (1998) · Academy of Management Journal · doi:10.2307/256963 · Cited by 847 · see citing works
Climate strength: a new direction for climate research.
Schneider, B., Salvaggio, A. N., & Ferrer, M. S. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.2.220 · Cited by 775 · see citing works
How Organizational Complaint Handling Drives Customer Loyalty: An Analysis of the Mechanistic and the Organic Approach
Homburg, C., & Fürst, A. (2005) · Journal of Marketing · doi:10.1509/jmkg.69.3.95.66367 · Cited by 779 · see citing works
RELATIONS BETWEEN WORK TEAM CHARACTERISTICS AND EFFECTIVENESS: A REPLICATION AND EXTENSION
Campion, M. A., Papper, E. M., & Medsker, G. J. (1996) · Personnel Psychology · doi:10.1111/j.1744-6570.1996.tb01806.x · Cited by 754 · see citing works
Expatriate managers and the psychological contract.
Guzzo, R. A., Noonan, K. A., & Elron, E. (1994) · Journal of Applied Psychology · doi:10.1037/0021-9010.79.4.617 · Cited by 714 · see citing works
MEASUREMENT ERROR IN RESEARCH ON HUMAN RESOURCES and FIRM PERFORMANCE: HOW MUCH ERROR IS THERE AND HOW DOES IT INFLUENCE EFFECT SIZE ESTIMATES?
Gerhart, B. A., Wright, P. M., MAHAN, G. C. M., & Snell, S. A. (2000) · Personnel Psychology · doi:10.1111/j.1744-6570.2000.tb02418.x · Cited by 679 · see citing works
CORRELATES OF CHARISMATIC LEADER BEHAVIOR IN MILITARY UNITS: SUBORDINATES' ATTITUDES, UNIT CHARACTERISTICS, AND SUPERIORS' APPRAISALS OF LEADER PERFORMANCE.
Shamir, B., Zakay, E., Breinin, E., & Popper, M. (1998) · Academy of Management Journal · doi:10.2307/257080 · Cited by 674 · see citing works
Handbook of Industrial, Work & Organizational Psychology
Anderson, N. R., Öneş, D. S., Sinangil, H. K., & Viswesvaran, C. (2001) · Sage eBooks · Cited by 632 · see citing works
A Multiple-Goal, Multilevel Model of Feedback Effects on the Regulation of Individual and Team Performance.
DeShon, R. P., Kozlowski, S. W. J., Schmidt, A. M., Milner, K. R., & Wiechmann, D. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.6.1035 · Cited by 608 · see citing works
Self-citation
Transforming service employees and climate: A multilevel, multisource examination of transformational leadership in building long-term service relationships.
Liao, H., & Chuang, A. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.4.1006 · Cited by 585 · see citing works
Embedding Ethical Leadership within and across Organization Levels
Schaubroeck, J. M., Hannah, S. T., Avolio, B. J., Kozlowski, S. W. J., Lord, R. G., Treviño, L. K., Dimotakis, N., & Peng, A. C. (2012) · Academy of Management Journal · doi:10.5465/amj.2011.0064 · Cited by 577 · see citing works
Self-citation
An empirical investigation of KM styles and their effect on corporate performance
Choi, B., & Lee, H. (2003) · Information & Management · doi:10.1016/s0378-7206(02)00060-5 · Cited by 567 · see citing works
The Impact of Justice Climate and Justice Orientation on Work Outcomes: A Cross-Level Multifoci Framework.
Liao, H., & Rupp, D. E. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.2.242 · Cited by 563 · see citing works
Climate quality and climate consensus as mediators of the relationship between organizational antecedents and outcomes.
Lindell, M. K., & Brandt, C. J. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.3.331 · Cited by 559 · see citing works
A Meta-Analysis of Cohesion and Performance
Gully, S. M., Devine, D. J., & Whitney, D. J. (1995) · Small Group Research · doi:10.1177/1046496495264003 · Cited by 527 · see citing works
Organization structure as a moderator of the relationship between procedural justice, interactional justice, perceived organizational support, and supervisory trust.
Ambrose, M. L., & Schminke, M. J. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.2.295 · Cited by 524 · see citing works
Team effectiveness: Beyond skills and cognitive ability.
Neuman, G. A., & Wright, J. A. (1999) · Journal of Applied Psychology · doi:10.1037/0021-9010.84.3.376 · Cited by 494 · see citing works
Knowledge management strategy and its link to knowledge creation process
Choi, B., & Lee, H. (2002) · Expert Systems with Applications · doi:10.1016/s0957-4174(02)00038-6 · Cited by 466 · see citing works
On Average Deviation Indices for Estimating Interrater Agreement
Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999) · Organizational Research Methods · doi:10.1177/109442819921004 · Cited by 462 · see citing works
The effect of leader moral development on ethical climate and employee attitudes
Schminke, M. J., Ambrose, M. L., & Neubaum, D. O. (2005) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2005.03.006 · Cited by 455 · see citing works
RETRACTED: Psychological processes linking authentic leadership to follower behaviors
Walumbwa, F. O., Wang, P., Wang, H., Schaubroeck, J. M., & Avolio, B. J. (2010) · The Leadership Quarterly · doi:10.1016/j.leaqua.2010.07.015 · Cited by 458 · see citing works
Retracted
Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide
Burke, M. J., & Dunlap, W. P. (2002) · Organizational Research Methods · doi:10.1177/1094428102005002002 · Cited by 448 · see citing works
Transformational leadership and innovative work behavior
Afsar, B., & Umrani, W. A. (2019) · European Journal of Innovation Management · doi:10.1108/ejim-12-2018-0257 · Cited by 511 · see citing works
Change in job conditions, change in psychological distress, and gender: a longitudinal study of dual-earner couples
BARNETT, R. C., & Brennan, R. T. (1997) · Journal of Organizational Behavior · doi:10.1002/(sici)1099-1379(199705)18:3 · Cited by 96 · see citing works
Untitled work
(2006) · eCommons (Cornell University) · Cited by 437 · see citing works
Organizational climate and culture: Reflections on the history of the constructs in the Journal of Applied Psychology.
Schneider, B., González‐Romá, V., Ostroff, C. L., & West, M. A. (2017) · Journal of Applied Psychology · doi:10.1037/apl0000090 · Cited by 452 · see citing works
The External Leadership of Self-Managing Teams: Intervening in the Context of Novel and Disruptive Events.
Morgeson, F. P. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.3.497 · Cited by 427 · see citing works
Advancing Multilevel Research Design
Kozlowski, S. W. J., Chao, G. T., Grand, J. A., Braun, M. T., & Kuljanin, G. (2013) · Organizational Research Methods · doi:10.1177/1094428113493119 · Cited by 421 · see citing works
Self-citation
Interrater Reliability and Agreement
Tinsley, H. E. A., & Weiss, D. J. (2000) · Elsevier eBooks · doi:10.1016/b978-012691360-6/50005-7 · Cited by 401 · see citing works
SATISFACTION, CITIZENSHIP BEHAVIORS, AND PERFORMANCE IN WORK UNITS: A META-ANALYSIS OF COLLECTIVE CONSTRUCT RELATIONS
Whitman, D. S., ROOY, D. L. V., & Viswesvaran, C. (2010) · Personnel Psychology · doi:10.1111/j.1744-6570.2009.01162.x · Cited by 382 · see citing works
INDIVIDUAL AND SITUATIONAL INFLUENCES ON THE DEVELOPMENT OF SELF‐EFFICACY: IMPLICATIONS FOR TRAINING EFFECTIVENESS
Mathieu, J. E., Martineau, J., & Tannenbaum, S. I. (1993) · Personnel Psychology · doi:10.1111/j.1744-6570.1993.tb00870.x · Cited by 369 · see citing works
THE INFLUENCE OF PROPORTIONAL AND PERCEPTUAL CONFLICT COMPOSITION ON TEAM PERFORMANCE
Jehn, K. A., & Chatman, J. A. (2000) · International Journal of Conflict Management · doi:10.1108/eb022835 · Cited by 363 · see citing works
ATTITUDES AND EFFECTIVENESS: EXAMINING RELATIONS AT AN ORGANIZATIONAL LEVEL
Ryan, A. M., Schmit, M. J., & JOHNSON, R. (1996) · Personnel Psychology · doi:10.1111/j.1744-6570.1996.tb02452.x · Cited by 363 · see citing works
Within-group agreement: On the use (and misuse) of r and r in leadership research and some best practice guidelines
Biemann, T., Cole, M. S., & Voelpel, S. C. (2011) · The Leadership Quarterly · doi:10.1016/j.leaqua.2011.11.006 · Cited by 364 · see citing works
A trickle-down model of organizational justice: Relating employees' and customers' perceptions of and reactions to fairness.
Masterson, S. S. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.4.594 · Cited by 361 · see citing works
Cohesiveness and organizational citizenship behavior: A multilevel analysis using work groups and individuals
Kidwell, R. E., Mossholder, K. W., & Bennett, N. (1997) · Journal of Management · doi:10.1177/014920639702300605 · Cited by 357 · see citing works
A social identity perspective on leadership and employee creativity
Hirst, G., van Dick, R., & van Knippenberg, D. (2009) · Journal of Organizational Behavior · doi:10.1002/job.600 · Cited by 356 · see citing works
The effect of organizational structure on perceptions of procedural fairness.
Schminke, M. J., Ambrose, M. L., & Cropanzano, R. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.2.294 · Cited by 335 · see citing works
Affective mechanisms linking dysfunctional behavior to performance in work teams: A moderated mediation study.
Cole, M. S., Walter, F. H., & Bruch, H. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.5.945 · Cited by 339 · see citing works
Measuring Normative Beliefs and Shared Behavioral Expectations in Organizations: The Reliability and Validity of the Organizational Culture Inventory
Cooke, R. A., & Szumal, J. L. (1993) · Psychological Reports · doi:10.2466/pr0.1993.72.3c.1299 · Cited by 336 · see citing works
Individual Moral Development and Ethical Climate: The Influence of Person–Organization Fit on Job Attitudes
Ambrose, M. L., Arnaud, A., & Schminke, M. J. (2007) · Journal of Business Ethics · doi:10.1007/s10551-007-9352-1 · Cited by 335 · see citing works
MAXIMIZING THE BENEFITS OF TASK CONFLICT: THE ROLE OF CONFLICT MANAGEMENT
DeChurch, L. A., & Marks, M. A. (2001) · International Journal of Conflict Management · doi:10.1108/eb022847 · Cited by 331 · see citing works
Group cohesion: From "field of forces" to multidimensional construct.
Dion, K. L. (2000) · Group Dynamics Theory Research and Practice · doi:10.1037/1089-2699.4.1.7 · Cited by 319 · see citing works
A FRAMEWORK FOR CONDUCTING MULTI-LEVEL CONSTRUCT VALIDATION
Chen, G., Mathieu, J. E., & Bliese, P. D. (2004) · Research in multi-level issues · doi:10.1016/s1475-9144(04)03013-9 · Cited by 317 · see citing works
The Dynamics of Emergence: Cognition and Cohesion in Work Teams
Kozlowski, S. W. J., & Chao, G. T. (2012) · Managerial and Decision Economics · doi:10.1002/mde.2552 · Cited by 316 · see citing works
Self-citation
Responsible leadership and employee's proenvironmental behavior: The role of organizational commitment, green shared vision, and internal environmental locus of control
Afsar, B., Maqsoom, A., Shahjehan, A., Afridi, S. A., Nawaz, A., & Fazliani, H. (2019) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.1806 · Cited by 311 · see citing works
Perceived team and player efficacy in hockey.
Feltz, D. L., & Lirgg, C. (1998) · Journal of Applied Psychology · doi:10.1037/0021-9010.83.4.557 · Cited by 302 · see citing works
Interrater Agreement Reconsidered: An Alternative to the rwg Indices
Brown, R. D., & Hauenstein, N. M. A. (2005) · Organizational Research Methods · doi:10.1177/1094428105275376 · Cited by 299 · see citing works
Work Outcomes and Job Design for Contract Versus Permanent Information Systems Professionals On Software Development Teams1, 2
Ang, S., & Slaughter, S. A. (2001) · MIS Quarterly · doi:10.2307/3250920 · Cited by 296 · see citing works
The Etiology of the Multilevel Paradigm in Management Research
Mathieu, J. E., & Chen, G. (2010) · Journal of Management · doi:10.1177/0149206310364663 · Cited by 295 · see citing works
Another look at the locus of control-organizational commitment relationship: it depends on the form of commitment
Coleman, D. F., Irving, G. P., & Cooper, C. L. (1999) · Journal of Organizational Behavior · doi:10.1002/(sici)1099-1379(199911)20:6 · Cited by 102 · see citing works
AN EXAMINATION OF THE RELATIONSHIP BETWEEN WORK GROUP CHARACTERISTICS AND PERFORMANCE: ONCE MORE INTO THE BREECH
HYATT, D. E., & Ruddy, T. M. (1997) · Personnel Psychology · doi:10.1111/j.1744-6570.1997.tb00703.x · Cited by 291 · see citing works
Organizational efforts to affirm sexual diversity: A cross-level examination.
Button, S. B. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.1.17 · Cited by 281 · see citing works
Advancing research on team process dynamics
Kozlowski, S. W. J. (2015) · Organizational Psychology Review · doi:10.1177/2041386614533586 · Cited by 278 · see citing works
Self-citation
Team structure, team climate and the quality of care in primary care: an observational study
Bower, P. (2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.4.273 · Cited by 277 · see citing works
What Drives Key Informant Accuracy?
Homburg, C., Klarmann, M., Reimann, M., & Schilke, O. (2012) · Journal of Marketing Research · doi:10.1509/jmr.09.0174 · Cited by 276 · see citing works
The impact of financial and nonfinancial incentives on business-unit outcomes over time.
Peterson, S. J., & Luthans, F. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.1.156 · Cited by 269 · see citing works
Variability Within Organizations: Implications for Strategic Human Resource ManagementPreprint
Nishii, L. H., & Wright, P. M. (2007) · eCommons (Cornell University) · Cited by 267 · see citing works
Shared Authentic Leadership and New Venture Performance
Hmieleski, K. M., Cole, M. S., & Baron, R. A. (2011) · Journal of Management · doi:10.1177/0149206311415419 · Cited by 265 · see citing works
Shared leadership: Toward a multi-level theory of leadership
Pearce, C. L., & Sims, H. P. (2004) · Advances in interdisciplinary studies of work teams · doi:10.1016/s1572-0977(00)07008-4 · Cited by 263 · see citing works
MEASUREMENT ERROR IN RESEARCH ON HUMAN RESOURCES AND FIRM PERFORMANCE: ADDITIONAL DATA AND SUGGESTIONS FOR FUTURE RESEARCH
Wright, P. M., Gardner, T. M., Moynihan, L. M., Park, H. J., Gerhart, B. A., & Delery, J. E. (2001) · Personnel Psychology · doi:10.1111/j.1744-6570.2001.tb00235.x · Cited by 260 · see citing works
A FIVE‐YEAR STUDY OF UPWARD FEEDBACK: WHAT MANAGERS DO WITH THEIR RESULTS MATTERS
Walker, A. G., & Smither, J. W. (1999) · Personnel Psychology · doi:10.1111/j.1744-6570.1999.tb00166.x · Cited by 257 · see citing works
The Restriction of Variance Hypothesis and Interrater Reliability and Agreement: Are Ratings from Multiple Sources Really Dissimilar?
LeBreton, J. M., Burgess, J. R. D., Kaiser, R. B., Atchley, E. K., & James, L. R. (2003) · Organizational Research Methods · doi:10.1177/1094428102239427 · Cited by 255 · see citing works
A Revised Index of Interrater Agreement for Multi-Item Ratings of a Single Target
Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999) · Applied Psychological Measurement · doi:10.1177/01466219922031257 · Cited by 255 · see citing works
Data analytic methods for the analysis of multilevel questions
Castro, S. L. (2002) · The Leadership Quarterly · doi:10.1016/s1048-9843(01)00105-9 · Cited by 250 · see citing works
Integrating Knowledge Activities for Team Innovation: Effects of Transformational Leadership
Jiang, Y., & Chen, C. C. (2016) · Journal of Management · doi:10.1177/0149206316628641 · Cited by 259 · see citing works
A multidimensional approach to the group cohesion-group performance relationship
Chang, A., & Bordia, P. (2001) · Small Group Research · doi:10.1177/104649640103200401 · Cited by 243 · see citing works
The differential effect of team members' trust on team performance: The mediation role of team cohesion
Mach, M., Dolan, S. L., & Tzafrir, S. S. (2009) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317909x473903 · Cited by 247 · see citing works
Looking inside and out: The impact of employee and community demographic composition on organizational diversity climate.
Pugh, S. D., Dietz, J., Brief, A. P., & Wiley, J. W. (2008) · Journal of Applied Psychology · doi:10.1037/a0012696 · Cited by 243 · see citing works
Organization structure and fairness perceptions: The moderating effects of organizational level
Schminke, M. J., Cropanzano, R., & Rupp, D. E. (2002) · Organizational Behavior and Human Decision Processes · doi:10.1016/s0749-5978(02)00034-1 · Cited by 242 · see citing works
A HIERARCHICAL CONSTRUCT OF SELF‐MANAGEMENT LEADERSHIP AND ITS RELATIONSHIP TO QUALITY OF WORK LIFE AND PERCEIVED WORK GROUP EFFECTIVENESS
Cohen, S. G., Chang, L., & Ledford, G. E. (1997) · Personnel Psychology · doi:10.1111/j.1744-6570.1997.tb00909.x · Cited by 230 · see citing works
SERVICE CLIMATE EFFECTS ON CUSTOMER ATTITUDES: AN EXAMINATION OF BOUNDARY CONDITIONS.
Dietz, J., Pugh, S. D., & Wiley, J. W. (2004) · Academy of Management Journal · doi:10.2307/20159561 · Cited by 224 · see citing works

Authors: Kozlowski, S. W., & Hattrup, K.
Affiliations: Michigan State University (US)
ORCID: Kozlowski, Steve W.
Venue: Journal of Applied Psychology
Publisher: American Psychological Association (APA)
Published: 1992-04
Volume: 77
Issue: 2
Pages: 161-167
ISSN: 0021-9010 (ISSN-L)
1939-1854
DOI: 10.1037/0021-9010.77.2.161
OpenAlex: W2069294908
Cited by: 514
References: 33
Citations by year: 2026: 8 · 2025: 6 · 2024: 9 · 2023: 12 · 2022: 5 · 2021: 23 · 2020: 11 · 2019: 15 · 2018: 16 · 2017: 15 · 2016: 19 · 2015: 27 · 2014: 24 · 2013: 17 · 2012: 26
FWCI: 12.78
Type: journal article
Language: en
Open access: closed
Locations: Journal of Applied Psychology (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Sociology and Political Science › Delphi Technique in Research
Keywords: interobserver agreement, confidence intervals, classical test theory, reliability statistics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

A disagreement about within-group agreement: disentangling issues of consistency versus consensus

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks