rwg: an assessment of within-group interrater agreement

James, Lawrence R.; Demaree, Robert G.; Wolf, Gerrit

doi:10.1037/0021-9010.78.2.306

Cited by 2,221Reliability and Agreement in MeasurementPsychometric Methodologies and TestingTeam Dynamics and Performance

rwg: an assessment of within-group interrater agreement

James, Lawrence R., Demaree, Robert G., Wolf, Gerrit·1993·Journal of Applied Psychology

DOI 10.1037/0021-9010.78.2.306

Open this paper in CoChat — free account

Abstract

Schmidt and Hunter (1989) critiqued the within-group interrater reliability statistic (r wg ) described by James, Demaree, and Wolf (1984). Kozlowski and Hattrup (1992) responded to the Schmidt and Hunter critique and argued that r wg is a suitable index of interrater agreement. This article focuses on the interpretation of r wg as a measure of agreement among judges' ratings of a single target. A new derivation of r wg is given that underscores this interpretation

Integrity signals

Registry

Crossref · 10.1037/0021-9010.78.2.306

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The Multidimensional Pain Inventory has good construct and discriminant validity, internal consistency, and test-retest reliability.
“The MPI has been shown to have good construct and discriminant validity, internal consistency, and test-retest reliability (8, 21).”
From Empathic and nonempathic interaction in chronic pain couples. (2008) · cites this paper as [21]

Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137 · see citing works
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,009 · see citing works
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774 · see citing works
Self-citation
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819 · see citing works
Employee and customer perceptions of service in banks: Replication and extension.
Schneider, B., & Bowen, D. E. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.3.423 · Cited by 1,344 · see citing works
Superiors' evaluations and subordinates' perceptions of transformational and transactional leadership.
Hater, J. J., & Bass, B. M. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.4.695 · Cited by 1,136 · see citing works
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898 · see citing works
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 855 · see citing works
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 818 · see citing works
Further Evidence on Response Sets and Test Design
Cronbach, L. J. (1950) · Educational and Psychological Measurement · doi:10.1177/001316445001000101 · Cited by 731 · see citing works
A disagreement about within-group agreement: disentangling issues of consistency versus consensus
Kozlowski, S. W., & Hattrup, K. (1992) · Journal of Applied Psychology · doi:10.1037/0021-9010.77.2.161 · Cited by 514 · see citing works
Interobserver agreement, reliability, and generalizability of data collected in observational studies.
Mitchell, S. K. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.376 · Cited by 445 · see citing works
Item response theory : application to psychological measurement
Hulin, C. L., Drasgow, F., & Parsons, C. K. (1983) · Dow Jones-Irwin eBooks · Cited by 361 · see citing works
AN EXPLORATION OF CLIMATES FOR TECHNICAL UPDATING AND PERFORMANCE
Kozlowski, S. W. J., & Hults, B. M. (1987) · Personnel Psychology · doi:10.1111/j.1744-6570.1987.tb00614.x · Cited by 319 · see citing works
Intraclass correlations: There's more there than meets the eye.
Lahey, M. A., Downey, R. G., & Saal, F. E. (1983) · Psychological Bulletin · doi:10.1037/0033-2909.93.3.586 · Cited by 222 · see citing works
A Note on Estimating the Reliability of Categorical Data
Finn, R. H. (1970) · Educational and Psychological Measurement · doi:10.1177/001316447003000106 · Cited by 207 · see citing works
Judgment of counseling process: Reliability, agreement, and error.
Lawlis, G. F., & Lu, E. (1972) · Psychological Bulletin · doi:10.1037/h0032935 · Cited by 174 · see citing works
Interrater reliability coefficients cannot be computed when only one stimulus is rated.
Schmidt, F. L., & Hunter, J. E. (1989) · Journal of Applied Psychology · doi:10.1037/0021-9010.74.2.368 · Cited by 122 · see citing works
Interobserver relational agreement.
Stine, W. W. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.106.2.341 · Cited by 26 · see citing works
Reply to Algina.
Bartko, J. J. (1978) · Psychological Bulletin · doi:10.1037//0033-2909.85.1.139 · Cited by 13 · see citing works

Flagged first, then by citations

Sorted by citations

KNOWLEDGE TRANSFER IN INTRAORGANIZATIONAL NETWORKS: EFFECTS OF NETWORK POSITION AND ABSORPTIVE CAPACITY ON BUSINESS UNIT INNOVATION AND PERFORMANCE.
Tsai, W. (2001) · Academy of Management Journal · doi:10.2307/3069443 · Cited by 4,246 · see citing works
THE ANTECEDENTS, CONSEQUENCES, AND MEDIATING ROLE OF ORGANIZATIONAL AMBIDEXTERITY.
Gibson, C. B., & Birkinshaw, J. (2004) · Academy of Management Journal · doi:10.2307/20159573 · Cited by 3,951 · see citing works
Exploratory Innovation, Exploitative Innovation, and Performance: Effects of Organizational Antecedents and Environmental Moderators
Jansen, J. J. P., Van Den Bosch, F. A. J., & Volberda, H. W. (2006) · Management Science · doi:10.1287/mnsc.1060.0576 · Cited by 3,705 · see citing works
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608 · see citing works
What's the difference? diversity constructs as separation, variety, or disparity in organizations
Harrison, D. A., & Klein, K. J. (2007) · Academy of Management Review · doi:10.5465/amr.2007.26586096 · Cited by 3,147 · see citing works
ADOPTION OF AN ORGANIZATIONAL PRACTICE BY SUBSIDIARIES OF MULTINATIONAL CORPORATIONS: INSTITUTIONAL AND RELATIONAL EFFECTS.
Kostova, T., & Roth, K. (2002) · Academy of Management Journal · doi:10.2307/3069293 · Cited by 2,677 · see citing works
Untitled work
(2000) · Cited by 2,531 · see citing works
HUMAN RESOURCE MANAGEMENT, MANUFACTURING STRATEGY, AND FIRM PERFORMANCE.
Youndt, M. A., Snell, S. A., J. W. DEAN, Jr., & Lepak, D. P. (1996) · Academy of Management Journal · doi:10.2307/256714 · Cited by 2,274 · see citing works
Knowledge Management Enablers, Processes, and Organizational Performance: An Integrative View and Empirical Examination
Lee, H., & Choi, B. (2003) · Journal of Management Information Systems · doi:10.1080/07421222.2003.11045756 · Cited by 2,269 · see citing works
Managing Potential and Realized Absorptive Capacity: How do Organizational Antecedents Matter?
Jansen, J. J. P., Bosch, F. A. J. V. D., & Volberda, H. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.19573106 · Cited by 2,240 · see citing works
Ambidexterity and Performance in Small-to Medium-Sized Firms: The Pivotal Role of Top Management Team Behavioral Integration
Lubatkin, M., Simsek, Z., Ling, Y., & Veiga, J. F. (2006) · Journal of Management · doi:10.1177/0149206306290712 · Cited by 2,230 · see citing works
Servant leadership: Development of a multidimensional measure and multi-level assessment
Liden, R. C., Wayne, S. J., Zhao, H., & Henderson, D. (2008) · The Leadership Quarterly · doi:10.1016/j.leaqua.2008.01.006 · Cited by 2,058 · see citing works
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053 · see citing works
Social Structure of “Coopetition” Within a Multiunit Organization: Coordination, Competition, and Intraorganizational Knowledge Sharing
Tsai, W. (2002) · Organization Science · doi:10.1287/orsc.13.2.179.536 · Cited by 1,904 · see citing works
Let's put the person back into entrepreneurship research: A meta-analysis on the relationship between business owners' personality traits, business creation, and success
Rauch, A., & Fresé, M. (2007) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320701595438 · Cited by 1,879 · see citing works
Transformational leadership and organizational commitment: mediating role of psychological empowerment and moderating role of structural distance
Avolio, B. J., Zhu, W., Koh, W. L., & Bhatia, P. (2004) · Journal of Organizational Behavior · doi:10.1002/job.283 · Cited by 1,702 · see citing works
Levels issues in theory development, data collection and analysis
Klein, K. J., Dansereau, F., & Hall, R. J. (1994) · The Academy of Management Review · doi:10.2307/258703 · Cited by 1,607 · see citing works
BEYOND SELF-MANAGEMENT: ANTECEDENTS AND CONSEQUENCES OF TEAM EMPOWERMENT.
Kirkman, B. L., & Rosen, B. (1999) · Academy of Management Journal · doi:10.2307/256874 · Cited by 2,110 · see citing works
An Overview of the Logic and Rationale of Hierarchical Linear Models
Hofmann, D. A. (1997) · Journal of Management · doi:10.1177/014920639702300602 · Cited by 1,584 · see citing works
A group-level model of safety climate: Testing the effect of group climate on microaccidents in manufacturing jobs.
Zohar, D. M. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.4.587 · Cited by 1,506 · see citing works
High-Performance Human Resource Practices, Citizenship Behavior, and Organizational Performance: A Relational Perspective
Sun, L., Aryee, S., & Law, K. S. (2007) · Academy of Management Journal · doi:10.5465/amj.2007.25525821 · Cited by 1,506 · see citing works
TIME, TEAMS, AND TASK PERFORMANCE: CHANGING EFFECTS OF SURFACE- AND DEEP-LEVEL DIVERSITY ON GROUP FUNCTIONING.
Price, K. H., Harrison, D. A., Gavin, J. H., & Florey, A. T. (2002) · Academy of Management Journal · doi:10.2307/3069328 · Cited by 1,594 · see citing works
Five-factor model of personality and transformational leadership
Judge, T. A., & Bono, J. E. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.5.751 · Cited by 1,382 · see citing works
Validating the organizational climate measure: links to managerial practices, productivity and innovation
Patterson, M. G., West, M. A., Shackleton, V. J., Dawson, J., Lawthom, R., Maitlis, S., Robinson, D. L., & Wallace, A. M. R. (2005) · Journal of Organizational Behavior · doi:10.1002/job.312 · Cited by 1,262 · see citing works
Intellectual Capital Profiles: An Examination of Investments and Returns
Youndt, M., Subramaniam, M., & Snell, S. (2004) · Journal of Management Studies · doi:10.1111/j.1467-6486.2004.00435.x · Cited by 1,278 · see citing works
The Structure and Function of Collective Constructs: Implications for Multilevel Research and Theory Development
Morgeson, F. P., & Hofmann, D. A. (1999) · Academy of Management Review · doi:10.5465/amr.1999.1893935 · Cited by 1,224 · see citing works
Measurement reliability and agreement in psychiatry
Shrout, P. E. (1998) · Statistical Methods in Medical Research · doi:10.1177/096228029800700306 · Cited by 1,211 · see citing works
Coordinating Expertise in Software Development Teams
Faraj, S., & Sproull, L. (2000) · Management Science · doi:10.1287/mnsc.46.12.1554.12072 · Cited by 1,689 · see citing works
Unpacking the Concept of Virtuality: The Effects of Geographic Dispersion, Electronic Dependence, Dynamic Structure, and National Diversity on Team Innovation
Gibson, C. B., & Gibbs, J. L. (2006) · Administrative Science Quarterly · doi:10.2189/asqu.51.3.451 · Cited by 1,152 · see citing works
Climate as a moderator of the relationship between leader-member exchange and content specific citizenship: Safety climate as an exemplar.
Hofmann, D. A., Morgeson, F. P., & Gerras, S. J. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.1.170 · Cited by 1,091 · see citing works
DOES DECISION PROCESS MATTER? A STUDY OF STRATEGIC DECISION-MAKING EFFECTIVENESS.
Dean, J. W., & Sharfman, M. P. (1996) · Academy of Management Journal · doi:10.2307/256784 · Cited by 1,071 · see citing works
Relationships between psychological climate perceptions and work outcomes: a meta‐analytic review
Parker, C. P., Baltes, B. B., Young, S. A., Huff, J. W., Altmann, R. A., Lacost, H. A., & Roberts, J. E. (2003) · Journal of Organizational Behavior · doi:10.1002/job.198 · Cited by 1,078 · see citing works
Opportunity Recognition as the Detection of Meaningful Patterns: Evidence from Comparisons of Novice and Experienced Entrepreneurs
Baron, R. A., & Ensley, M. D. (2006) · Management Science · doi:10.1287/mnsc.1060.0538 · Cited by 1,331 · see citing works
Anaesthetists’ Non-Technical Skills (ANTS): evaluation of a behavioural marker system † †Declaration of interest: The ANTS system was developed under research funding from the Scottish Council for Postgraduate Medical and Dental Education, now part of NHS Education for Scotland, through grants to the University of Aberdeen from September 1999 to August 2003. The views presented in this paper are those of the authors and should not be taken to represent the position or policy of the funding body.
Fletcher, G., Flin, R., McGeorge, P., Glavin, R., Maran, N., & Patey, R. (2003) · British Journal of Anaesthesia · doi:10.1093/bja/aeg112 · Cited by 958 · see citing works
Strategic leadership for exploration and exploitation: The moderating role of environmental dynamism
Jansen, J. J. P., Vera, D. M., & Crossan, M. M. (2008) · The Leadership Quarterly · doi:10.1016/j.leaqua.2008.11.008 · Cited by 959 · see citing works
Cognition-based and affect-based trust as mediators of leader behavior influences on team performance
Schaubroeck, J., Lam, S. S. K., & Peng, A. C. (2011) · Journal of Applied Psychology · doi:10.1037/a0022625 · Cited by 933 · see citing works
THE IMPACT OF TEAM EMPOWERMENT ON VIRTUAL TEAM PERFORMANCE: THE MODERATING ROLE OF FACE-TO-FACE INTERACTION.
Kirkman, B. L., Rosen, B., Tesluk, P. E., & Gibson, C. B. (2004) · Academy of Management Journal · doi:10.2307/20159571 · Cited by 889 · see citing works
MONKEY SEE, MONKEY DO: THE INFLUENCE OF WORK GROUPS ON THE ANTISOCIAL BEHAVIOR OF EMPLOYEES.
Robinson, S. L., & O’Leary-Kelly, A. M. (1998) · Academy of Management Journal · doi:10.2307/256963 · Cited by 847 · see citing works
A CROSS‐LEVEL INVESTIGATION OF FACTORS INFLUENCING UNSAFE BEHAVIORS AND ACCIDENTS
Hofmann, D. A., & Stetzer, A. (1996) · Personnel Psychology · doi:10.1111/j.1744-6570.1996.tb01802.x · Cited by 838 · see citing works
Family Control and Family Firm Valuation by Family CEOs: The Importance of Intentions for Transgenerational Control
Zellweger, T. M., Kellermanns, F. W., Chrisman, J. J., & Chua, J. H. (2011) · Organization Science · doi:10.1287/orsc.1110.0665 · Cited by 863 · see citing works
APPLICANT PERSONALITY, ORGANIZATIONAL CULTURE, AND ORGANIZATION ATTRACTION
Judge, T. A., & Cable, D. M. (1997) · Personnel Psychology · doi:10.1111/j.1744-6570.1997.tb00912.x · Cited by 830 · see citing works
Effectual versus predictive logics in entrepreneurial decision-making: Differences between experts and novices
Dew, N., Read, S., Sarasvathy, S. D., & Wiltbank, R. (2008) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2008.02.002 · Cited by 830 · see citing works
The effects of leadership dimensions, safety climate, and assigned priorities on minor injuries in work groups
Zohar, D. M. (2002) · Journal of Organizational Behavior · doi:10.1002/job.130 · Cited by 819 · see citing works
The importance of vertical and shared leadership within new venture top management teams: Implications for the performance of startups
Ensley, M. D., Hmieleski, K. M., & Pearce, C. L. (2006) · The Leadership Quarterly · doi:10.1016/j.leaqua.2006.02.002 · Cited by 820 · see citing works
A Healthy Divide: Subgroups as a Stimulus for Team Learning Behavior
Gibson, C. B., & Vermeulen, F. (2003) · Administrative Science Quarterly · doi:10.2307/3556657 · Cited by 764 · see citing works
The cross‐level effects of culture and climate in human service teams
Glisson, C., & James, L. R. (2002) · Journal of Organizational Behavior · doi:10.1002/job.162 · Cited by 758 · see citing works
Self-citation
Recalculation of the Critical Values for Lawshe’s Content Validity Ratio
Wilson, F. R., Pan, W., & Schumsky, D. A. (2012) · Measurement and Evaluation in Counseling and Development · doi:10.1177/0748175612440286 · Cited by 703 · see citing works
Using the five-factor model to represent the DSM-IV personality disorders: An expert consensus approach.
Lynam, D. R., & Widiger, T. A. (2001) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.110.3.401 · Cited by 688 · see citing works
EMPLOYEE SILENCE ON CRITICAL WORK ISSUES: THE CROSS LEVEL EFFECTS OF PROCEDURAL JUSTICE CLIMATE
Tangirala, S., & Ramanujam, R. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00105.x · Cited by 689 · see citing works
MEASUREMENT ERROR IN RESEARCH ON HUMAN RESOURCES and FIRM PERFORMANCE: HOW MUCH ERROR IS THERE AND HOW DOES IT INFLUENCE EFFECT SIZE ESTIMATES?
Gerhart, B. A., Wright, P. M., MAHAN, G. C. M., & Snell, S. A. (2000) · Personnel Psychology · doi:10.1111/j.1744-6570.2000.tb02418.x · Cited by 679 · see citing works
Change Readiness: A Multilevel Review
Rafferty, A. E., Jimmieson, N. L., & Armenakis, A. A. (2013) · Journal of Management · doi:10.1177/0149206312457417 · Cited by 679 · see citing works
The Impact of Information Technology and Transactive Memory Systems on Knowledge Sharing, Application, and Team Performance: A Field Study1
Choi, S. Y., Lee, H., & Yoo, Y. (2010) · MIS Quarterly · doi:10.2307/25750708 · Cited by 679 · see citing works
Human Resource Configurations, Intellectual Capital, and Organizational Performance.
Youndt, M. A., & Snell, S. A. (2004) · Journal of managerial issues · Cited by 654 · see citing works
Handbook of Industrial, Work & Organizational Psychology
Anderson, N. R., Öneş, D. S., Sinangil, H. K., & Viswesvaran, C. (2001) · Sage eBooks · Cited by 632 · see citing works
Organizational and psychological climate: A review of theory and research
James, L., Choi, C. C., Ko, C.-H. E., McNeil, P., Minton, M. K., Wright, M., & Kim, K.-I. (2008) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320701662550 · Cited by 621 · see citing works
Self-citation
Transformational Leadership's Role in Promoting Corporate Entrepreneurship: Examining the CEO-TMT Interface
Ling, Y., Şimşek, Z., Lubatkin, M., & Veiga, J. F. (2008) · Academy of Management Journal · doi:10.5465/amj.2008.32626023 · Cited by 623 · see citing works
Do bridging ties complement strong ties? An empirical examination of alliance ambidexterity
Tiwana, A. (2007) · Strategic Management Journal · doi:10.1002/smj.666 · Cited by 622 · see citing works
Composition, process, and performance in self-managed groups: The role of personality.
Barry, B., & Stewart, G. L. (1997) · Journal of Applied Psychology · doi:10.1037/0021-9010.82.1.62 · Cited by 619 · see citing works
Assessing the Organizational Social Context (OSC) of Mental Health Services: Implications for Research and Practice
Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., Green, P., & The Research Network on Youth Mental Health (2008) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-007-0148-5 · Cited by 608 · see citing works
Senior Team Attributes and Organizational Ambidexterity: The Moderating Role of Transformational Leadership
Jansen, J. J. P., George, G., Bosch, F. A. J. V. D., & Volberda, H. (2008) · Journal of Management Studies · doi:10.1111/j.1467-6486.2008.00775.x · Cited by 613 · see citing works
The Antecedents, Consequences, and Mediating Role of Organizational Ambidexterity
Gibson, C. B., & Birkinshaw, J. (2004) · Academy of Management Journal · doi:10.5465/20159573 · Cited by 636 · see citing works
Leader-member exchange, differentiation, and task interdependence: Implications for individual and group performance
Liden, R. C., Erdogan, B., Wayne, S. J., & Sparrowe, R. T. (2006) · Journal of Organizational Behavior · doi:10.1002/job.409 · Cited by 595 · see citing works
Knowledge transfer between groups via personnel rotation: Effects of social identity and knowledge quality
Kane, A. A., Argote, L., & Levine, J. M. (2004) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2004.09.002 · Cited by 594 · see citing works
TIME, TEAMS, AND TASK PERFORMANCE: CHANGING EFFECTS OF SURFACE- AND DEEP-LEVEL DIVERSITY ON GROUP FUNCTIONING.
Price, K. H., Harrison, D. A., Gavin, J. H., & Florey, A. T. (2002) · Academy of Management Journal · doi:10.5465/3069328 · Cited by 583 · see citing works
How Does Trust Affect the Performance of Ongoing Teams? The Mediating Role of Reflexivity, Monitoring, and Effort
Jong, B. A. D., & Elfring, T. (2010) · Academy of Management Journal · doi:10.5465/amj.2010.51468649 · Cited by 578 · see citing works
INDIVIDUAL CREATIVITY AND GROUP ABILITY TO UTILIZE INDIVIDUAL CREATIVE RESOURCES: A MULTILEVEL MODEL.
Taggar, S. (2002) · Academy of Management Journal · doi:10.2307/3069349 · Cited by 834 · see citing works
Organizational climate and company productivity: The role of employee affect and employee level
Patterson, M. G., Warr, P. B., & West, M. A. (2004) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317904774202144 · Cited by 575 · see citing works
Understanding the dynamics of new venture top management teams: cohesion, conflict, and new venture performance
Ensley, M. D., Pearson, A. W., & Amason, A. C. (2002) · Journal of Business Venturing · doi:10.1016/s0883-9026(00)00065-3 · Cited by 565 · see citing works
An empirical investigation of KM styles and their effect on corporate performance
Choi, B., & Lee, H. (2003) · Information & Management · doi:10.1016/s0378-7206(02)00060-5 · Cited by 567 · see citing works
The evolving theory of quality management: The role of Six Sigma
Zu, X., Fredendall, L. D., & Douglas, T. J. (2008) · Journal of Operations Management · doi:10.1016/j.jom.2008.02.001 · Cited by 566 · see citing works
The Impact of Justice Climate and Justice Orientation on Work Outcomes: A Cross-Level Multifoci Framework.
Liao, H., & Rupp, D. E. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.2.242 · Cited by 563 · see citing works
The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership
Aarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014) · Implementation Science · doi:10.1186/1748-5908-9-45 · Cited by 559 · see citing works
Climate quality and climate consensus as mediators of the relationship between organizational antecedents and outcomes.
Lindell, M. K., & Brandt, C. J. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.3.331 · Cited by 559 · see citing works
Modifying supervisory practices to improve subunit safety: A leadership-based intervention model.
Zohar, D. M. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.1.156 · Cited by 550 · see citing works
Implementing New Practices: An Empirical Study of Organizational Learning in Hospital Intensive Care Units
Tucker, A. L., Nembhard, I. M., & Edmondson, A. C. (2007) · Management Science · doi:10.1287/mnsc.1060.0692 · Cited by 544 · see citing works
Climate as a Social-Cognitive Construction of Supervisory Safety Practices: Scripts as Proxy of Behavior Patterns.
Zohar, D. M., & Luria, G. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.2.322 · Cited by 540 · see citing works
Applying multilevel confirmatory factor analysis techniques to the study of leadership
Dyer, N., Hanges, P. J., & Hall, R. J. (2005) · The Leadership Quarterly · doi:10.1016/j.leaqua.2004.09.009 · Cited by 538 · see citing works
DO THEY DO WHAT THEY BELIEVE THEY CAN? GROUP EFFICACY AND GROUP EFFECTIVENESS ACROSS TASKS AND CULTURES.
Gibson, C. B. (1999) · Academy of Management Journal · doi:10.2307/257089 · Cited by 532 · see citing works
Too real for comfort? Uncanny responses to computer generated faces
MacDorman, K. F., Green, R. D., Ho, C.-C., & Koch, C. T. (2009) · Computers in Human Behavior · doi:10.1016/j.chb.2008.12.026 · Cited by 533 · see citing works
Authentic Leadership and Positive Psychological Capital
Clapp‐Smith, R., Vogelgesang, G. R., & Avey, J. B. (2008) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051808326596 · Cited by 551 · see citing works
Catching moods and hitting runs: Mood linkage and subjective performance in professional sport teams.
Totterdell, P. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.6.848 · Cited by 525 · see citing works
Organization structure as a moderator of the relationship between procedural justice, interactional justice, perceived organizational support, and supervisory trust.
Ambrose, M. L., & Schminke, M. J. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.2.295 · Cited by 524 · see citing works
Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings
Gaba, D. M., Howard, S. K., Flanagan, B. T., Smith, B. E., Fish, K. J., & Botney, R. (1998) · Anesthesiology · doi:10.1097/00000542-199807000-00005 · Cited by 512 · see citing works
Work team trust and effectiveness
Costa, A. C. (2003) · Personnel Review · doi:10.1108/00483480310488360 · Cited by 527 · see citing works
Surgeons’ Non‐technical Skills in the Operating Room: Reliability Testing of the NOTSS Behavior Rating System
Yule, S., Flin, R. H., Maran, N., ROWLEY, D. I., Youngson, G. G., & Paterson‐Brown, S. (2008) · World Journal of Surgery · doi:10.1007/s00268-007-9320-z · Cited by 507 · see citing works
Hotels' environmental policies and employee personal environmental beliefs: Interactions and outcomes
Chou, C.-J. (2013) · Tourism Management · doi:10.1016/j.tourman.2013.08.001 · Cited by 503 · see citing works
Translating Team Creativity to Innovation Implementation
Somech, A., & Drach‐Zahavy, A. (2011) · Journal of Management · doi:10.1177/0149206310394187 · Cited by 504 · see citing works
Developing and testing a measure for the ethical culture of organizations: the corporate ethical virtues model
Kaptein, M. (2008) · Journal of Organizational Behavior · doi:10.1002/job.520 · Cited by 522 · see citing works
How Ethical Leadership Influence Employees’ Innovative Work Behavior: A Perspective of Intrinsic Motivation
Tu, Y., & Lu, X. (2012) · Journal of Business Ethics · doi:10.1007/s10551-012-1455-7 · Cited by 483 · see citing works
Abusive supervision and subordinates’ organization deviance
Tepper, B. J., Henle, C. A., Lambert, L. S., Giacalone, R. A., & Duffy, M. K. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.4.721 · Cited by 468 · see citing works
Knowledge management strategy and its link to knowledge creation process
Choi, B., & Lee, H. (2002) · Expert Systems with Applications · doi:10.1016/s0957-4174(02)00038-6 · Cited by 466 · see citing works
On Average Deviation Indices for Estimating Interrater Agreement
Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999) · Organizational Research Methods · doi:10.1177/109442819921004 · Cited by 462 · see citing works
The relationship between individual creativity and team creativity: aggregating across people and time
Pirola‐Merlo, A., & Mann, L. (2004) · Journal of Organizational Behavior · doi:10.1002/job.240 · Cited by 460 · see citing works
Relation of principal transformational leadership to school staff job satisfaction, staff turnover, and school performance
Griffith, J. E. (2004) · Journal of Educational Administration · doi:10.1108/09578230410534667 · Cited by 456 · see citing works
Innovativeness in family firms: a family influence perspective
Kellermanns, F. W., Eddleston, K. A., Sarathy, R., & Murphy, F. (2010) · Small Business Economics · doi:10.1007/s11187-010-9268-5 · Cited by 459 · see citing works
The effect of leader moral development on ethical climate and employee attitudes
Schminke, M. J., Ambrose, M. L., & Neubaum, D. O. (2005) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2005.03.006 · Cited by 455 · see citing works
Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide
Burke, M. J., & Dunlap, W. P. (2002) · Organizational Research Methods · doi:10.1177/1094428102005002002 · Cited by 448 · see citing works
“Yes, I Can, I Feel Good, and I Just Do It!” On Gain Cycles and Spirals of Efficacy Beliefs, Affect, and Engagement
Salanova, M., Llorens, S., & Schaufeli, W. B. (2010) · Applied Psychology · doi:10.1111/j.1464-0597.2010.00435.x · Cited by 461 · see citing works
Human resources management and firm performance: The differential role of managerial affective and continuance commitment.
Gong, Y., Law, K. S., Chang, S., & Xin, K. (2009) · Journal of Applied Psychology · doi:10.1037/a0013116 · Cited by 447 · see citing works
Top Management Attention to Innovation: The Role of Search Selection and Intensity in New Product Introductions
Li, Q., Maggitti, P. G., Smith, K. G., Tesluk, P. E., & Katila, R. (2012) · Academy of Management Journal · doi:10.5465/amj.2010.0844 · Cited by 439 · see citing works

Authors: James, L. R., Demaree, R. G., & Wolf, G.
Affiliations: University of Tennessee at Knoxville (US)
Venue: Journal of Applied Psychology
Publisher: American Psychological Association (APA)
Published: 1993-04
Volume: 78
Issue: 2
Pages: 306-309
ISSN: 0021-9010 (ISSN-L)
1939-1854
DOI: 10.1037/0021-9010.78.2.306
OpenAlex: W2022950201
Cited by: 2,221
References: 21
Citations by year: 2026: 54 · 2025: 88 · 2024: 76 · 2023: 85 · 2022: 88 · 2021: 97 · 2020: 95 · 2019: 98 · 2018: 108 · 2017: 82 · 2016: 86 · 2015: 100 · 2014: 111 · 2013: 110 · 2012: 96
FWCI: 12.64
Type: journal article
Language: en
Open access: closed
Locations: Journal of Applied Psychology (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
Keywords: inter-rater reliability, interobserver agreement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

rwg: an assessment of within-group interrater agreement

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks