Cited by 2,221Reliability and Agreement in MeasurementPsychometric Methodologies and TestingTeam Dynamics and Performance

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

rwg: an assessment of within-group interrater agreement

James, Lawrence R., Demaree, Robert G., Wolf, Gerrit·1993·Journal of Applied Psychology

DOI 10.1037/0021-9010.78.2.306

Open this paper in CoChat — free account

Abstract

Schmidt and Hunter (1989) critiqued the within-group interrater reliability statistic (r wg ) described by James, Demaree, and Wolf (1984). Kozlowski and Hattrup (1992) responded to the Schmidt and Hunter critique and argued that r wg is a suitable index of interrater agreement. This article focuses on the interpretation of r wg as a measure of agreement among judges' ratings of a single target. A new derivation of r wg is given that underscores this interpretation

Integrity signals

Registry

Crossref · 10.1037/0021-9010.78.2.306

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,009
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
Self-citation
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Employee and customer perceptions of service in banks: Replication and extension.
Schneider, B., & Bowen, D. E. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.3.423 · Cited by 1,344
Superiors' evaluations and subordinates' perceptions of transformational and transactional leadership.
Hater, J. J., & Bass, B. M. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.4.695 · Cited by 1,136
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 883
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 830

Flagged first, then by citations

Show all 21 references

Sorted by citations

KNOWLEDGE TRANSFER IN INTRAORGANIZATIONAL NETWORKS: EFFECTS OF NETWORK POSITION AND ABSORPTIVE CAPACITY ON BUSINESS UNIT INNOVATION AND PERFORMANCE.
Tsai, W. (2001) · Academy of Management Journal · doi:10.2307/3069443 · Cited by 4,352
THE ANTECEDENTS, CONSEQUENCES, AND MEDIATING ROLE OF ORGANIZATIONAL AMBIDEXTERITY.
Gibson, C. B., & Birkinshaw, J. (2004) · Academy of Management Journal · doi:10.2307/20159573 · Cited by 3,951
Exploratory Innovation, Exploitative Innovation, and Performance: Effects of Organizational Antecedents and Environmental Moderators
Jansen, J. J. P., Van Den Bosch, F. A. J., & Volberda, H. W. (2006) · Management Science · doi:10.1287/mnsc.1060.0576 · Cited by 3,705
“We calculated an interrater agreement score (rwg) for data on these variables (James et al. 1993).”
Method
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608
“Schmidt and Hunter (1989) critiqued the rWG and rWGðJÞ indices, largely based on semantic confusion arising from earlier writers’ labels of the rWG indices as reliability coefficients (James et al., 1984) versus agreement coefficients (James et al., 1993; Kozlowski & Hattrup, 1992).”
BackgroundMethodInfluential
What's the difference? diversity constructs as separation, variety, or disparity in organizations
Harrison, D. A., & Klein, K. J. (2007) · Academy of Management Review · doi:10.5465/amr.2007.26586096 · Cited by 3,297
ADOPTION OF AN ORGANIZATIONAL PRACTICE BY SUBSIDIARIES OF MULTINATIONAL CORPORATIONS: INSTITUTIONAL AND RELATIONAL EFFECTS.
Kostova, T., & Roth, K. (2002) · Academy of Management Journal · doi:10.2307/3069293 · Cited by 2,775
Untitled work
(2000) · Cited by 2,531
HUMAN RESOURCE MANAGEMENT, MANUFACTURING STRATEGY, AND FIRM PERFORMANCE.
Youndt, M. A., Snell, S. A., J. W. DEAN, Jr., & Lepak, D. P. (1996) · Academy of Management Journal · doi:10.2307/256714 · Cited by 2,274
Knowledge Management Enablers, Processes, and Organizational Performance: An Integrative View and Empirical Examination
Lee, H., & Choi, B. (2003) · Journal of Management Information Systems · doi:10.1080/07421222.2003.11045756 · Cited by 2,269
“[58] developed indices appropriate for within-group agreement for a set of raters rating a single target with a single item (rwg(1)) or multiple-item scale (rwg(J)).”
Background
Managing Potential and Realized Absorptive Capacity: How do Organizational Antecedents Matter?
Jansen, J. J. P., Bosch, F. A. J. V. D., & Volberda, H. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.19573106 · Cited by 2,326
Ambidexterity and Performance in Small-to Medium-Sized Firms: The Pivotal Role of Top Management Team Behavioral Integration
Lubatkin, M., Simsek, Z., Ling, Y., & Veiga, J. F. (2006) · Journal of Management · doi:10.1177/0149206306290712 · Cited by 2,230
“We also used an interrater reliability coefficient developed by James, Demaree, and Wolf (1993) to examine the intragroup reliability (rwg) of responses.”
Method
Servant leadership: Development of a multidimensional measure and multi-level assessment
Liden, R. C., Wayne, S. J., Zhao, H., & Henderson, D. (2008) · The Leadership Quarterly · doi:10.1016/j.leaqua.2008.01.006 · Cited by 2,058
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053
“…the interpretation of rwg as an IRR coefficient; Kozlowski and Hattrup (1992) argued that rwg is properly interpreted as an IRA index, not an IRR coefficient; James, Demaree, and Wolf (1993) concurred; and rwg is now more widely (and correctly) interpreted as an IRA index, not an IRR coefficient.”
Background
Social Structure of “Coopetition” Within a Multiunit Organization: Coordination, Competition, and Intraorganizational Knowledge Sharing
Tsai, W. (2002) · Organization Science · doi:10.1287/orsc.13.2.179.536 · Cited by 1,960
“For nonrelational data (such as centralization), I used the methods prescribed by James et al. (1984, 1993) to calculate interrater agreement (see e.g., Kozlowski and Hattrup (1992) for a further discussion on the measure of interrater agreement).”
Method
Let's put the person back into entrepreneurship research: A meta-analysis on the relationship between business owners' personality traits, business creation, and success
Rauch, A., & Fresé, M. (2007) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320701595438 · Cited by 1,879
“We only analysed traits that showed high interrater agreement (James, Demaree, & Wolf, 1993 ).”
Method
Transformational leadership and organizational commitment: mediating role of psychological empowerment and moderating role of structural distance
Avolio, B. J., Zhu, W., Koh, W. L., & Bhatia, P. (2004) · Journal of Organizational Behavior · doi:10.1002/job.283 · Cited by 1,702
Levels issues in theory development, data collection and analysis
Klein, K. J., Dansereau, F., & Hall, R. J. (1994) · The Academy of Management Review · doi:10.2307/258703 · Cited by 1,607
BEYOND SELF-MANAGEMENT: ANTECEDENTS AND CONSEQUENCES OF TEAM EMPOWERMENT.
Kirkman, B. L., & Rosen, B. (1999) · Academy of Management Journal · doi:10.2307/256874 · Cited by 2,110
An Overview of the Logic and Rationale of Hierarchical Linear Models
Hofmann, D. (1997) · Journal of Management · doi:10.1177/014920639702300602 · Cited by 1,681
A group-level model of safety climate: Testing the effect of group climate on microaccidents in manufacturing jobs.
Zohar, D. M. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.4.587 · Cited by 1,559
High-Performance Human Resource Practices, Citizenship Behavior, and Organizational Performance: A Relational Perspective
Sun, L., Aryee, S., & Law, K. S. (2007) · Academy of Management Journal · doi:10.5465/amj.2007.25525821 · Cited by 1,506
TIME, TEAMS, AND TASK PERFORMANCE: CHANGING EFFECTS OF SURFACE- AND DEEP-LEVEL DIVERSITY ON GROUP FUNCTIONING.
Price, K. H., Harrison, D. A., Gavin, J. H., & Florey, A. T. (2002) · Academy of Management Journal · doi:10.2307/3069328 · Cited by 1,594
Five-factor model of personality and transformational leadership
Judge, T. A., & Bono, J. E. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.5.751 · Cited by 1,382
Validating the organizational climate measure: links to managerial practices, productivity and innovation
Patterson, M., West, M. A., Shackleton, V., Dawson, J., Lawthom, R., Maitlis, S., Robinson, D. L., & Wallace, A. M. (2005) · Journal of Organizational Behavior · doi:10.1002/job.312 · Cited by 1,318
“The within-group agreement index of multiple item scales developed by James, Demaree, and Wolf (1984, 1993) is a technique for assessing agreement among the judgments made on a single variable in regard to a single target.”
Method
Intellectual Capital Profiles: An Examination of Investments and Returns*
Youndt, M. A., Subramaniam, M., & Snell, S. A. (2004) · Journal of Management Studies · doi:10.1111/j.1467-6486.2004.00435.x · Cited by 1,280
“To make certain the data could be aggregated to the organizational level, Rwg, or interrater agreement, was calculated for each of the variables according to the procedures prescribed by James et al. (1993) .”
Method
The Structure and Function of Collective Constructs: Implications for Multilevel Research and Theory Development
Morgeson, F. P., & Hofmann, D. A. (1999) · Academy of Management Review · doi:10.5465/amr.1999.1893935 · Cited by 1,224
Measurement reliability and agreement in psychiatry
Shrout, P. E. (1998) · Statistical Methods in Medical Research · doi:10.1177/096228029800700306 · Cited by 1,211
Coordinating Expertise in Software Development Teams
Faraj, S., & Sproull, L. (2000) · Management Science · doi:10.1287/mnsc.46.12.1554.12072 · Cited by 1,689
Unpacking the Concept of Virtuality: The Effects of Geographic Dispersion, Electronic Dependence, Dynamic Structure, and National Diversity on Team Innovation
Gibson, C. B., & Gibbs, J. L. (2006) · Administrative Science Quarterly · doi:10.2189/asqu.51.3.451 · Cited by 1,152
“Finally, rwg was used to assess internal agreement within a team for each variable, ranging from 0 (no agreement) to 1 (complete agreement) (James, Demaree, and Wolf, 1993).”
Method
Climate as a moderator of the relationship between leader-member exchange and content specific citizenship: Safety climate as an exemplar.
Hofmann, D. A., Morgeson, F. P., & Gerras, S. J. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.1.170 · Cited by 1,091
“Both Katz and Kahn (1978) and Graen (1976) suggested that individuals accomplish work in organizations by engaging in roles that are expected of them by other individuals within the organization.”
Background
DOES DECISION PROCESS MATTER? A STUDY OF STRATEGIC DECISION-MAKING EFFECTIVENESS.
Dean, J. W., & Sharfman, M. P. (1996) · Academy of Management Journal · doi:10.2307/256784 · Cited by 1,071
Relationships between psychological climate perceptions and work outcomes: a meta‐analytic review
Parker, C. P., Baltes, B. B., Young, S. A., Huff, J. W., Altmann, R. A., Lacost, H. A., & Roberts, J. E. (2003) · Journal of Organizational Behavior · doi:10.1002/job.198 · Cited by 1,079
Opportunity Recognition as the Detection of Meaningful Patterns: Evidence from Comparisons of Novice and Experienced Entrepreneurs
Baron, R. A., & Ensley, M. D. (2006) · Management Science · doi:10.1287/mnsc.1060.0538 · Cited by 1,331
Anaesthetists’ Non-Technical Skills (ANTS): evaluation of a behavioural marker system † †Declaration of interest: The ANTS system was developed under research funding from the Scottish Council for Postgraduate Medical and Dental Education, now part of NHS Education for Scotland, through grants to the University of Aberdeen from September 1999 to August 2003. The views presented in this paper are those of the authors and should not be taken to represent the position or policy of the funding body.
Fletcher, G., Flin, R., McGeorge, P., Glavin, R., Maran, N., & Patey, R. (2003) · British Journal of Anaesthesia · doi:10.1093/bja/aeg112 · Cited by 958
Strategic leadership for exploration and exploitation: The moderating role of environmental dynamism
Jansen, J. J. P., Vera, D. M., & Crossan, M. M. (2008) · The Leadership Quarterly · doi:10.1016/j.leaqua.2008.11.008 · Cited by 959
“We calculated an inter-rater agreement score (rwg) for each variable (James et al., 1993).”
Method
Cognition-based and affect-based trust as mediators of leader behavior influences on team performance
Schaubroeck, J., Lam, S. S. K., & Peng, A. C. (2011) · Journal of Applied Psychology · doi:10.1037/a0022625 · Cited by 933
THE IMPACT OF TEAM EMPOWERMENT ON VIRTUAL TEAM PERFORMANCE: THE MODERATING ROLE OF FACE-TO-FACE INTERACTION.
Kirkman, B. L., Rosen, B., Tesluk, P. E., & Gibson, C. B. (2004) · Academy of Management Journal · doi:10.2307/20159571 · Cited by 889
MONKEY SEE, MONKEY DO: THE INFLUENCE OF WORK GROUPS ON THE ANTISOCIAL BEHAVIOR OF EMPLOYEES.
Robinson, S. L., & O’Leary-Kelly, A. M. (1998) · Academy of Management Journal · doi:10.2307/256963 · Cited by 872
A CROSS‐LEVEL INVESTIGATION OF FACTORS INFLUENCING UNSAFE BEHAVIORS AND ACCIDENTS
Hofmann, D. A., & Stetzer, A. (1996) · Personnel Psychology · doi:10.1111/j.1744-6570.1996.tb01802.x · Cited by 838
Family Control and Family Firm Valuation by Family CEOs: The Importance of Intentions for Transgenerational Control
Zellweger, T. M., Kellermanns, F. W., Chrisman, J. J., & Chua, J. H. (2011) · Organization Science · doi:10.1287/orsc.1110.0665 · Cited by 918
“For this subset, we calculated the coefficient of agreement (rwg5 (James et al. 1984, 1993).”
Method
APPLICANT PERSONALITY, ORGANIZATIONAL CULTURE, AND ORGANIZATION ATTRACTION
Judge, T. A., & Cable, D. M. (1997) · Personnel Psychology · doi:10.1111/j.1744-6570.1997.tb00912.x · Cited by 830
Effectual versus predictive logics in entrepreneurial decision-making: Differences between experts and novices
Dew, N., Read, S., Sarasvathy, S. D., & Wiltbank, R. (2008) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2008.02.002 · Cited by 898
“The two sets of codings were compared to examine inter-rater agreement (James et al., 1993) and the results revealed a strong mean agreement of .82 across the variables used in this study, with no specific variable scoring lower than .67.”
BackgroundMethod
The effects of leadership dimensions, safety climate, and assigned priorities on minor injuries in work groups
Zohar, D. M. (2002) · Journal of Organizational Behavior · doi:10.1002/job.130 · Cited by 819
The importance of vertical and shared leadership within new venture top management teams: Implications for the performance of startups
Ensley, M. D., Hmieleski, K. M., & Pearce, C. L. (2006) · The Leadership Quarterly · doi:10.1016/j.leaqua.2006.02.002 · Cited by 820
A Healthy Divide: Subgroups as a Stimulus for Team Learning Behavior
Gibson, C. B., & Vermeulen, F. (2003) · Administrative Science Quarterly · doi:10.2307/3556657 · Cited by 786
The cross‐level effects of culture and climate in human service teams
Glisson, C., & James, L. R. (2002) · Journal of Organizational Behavior · doi:10.1002/job.162 · Cited by 758
Self-citation
Recalculation of the Critical Values for Lawshe’s Content Validity Ratio
Wilson, F. R., Pan, W., & Schumsky, D. A. (2012) · Measurement and Evaluation in Counseling and Development · doi:10.1177/0748175612440286 · Cited by 731
Using the five-factor model to represent the DSM-IV personality disorders: An expert consensus approach.
Lynam, D. R., & Widiger, T. A. (2001) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.110.3.401 · Cited by 688
“This coefficient has been offered as a means of "assessing agreement among the judgements made by a single group of judges on a single variable in regard to a single target" (James, Demaree, & Wolf, 1984, as cited in James, Demaree, & Wolf, 1993 ) and represents the proportional reduction in error variance relative to…”
Method
EMPLOYEE SILENCE ON CRITICAL WORK ISSUES: THE CROSS LEVEL EFFECTS OF PROCEDURAL JUSTICE CLIMATE
Tangirala, S., & Ramanujam, R. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00105.x · Cited by 714
MEASUREMENT ERROR IN RESEARCH ON HUMAN RESOURCES and FIRM PERFORMANCE: HOW MUCH ERROR IS THERE AND HOW DOES IT INFLUENCE EFFECT SIZE ESTIMATES?
Gerhart, B. A., Wright, P. M., MAHAN, G. C. M., & Snell, S. A. (2000) · Personnel Psychology · doi:10.1111/j.1744-6570.2000.tb02418.x · Cited by 679
Change Readiness: A Multilevel Review
Rafferty, A. E., Jimmieson, N. L., & Armenakis, A. A. (2013) · Journal of Management · doi:10.1177/0149206312457417 · Cited by 679
The Impact of Information Technology and Transactive Memory Systems on Knowledge Sharing, Application, and Team Performance: A Field Study1
Choi, S. Y., Lee, H., & Yoo, Y. (2010) · MIS Quarterly · doi:10.2307/25750708 · Cited by 697
Human Resource Configurations, Intellectual Capital, and Organizational Performance.
Youndt, M. A., & Snell, S. A. (2004) · Journal of managerial issues · Cited by 654
Handbook of Industrial, Work & Organizational Psychology
Anderson, N. R., Öneş, D. S., Sinangil, H. K., & Viswesvaran, C. (2001) · Sage eBooks · Cited by 632
Organizational and psychological climate: A review of theory and research
James, L. R., Choi, C. C., Ko, C.-H. E., McNeil, P. K., Minton, M. K., Wright, M. A., & Kim, K. (2008) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320701662550 · Cited by 691
“They questioned the theoretical basis of rwg(j) (James et al., 1984, 1993), which is an index of agreement among judges’ ratings of a single target on multiple items.”
BackgroundSelf-citation
Transformational Leadership's Role in Promoting Corporate Entrepreneurship: Examining the CEO-TMT Interface
Ling, Y., Şimşek, Z., Lubatkin, M., & Veiga, J. F. (2008) · Academy of Management Journal · doi:10.5465/amj.2008.32626023 · Cited by 623
Do bridging ties complement strong ties? An empirical examination of alliance ambidexterity
Tiwana, A. (2007) · Strategic Management Journal · doi:10.1002/smj.666 · Cited by 640
“Before such aggregation can be done, it is necessary to assess within-project interrater agreement using the Intraclass Correlation Coefficient (James, Demaree, and Wolf, 1993).”
MethodInfluential
Composition, process, and performance in self-managed groups: The role of personality.
Barry, B., & Stewart, G. L. (1997) · Journal of Applied Psychology · doi:10.1037/0021-9010.82.1.62 · Cited by 619
Assessing the Organizational Social Context (OSC) of Mental Health Services: Implications for Research and Practice
Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., Green, P., & The Research Network on Youth Mental Health (2008) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-007-0148-5 · Cited by 608
Senior Team Attributes and Organizational Ambidexterity: The Moderating Role of Transformational Leadership
Jansen, J. J. P., George, G., Bosch, F. A. J. V. D., & Volberda, H. (2008) · Journal of Management Studies · doi:10.1111/j.1467-6486.2008.00775.x · Cited by 613
The Antecedents, Consequences, and Mediating Role of Organizational Ambidexterity
Gibson, C. B., & Birkinshaw, J. (2004) · Academy of Management Journal · doi:10.5465/20159573 · Cited by 636
Leader-member exchange, differentiation, and task interdependence: Implications for individual and group performance
Liden, R. C., Erdogan, B., Wayne, S. J., & Sparrowe, R. T. (2006) · Journal of Organizational Behavior · doi:10.1002/job.409 · Cited by 595
Knowledge transfer between groups via personnel rotation: Effects of social identity and knowledge quality
Kane, A. A., Argote, L., & Levine, J. M. (2004) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2004.09.002 · Cited by 594
TIME, TEAMS, AND TASK PERFORMANCE: CHANGING EFFECTS OF SURFACE- AND DEEP-LEVEL DIVERSITY ON GROUP FUNCTIONING.
Price, K. H., Harrison, D. A., Gavin, J. H., & Florey, A. T. (2002) · Academy of Management Journal · doi:10.5465/3069328 · Cited by 583
How Does Trust Affect the Performance of Ongoing Teams? The Mediating Role of Reflexivity, Monitoring, and Effort
Jong, B. A. D., & Elfring, T. (2010) · Academy of Management Journal · doi:10.5465/amj.2010.51468649 · Cited by 616
INDIVIDUAL CREATIVITY AND GROUP ABILITY TO UTILIZE INDIVIDUAL CREATIVE RESOURCES: A MULTILEVEL MODEL.
Taggar, S. (2002) · Academy of Management Journal · doi:10.2307/3069349 · Cited by 834
Organizational climate and company productivity: The role of employee affect and employee level
Patterson, M. G., Warr, P. B., & West, M. A. (2004) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317904774202144 · Cited by 575
Understanding the dynamics of new venture top management teams: cohesion, conflict, and new venture performance
Ensley, M. D., Pearson, A. W., & Amason, A. C. (2002) · Journal of Business Venturing · doi:10.1016/s0883-9026(00)00065-3 · Cited by 592
“Originally developed as a measure of within team reliability, James et al. (1993) noted that it was really a measure of within team agreement.”
Background
An empirical investigation of KM styles and their effect on corporate performance
Choi, B., & Lee, H. (2003) · Information & Management · doi:10.1016/s0378-7206(02)00060-5 · Cited by 642
The evolving theory of quality management: The role of Six Sigma
Zu, X., Fredendall, L. D., & Douglas, T. J. (2008) · Journal of Operations Management · doi:10.1016/j.jom.2008.02.001 · Cited by 585
“70 (James et al., 1993), and the AD values range from 0.”
Background
The Impact of Justice Climate and Justice Orientation on Work Outcomes: A Cross-Level Multifoci Framework.
Liao, H., & Rupp, D. E. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.2.242 · Cited by 563
The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership
Aarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014) · Implementation Science · doi:10.1186/1748-5908-9-45 · Cited by 559
“We next examined the average agreement within clinical work group for individual items and scales using awg(1) and awg(J), respectively [48-50].”
BackgroundMethod
Climate quality and climate consensus as mediators of the relationship between organizational antecedents and outcomes.
Lindell, M. K., & Brandt, C. J. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.3.331 · Cited by 570
Modifying supervisory practices to improve subunit safety: A leadership-based intervention model.
Zohar, D. M. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.1.156 · Cited by 550
Implementing New Practices: An Empirical Study of Organizational Learning in Hospital Intensive Care Units
Tucker, A. L., Nembhard, I. M., & Edmondson, A. C. (2007) · Management Science · doi:10.1287/mnsc.1060.0692 · Cited by 567
Climate as a Social-Cognitive Construction of Supervisory Safety Practices: Scripts as Proxy of Behavior Patterns.
Zohar, D. M., & Luria, G. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.2.322 · Cited by 561
“Homogeneity of climate perceptions was assessed with rwg (James et al., 1993), intraclass correlation (ICC[1]), and reliability of the mean (ICC[2]; James, 1982; Shrout & Fleiss, 1979).”
Method
Applying multilevel confirmatory factor analysis techniques to the study of leadership
Dyer, N., Hanges, P. J., & Hall, R. J. (2005) · The Leadership Quarterly · doi:10.1016/j.leaqua.2004.09.009 · Cited by 538
DO THEY DO WHAT THEY BELIEVE THEY CAN? GROUP EFFICACY AND GROUP EFFECTIVENESS ACROSS TASKS AND CULTURES.
Gibson, C. B. (1999) · Academy of Management Journal · doi:10.2307/257089 · Cited by 532
“The James et al. (1984) within-group agreement (rwJ) of the ratings was .”
Background
Too real for comfort? Uncanny responses to computer generated faces
MacDorman, K. F., Green, R. D., Ho, C.-C., & Koch, C. T. (2009) · Computers in Human Behavior · doi:10.1016/j.chb.2008.12.026 · Cited by 533
Authentic Leadership and Positive Psychological Capital
Clapp‐Smith, R., Vogelgesang, G. R., & Avey, J. B. (2008) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051808326596 · Cited by 551
Catching moods and hitting runs: Mood linkage and subjective performance in professional sport teams.
Totterdell, P. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.6.848 · Cited by 525
Organization structure as a moderator of the relationship between procedural justice, interactional justice, perceived organizational support, and supervisory trust.
Ambrose, M. L., & Schminke, M. J. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.2.295 · Cited by 550
Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings
Gaba, D. M., Howard, S. K., Flanagan, B. T., Smith, B. E., Fish, K. J., & Botney, R. (1998) · Anesthesiology · doi:10.1097/00000542-199807000-00005 · Cited by 512
Work team trust and effectiveness
Costa, A. C. (2003) · Personnel Review · doi:10.1108/00483480310488360 · Cited by 569
Surgeons’ Non‐technical Skills in the Operating Room: Reliability Testing of the NOTSS Behavior Rating System
Yule, S., Flin, R. H., Maran, N., ROWLEY, D. I., Youngson, G. G., & Paterson‐Brown, S. (2008) · World Journal of Surgery · doi:10.1007/s00268-007-9320-z · Cited by 507
Hotels' environmental policies and employee personal environmental beliefs: Interactions and outcomes
Chou, C.-J. (2013) · Tourism Management · doi:10.1016/j.tourman.2013.08.001 · Cited by 544
Translating Team Creativity to Innovation Implementation
Somech, A., & Drach‐Zahavy, A. (2011) · Journal of Management · doi:10.1177/0149206310394187 · Cited by 545
“A value of .70 or above is suggested as a “good” amount of within-group interrater agreement (James et al., 1993).”
Background
Developing and testing a measure for the ethical culture of organizations: the corporate ethical virtues model
Kaptein, M. (2008) · Journal of Organizational Behavior · doi:10.1002/job.520 · Cited by 522
“The goodness-of-fit index (GFI), as developed by Jöreskog and Sörbom (1985; 1993), indicates how well the observed covariance matrix of the analyzed items is reproduced by the estimated covariance matrix. Values larger than 0.9 are required for good-fitting models. As the GFI is sensitive to the number of estimated pa…”
MethodInfluential
How Ethical Leadership Influence Employees’ Innovative Work Behavior: A Perspective of Intrinsic Motivation
Tu, Y., & Lu, X. (2012) · Journal of Business Ethics · doi:10.1007/s10551-012-1455-7 · Cited by 528
“In this study, the rwg were 0.714 for group ethical leadership and 0.785 for intrinsic motivation, both of which were above the critical value of 0.70 and suggested as good with respect to within-group agreement (James et al. 1993).”
BackgroundResultInfluential
Abusive supervision and subordinates’ organization deviance
Tepper, B. J., Henle, C. A., Lambert, L. S., Giacalone, R. A., & Duffy, M. K. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.4.721 · Cited by 468
Knowledge management strategy and its link to knowledge creation process
Choi, B., & Lee, H. (2002) · Expert Systems with Applications · doi:10.1016/s0957-4174(02)00038-6 · Cited by 466
“the extent to which raters make the same ratings ( James, Demaree, & Wolf, 1993 ).”
Background
On Average Deviation Indices for Estimating Interrater Agreement
Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999) · Organizational Research Methods · doi:10.1177/109442819921004 · Cited by 462
“We refer readers interested in comparisons of other interrater agreement and interrater reliability indices to George and James (1993) and James et al. (1984, 1993).”
BackgroundMethod
The relationship between individual creativity and team creativity: aggregating across people and time
Pirola‐Merlo, A., & Mann, L. (2004) · Journal of Organizational Behavior · doi:10.1002/job.240 · Cited by 460
Relation of principal transformational leadership to school staff job satisfaction, staff turnover, and school performance
Griffith, J. E. (2004) · Journal of Educational Administration · doi:10.1108/09578230410534667 · Cited by 503
Innovativeness in family firms: a family influence perspective
Kellermanns, F. W., Eddleston, K. A., Sarathy, R., & Murphy, F. (2010) · Small Business Economics · doi:10.1007/s11187-010-9268-5 · Cited by 459
The effect of leader moral development on ethical climate and employee attitudes
Schminke, M. J., Ambrose, M. L., & Neubaum, D. O. (2005) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2005.03.006 · Cited by 455
Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide
Burke, M. J., & Dunlap, W. P. (2002) · Organizational Research Methods · doi:10.1177/1094428102005002002 · Cited by 448
“Yes, I Can, I Feel Good, and I Just Do It!” On Gain Cycles and Spirals of Efficacy Beliefs, Affect, and Engagement
Salanova, M., Llorens, S., & Schaufeli, W. B. (2010) · Applied Psychology · doi:10.1111/j.1464-0597.2010.00435.x · Cited by 461
Human resources management and firm performance: The differential role of managerial affective and continuance commitment.
Gong, Y., Law, K. S., Chang, S., & Xin, K. (2009) · Journal of Applied Psychology · doi:10.1037/a0013116 · Cited by 447
Top Management Attention to Innovation: The Role of Search Selection and Intensity in New Product Introductions
Li, Q., Maggitti, P. G., Smith, K. G., Tesluk, P. E., & Katila, R. (2012) · Academy of Management Journal · doi:10.5465/amj.2010.0844 · Cited by 439

Authors: James, L. R., Demaree, R. G., & Wolf, G.
Affiliations: University of Tennessee at Knoxville (US)
Venue: Journal of Applied Psychology
Publisher: American Psychological Association (APA)
Published: 1993-04
Volume: 78
Issue: 2
Pages: 306-309
ISSN: 0021-9010 (ISSN-L)
1939-1854
DOI: 10.1037/0021-9010.78.2.306
OpenAlex: W2022950201
Cited by: 2,221
References: 21
Citations by year: 2026: 54 · 2025: 88 · 2024: 76 · 2023: 85 · 2022: 88 · 2021: 97 · 2020: 95 · 2019: 98 · 2018: 108 · 2017: 82 · 2016: 86 · 2015: 100 · 2014: 111 · 2013: 110 · 2012: 96
FWCI: 12.64
Type: journal article
Language: en
Open access: closed
Locations: Journal of Applied Psychology (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
Keywords: inter-rater reliability, interobserver agreement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

rwg: an assessment of within-group interrater agreement

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks