Cited by 2,221Reliability and Agreement in MeasurementPsychometric Methodologies and TestingTeam Dynamics and Performance

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

rwg: an assessment of within-group interrater agreement

James, Lawrence R., Demaree, Robert G., Wolf, Gerrit·1993·Journal of Applied Psychology

DOI 10.1037/0021-9010.78.2.306

Open this paper in CoChat — free account

Abstract

Schmidt and Hunter (1989) critiqued the within-group interrater reliability statistic (r wg ) described by James, Demaree, and Wolf (1984). Kozlowski and Hattrup (1992) responded to the Schmidt and Hunter critique and argued that r wg is a suitable index of interrater agreement. This article focuses on the interpretation of r wg as a measure of agreement among judges' ratings of a single target. A new derivation of r wg is given that underscores this interpretation

Integrity signals

Registry

Crossref · 10.1037/0021-9010.78.2.306

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,009
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
Self-citation
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Employee and customer perceptions of service in banks: Replication and extension.
Schneider, B., & Bowen, D. E. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.3.423 · Cited by 1,344
Superiors' evaluations and subordinates' perceptions of transformational and transactional leadership.
Hater, J. J., & Bass, B. M. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.4.695 · Cited by 1,136
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 883
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 830

Flagged first, then by citations

Show all 21 references

Sorted by citations

Items 201–300 of 1,001

Context Matters
Laschinger, H. K. S., Finegan, J., & Wilk, P. (2009) · JONA The Journal of Nursing Administration · doi:10.1097/nna.0b013e3181a23d2b · Cited by 258
“We calculated the rwg, a measure that compares the amount of within-group agreement with a null distribution.(52,53) Because the withingroup agreement was greater than what would be expected by chance (the average rwg for structural empowerment and for LMX quality were 0.”
Method
Organizational structure, context, customer orientation, and performance: lessons from Chinese state‐owned enterprises
Lin, X., & Germain, R. N. (2003) · Strategic Management Journal · doi:10.1002/smj.348 · Cited by 257
Collective Efficacy: A Multilevel Analysis
Watson, C. B., Chemers, M. M., & Preiser, N. (2001) · Personality and Social Psychology Bulletin · doi:10.1177/0146167201278012 · Cited by 257
A FIVE‐YEAR STUDY OF UPWARD FEEDBACK: WHAT MANAGERS DO WITH THEIR RESULTS MATTERS
Walker, A. G., & Smither, J. W. (1999) · Personnel Psychology · doi:10.1111/j.1744-6570.1999.tb00166.x · Cited by 257
Contingency fit, institutional fit, and firm performance: a metafit approach to organization–environment relationships
Volberda, H. W., van der Weerdt, N., Verwaal, E., Stienstra, M., & Verdu, A. J. (2012) · Organization Science · doi:10.1287/orsc.1110.0687 · Cited by 254
“Both the interrater agreement score (James et al. 1993) and the interrater reliability score (Jones et al.”
BackgroundMethodInfluential
The Restriction of Variance Hypothesis and Interrater Reliability and Agreement: Are Ratings from Multiple Sources Really Dissimilar?
LeBreton, J. M., Burgess, J. R. D., Kaiser, R. B., Atchley, E. K., & James, L. R. (2003) · Organizational Research Methods · doi:10.1177/1094428102239427 · Cited by 272
“As James et al. (1993) noted, when these raters are in perfect agreement, SX 2 = 0.”
BackgroundSelf-citation
Team Cognition: Development and Evolution in Software Project Teams
He, J., Butler, B. S., & King, W. R. (2007) · Journal of Management Information Systems · doi:10.2753/mis0742-1222240210 · Cited by 255
A Revised Index of Interrater Agreement for Multi-Item Ratings of a Single Target
Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999) · Applied Psychological Measurement · doi:10.1177/01466219922031257 · Cited by 255
“Negative values are not a problem for r∗ wg andr∗ wg(J ) if they are accepted as indices of agreement (James et al., 1993; Kozlowski & Hattrup, 1992; Lindell & Brandt, 1997).”
Background
The Shared Leadership Process in Decision-Making Teams
Bergman, J. Z., Rentsch, J. R., Small, E. E., Davenport, S. W., & Bergman, S. M. (2011) · The Journal of Social Psychology · doi:10.1080/00224545.2010.538763 · Cited by 255
“The level of within-team agreement for intragroup trust, cohesion, task conflict, and socio-emotional conflict were assessed using r wg ( James, Demaree, & Wolf, 1984 ).”
Method
Overcoming Resistance to Change and Enhancing Creative Performance
Hon, A. H. Y., Bloom, M., & Crant, J. M. (2011) · Journal of Management · doi:10.1177/0149206311415418 · Cited by 254
Investigating the Antecedents of Team-Based Clan Control: Adding Social Capital as a Predictor
Kirsch, L. J., Ko, D., & Haney, M. H. (2009) · Organization Science · doi:10.1287/orsc.1090.0458 · Cited by 263
The Effects of Family Firm Specific Sources of TMT Diversity: The Moderating Role of Information Exchange Frequency
Yan, L., & Kellermanns, F. W. (2009) · Journal of Management Studies · doi:10.1111/j.1467-6486.2009.00893.x · Cited by 252
Measuring dimensions of manufacturing flexibility
Koste, L. L., Malhotra, M. K., & Sharma, S. (2004) · Journal of Operations Management · doi:10.1016/j.jom.2004.01.001 · Cited by 252
Top management teams within multinational corporations: Effects of cultural heterogeneity
Elron, E. (1997) · The Leadership Quarterly · doi:10.1016/s1048-9843(97)90021-7 · Cited by 251
Data analytic methods for the analysis of multilevel questions
Castro, S. L. (2002) · The Leadership Quarterly · doi:10.1016/s1048-9843(01)00105-9 · Cited by 250
Human Resource Management and Firm Performance: Testing a Contingency Model of Executive Controls
Snell, S. A., & Youndt, M. A. (1995) · Journal of Management · doi:10.1016/0149-2063(95)90007-1 · Cited by 246
Organizational injury rate underreporting: The moderating effect of organizational safety climate.
Probst, T. M., Brubaker, T. L., & Barsotti, A. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.5.1147 · Cited by 244
Internal and external antecedents of SMEs' innovation ambidexterity outcomes
Chang, Y., Hughes, M., & Hotho, S. (2011) · Management Decision · doi:10.1108/00251741111183816 · Cited by 252
Organization structure and fairness perceptions: The moderating effects of organizational level
Schminke, M. J., Cropanzano, R., & Rupp, D. E. (2002) · Organizational Behavior and Human Decision Processes · doi:10.1016/s0749-5978(02)00034-1 · Cited by 242
Team Member Change, Flux in Coordination, and Performance: Effects of Strategic Core Roles, Information Transfer, and Cognitive Ability
Summers, J. K., Humphrey, S. E., & Ferris, G. R. (2012) · Academy of Management Journal · doi:10.5465/amj.2010.0175 · Cited by 256
Promoting Organizational Citizenship Behavior in Schools: The Impact of Individual and Organizational Characteristics
Somech, A., & Ron, I. (2007) · Educational Administration Quarterly · doi:10.1177/0013161x06291254 · Cited by 237
Organizational ethical climate, perceived organizational support, and employee silence: A cross-level investigation
Wang, Y.-D., & Hsieh, H.-H. (2012) · Human Relations · doi:10.1177/0018726712460706 · Cited by 237
“To justify the aggregation, we calculated the within-group agreement statistic (rwg) (James et al., 1993) for each of the five ethical climates and for each company.”
BackgroundMethod
Ölçek Geliştirme Çalışmalarında Kullanılan Kapsam Geçerliği İçin Bir Yol Haritası
Yeşilyurt, S., & Çapraz, C. (2018) · Erzincan Üniversitesi Eğitim Fakültesi Dergisi · doi:10.17556/erziefd.297741 · Cited by 235
Social capital and creativity in R&D project teams
Chen, M., Chang, Y., & Hung, S. (2007) · R and D Management · doi:10.1111/j.1467-9310.2007.00494.x · Cited by 234
Leadership, Collective Personality, and Performance.
Hofmann, D. A., & Jones, L. M. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.3.509 · Cited by 234
What Drives Key Informant Accuracy?
Homburg, C., Klarmann, M., Reimann, M., & Schilke, O. (2012) · Journal of Marketing Research · doi:10.1509/jmr.09.0174 · Cited by 276
“In addition, although it does not measure reliability in a strict sense (Kozlowski and Hattrup 1992), the rwg(J) agreement index (James, Demaree, and Wolf 1984) offers a heuristic approach to assessing random error (James, Demaree, and Wolf 1993).”
Background
Staff burnout and patient satisfaction: Evidence of relationships at the care unit level.
Garman, A. N., Corrigan, P. W., & Morris, S. B. (2002) · Journal of Occupational Health Psychology · doi:10.1037/1076-8998.7.3.235 · Cited by 232
“A group was viewed as having adequate level agreement if rwg was greater than .7 ( James et al., 1993 ).”
BackgroundMethod
A HIERARCHICAL CONSTRUCT OF SELF‐MANAGEMENT LEADERSHIP AND ITS RELATIONSHIP TO QUALITY OF WORK LIFE AND PERCEIVED WORK GROUP EFFECTIVENESS
Cohen, S. G., Chang, L., & Ledford, G. E. (1997) · Personnel Psychology · doi:10.1111/j.1744-6570.1997.tb00909.x · Cited by 230
EFFECTS OF TRANSFORMATIONAL LEADERSHIP ON SUBORDINATE MOTIVATION, EMPOWERING NORMS, AND ORGANIZATIONAL PRODUCTIVITY
Masi, R., & Cooke, R. A. (2000) · The International Journal of Organizational Analysis · doi:10.1108/eb028909 · Cited by 228
The trickle-down effect of servant leadership on frontline employee service behaviors and performance: A multilevel study of Chinese hotels
Ling, Q., Lin, M., & Wu, X. (2015) · Tourism Management · doi:10.1016/j.tourman.2015.07.008 · Cited by 232
An empirical assessment of the influence of customer emotions and contact employee performance on encounter and relationship satisfaction
van Dolen, W., de Ruyter, K., & Lemmink, J. G. A. M. (2002) · Journal of Business Research · doi:10.1016/s0148-2963(02)00277-1 · Cited by 234
“Empirical justification for aggregation was tested by means of an estimate rWG( J), as suggested by James et al. (1993).”
Method
Team Goal Commitment and Team Effectiveness: The Role of Task Interdependence and Supportive Behaviors.
Aubé, C., & Rousseau, V. (2005) · Group Dynamics Theory Research and Practice · doi:10.1037/1089-2699.9.3.189 · Cited by 275
Predictors of Formal Control Usage in IT Outsourcing Partnerships
Rustagi, S., King, W. R., & Kirsch, L. J. (2008) · Information Systems Research · doi:10.1287/isre.1080.0169 · Cited by 224
“The r wg scores for cost of the outsourcing arrangement and the current duration of the outsourcing arrangement were 0.76 and 0.71, respectively, which are within acceptable ranges (James et al. 1993).”
BackgroundInfluential
Dynamic Managerial Capabilities and the Multibusiness Team: The Role of Episodic Teams in Executive Leadership Groups
Martin, J. A. (2010) · Organization Science · doi:10.1287/orsc.1090.0515 · Cited by 234
Psychological Ownership: How Having Control Matters
Liu, J., Wang, H., Hui, C., & Lee, C. (2011) · Journal of Management Studies · doi:10.1111/j.1467-6486.2011.01028.x · Cited by 220
The effects of job complexity and autonomy on cohesiveness in collectivistic and individualistic work groups: a cross‐cultural analysis
Man, D. C., & Lam, S. S. K. (2003) · Journal of Organizational Behavior · doi:10.1002/job.227 · Cited by 215
How do high performance work systems influence organizational innovation in professional service firms?
Fu, N., Flood, P. C., Bosak, J., Morris, T., & O’Regan, P. (2015) · Employee Relations · doi:10.1108/er-10-2013-0155 · Cited by 234
“Results Table 2 presents descriptive statistics, including the means, standard deviations, Rwgs, ICC(1)s, ICC(2)s, correlations, and inter-item reliabilities.”
MethodInfluential
Military leadership: A context specific review
Wong, L., Bliese, P. D., & McGurk, D. (2003) · The Leadership Quarterly · doi:10.1016/j.leaqua.2003.08.001 · Cited by 219
Directive Versus Participative Leadership: Two Complementary Approaches to Managing School Effectiveness
Somech, A. (2005) · Educational Administration Quarterly · doi:10.1177/0013161x05279448 · Cited by 250
“A value of .70 or above is suggested as a “good” amount of within-group interrater agreement (James et al., 1993).”
Background
The role of transformational leadership in enhancing team reflexivity
Schippers, M. C., Den Hartog, D. N., Koopman, P. L., & van Knippenberg, D. (2008) · Human Relations · doi:10.1177/0018726708096639 · Cited by 215
“A value of .70 or above is suggested as “good” with respect to within-group interrater agreement (James et al., 1993).”
A Social Exchange Perspective of Employee–Organization Relationships and Employee Unethical Pro-organizational Behavior: The Moderating Role of Individual Moral Identity
Wang, T., Long, L., Zhang, Y., & He, W. (2018) · Journal of Business Ethics · doi:10.1007/s10551-018-3782-9 · Cited by 238
Visionary communication qualities as mediators of the relationship between narcissism and attributions of leader charisma.
GALVIN, B. M., WALDMAN, D. A., & BALTHAZARD, P. (2010) · Personnel Psychology · doi:10.1111/j.1744-6570.2010.01179.x · Cited by 212
Knowledge effectiveness, social context and innovation
Brachos, D., Kostopoulos, K. C., Söderquist, K. E., & Prastacos, G. P. (2007) · Journal of Knowledge Management · doi:10.1108/13673270710819780 · Cited by 212
“First, the ‘‘interrater agreement score’’ (rwg(j)) (James et al., 1993) was calculated for each variable.”
Method
Reliability and agreement of student ratings of the classroom environment: A reanalysis of TIMSS data
Lüdtke, O., Trautwein, U., Kunter, M., & Baumert, J. (2007) · Learning Environments Research · doi:10.1007/s10984-006-9014-8 · Cited by 211
“Following critical analysis by Schmidt and Hunter (1989), it became standard practice to see the rWG as a measure of interrater agreement (James et al., 1993; Kozlowski & Hattrup, 1992).”
Method
Critical success factors for human resource outcomes in Kaizen events: An empirical study
Farris, J. A., Aken, E. M. V., Doolen, T. L., & Worley, J. M. (2008) · International Journal of Production Economics · doi:10.1016/j.ijpe.2008.08.051 · Cited by 224
Communication intensity, goal congruence, and uncertainty in buyer–supplier new product development
Yan, T., & Dooley, K. J. (2013) · Journal of Operations Management · doi:10.1016/j.jom.2013.10.001 · Cited by 212
“To assess inter-rater reliability and agreement, we computed the mean of item-level inter-rater correlations, within-group inter-rater reliabilities rwg (James et al., 1993), and intraclass correlation coefficients ICC(1) and ICC(2) (Bliese, 2000).”
Collective Efficacy and Team Performance: A Longitudinal Study of Collegiate Football Teams.
Myers, N. D., Feltz, D. L., & Short, S. E. (2004) · Group Dynamics Theory Research and Practice · doi:10.1037/1089-2699.8.2.126 · Cited by 210
THE SOURCES AND ACCURACY OF JOB APPLICANTS' BELIEFS ABOUT ORGANIZATIONAL CULTURE.
Cable, D. M., Aiman‐Smith, L., Mulvey, P. W., & Edwards, J. R. (2000) · Academy of Management Journal · doi:10.2307/1556336 · Cited by 210
Empowerment in the Hospitality Industry: an Exploration of Antecedents and Outcomes
Sparrowe, R. T. (1994) · Journal of Hospitality & Tourism Research · doi:10.1177/109634809401700306 · Cited by 209
Employment relationships and firm performance: evidence from an emerging economy
Wang, D., Tsui, A. S., Zhang, Y., & Ma, L. (2003) · Journal of Organizational Behavior · doi:10.1002/job.213 · Cited by 208
Inclusive leadership and team innovation: The role of team voice and performance pressure
Ye, Q., Wang, D., & Guo, W. (2019) · European Management Journal · doi:10.1016/j.emj.2019.01.006 · Cited by 224
PREVIOUS WORK EXPERIENCE AND ORGANIZATIONAL SOCIALIZATION: A LONGITUDINAL EXAMINATION.
Adkins, C. L. (1995) · Academy of Management Journal · doi:10.2307/256748 · Cited by 208
Explaining affective linkages in teams: Individual differences in susceptibility to contagion and individualism-collectivism.
Ilieș, R., Wagner, D. T., & Morgeson, F. P. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.4.1140 · Cited by 208
“Therefore, we set out to determine if there was a reasonable level of agreement within teams by computing an index of interrater agreement (rwg; James, Demaree, & Wolf, 1993 ).”
Me or we? The role of personality and justice as other-centered antecedents to innovative citizenship behaviors within organizations.
Moon, H. K., Kamdar, D., Mayer, D. M., & Takeuchi, R. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.1.84 · Cited by 211
The Relation of School Structure and Social Environment to Parent Involvement in Elementary Schools
Griffith, J. E. (1998) · The Elementary School Journal · doi:10.1086/461916 · Cited by 219
Navigating in a hypercompetitive environment: the roles of action aggressiveness and TMT integration
Chen, M., Lin, H., & Michel, J. G. (2010) · Strategic Management Journal · doi:10.1002/smj.891 · Cited by 207
New to the game and questioning the rules: The experiences and beliefs of founders who start imitative versus innovative firms
Cliff, J. E., Jennings, P. D., & Greenwood, R. (2005) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2005.02.010 · Cited by 205
The Role of Procedural Justice and Power Distance in the Relationship Between High Performance Work Systems and Employee Attitudes: A Multilevel Perspective
Wu, P., & Chaturvedi, S. (2009) · Journal of Management · doi:10.1177/0149206308331097 · Cited by 221
Linking learning and effective process implementation to mass customization capability
Huang, X., Kristal, M. M., & Schroeder, R. G. (2007) · Journal of Operations Management · doi:10.1016/j.jom.2007.11.002 · Cited by 206
Measuring trust in teams: Development and validation of a multifaceted measure of formative and reflective indicators of team trust
Costa, A. C., & Anderson, N. R. (2010) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320903272083 · Cited by 244
“Average rwg(j) values of .70 and above are indicative of sufficient agreement level to suggest shared climates within teams (James et al., 1993).”
BackgroundMethodInfluential
Investigating the Role of Leadership and Organizational Culture in Fostering Innovation Ambidexterity
Lin, H., & McDonough, E. F. (2011) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2010.2092781 · Cited by 205
Adaptive versus Proactive Behavior in Service Recovery: The Role of Self-Managing Teams
de Jong, A., & de Ruyter, K. (2004) · Decision Sciences · doi:10.1111/j.0011-7315.2004.02513.x · Cited by 222
Offshoring and firm innovation: The moderating role of top management team attributes
Mihalache, O. R., Jansen, J. J. P., Bosch, F. A. J. V. D., & Volberda, H. (2012) · Strategic Management Journal · doi:10.1002/smj.1983 · Cited by 201
Recruiters’ Perceptions and Use of Applicant Résumé Information: Screening the Recent Graduate
Cole, M. S., Rubin, R. S., Feild, H. S., & Giles, W. F. (2007) · Applied Psychology · doi:10.1111/j.1464-0597.2007.00288.x · Cited by 200
The contingent effects of leadership on team collaboration in virtual teams
Huang, R., Kahai, S. S., & Jestice, R. J. (2010) · Computers in Human Behavior · doi:10.1016/j.chb.2010.03.014 · Cited by 201
Examining the cross-level relationship between shared leadership and learning in teams: Evidence from China
Liu, S., Hu, J. J., Li, Y., Wang, Z., & Lin, X. (2013) · The Leadership Quarterly · doi:10.1016/j.leaqua.2013.08.006 · Cited by 228
Cohesiveness and organizational citizenship behavior: A multilevel analysis using work groups and individuals
Kidwell, R. E., Mossholder, K. W., & Bennett, N. (1997) · Journal of Management · doi:10.1177/014920639702300605 · Cited by 357
Team Virtues and Performance: An Examination of Transparency, Behavioral Integrity, and Trust
Palanski, M. E., Kahai, S. S., & Yammarino, F. J. (2010) · Journal of Business Ethics · doi:10.1007/s10551-010-0650-7 · Cited by 202
“The rwg statistic (James et al., 1993) is calculated by comparing the observed group variance to an expected random variance and is useful for analyzing the variance within a single group.”
Method
The occurrences and correlates of bullying and harassment in the restaurant sector
Mathisen, G. E., Einarsen, S. V., & Mykletun, R. J. (2008) · Scandinavian Journal of Psychology · doi:10.1111/j.1467-9450.2007.00602.x · Cited by 208
To reflect or not to reflect: Prior team performance as a boundary condition of the effects of reflexivity on learning and final team performance
Schippers, M. C., Homan, A. C., & van Knippenberg, D. (2012) · Journal of Organizational Behavior · doi:10.1002/job.1784 · Cited by 199
Procedural Justice Climate and Group Power Distance: An Examination of Cross-Level Interaction Effects
Yang, J., Mossholder, K. W., & Peng, T. K. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.3.681 · Cited by 196
How Authentic Leadership Influences Team Performance: The Mediating Role of Team Reflexivity
Lyubovnikova, J., Legood, A., Turner, N., & Mamakouka, A. (2015) · Journal of Business Ethics · doi:10.1007/s10551-015-2692-3 · Cited by 239
THE ROLE OF RELATIONAL TRUST IN BANK--SMALL FIRM RELATIONSHIPS.
Saparito, P. A., Chen, C. C., & Sapienza, H. J. (2004) · Academy of Management Journal · doi:10.2307/20159589 · Cited by 222
The Impact of Demographic Heterogeneity and Team Leader-Team Member Demographic Fit on Team Empowerment and Effectiveness
Kirkman, B. L., Tesluk, P. E., & Rosen, B. (2004) · Group & Organization Management · doi:10.1177/1059601103257412 · Cited by 202
RETRACTED: Impact of spiritual leadership on unit performance
Fry, L. W., Hannah, S. T., Noel, M., & Walumbwa, F. O. (2011) · The Leadership Quarterly · doi:10.1016/j.leaqua.2011.02.002 · Cited by 219
Retracted
Dispositional antecedents and outcomes of political skill in organizations: A four-study investigation with convergence
Liu, Y., Ferris, G. R., Zinko, R., Perrewé, P. L., Weitz, B., & Xu, J. (2007) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2007.04.003 · Cited by 192
Overcoming roadblocks to effectiveness: Incorporating management of performance barriers into models of work group effectiveness.
Tesluk, P. E., & Mathieu, J. E. (1999) · Journal of Applied Psychology · doi:10.1037/0021-9010.84.2.200 · Cited by 194
Shared Leadership in Teams
Small, E. E., & Rentsch, J. R. (2010) · Journal of Personnel Psychology · doi:10.1027/1866-5888/a000017 · Cited by 205
Innovation in globally distributed teams: The role of LMX, communication frequency, and member influence on team decisions.
Gajendran, R. S., & Joshi, A. (2012) · Journal of Applied Psychology · doi:10.1037/a0028958 · Cited by 219
The Impact of Team Members’ Cultural Values on Productivity, Cooperation, and Empowerment in Self-Managing Work Teams
Kirkman, B. L., & Shapiro, D. L. (2001) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022101032005005 · Cited by 189
Behaving safely under pressure: The effects of job demands, resources, and safety climate on employee physical and psychosocial safety behavior
Bronkhorst, B. (2015) · Journal of Safety Research · doi:10.1016/j.jsr.2015.09.002 · Cited by 229
Playing Follow the Leader: Status-Determining Traits in Relation to Collective Efficacy across Cultures
Earley, P. C. (1999) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1999.2863 · Cited by 190
Falling from Great (and Not-So-Great) Heights: How Initial Status Position Influences Performance after Status Loss
Marr, J. C., & Thau, S. (2013) · Academy of Management Journal · doi:10.5465/amj.2011.0909 · Cited by 200
Assessing the Organizational Social Context (OSC) of child welfare systems: Implications for research and practice
Glisson, C. A., Green, P., & Williams, N. J. (2012) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2012.06.002 · Cited by 193
Withholding inputs in team contexts: Member composition, interaction processes, evaluation structure, and social loafing.
Price, K. H., Harrison, D. A., & Gavin, J. H. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.6.1375 · Cited by 190
Transformational Leadership and Work-Related Attitudes: The Moderating Effects of Collective and Self-Efficacy Across Cultures
Walumbwa, F. O., LAWLER, J. J., Avolio, B. J., Wang, P., & Shi, K. (2005) · Journal of Leadership & Organizational Studies · doi:10.1177/107179190501100301 · Cited by 186
Organizational climate, services, and outcomes in child welfare systems
Glisson, C., & Green, P. (2011) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2011.04.009 · Cited by 185
A multi‐level investigation of psychological contract breach and organizational identification through the lens of perceived organizational membership: Testing a moderated–mediated model
Epitropaki, O. (2012) · Journal of Organizational Behavior · doi:10.1002/job.1793 · Cited by 184
An Empirical Study of System Improvement by Frontline Employees in Hospital Units
Tucker, A. L. (2007) · Manufacturing & Service Operations Management · doi:10.1287/msom.1060.0156 · Cited by 186
“outlined by James and colleagues (James et al. 1993).”
Flowing Together: A Longitudinal Study of Collective Efficacy and Collective Flow Among Workgroups
Salanova, M., Rodríguez-Sánchez, A. M., Schaufeli, W. B., & Cifré, E. (2014) · The Journal of Psychology · doi:10.1080/00223980.2013.806290 · Cited by 208
Treatment Errors in Healthcare: A Safety Climate Approach
Naveh, E., Katz-Navon, T., & Stern, Z. (2005) · Management Sciences · doi:10.1287/mnsc.1050.0372 · Cited by 169
“We tested for the homogeneity of responses by calculating the Rwg coefficients of homogeneity (James et al. 1984) of the four dimensions for each of the participating medical units.”
Method
Decision Comprehensiveness and Corporate Entrepreneurship: The Moderating Role of Managerial Uncertainty Preferences and Environmental Dynamism
Heavey, C., Şimşek, Z., Roche, F., & Kelly, A. (2009) · Journal of Management Studies · doi:10.1111/j.1467-6486.2009.00858.x · Cited by 181
When Is Empowerment Effective? The Role of Leader-Leader Exchange in Empowering Leadership, Cynicism, and Time Theft
Lorinkova, N., & Perry, S. J. (2014) · Journal of Management · doi:10.1177/0149206314560411 · Cited by 207
Development and validation of safety climate scales for lone workers using truck drivers as exemplar
Huang, Y.-H., Zohar, D. M., Robertson, M. M., Garabet, A., Lee, J., & Murphy, L. A. (2012) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2012.08.011 · Cited by 187
“70 (James et al., 1993); ICC(1) > .”
BackgroundInfluential
Predictors of Community Therapists’ Use of Therapy Techniques in a Large Public Mental Health System
Beidas, R. S., Marcus, S. C., Aarons, G. A., Hoagwood, K., Schoenwald, S. K., Evans, A. C., Hurford, M. O., Hadley, T. R., Barg, F. K., Walsh, L., et al. (2015) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2014.3736 · Cited by 193
Signalling theory as a framework for analysing human resource management processes and integrating human resource attribution theories: A conceptual analysis and empirical exploration
Guest, D. E., Sanders, K., Rodrigues, R., & Oliveira, T. C. (2020) · Human Resource Management Journal · doi:10.1111/1748-8583.12326 · Cited by 189
Ethical Leadership and Corporate Social Responsibility in China: A Multilevel Study of Their Effects on Trust and Organizational Citizenship Behavior
Tourigny, L., Han, J., Baba, V. V., & Pan, P. (2017) · Journal of Business Ethics · doi:10.1007/s10551-017-3745-6 · Cited by 195
“Then, we computed Rwg(j) statistic (James 1982; James et al. 1993) and ICC(2) to demonstrate acceptable within groups or inter-rater agreement (Klein and Kozlowski 2000).”
BackgroundMethod
Variation in caregiver perceptions of teamwork climate in labor and delivery units
Sexton, J. B., Holzmueller, C. G., Pronovost, P. J., Thomas, E. J., McFerran, S., Nunes, J., Thompson, D. A., Knight, A. P., Penning, D. H., & Fox, H. E. (2006) · Journal of Perinatology · doi:10.1038/sj.jp.7211556 · Cited by 175
When organizational climate is unambiguous, it is also strong.
Dickson, M. W., Resick, C. J., & Hanges, P. J. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.2.351 · Cited by 175
“We calculated index rwg(j)s (i.e., within-group interrater agreement for scales with j essentially parallel items; L. R. James, Demaree, & Wolf, 1984, 1993 ) and intraclass correlation coefficients (ICCs) ICC(1) and ICC(2) to determine whether there was statistical evidence to justify aggregation of individual respons…”
Method
Using Self-Categorization Theory to Understand Relational Demography–Based Variations in People's Responsiveness to Organizational Culture
Chatman, J. A., & Spataro, S. E. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.16928415 · Cited by 173

Authors: James, L. R., Demaree, R. G., & Wolf, G.
Affiliations: University of Tennessee at Knoxville (US)
Venue: Journal of Applied Psychology
Publisher: American Psychological Association (APA)
Published: 1993-04
Volume: 78
Issue: 2
Pages: 306-309
ISSN: 0021-9010 (ISSN-L)
1939-1854
DOI: 10.1037/0021-9010.78.2.306
OpenAlex: W2022950201
Cited by: 2,221
References: 21
Citations by year: 2026: 54 · 2025: 88 · 2024: 76 · 2023: 85 · 2022: 88 · 2021: 97 · 2020: 95 · 2019: 98 · 2018: 108 · 2017: 82 · 2016: 86 · 2015: 100 · 2014: 111 · 2013: 110 · 2012: 96
FWCI: 12.64
Type: journal article
Language: en
Open access: closed
Locations: Journal of Applied Psychology (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
Keywords: inter-rater reliability, interobserver agreement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

rwg: an assessment of within-group interrater agreement

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks