Cited by 2,221Reliability and Agreement in MeasurementPsychometric Methodologies and TestingTeam Dynamics and Performance

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

rwg: an assessment of within-group interrater agreement

James, Lawrence R., Demaree, Robert G., Wolf, Gerrit·1993·Journal of Applied Psychology

DOI 10.1037/0021-9010.78.2.306

Open this paper in CoChat — free account

Abstract

Schmidt and Hunter (1989) critiqued the within-group interrater reliability statistic (r wg ) described by James, Demaree, and Wolf (1984). Kozlowski and Hattrup (1992) responded to the Schmidt and Hunter critique and argued that r wg is a suitable index of interrater agreement. This article focuses on the interpretation of r wg as a measure of agreement among judges' ratings of a single target. A new derivation of r wg is given that underscores this interpretation

Integrity signals

Registry

Crossref · 10.1037/0021-9010.78.2.306

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,009
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
Self-citation
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Employee and customer perceptions of service in banks: Replication and extension.
Schneider, B., & Bowen, D. E. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.3.423 · Cited by 1,344
Superiors' evaluations and subordinates' perceptions of transformational and transactional leadership.
Hater, J. J., & Bass, B. M. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.4.695 · Cited by 1,136
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 883
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 830

Flagged first, then by citations

Show all 21 references

Sorted by citations

Items 401–500 of 1,001

AN ASSESSMENT AND REFINEMENT OF JEHN'S INTRAGROUP CONFLICT SCALE
Pearson, A. W., Ensley, M. D., & Amason, A. C. (2002) · International Journal of Conflict Management · doi:10.1108/eb022870 · Cited by 127
Does Work Group Cohesiveness Affect Individuals' Performance and Organizational Commitment?
Wech, B. A., Mossholder, K. W., Steel, R. P., & Bennett, N. (1998) · Small Group Research · doi:10.1177/1046496498294004 · Cited by 128
What's the Weather Like? The Effect of Team Learning Climate, Empowerment Climate, and Gender on Individuals' Technology Exploration and Use
Maruping, L. M., & Magni, M. (2012) · Journal of Management Information Systems · doi:10.2753/mis0742-1222290103 · Cited by 129
Why Organizational and Community Diversity Matter: Representativeness and the Emergence of Incivility and Organizational Performance
King, E., Dawson, J., West, M. A., Gilrane, V. L., Peddie, C. I., & Bastin, L. (2011) · Academy of Management Journal · doi:10.5465/amj.2010.0016 · Cited by 119
Differentiated transformational leadership and knowledge sharing: A cross-level investigation
Li, G., Shang, Y., Liu, H., & Xi, Y. (2013) · European Management Journal · doi:10.1016/j.emj.2013.10.004 · Cited by 136
When and why do customer solutions pay off in business markets?
Worm, S., Bharadwaj, S. G., Ulaga, W., & Reinartz, W. (2017) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-017-0529-6 · Cited by 128
“An index of inter-rater agreement of .97 (James et al. 1984, 1993) indicates strong consistency among raters.”
Socio‐demographic factors and shared leadership behaviors in dispersed teams: Implications for human resource management
Muethel, M., Gehrlein, S., & Hoegl, M. (2012) · Human Resource Management · doi:10.1002/hrm.21488 · Cited by 136
Global organizational communities of practice: The effects of nationality diversity, psychological safety, and media richness on community performance
Kirkman, B. L., Cordery, J. L., Mathieu, J. E., Rosen, B., & Kukenberger, M. R. (2013) · Human Relations · doi:10.1177/0018726712464076 · Cited by 140
“70 (James et al., 1993), which supports aggregation of members’ responses.”
BackgroundMethodInfluential
Empathic and nonempathic interaction in chronic pain couples.
Cano, A., Barterian, J. A., & Heller, J. B. (2008) · The Clinical Journal of Pain · doi:10.1097/ajp.0b013e31816753d8 · Cited by 125
Promoting patient care: work engagement as a mediator between ward service climate and patient‐centred care
Abdelhadi, N., & Drach‐Zahavy, A. (2011) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2011.05834.x · Cited by 135
“Tests of rwg (James et al. 1993) indicated agreement of responses at the unit level: the median value was: 0Æ83.”
Background
Exploring the effects of intragroup conflict and past performance feedback on team effectiveness
Passos, A., & Caetano, A. (2005) · Journal of Managerial Psychology · doi:10.1108/02683940510589028 · Cited by 124
Stewardship Climate Scale
Neubaum, D. O., Thomas, C. H., Dibrell, C., & Craig, J. B. L. (2016) · Family Business Review · doi:10.1177/0894486516673701 · Cited by 124
“Values of r WG should meet or exceed .70, while ICC(1) values as low as .05 may indicate a minimal group-level effect with values of .25 considered to be a strong effect (L. R. James et al., 1993; LeBreton & Senter, 2008).”
Background
Leadership and dynamic capabilities: the role of HR systems
Cabrales, Á. L., Bornay‐Barrachina, M., & Díaz-Fernández, M. (2017) · Personnel Review · doi:10.1108/pr-05-2015-0146 · Cited by 124
Safety climate in conditions of construction subcontracting: a multi‐level analysis
Lingard, H. C., Cooke, T., & Blismas, N. G. (2010) · Construction Management and Economics · doi:10.1080/01446190903480035 · Cited by 123
“It estimates whether responses from one participant are ‘similar’ to the responses provided by others in the same workgroup, thus reflecting the degree of ‘sharedness’ in group climate scores (James et al , 1993 ).”
Background
The Effects of Procedural Justice Climate on Work Group Performance
Naumann, S. E., & Bennett, N. (2002) · Small Group Research · doi:10.1177/10496402033003004 · Cited by 120
Nightmare Bosses: The Impact of Abusive Supervision on Employees’ Sleep, Emotions, and Creativity
Han, G. H., Harms, P. D., & Bai, Y. (2015) · Journal of Business Ethics · doi:10.1007/s10551-015-2859-y · Cited by 135
Personal Initiative and Job Performance Evaluations: Role of Political Skill in Opportunity Recognition and Capitalization
Wihler, A., Blickle, G., Ellen, B. P., Hochwarter, W. A., & Ferris, G. R. (2014) · Journal of Management · doi:10.1177/0149206314552451 · Cited by 125
“Another, popular estimate of interrater agreement, rwg, was provided by James, Demaree, and Wolf (1993).”
Background
The Impact of Group Interaction Styles on Problem-Solving Effectiveness
Cooke, R. A., & Szumal, J. L. (1994) · The Journal of Applied Behavioral Science · doi:10.1177/0021886394304005 · Cited by 122
Leader-member exchange and innovative work behavior
Saeed, B. B., Afsar, B., Cheema, S., & Javed, F. (2018) · European Journal of Innovation Management · doi:10.1108/ejim-11-2017-0158 · Cited by 134
Perceptions of Organizational Culture and Women's Advancement in Organizations: A Cross-Cultural Examination
Bajdo, L. M., & Dickson, M. W. (2001) · Sex Roles · doi:10.1023/a:1014365716222 · Cited by 122
“A measure of within-group interrater agreement, rwg ( James, Demaree, & Wolf, 1984, 1993 ), was used to ensure aggregatability to the organizational level of analysis.”
Method
Team effectiveness in China: Cooperative conflict for relationship building
Tjosvold, D., Poon, M. C. C., & Yu, Z. (2005) · Human Relations · doi:10.1177/0018726705053426 · Cited by 127
Measuring Job-Related Situational Strength and Assessing Its Interactive Effects With Personality on Voluntary Work Behavior
Meyer, R. D., Dalal, R. S., José, I. J., Hermida, R., Chen, T. R., Vega, R. P., Brooks, C. K., & Khare, V. P. (2011) · Journal of Management · doi:10.1177/0149206311425613 · Cited by 125
The setting of non-financial goals in the family firm: The influence of family climate and identification
Suárez, M. K. C., Déniz, M. D. L. C. D., & Martín‐Santana, J. D. (2014) · Journal of Family Business Strategy · doi:10.1016/j.jfbs.2014.05.003 · Cited by 121
Psychological safety and social support in groupware adoption: A multi-level assessment in education
Schepers, J., de Jong, A., Wetzels, M. G. M., & de Ruyter, K. (2007) · Computers & Education · doi:10.1016/j.compedu.2007.08.001 · Cited by 127
Enhancing employee creativity in the Chinese context: The mediating role of employee self-concordance
Hon, A. H. Y. (2010) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2010.06.002 · Cited by 122
When confidence is detrimental: Influence of overconfidence on leadership effectiveness
Shipman, A. S., & Mumford, M. D. (2011) · The Leadership Quarterly · doi:10.1016/j.leaqua.2011.05.006 · Cited by 117
Situational and Dispositional Influences on Nurses' Workplace Well-being
Laschinger, H. K. S., Finegan, J., & Wilk, P. (2011) · Nursing Research · doi:10.1097/nnr.0b013e318209782e · Cited by 115
Diversity and creativity in cross-national teams: The role of team knowledge sharing and inclusive climate
Bodla, A. A., Tang, N.-Y., Jiang, W., & Tian, L.-W. (2016) · Journal of Management & Organization · doi:10.1017/jmo.2016.34 · Cited by 109
Feeling Included and Valued: How Perceived Respect Affects Positive Team Identity and Willingness to Invest in the Team
Ellemers, N., Sleebos, E., Stam, D. A., & de Gilder, D. (2011) · British Journal of Management · doi:10.1111/j.1467-8551.2011.00784.x · Cited by 119
How Do Leaders and their Teams Bring about Organizational Learning and Outcomes?
Berson, Y., Da’as, R., & Waldman, D. A. (2013) · Personnel Psychology · doi:10.1111/peps.12071 · Cited by 127
EFFICACY DISPERSION IN TEAMS: MOVING BEYOND AGREEMENT AND AGGREGATION
DeRue, D. S., Hollenbeck, J. R., Ilgen, D., & Feltz, D. L. (2010) · Personnel Psychology · doi:10.1111/j.1744-6570.2009.01161.x · Cited by 128
Recent Issues Regarding rWG, rWG, rWG(J), and rWG(J)
LeBreton, J. M., James, L. R., & Lindell, M. K. (2004) · Organizational Research Methods · doi:10.1177/1094428104272181 · Cited by 118
“However, the formation of a linear composite does not require an a priori assumption that the Spearman-Brown formula applies to rWG, which was the concern expressed by Lindell and Brandt (1999). The derivation of an agreement index for composite scores within a group extrapolates the logic for one item (see James et a…”
BackgroundMethodInfluentialSelf-citation
Effects of guided reflexivity and team feedback on team performance improvement: The role of team regulatory processes and cognitive emergent states
Konradt, U., Schippers, M. C., Garbers, Y., & Steenfatt, C. (2015) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2015.1005608 · Cited by 132
Dependent Effect Sizes in Meta-Analysis: Incorporating the Degree of Interdependence.
Cheung, S. F., & Chan, D. K. ‐. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.5.780 · Cited by 118
Is What You See, What You Get?
Tyran, K. L., & Gibson, C. B. (2008) · Group & Organization Management · doi:10.1177/1059601106287111 · Cited by 111
“An additional estimate of within-group interrater agreement was made using the rwg analysis recommended by James et al. (1984, 1993).”
Method
Experiencing meaningfulness climate in teams: How spiritual leadership enhances team effectiveness when facing uncertain tasks
Yang, F., Huang, X., & Wu, L. (2018) · Human Resource Management · doi:10.1002/hrm.21943 · Cited by 121
The Decision to Adopt Evidence-Based and Other Innovative Mental Health Practices: Risky Business?
Panzano, P. C., & Roth, D. (2006) · Psychiatric Services · doi:10.1176/appi.ps.57.8.1153 · Cited by 115
The effects of service climate and the effective leadership behaviour of supervisors on frontline employee service quality: A multi‐level analysis
Hui, C.-C. H., Chiu, W. C. K., Yu, P. L. H., Cheng, K. H. C., & Tse, H. H. M. (2006) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317905x89391 · Cited by 114
“For example, Jermier and Kerr (1997) stated that they had never intended to treat the substitutes framework as a closed system, and that a better specification of the conceptual domain of substitutes is long overdue.”
Background
A Monte Carlo Study of the Effects of Common Method Variance on Significance Testing and Parameter Bias in Hierarchical Linear Modeling
Lai, X., Li, F., & Leung, K. Y. (2013) · Organizational Research Methods · doi:10.1177/1094428112469667 · Cited by 116
The role of organizational trust in safety climate's influence on organizational outcomes
Kath, L. M., Magley, V. J., & Marmet, M. D. (2010) · Accident Analysis & Prevention · doi:10.1016/j.aap.2009.11.010 · Cited by 123
Innovation in top management teams: Minority dissent, transformational leadership, and radical innovations
Nijstad, B. A., Berger-Selman, F., & Dreu, C. K. W. D. (2012) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2012.734038 · Cited by 119
An analysis of shared leadership, diversity, and team creativity in an e-learning environment
Lee, D. S., Lee, K. C., Seo, Y. W., & Choi, D. Y. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2013.10.064 · Cited by 117
Analysing the job characteristics model: new support from a cross-section of establishments
DeVaro, J., Li, R., & Brookshire, D. (2007) · The International Journal of Human Resource Management · doi:10.1080/09585190701321211 · Cited by 115
“, 1984 , 1993 ) is the most widely encountered measure of within-group agreement in the organizational literature (Bliese, 2000 ), we compute this for the 14 items in the worker survey.”
Background
Perceived Environmental Dynamism, Relative Competitive Performance, and Top Management Team Heterogeneity: Examining Correlates of Upper Echelons’ Advice-Seeking
Heyden, M. L. M., Doorn, S. V., Reimer, M., Bosch, F. A. J. V. D., & Volberda, H. (2013) · Organization Studies · doi:10.1177/0170840612470229 · Cited by 119
ADVOCACY, PERFORMANCE, AND THRESHOLD INFLUENCES ON DECISIONS TO TERMINATE NEW PRODUCT DEVELOPMENT.
Green, S. G., Welsh, M. A., & Dehler, G. E. (2003) · Academy of Management Journal · doi:10.2307/30040636 · Cited by 113
Control mechanisms across a buyer–supplier relationship quality matrix
Liu, Y., Li, Y., & Zhang, L. (2009) · Journal of Business Research · doi:10.1016/j.jbusres.2009.01.005 · Cited by 115
How relational capital and knowledge combination capability enhance the performance of work units in a high technology industry
Carmeli, A., & Azeroual, B. (2009) · Strategic Entrepreneurship Journal · doi:10.1002/sej.63 · Cited by 118
“Finally, we also used an inter-rater reliability coeffi cient developed by James, Demaree, and Wolf (1993) to examine the intragroup reliability (rwg) of responses.”
Method
How Lead Founder Personality Affects New Venture Performance
de Jong, A., Song, M., & Song, L. (2011) · Journal of Management · doi:10.1177/0149206311407509 · Cited by 125
“99, suggest that there is convincing evidence that founding TMT members agree on the level of team conflict (James et al., 1993).”
Background
Emergency Room Culture and the Emotional Support Component of Family-Centered Care
Hemmelgarn, A. L., Glisson, C., & Dukes, D. (2001) · Children's Health Care · doi:10.1207/s15326888chc3002_2 · Cited by 112
Informal Learning Environments and Impact on Interest in STEM Careers
Maiorca, C., Roberts, T., Jackson, C. D., Bush, S. B., Delaney, A., Mohr‐Schroeder, M. J., & Yao, S. G. (2020) · International Journal of Science and Mathematics Education · doi:10.1007/s10763-019-10038-9 · Cited by 116
Expertise, extraversion and group interaction styles as performance indicators in virtual teams
Balthazard, P. A., Potter, R. E., & Warren, J. (2004) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/968464.968469 · Cited by 111
Opportunistic behaviors in franchise chains: the role of cohesion among franchisees
Akremi, A. E., Mignonac, K., & Perrigot, R. (2010) · Strategic Management Journal · doi:10.1002/smj.912 · Cited by 111
Leadership and Social Identification in Military Units: Direct and Indirect Relationships1
Shamir, B., Zakay, E., Brainin, E., & Popper, M. (2000) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2000.tb02499.x · Cited by 111
Using social exchange theory to predict the effects of high‐performance human resource practices on corporate entrepreneurship: Evidence from China
Zhang, Z., & Jia, M. (2010) · Human Resource Management · doi:10.1002/hrm.20378 · Cited by 121
Understanding The Curvilinear Relationships between LMX Differentiation and Team Coordination and Performance
Sui, Y., Wang, H., Kirkman, B. L., & Li, N. (2015) · Personnel Psychology · doi:10.1111/peps.12115 · Cited by 115
When managers become leaders: The role of manager network centralities, social power, and followers' perception of leadership
Chiu, C., Balkundi, P., & Weinberg, F. J. (2016) · The Leadership Quarterly · doi:10.1016/j.leaqua.2016.05.004 · Cited by 137
From Employee-experienced High-involvement Work System to Innovation: An Emergence-based Human Resource Management Framework
Li, Y., Wang, M., van Jaarsveld, D. D., Lee, G. K., & Ma, D. G. (2017) · Academy of Management Journal · doi:10.5465/amj.2015.1101 · Cited by 110
Climate for work group creativity and innovation: Norwegian validation of the team climate inventory (TCI)
Mathisen, G. E., Einarsen, S. V., JØRSTAD, K., & Brønnick, K. (2004) · Scandinavian Journal of Psychology · doi:10.1111/j.1467-9450.2004.00420.x · Cited by 110
Group Citizenship Behaviour Conceptualization and Preliminary Tests of its Antecedents and Consequences
Chen, X.-P., Lam, S. S. K., Naumann, S. E., & Schaubroeck, J. M. (2005) · Management and Organization Review · doi:10.1111/j.1740-8784.2005.00012.x · Cited by 110
Why is Your Team More Creative Than Mine? The Influence of Shared Mental Models on Intra‐group Conflict, Team Creativity and Effectiveness
Santos, C. M., Uitdewilligen, S., & Passos, A. (2015) · Creativity and Innovation Management · doi:10.1111/caim.12129 · Cited by 108
Entrepreneurial universities and research ambidexterity: A multilevel analysis
Chang, Y., Yang, P. Y., Martin, B. R., Chi, H., & Tsai-Lin, T.-F. (2016) · Technovation · doi:10.1016/j.technovation.2016.02.006 · Cited by 111
Human capital and human resource management to achieve ambidextrous learning: A structural perspective
Díaz-Fernández, M., Pasamar, S., & Cabrera, R. V. (2016) · BRQ Business Research Quarterly · doi:10.1016/j.brq.2016.03.002 · Cited by 121
The Impact of Organizational Structure on Mass Customization Capability: A Contingency View
Huang, X., Kristal, M. M., & Schroeder, R. G. (2009) · Production and Operations Management · doi:10.1111/j.1937-5956.2009.01117.x · Cited by 109
“The Inter-rater reliability analysis at the individual-respondent level showed that all of our constructs have inter-rater agreement coefﬁcients above 0.70, indicating an acceptable internal consistency of the scale items across informants (Boyer and Verma 2000, James et al. 1984).”
MethodInfluential
Virtual team interaction styles: assessment and effects
Potter, R. E., & Balthazard, P. A. (2002) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2002.1001 · Cited by 109
A Tale of Two Factions: Why and When Factional Demographic Faultlines Hurt Board Performance
Veltrop, D. B., Hermes, N., Postma, T. J. B. M., & de Haan, J. (2014) · Corporate Governance An International Review · doi:10.1111/corg.12098 · Cited by 109
“We calculated the r wg ( j ) inter-agreement coefﬁcient for multi-item indices (James et al., 1984; James, Demaree, & Wolf, 1993) and intra-class correlation coefﬁcient (ICC 1 ) to determine whether aggregation to the group level was accurate.”
Method
Self-reliance: A Gender Perspective on its Relationship to Communality and Leadership Evaluations
Schaumberg, R. L., & Flynn, F. J. (2016) · Academy of Management Journal · doi:10.5465/amj.2015.0018 · Cited by 121
Socio‐Emotional Wealth Separation and Decision‐Making Quality in Family Firm TMTs: The Moderating Role of Psychological Safety
Vandekerkhof, P., Steijvers, T., Hendriks, W., & Voordeckers, W. (2017) · Journal of Management Studies · doi:10.1111/joms.12277 · Cited by 112
Psychological Capital, Team Resources and Organizational Citizenship Behavior
Bogler, R., & Somech, A. (2019) · The Journal of Psychology · doi:10.1080/00223980.2019.1614515 · Cited by 122
Issues in Multilevel Research: Theory Development, Measurement, and Analysis
Hofmann, D. A. (2004) · doi:10.1002/9780470756669.ch12 · Cited by 108
Team Dispersion and Performance: The Role of Team Communication and Transformational Leadership
Eisenberg, J., Post, C., & DiTomaso, N. (2019) · Small Group Research · doi:10.1177/1046496419827376 · Cited by 108
The Effects of Group Conflict and Work Stress on Employee Performance
Hon, A. H. Y., & Chan, W. W. H. (2013) · Cornell Hospitality Quarterly · doi:10.1177/1938965513476367 · Cited by 107
“Looking at the team members’ assessments of conflict, interrater agreement coefficients (r wg ; James, Demaree, and Wolf 1993) for the two team-level variables—team task conflict (.93) and team relationship conflict (.90)—indicated high agreement.”
Background
Balancing knowledge strategy: codification and personalization during product development
Kumar, A. S., & Ganesh, L. S. (2011) · Journal of Knowledge Management · doi:10.1108/13673271111108738 · Cited by 107
A Multilevel Model of Teachers’ Job Performance: Understanding the Effects of Trait Emotional Intelligence, Job Satisfaction, and Organizational Trust
Li, M., Pérez-Díaz, P. A., Mao, Y., & Petrides, K. V. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.02420 · Cited by 134
What’s Love Got to Do With It? Team Entrepreneurial Passion and Performance in New Venture Teams
Santos, S. C., & Cardon, M. S. (2018) · Entrepreneurship Theory and Practice · doi:10.1177/1042258718812185 · Cited by 119
Emotional intelligence and leadership in adolescents
Charbonneau, D., & Nicol, A. A. M. (2002) · Personality and Individual Differences · doi:10.1016/s0191-8869(01)00216-1 · Cited by 107
Relationships of participative leadership with relational demography variables: a multi‐level perspective
Somech, A. (2003) · Journal of Organizational Behavior · doi:10.1002/job.225 · Cited by 109
“For example, Johnson and Swap (1982) found that women and men differed in the kind of information they used in developing trust in others.”
BackgroundResult
The role of relational orientation as measured by face and renqing in innovative behavior in China: An indigenous analysis
Leung, K. Y., Chen, Z., Zhou, F., & Lim, K. H. (2011) · Asia Pacific Journal of Management · doi:10.1007/s10490-011-9277-1 · Cited by 107
Team effectiveness under stress: a structural contingency approach
Drach‐Zahavy, A., & Freund, A. (2006) · Journal of Organizational Behavior · doi:10.1002/job.430 · Cited by 106
Organizational Behaviour across Cultures
Fischer, R. D., Ferreira, M. C., Assmar, E. M. L., Redford, P., & Harb, C. (2005) · International Journal of Cross Cultural Management · doi:10.1177/1470595805050823 · Cited by 106
“Statistical techniques to do this are available (James et al., 1984, 1993; Shrout and Fleiss, 1979).”
Method
The moderating role of organizational culture in the relationship between organizational justice and organizational citizenship behaviors
Erkutlu, H. (2011) · Leadership & Organization Development Journal · doi:10.1108/01437731111161058 · Cited by 111
“For the peer reports of respect for people and team orientation dimensions, median within-group inter-rater agreement values (rwg; James et al., 1993) were 0.”
Background
Modelling leadership—Implicit leadership theories in Sweden
Holmberg, I., & Åkerblom, S. (2006) · Scandinavian Journal of Management · doi:10.1016/j.scaman.2006.10.002 · Cited by 106
Group absenteeism and positive affective tone: a longitudinal study
Mason, C., & Griffin, M. A. (2003) · Journal of Organizational Behavior · doi:10.1002/job.210 · Cited by 105
“James, Demaree, and Wolf’s ( James, Demaree, & Wolf, 1984, 1993 ) rwg(j) statistic was used to assess the level of within-group agreement.”
Team Self-Managing Behaviors and Team Effectiveness: The Moderating Effect of Task Routineness
Rousseau, V., & Aubé, C. (2010) · Group & Organization Management · doi:10.1177/1059601110390835 · Cited by 107
The role of within‐group agreement in understanding transformational leadership
Feinberg, B., Ostroff, C. L., & Burke, W. W. (2005) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317905x26156 · Cited by 105
INFORMATION SHARING AND GROUP CONFLICT: GOING BEYOND DECISION MAKING TO UNDERSTAND THE EFFECTS OF INFORMATION SHARING ON GROUP PERFORMANCE
Moye, N. A., & Langfred, C. W. (2004) · International Journal of Conflict Management · doi:10.1108/eb022919 · Cited by 104
Validation of a Multi‐Dimensional Measure of Strategy Development Processes
Bailey, A., Johnson, G., & Daniels, K. (2000) · British Journal of Management · doi:10.1111/1467-8551.t01-1-00157 · Cited by 105
Innovation and Conflict Management in Work Teams: The Effects of Team Identification and Task and Relationship Conflict
Desivilya, H. S., Somech, A., & Lidgoster, H. (2010) · Negotiation and Conflict Management Research · doi:10.1111/j.1750-4716.2009.00048.x · Cited by 105
“A value of .70 or above is suggested as a ‘‘good’’ amount of withingroup inter-rater agreement (James et al., 1993).”
Background
An Overview of Interrater Agreement on Likert Scales for Researchers and Practitioners
O’Neill, T. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00777 · Cited by 127
“One recommended procedure is to set inadmissible values to 0 (James et al., 1993).”
Background
Rationale and construct validity evidence for a measure of perceived organizational family support (POFS): Because purported practices may not reflect reality
Jahn, E. W., Thompson, C. A., & Kopelman, R. E. (2003) · Community Work & Family · doi:10.1080/13668800302545 · Cited by 104
“In the present research, we argue that POFS is a conceptually meaningful construct that can be assessed by individuals and aggregated across individuals, in a manner akin to psychological climate (James & Jones, 1974; James et al., 1984, 1993).”
BackgroundMethodInfluential
Group Composition, Creative Synergy, and Group Performance
Taggar, S. (2001) · The Journal of Creative Behavior · doi:10.1002/j.2162-6057.2001.tb01050.x · Cited by 104
Managing a new collaborative entity in business organizations: Understanding organizational communities of practice effectiveness.
Kirkman, B. L., Mathieu, J. E., Cordery, J. L., Rosen, B., & Kukenberger, M. R. (2011) · Journal of Applied Psychology · doi:10.1037/a0024198 · Cited by 110
Government-led critical success factors in PPP infrastructure development
Wibowo, A., & Alfen, H. W. (2015) · Built Environment Project and Asset Management · doi:10.1108/bepam-03-2014-0016 · Cited by 103
Team Goal Orientation and Team Performance: The Mediating Role of Team Planning
Mehta, A., Feild, H. S., Armenakis, A. A., & Mehta, N. (2008) · Journal of Management · doi:10.1177/0149206308326773 · Cited by 106
Family firms are indeed better places to work than non-family firms! Socioemotional wealth and employees’ perceived organizational caring
Christensen-Salem, A., Mesquita, L. F., Hashimoto, M., Hom, P. W., & Gómez‐Mejía, L. R. (2021) · Journal of Family Business Strategy · doi:10.1016/j.jfbs.2020.100412 · Cited by 123
Therapist Adherence and Organizational Effects on Change in Youth Behavior Problems One Year After Multisystemic Therapy
Schoenwald, S. K., Carter, R. E., Chapman, J. E., & Sheidow, A. J. (2008) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0181-z · Cited by 108
“Examination of interrater agreement on the organizational constructs using rWG ( James et al. 1993; LeBreton et al. 2005) and intraclass correlations indicated there was a non-trivial level of consensus in therapist ratings but that a substantial portion of the variance was also attributable to the therapist providing…”
MethodInfluential
Multilevel and Cross-Level Effects of Workplace Attitudes and Group Member Relations on Interpersonal Helping Behavior
Choi, J. N. (2006) · Human Performance · doi:10.1207/s15327043hup1904_4 · Cited by 103
Are general and strategic measures of organizational context and leadership associated with knowledge and attitudes toward evidence-based practices in public behavioral health settings? A cross-sectional observational study
Powell, B. J., Mandell, D. S., Hadley, T. R., Rubin, R. M., Evans, A. C., Hurford, M. O., & Beidas, R. S. (2017) · Implementation Science · doi:10.1186/s13012-017-0593-9 · Cited by 114
“60 level [74, 75]; thus, individual participant responses within organizations were averaged.”
BackgroundMethod
Multilevel Research in the Field of Organizational Behavior
Costa, P., Graça, A. M., Marques‐Quinteiro, P., Santos, C. M., Caetano, A., & Passos, A. (2013) · SAGE Open · doi:10.1177/2158244013498244 · Cited by 101
Team Autonomy, Performance, and Member Job Strain: Uncovering the Teamwork KSA Link
Leach, D. J., Wall, T. D., Rogelberg, S. G., & Jackson, P. R. (2005) · Applied Psychology · doi:10.1111/j.1464-0597.2005.00193.x · Cited by 106
Learning from conflicts? The relations between task and relationship conflicts, team learning and team performance
van Woerkom, M., & Engen, M. L. V. (2009) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320802569514 · Cited by 108

Authors: James, L. R., Demaree, R. G., & Wolf, G.
Affiliations: University of Tennessee at Knoxville (US)
Venue: Journal of Applied Psychology
Publisher: American Psychological Association (APA)
Published: 1993-04
Volume: 78
Issue: 2
Pages: 306-309
ISSN: 0021-9010 (ISSN-L)
1939-1854
DOI: 10.1037/0021-9010.78.2.306
OpenAlex: W2022950201
Cited by: 2,221
References: 21
Citations by year: 2026: 54 · 2025: 88 · 2024: 76 · 2023: 85 · 2022: 88 · 2021: 97 · 2020: 95 · 2019: 98 · 2018: 108 · 2017: 82 · 2016: 86 · 2015: 100 · 2014: 111 · 2013: 110 · 2012: 96
FWCI: 12.64
Type: journal article
Language: en
Open access: closed
Locations: Journal of Applied Psychology (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
Keywords: inter-rater reliability, interobserver agreement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

rwg: an assessment of within-group interrater agreement

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks