Cited by 2,221Reliability and Agreement in MeasurementPsychometric Methodologies and TestingTeam Dynamics and Performance

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

rwg: an assessment of within-group interrater agreement

James, Lawrence R., Demaree, Robert G., Wolf, Gerrit·1993·Journal of Applied Psychology

DOI 10.1037/0021-9010.78.2.306

Open this paper in CoChat — free account

Abstract

Schmidt and Hunter (1989) critiqued the within-group interrater reliability statistic (r wg ) described by James, Demaree, and Wolf (1984). Kozlowski and Hattrup (1992) responded to the Schmidt and Hunter critique and argued that r wg is a suitable index of interrater agreement. This article focuses on the interpretation of r wg as a measure of agreement among judges' ratings of a single target. A new derivation of r wg is given that underscores this interpretation

Integrity signals

Registry

Crossref · 10.1037/0021-9010.78.2.306

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,009
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
Self-citation
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Employee and customer perceptions of service in banks: Replication and extension.
Schneider, B., & Bowen, D. E. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.3.423 · Cited by 1,344
Superiors' evaluations and subordinates' perceptions of transformational and transactional leadership.
Hater, J. J., & Bass, B. M. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.4.695 · Cited by 1,136
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 883
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 830

Flagged first, then by citations

Show all 21 references

Sorted by citations

Items 301–400 of 1,001

The brand management system and service firm competitiveness
Vijande, M. L. S., Lanza, A. B. D. R., Álvarez, L. S., & Díaz‐Martín, A. M. (2012) · Journal of Business Research · doi:10.1016/j.jbusres.2012.07.007 · Cited by 174
A Systematic Approach to Scale Development in Tourist Shopping Satisfaction
Wong, I. A., & Wan, Y. K. P. (2012) · Journal of Travel Research · doi:10.1177/0047287512457263 · Cited by 184
`That is my job'
Parker, S. K. (2007) · Human Relations · doi:10.1177/0018726707076684 · Cited by 172
“The mean rwg(j) statistic (James et al., 1993) was .80, which exceeds Glick’s (1985) recommendation of .60 as a cut-off for acceptable inter-rater agreement values.”
Method
Government institutional support, entrepreneurial orientation, strategic renewal, and firm performance in transitional China
Shu, C., Clercq, D. D., Zhou, Y., & Liu, C. (2019) · International Journal of Entrepreneurial Behaviour & Research · doi:10.1108/ijebr-07-2018-0465 · Cited by 182
THE ROLE OF AFFECT AND LEADERSHIP DURING ORGANIZATIONAL CHANGE
Seo, M., Taylor, M. S., Hill, N. S., Zhang, X., Tesluk, P. E., & Lorinkova, N. (2012) · Personnel Psychology · doi:10.1111/j.1744-6570.2011.01240.x · Cited by 194
Institutional and individual factors affecting scientists' data‐sharing behaviors: A multilevel analysis
Kim, Y., & Stanton, J. M. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23424 · Cited by 168
On the entrepreneurial genesis of new markets: effectual transformations versus causal search and selection
Dew, N., Read, S., Sarasvathy, S. D., & Wiltbank, R. (2010) · Journal of Evolutionary Economics · doi:10.1007/s00191-010-0185-1 · Cited by 168
Statistical properties of the rWG(J) index of agreement.
Cohen, A., Doveh, E., & Eick, U. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.3.297 · Cited by 167
High-Performance Work System, Work Well-Being, and Employee Creativity: Cross-Level Moderating Role of Transformational Leadership
Miao, R., & Cao, Y. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16091640 · Cited by 203
“70 [83], thereby supporting the standpoint of strategic researchers that meaningful variability exists in employees’ individual experiences of human resource system [3].”
Background
What leaders say versus what they do: Inclusive leadership, policy-practice decoupling, and the anomaly of climate for inclusion
Mor Barak, M. E., Luria, G., & Brimhall, K. C. (2022) · Group & Organization Management · doi:10.1177/10596011211005916 · Cited by 167
The effect of servant leadership on customer value co-creation: A cross-level analysis of key mediating roles
Hsiao, C., Lee, Y.-H., & Chen, W.-J. (2015) · Tourism Management · doi:10.1016/j.tourman.2015.02.012 · Cited by 166
Ratings of Organizational Citizenship Behavior: Does the Source Make a Difference?
Allen, T. D., Barnard, S., Rush, M. C., & Russell, J. E. A. (2000) · Human Resource Management Review · doi:10.1016/s1053-4822(99)00041-8 · Cited by 168
Creative Climate: A Leadership Lever for Innovation
Isaksen, S. G., & AKKERMANS, H. J. (2011) · The Journal of Creative Behavior · doi:10.1002/j.2162-6057.2011.tb01425.x · Cited by 165
“After several critiques and debates (Schmidt & Hunter, 1989; James et al., 1993), it has been accepted that r wg is a measure of agreement (Kozlowski & Hattrup, 1992).”
Group affective tone: A review and future research directions
Collins, A. L., Lawrence, S. A., Troth, A. C., & Jordan, P. J. (2013) · Journal of Organizational Behavior · doi:10.1002/job.1887 · Cited by 176
Shaping Environments Conductive to Creativity
Hon, A. H. Y. (2011) · Cornell Hospitality Quarterly · doi:10.1177/1938965511424725 · Cited by 193
“The interrater agreement (r wg ) coefficients (James, Demaree, and Wolf 1993) for climate for creativity (.92), supportive coworkers (.95), empowering leadership (.91), task conflict (.87), and personal conflict (.81) indicated high interrater agreement.”
Background
HOW SUPERVISORS INFLUENCE PERFORMANCE: A MULTILEVEL STUDY OF COACHING AND GROUP MANAGEMENT IN TECHNOLOGY‐MEDIATED SERVICES
Liu, X. H., & Batt, R. (2010) · Personnel Psychology · doi:10.1111/j.1744-6570.2010.01170.x · Cited by 181
Emulation in Academia: Balancing Structure and Identity
Labianca, G., Fairbank, J. F., Thomas, J. B., Gioia, D. A., & Umphress, E. E. (2001) · Organization Science · doi:10.1287/orsc.12.3.312.10101 · Cited by 160
Internal and External Social Capital for Radical Product Innovation: Do They Always Work Well Together?
Cuevas‐Rodríguez, G., Cabello‐Medina, C., & Lavado, A. C. (2013) · British Journal of Management · doi:10.1111/1467-8551.12002 · Cited by 161
A team level investigation of the relationship between Leader–Member Exchange (LMX) differentiation, and commitment and performance
Blanc, P. M. L., & González‐Romá, V. (2012) · The Leadership Quarterly · doi:10.1016/j.leaqua.2011.12.006 · Cited by 165
Me and us: differential relationships among goal‐setting training, efficacy and effectiveness at the individual and team level
Gibson, C. B. (2001) · Journal of Organizational Behavior · doi:10.1002/job.114 · Cited by 161
Person-Group Fit
Seong, J. Y., Kristof‐Brown, A. L., Park, W.-W., Hong, D., & Shin, Y. (2012) · Journal of Management · doi:10.1177/0149206312453738 · Cited by 196
“We calculated the james, Demaree, and Wolf (1984, 1993) within-group agreement index (rwg) for each of the proximal outcome variables for each team.”
Method
Integration mechanisms and R&D project performance
Sicotte, H., & Langley, A. (2000) · Journal of Engineering and Technology Management · doi:10.1016/s0923-4748(99)00018-1 · Cited by 160
Spotlight on the Followers: An Examination of Moderators of Relationships Between Transformational Leadership and Subordinates’ Citizenship and Taking Charge
Li, N., Chiaburu, D. S., Kirkman, B. L., & Xie, Z. (2012) · Personnel Psychology · doi:10.1111/peps.12014 · Cited by 166
When team members’ values differ: The moderating role of team leadership
Klein, K. J., Knight, A. P., Ziegert, J. C., Lim, B. C., & Saltz, J. L. (2011) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2010.08.004 · Cited by 160
How authentic leadership and inclusion benefit organizations
Cottrill, K., Lopez, P. D. J., & Hoffman, C. C. (2014) · Equality Diversity and Inclusion An International Journal · doi:10.1108/edi-05-2012-0041 · Cited by 133
Ethical Leadership and Employee Green Behavior: A Multilevel Moderated Mediation Analysis
Saleem, M., Qadeer, F., Mahmood, F., Ariza‐Montes, A., & Han, H. (2020) · Sustainability · doi:10.3390/su12083314 · Cited by 185
Behavioral complexity in leadership: The psychometric properties of a new instrument to measure behavioral repertoire ☆
Lawrence, K. A., Lenk, P., & Quinn, R. (2009) · The Leadership Quarterly · doi:10.1016/j.leaqua.2009.01.014 · Cited by 167
How to enable employee creativity in a team context: A cross-level mediating process of transformational leadership
Bai, Y., Lin, L., & Li, P. P. (2016) · Journal of Business Research · doi:10.1016/j.jbusres.2016.02.025 · Cited by 182
Entrepreneurial Orientation and Firm Performance: Drawing Attention to the Senior Team
Doorn, S. V., Jansen, J. J. P., Bosch, F. A. J. V. D., & Volberda, H. (2013) · Journal of Product Innovation Management · doi:10.1111/jpim.12032 · Cited by 159
The popularity contest at work: Who wins, why, and what do they receive?
Scott, B. A., & Judge, T. A. (2009) · Journal of Applied Psychology · doi:10.1037/a0012951 · Cited by 167
Embedding Leader Characteristics: An Examination of Homogeneity of Personality and Values in Organizations.
Giberson, T. R., Resick, C. J., & Dickson, M. W. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.5.1002 · Cited by 157
Unit-Level OCB and Unit Effectiveness: Examining the Incremental Effect of Helping Behavior
Ehrhart, M. G., Bliese, P. D., & Thomas, J. L. (2006) · Human Performance · doi:10.1207/s15327043hup1902_4 · Cited by 156
“To test within-group agreement, we used rwg(j), as described by James, Demaree, and Wolf (1984, 1993), and later discussed by Kozlowski and Hattrup (1992) and Bliese (2000).”
Method
A dual perspective on risk perception and its effect on safety behavior: A moderated mediation model of safety motivation, and supervisor’s and coworkers’ safety climate
Xia, N., Xie, Q., Hu, X., Wang, X., & Meng, H. (2019) · Accident Analysis & Prevention · doi:10.1016/j.aap.2019.105350 · Cited by 177
Servant Versus Authentic Leadership: Assessing Effectiveness in China’s Hospitality Industry
Ling, Q., Liu, F., & Wu, X. (2017) · Cornell Hospitality Quarterly · doi:10.1177/1938965516641515 · Cited by 168
“Following James, Demaree, and Wolf (1984, 1993), we computed rwg.”
Method
Understanding the effect of knowledge management strategies on knowledge management performance: A contingency perspective
Kim, T. H., Lee, J.-N., Chun, J. U., & Benbasat, I. (2014) · Information & Management · doi:10.1016/j.im.2014.03.001 · Cited by 165
“rwg(j) is an index of within-unit interrater agreement [58].”
Background
The De-adoption of Innovative Mental Health Practices (IMHP): Why Organizations Choose not to Sustain an IMHP
Massatti, R. R., Sweeney, H. A., Panzano, P. C., & Roth, D. (2007) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-007-0141-z · Cited by 153
“With regard to agreement, rwg values for each measure and each organization were examined as an index of agreement among responses within each project. ( James et al. 1993 ).”
MethodInfluential
How Psychological Safety Affects Team Performance: Mediating Role of Efficacy and Learning Behavior
Kim, S., Lee, H., & Connerton, T. P. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01581 · Cited by 217
The Role of Relational Trust in Bank–Small Firm Relationships
Saparito, P. A., Chen, C. C., & Sapienza, H. J. (2004) · Academy of Management Journal · doi:10.5465/20159589 · Cited by 153
Integrated manufacturing, empowerment, and company performance
Patterson, M. G., West, M. A., & Wall, T. D. (2004) · Journal of Organizational Behavior · doi:10.1002/job.261 · Cited by 152
Unpacking the cross‐level effects of tenure diversity, explicit knowledge, and knowledge sharing on individual creativity
Gilson, L. L., Lim, H. S., Luciano, M. M., & Choi, J. N. (2013) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12011 · Cited by 166
Informational or emotional appeals in crowdfunding message strategy: an empirical investigation of backers’ support decisions
Xiang, D., Zhang, L., Tao, Q., Wang, Y., & Ma, S. (2019) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00638-w · Cited by 151
Does Knowledge Mediate the Effect of Context on Performance? Some Initial Evidence
Dröge, C., Claycomb, C., & Germain, R. N. (2003) · Decision Sciences · doi:10.1111/j.1540-5414.2003.02324.x · Cited by 155
Leveraging the employee voice: a multi-level social learning perspective of ethical leadership
Bai, Y., Lin, L., & Liu, J. T. (2017) · The International Journal of Human Resource Management · doi:10.1080/09585192.2017.1308414 · Cited by 169
Strategy alignment between HRM, KM, and corporate development
Shih, H., & Chiang, Y. (2005) · International Journal of Manpower · doi:10.1108/01437720510625476 · Cited by 149
“Interrater agreement, suggested by James et al. (1993)”
Method
Performance Implications of Peer Monitoring
Loughry, M. L., & Tosi, H. L. (2008) · Organization Science · doi:10.1287/orsc.1080.0356 · Cited by 149
“We used the interrater agreement index rwg (James et al. 1993) to”
Method
Comparing Group and Individual Level Assessments of Job Characteristics in Testing the Job Demand-Control Model: A Multilevel Approach
JONGE, J. D., van Breukelen, G., Landeweerd, J. A., & Nijhuis, F. J. N. (1999) · Human Relations · doi:10.1177/001872679905200106 · Cited by 153
Interorganizational collaboration and firm innovativeness: Unpacking the role of the organizational environment
Alexiev, A. S., Volberda, H., & Bosch, F. A. J. V. D. (2015) · Journal of Business Research · doi:10.1016/j.jbusres.2015.09.002 · Cited by 149
Is Leader–Member Exchange Differentiation Beneficial or Detrimental for Group Effectiveness? A Meta-Analytic Investigation and Theoretical Integration
Yu, A., Matta, F. K., & Cornfield, B. (2017) · Academy of Management Journal · doi:10.5465/amj.2016.1212 · Cited by 163
ASSESSING THE INCREMENTAL VALIDITY OF TEAM CONSENSUS RATINGS OVER AGGREGATION OF INDIVIDUAL‐LEVEL DATA IN PREDICTING TEAM EFFECTIVENESS
Kirkman, B. L., Tesluk, P. E., & Rosen, B. (2001) · Personnel Psychology · doi:10.1111/j.1744-6570.2001.tb00226.x · Cited by 147
GENERALIZATION OF EMPLOYEE INVOLVEMENT TRAINING TO THE JOB SETTING: INDIVIDUAL AND SITUATIONAL EFFECTS
Tesluk, P. E., Farr, J. L., Mathieu, J. E., & Vance, R. J. (1995) · Personnel Psychology · doi:10.1111/j.1744-6570.1995.tb01773.x · Cited by 147
A Social Composition View of Team Creativity: The Role of Member Nationality-Heterogeneous Ties Outside of the Team
Perry-Smith, J. E., & Shalley, C. E. (2014) · Organization Science · doi:10.1287/orsc.2014.0912 · Cited by 146
“The interrater reliability for the experts’ ratings of creativity was calculated using rwg425 (James et al. 1993) with a mean rwg425 = 0082 and a median rwg425 = 0088.”
Method
Team mental models and their potential to improve teamwork and safety: A review and implications for future research in healthcare
Burtscher, M. J., & Manser, T. (2012) · Safety Science · doi:10.1016/j.ssci.2011.12.033 · Cited by 148
Antecedents of shared leadership: empowering leadership and interdependence
Fausing, M. S., Jønsson, T., Lewandowski, J., & Bligh, M. C. (2015) · Leadership & Organization Development Journal · doi:10.1108/lodj-06-2013-0075 · Cited by 145
Organizational commitment—A missing link between leadership behavior and organizational performance?
Steyrer, J. M., Schiffinger, M., & Lang, R. (2008) · Scandinavian Journal of Management · doi:10.1016/j.scaman.2008.04.002 · Cited by 144
High‐Performance Work Systems in Professional Service Firms: Examining the Practices‐Resources‐Uses‐Performance Linkage
Fu, N., Flood, P. C., Bosak, J., Rousseau, D. M., Morris, T., & O’Regan, P. (2015) · Human Resource Management · doi:10.1002/hrm.21767 · Cited by 161
“meta-analysis, Jiang et al. (2012) find that HPWS are related to higher levels of human capital, effects found in both Japan (Takeuchi et al.”
Background
Differentiating transformational and non-transformational leaders on the basis of neurological imaging
Balthazard, P. A., Waldman, D. A., Thatcher, R. W., & Hannah, S. T. (2011) · The Leadership Quarterly · doi:10.1016/j.leaqua.2011.08.002 · Cited by 140
Can confidence come too soon? Collective efficacy, conflict and group performance over time
Goncalo, J. A., Polman, E., & Maslach, C. (2010) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2010.05.001 · Cited by 154
Assessing the impact of the manufacturing executive’s role on business performance through strategic alignment
Papke‐Shields, K. E., & Malhotra, M. K. (2000) · Journal of Operations Management · doi:10.1016/s0272-6963(00)00050-4 · Cited by 142
The Emergence of Emergent Leadership: A Comprehensive Framework and Directions for Future Research
Hanna, A. A., Smith, T. A., Kirkman, B. L., & Griffin, R. W. (2020) · Journal of Management · doi:10.1177/0149206320965683 · Cited by 167
Do you see what we see? The complex effects of perceptual distance between leaders and teams.
Gibson, C. B., Cooper, C. D., & Conger, J. A. (2009) · Journal of Applied Psychology · doi:10.1037/a0013073 · Cited by 149
An empirical study of transformational leadership, team performance and service quality in retail banks
Lee, P. K. C., Cheng, T. C. E., Yeung, A. C. L., & Lai, K. (2011) · Omega · doi:10.1016/j.omega.2011.02.001 · Cited by 141
“70, demonstrating a high level of agreement among different members within teams [75].”
Background
Stability and Change in Managerial Work Values: A Longitudinal Study of China, Hong Kong, and the U.S.
Ralston, D. A., Pounder, J. S., Lo, C. W., Wong, Y., Egri, C. P., & Stauffer, J. R. (2006) · Management and Organization Review · doi:10.1111/j.1740-8784.2006.00031.x · Cited by 143
“Prior to conducting the hierarchical cluster analyses, we calculated the inter-rater agreement (r W o) f° r each values dimension according to the methods prescribed by James, Demaree, and Wolf (1993).”
Strategic Manufacturing Planning Systems and Their Linkage to Planning System Success
Papke‐Shields, K. E., Malhotra, M. K., & Grover, V. (2002) · Decision Sciences · doi:10.1111/j.1540-5915.2002.tb01634.x · Cited by 140
Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes
Roberson, Q. M., Sturman, M. C., & Simons, T. L. (2007) · Organizational Research Methods · doi:10.1177/1094428106294746 · Cited by 140
“For example, while Burke et al. (1999) showed correlations between average deviation and interrater agreement indices (rwg; James et al., 1993) to be between 0.79 and 0.93, such results also suggest that 14% – 38% of their variance is not shared.”
BackgroundResultInfluential
Is the Most Effective Team Leadership Shared?
Hoch, J. E., Pearce, C. L., & Welzel, L. (2010) · Journal of Personnel Psychology · doi:10.1027/1866-5888/a000020 · Cited by 141
“With an average rwg of .78 (James, 1982; James et al., 1993), as well as an ICC(1) of .32 and an ICC(2) of .70 aggregation of the data was thus justified and we conducted our main analyses at the team level of analysis of the data.”
Method
The role of leaders in shaping formal team norms
Taggar, S., & Ellis, R. J. (2007) · The Leadership Quarterly · doi:10.1016/j.leaqua.2007.01.002 · Cited by 140
Reciprocal Relationships Between Collective Efficacy and Team Performance in Women's Ice Hockey.
Myers, N. D., Payment, C. A., & Feltz, D. L. (2004) · Group Dynamics Theory Research and Practice · doi:10.1037/1089-2699.8.3.182 · Cited by 148
A longitudinal study of safety climate on the Norwegian continental shelf
Tharaldsen, J., Olsen, E., & Rundmo, T. (2007) · Safety Science · doi:10.1016/j.ssci.2007.05.006 · Cited by 139
Preferences for group dominance track and mediate the effects of macro-level social inequality and violence across societies
Kunst, J. R., Fischer, R. D., Sidanius, J., & Thomsen, L. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1616572114 · Cited by 150
“70 (65) in all states (SI Appendix, Table S4).”
Background
Crafting in context: Exploring when job crafting is dysfunctional for performance effectiveness.
Dierdorff, E. C., & Jensen, J. M. (2017) · Journal of Applied Psychology · doi:10.1037/apl0000295 · Cited by 139
A multilevel investigation of the link between ethical leadership behaviour and employees green behaviour in the hospitality industry
Wood, B. P., Eid, R., & Agag, G. (2021) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2021.102993 · Cited by 138
Examining Employee Involvement in the Context of Participative Work Environments
Tesluk, P. E., Vance, R. J., & Mathieu, J. E. (1999) · Group & Organization Management · doi:10.1177/1059601199243003 · Cited by 145
Climate strength – How leaders form consensus
Luria, G. (2008) · The Leadership Quarterly · doi:10.1016/j.leaqua.2007.12.004 · Cited by 137
Family background, entry selectivity and opportunities to learn: What matters in primary teacher education? An international comparison of fifteen countries
Blömeke, S., Suhl, U., Kaiser, G., & Döhrmann, M. (2011) · Teaching and Teacher Education · doi:10.1016/j.tate.2011.08.006 · Cited by 137
The Direct and Indirect Impact of Gender Diversity in New Venture Teams on Innovation Performance
Dai, Y., Byun, G., & Ding, F. (2018) · Entrepreneurship Theory and Practice · doi:10.1177/1042258718807696 · Cited by 151
Multilevel Investigation of Adaptive Performance
Han, T. Y., & Williams, K. J. (2008) · Group & Organization Management · doi:10.1177/1059601108326799 · Cited by 136
Roadmapping for technology push and partnership: A contribution for open innovation environments
Caetano, M., & Amaral, D. C. (2011) · Technovation · doi:10.1016/j.technovation.2011.01.005 · Cited by 135
Relation between teachers' early expectations and students' later perceived competence in physical education classes: Autonomy-supportive climate as a moderator.
Trouilloud, D. O., Sarrazin, P. G., Bressoux, P., & Bois, J. E. (2006) · Journal of Educational Psychology · doi:10.1037/0022-0663.98.1.75 · Cited by 135
Teachers’ organizational citizenship behavior: Examining the boundary between in-role behavior and extra-role behavior from the perspective of teachers, principals and parents
Belogolovsky, E., & Somech, A. (2009) · Teaching and Teacher Education · doi:10.1016/j.tate.2009.10.032 · Cited by 137
Flow experience and team performance: The role of team goal commitment and information exchange
Aubé, C., Brunelle, É., & Rousseau, V. (2013) · Motivation and Emotion · doi:10.1007/s11031-013-9365-2 · Cited by 133
“In order to ensure within-group interrater agreement was high enough to justify data aggregation, rwg indices were calculated (James et al. 1984, 1993).”
Team Creative Performance
Hon, A. H. Y., & Chan, W. W. H. (2012) · Cornell Hospitality Quarterly · doi:10.1177/1938965512455859 · Cited by 141
“Likewise, ICC(1) values were all greater than the conventional cut-off value of .05 and all were significant, indicating that between-team variance was greater than within-team variance (James et al. 1993).”
Background
Trust‐effectiveness patterns in schools
Forsyth, P. B., Barnes, L. L. B., & Adams, C. M. (2006) · Journal of Educational Administration · doi:10.1108/09578230610652024 · Cited by 144
Interorganizational Teams as Boundary Spanners Between Supplier and Customer Companies
Stock, R. (2006) · Journal of the Academy of Marketing Science · doi:10.1177/0092070306288765 · Cited by 133
Achieving Service-Sales Ambidexterity
Yu, T., Patterson, P. G., & de Ruyter, K. (2012) · Journal of Service Research · doi:10.1177/1094670512453878 · Cited by 132
The Relationship Between Supervisor Personality, Supervisors’ Perceived Stress and Workplace Bullying
Mathisen, G. E., Einarsen, S. V., & Mykletun, R. J. (2010) · Journal of Business Ethics · doi:10.1007/s10551-010-0674-z · Cited by 132
Multiple Informant Methodology: A Critical Review and Recommendations
Wagner, S. M., Rau, C., & Lindemann, E. (2010) · Sociological Methods & Research · doi:10.1177/0049124110366231 · Cited by 143
“Finn (1970) and James et al. (1984, 1993) assumed that in the case of no agreement the answers are entirely due to random measurement errors, which means that each possible answer is equally likely and therefore that the answers are distributed uniformly in a rectangular fashion.”
Background
New product portfolio performance in family firms
Kraiczy, N. D., Hack, A., & Kellermanns, F. W. (2013) · Journal of Business Research · doi:10.1016/j.jbusres.2013.06.005 · Cited by 131
Enhancing Entrepreneurial Orientation in Dynamic Environments: The Interplay between Top Management Team Advice-Seeking and Absorptive Capacity
Doorn, S. V., Heyden, M. L. M., & Volberda, H. (2016) · Long Range Planning · doi:10.1016/j.lrp.2016.06.003 · Cited by 134
Assessing interrater agreement on the job relevance of a test: A comparison of CVI, T, rWG(J)}, and r*WG(J)} indexes.
Lindell, M. K., & Brandt, C. J. (1999) · Journal of Applied Psychology · doi:10.1037/0021-9010.84.4.640 · Cited by 131
When the dark ones gain power: perceived position power strengthens the effect of supervisor Machiavellianism on abusive supervision in work teams.
Wisse, B., & Sleebos, E. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.05.019 · Cited by 175
Evaluation of the Scrub Practitioners’ List of Intraoperative Non-Technical Skills (SPLINTS) system
Mitchell, L., Flin, R. H., Yule, S., Mitchell, J., Coutts, K., & Youngson, G. G. (2011) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.08.012 · Cited by 130
SHRM and product innovation: testing the moderating effects of organizational culture and structure in Chinese firms
Wei, L., Liu, J., & Herndon, N. C. (2011) · The International Journal of Human Resource Management · doi:10.1080/09585192.2011.538965 · Cited by 134
“Three indices were calculated for justifying the propriety of the aggregation of culture measures and examining if there was significant between-company variance in each scale: the intraclass correlation coefficient [1] and [2] (ICC[1] and ICC[2]), and the within-group interrater reliability ( R wg) (Bartko 1976 ; Jam…”
Method
Organizational climate and climate strength in UK hospitals
Dawson, J., González‐Romá, V., Davis, A. J., & West, M. A. (2008) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320601046664 · Cited by 139
“) Table 2 shows values of ICC(2) and ICC(1) (intraclass correlations; to measure interrater reliability), and AD M and r WG(j) (to measure agreement) for each organization (James, Demaree, & Wolf, 1984 , 1993 ; a more familiar index for demonstrating agreement) for the three climate scales.”
Transformational and transactional leaderships, empowerment climate, and innovation performance: A multilevel analysis in the Chinese context
Si, S., & Wei, F. (2011) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2011.570445 · Cited by 130
“Both rwg (James, Demaree, & Wolf, 1993 ) and intraclass correlation (ICC; Bliese, 2000 ) as computed through a random intercepts model indicated the proportion of total variance between teams.”
BackgroundMethod
Too Unsafe to Monitor? How Board–CEO Cognitive Conflict and Chair Leadership Shape Outside Director Monitoring
Veltrop, D. B., Bezemer, P., Nicholson, G. J., & Pugliese, A. (2020) · Academy of Management Journal · doi:10.5465/amj.2017.1256 · Cited by 133
Toward a Better Understanding of the Link Between Ethical Climate and Job Satisfaction: A Multilevel Analysis
Wang, Y.-D., & Hsieh, H.-H. (2011) · Journal of Business Ethics · doi:10.1007/s10551-011-0984-9 · Cited by 130
“The mean rwgs were all greater than the conventionally accepted value of .70 (James et al. 1993), indicating a reasonable level of agreement.”
BackgroundMethodInfluential
A cross‐level analysis of organizational creativity climate and perceived innovation
Lin, C. Y., & Liu, F. (2012) · European Journal of Innovation Management · doi:10.1108/14601061211192834 · Cited by 129
“70 is generally considered as a sufficient agreement to warrant aggregation ( James et al. , 1984, 1993”
Background
Labor-Management Cooperation: Antecedents and Impact on Organizational Performance
Deery, S. J., & Iverson, R. D. (2005) · Industrial and Labor Relations Review · doi:10.1177/001979390505800404 · Cited by 129
The Bounded Rationality Bias in Managerial Valuation of Real Options: Theory and Evidence from IT Projects*
Tiwana, A., Wang, J., Keil, M., & Ahluwalia, P. (2007) · Decision Sciences · doi:10.1111/j.1540-5915.2007.00152.x · Cited by 129
“A weak relationship between the two stakeholders’ responses would indicate that the original assessments are susceptible to bias (James, Demaree, & Wolf, 1984; James, Demaree, & Wolf, 1993 ).”
Background
Shared leadership, diversity, and information sharing in teams
Hoch, J. E. (2014) · Journal of Managerial Psychology · doi:10.1108/jmp-02-2012-0053 · Cited by 128

Authors: James, L. R., Demaree, R. G., & Wolf, G.
Affiliations: University of Tennessee at Knoxville (US)
Venue: Journal of Applied Psychology
Publisher: American Psychological Association (APA)
Published: 1993-04
Volume: 78
Issue: 2
Pages: 306-309
ISSN: 0021-9010 (ISSN-L)
1939-1854
DOI: 10.1037/0021-9010.78.2.306
OpenAlex: W2022950201
Cited by: 2,221
References: 21
Citations by year: 2026: 54 · 2025: 88 · 2024: 76 · 2023: 85 · 2022: 88 · 2021: 97 · 2020: 95 · 2019: 98 · 2018: 108 · 2017: 82 · 2016: 86 · 2015: 100 · 2014: 111 · 2013: 110 · 2012: 96
FWCI: 12.64
Type: journal article
Language: en
Open access: closed
Locations: Journal of Applied Psychology (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
Keywords: inter-rater reliability, interobserver agreement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

rwg: an assessment of within-group interrater agreement

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks