Cited by 2,221Reliability and Agreement in MeasurementPsychometric Methodologies and TestingTeam Dynamics and Performance

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

rwg: an assessment of within-group interrater agreement

James, Lawrence R., Demaree, Robert G., Wolf, Gerrit·1993·Journal of Applied Psychology

DOI 10.1037/0021-9010.78.2.306

Open this paper in CoChat — free account

Abstract

Schmidt and Hunter (1989) critiqued the within-group interrater reliability statistic (r wg ) described by James, Demaree, and Wolf (1984). Kozlowski and Hattrup (1992) responded to the Schmidt and Hunter critique and argued that r wg is a suitable index of interrater agreement. This article focuses on the interpretation of r wg as a measure of agreement among judges' ratings of a single target. A new derivation of r wg is given that underscores this interpretation

Integrity signals

Registry

Crossref · 10.1037/0021-9010.78.2.306

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,009
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
Self-citation
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Employee and customer perceptions of service in banks: Replication and extension.
Schneider, B., & Bowen, D. E. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.3.423 · Cited by 1,344
Superiors' evaluations and subordinates' perceptions of transformational and transactional leadership.
Hater, J. J., & Bass, B. M. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.4.695 · Cited by 1,136
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 883
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 830

Flagged first, then by citations

Show all 21 references

Sorted by citations

Items 501–600 of 1,001

A New Look at the Role of Insiders in the Newcomer Socialization Process
Slaughter, J. E., & Zickar, M. J. (2006) · Group & Organization Management · doi:10.1177/1059601104273065 · Cited by 100
The dark side of creativity: Coworker envy and ostracism as a response to employee creativity
Breidenthal, A. P., Liu, D., Bai, Y., & Mao, Y. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.08.001 · Cited by 109
Readiness to Report Medical Treatment Errors
Naveh, E., Katz‐Navon, T., & Stern, Z. (2006) · Medical Care · doi:10.1097/01.mlr.0000197035.12311.88 · Cited by 100
“Although the departments in the sample were composed of physicians and nurses, the high ICCs and the rwg coefficients in each department demonstrated high levels of agreement among physicians and nurses.(42)”
BackgroundMethodResultInfluential
[Case Study] CityCenter and Cosmopolitan Construction Projects, Las Vegas, Nevada: Lessons learned from the use of multiple sources and mixed methods in a safety needs assessment
Gittleman, J. L., Gardner, P., Haile, E., Sampson, J., Cigularov, K. P., Ermann, E. D., Stafford, P., & Chen, P. Y. (2010) · Journal of Safety Research · doi:10.1016/j.jsr.2010.04.004 · Cited by 99
Implementing computer technology: a multiorganizational test of Klein and Sorra’s model
Holahan, P. J., Aronson, Z. H., Jurkat, M. P., & Schoorman, F. D. (2004) · Journal of Engineering and Technology Management · doi:10.1016/j.jengtecman.2003.12.003 · Cited by 100
“Rwg(j) provides a measure of within group agreement, also referred to as within group inter-rater reliability ( James et al., 1993, 1984 ).”
Background
Members’ Openness to Experience and Teams’ Creative Performance
Schilpzand, M. C., Herold, D. M., & Shalley, C. E. (2010) · Small Group Research · doi:10.1177/1046496410377509 · Cited by 105
Team heterogeneity and its relationship with team support and team effectiveness
Drach‐Zahavy, A., & Somech, A. (2002) · Journal of Educational Administration · doi:10.1108/09578230210415643 · Cited by 98
A multilevel model of emotional skills, communication performance, and task performance in teams
Troth, A. C., Jordan, P. J., Lawrence, S. A., & Tse, H. H. M. (2011) · Journal of Organizational Behavior · doi:10.1002/job.785 · Cited by 109
Toward an Organizational Model of Change in Elementary Schools
Schechter, C., & Qadach, M. (2011) · Educational Administration Quarterly · doi:10.1177/0013161x11419653 · Cited by 105
Transactive Memory Systems and Firm Performance: An Upper Echelons Perspective
Heavey, C., & Şimşek, Z. (2015) · Organization Science · doi:10.1287/orsc.2015.0979 · Cited by 96
Exploratory Innovation, Exploitative Innovation, and Ambidexterity: The Impact of Environmental and Organizational Antecedents
Jansen, J. J. P., Bosch, F. A. J. V. D., & Volberda, H. (2005) · Schmalenbach Business Review · doi:10.1007/bf03396721 · Cited by 119
Does top management team responsible leadership help employees go green? The role of green human resource management and environmental felt‐responsibility
Lu, H., Xu, W., Cai, S., Yang, F., & Chen, Q. (2022) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.2239 · Cited by 102
Data Aggregation in Multilevel Analysis: A Review of Conceptual and Statistical Issues
Dixon, M. A., & Cunningham, G. B. (2006) · Measurement in Physical Education and Exercise Science · doi:10.1207/s15327841mpee1002_2 · Cited by 96
“LeBreton, James, and Lindell (2005), following James et al. (1984, 1993), also provided an equation for calculating r wg(J), or agreement index for multi-item scales.”
BackgroundMethodInfluential
Novelty and new firm performance: The case of employment systems in knowledge-intensive service organizations
Jennings, P. D., Jennings, P. D., & Greenwood, R. (2008) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2008.03.003 · Cited by 96
Cognitive Frames, Learning Mechanisms, and Innovation Ambidexterity
Lin, H., & McDonough, E. F. (2014) · Journal of Product Innovation Management · doi:10.1111/jpim.12199 · Cited by 95
Testing a theory of organizational culture, climate and youth outcomes in child welfare systems: A United States national study
Williams, N. J., & Glisson, C. A. (2013) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2013.09.003 · Cited by 102
Strategic Flexibility in New High‐Technology Ventures
Dai, Y., Goodale, J. C., Byun, G., & Ding, F.-S. (2018) · Journal of Management Studies · doi:10.1111/joms.12288 · Cited by 83
Linking molar organizational climate and strategic implementation climate to clinicians’ use of evidence-based psychotherapy techniques: cross-sectional and lagged analyses from a 2-year observational study
Williams, N. J., Ehrhart, M. G., Aarons, G. A., Marcus, S. C., & Beidas, R. S. (2018) · Implementation Science · doi:10.1186/s13012-018-0781-2 · Cited by 95
It is all about the bottom line: Group bottom‐line mentality, psychological safety, and group creativity
Greenbaum, R. L., Bonner, J., Mawritz, M. B., Butts, M. M., & Smith, M. B. (2020) · Journal of Organizational Behavior · doi:10.1002/job.2445 · Cited by 97
Are family female directors catalysts of innovation in family small and medium enterprise s ?
Bauweraerts, J., Rondi, E., Rovelli, P., Massis, A. D., & Sciascia, S. (2022) · Strategic Entrepreneurship Journal · doi:10.1002/sej.1420 · Cited by 95
Imprinting Effects of Founding Core Teams on HR Values in New Ventures
Leung, A., der Foo, M., & Chaturvedi, S. (2013) · Entrepreneurship Theory and Practice · doi:10.1111/j.1540-6520.2012.00532.x · Cited by 95
Complex Relationships Among Personality Traits, Job Characteristics, and Work Behaviors
van den Berg, P. T., & Feij, J. A. (2003) · International Journal of Selection and Assessment · doi:10.1111/j.0965-075x.2003.00255.x · Cited by 95
The Impact of Cultural Value Diversity on Multicultural Team Performance☆
KIRKMAN, B., & SHAPIRO, D. (2005) · Advances in International Management · doi:10.1016/s0747-7929(05)18002-0 · Cited by 94
Do happy leaders lead better? Affective and attitudinal antecedents of transformational leadership
Jin, S., Seo, M., & Shapiro, D. L. (2015) · The Leadership Quarterly · doi:10.1016/j.leaqua.2015.09.002 · Cited by 118
Instructional leadership and teachers' intent to leave: The mediating role of collective teacher efficacy and shared vision
Qadach, M., Schechter, C., & Da’as, R. (2019) · Educational Management Administration & Leadership · doi:10.1177/1741143219836683 · Cited by 130
“70 or greater is suggested as an adequate amount of within-group interrater agreement (rWG) (James et al., 1993).”
Background
High commitment work systems and employee well‐being: The roles of workplace friendship and task interdependence
Zhang, Y., Sun, J., Shaffer, M. A., & Lin, C.-H. V. (2021) · Human Resource Management · doi:10.1002/hrm.22093 · Cited by 105
Influences on Strategic Decision-making in the Dutch Financial Services Industry
Brouthers, K. D., Brouthers, L. E., & Werner, S. (2000) · Journal of Management · doi:10.1177/014920630002600506 · Cited by 95
“James et al. (1993) suggest that responses from the same organization should be tested for interrater agreement.”
Background
Effects of unit empowerment and perceived support for professional nursing practice on unit effectiveness and individual nurse well-being: A time-lagged study
Laschinger, H. K. S., Nosko, A., Wilk, P., & Finegan, J. (2014) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.04.010 · Cited by 92
Family Background, Entry Selectivity and Opportunities to Learn: What Matters in Primary Teacher Education? An International Comparison of Fifteen Countries
Blömeke, S., Suhl, U., Kaiser, G., & Dohrmann, M. (2012) · Advances in mathematics education · doi:10.1007/978-94-007-6437-8_15 · Cited by 166
Masked translation priming with Japanese–English bilinguals: Interactions between cognate status, target frequency and L2 proficiency
Nakayama, M., Sears, C. R., Hino, Y., & Lupker, S. J. (2013) · Journal of Cognitive Psychology · doi:10.1080/20445911.2013.839560 · Cited by 100
Line Managers as Paradox Navigators in HRM Implementation: Balancing Consistency and Individual Responsiveness
Fu, N., Flood, P. C., Rousseau, D. M., & Morris, T. (2018) · Journal of Management · doi:10.1177/0149206318785241 · Cited by 95
The building blocks of shared leadership: The interactive effects of diversity types, team climate, and time
Kukenberger, M. R., & D’Innocenzo, L. (2019) · Personnel Psychology · doi:10.1111/peps.12318 · Cited by 103
Do high-performance human resource practices help corporate entrepreneurship? The mediating role of organizational citizenship behavior
Zhang, Z., Wan, D., & Jia, M. (2008) · The Journal of High Technology Management Research · doi:10.1016/j.hitech.2008.10.005 · Cited by 92
Job insecurity climate's influence on employees' job attitudes: Evidence from two European countries
Sora, B., Caballer, A., Silla, J. M. P., & Witte, H. D. (2008) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320802211968 · Cited by 92
“Both the Spanish sample and the Belgian sample obtained median and mean values of rwg(j) that were slightly higher than the critical value of .70 (James et al., 1993, 1984).”
BackgroundResult
The Top Management Team, Reflexivity, Knowledge Sharing and New Product Performance: A Study of the Irish Software Industry
MacCurtain, S., Flood, P. C., Ramamoorthy, N., West, M. A., & Dawson, J. (2010) · Creativity and Innovation Management · doi:10.1111/j.1467-8691.2010.00564.x · Cited by 105
Organizational safety climate and supervisor safety enforcement: Multilevel explorations of the causes of accident underreporting.
Probst, T. M. (2015) · Journal of Applied Psychology · doi:10.1037/a0039195 · Cited by 91
Perceived Organizational Politics, Engagement, and Stress: The Mediating Influence of Meaningful Work
Landells, E. M., & Albrecht, S. L. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01612 · Cited by 108
The impact of service employees’ technostress on customer satisfaction and delight: A dyadic analysis
Christ‐Brendemühl, S., & Schaarschmidt, M. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.06.021 · Cited by 110
Social media resources and capabilities as strategic determinants of social media performance
Marchand, A., Hennig‐Thurau, T., & Flemming, J. (2020) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2020.09.011 · Cited by 133
The mediating role of operations knowledge in the relationship of context with performance
Germain, R. N., Dröge, C., & Christensen, W. J. (2001) · Journal of Operations Management · doi:10.1016/s0272-6963(00)00067-x · Cited by 92
Male and female managers' ability to read emotions: Relationships with supervisor's performance ratings and subordinates' satisfaction ratings
Byron, K. L. (2007) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317907x174349 · Cited by 100
Development and validation of health service management competencies.
Liang, Z.-M., Howard, P., Leggat, S., & Bartram, T. (2018) · Journal of health organization and management · doi:10.1108/jhom-06-2017-0120 · Cited by 61
The Efficacy Advantage: Factors Related to the Formation of Group Efficacy1
Gibson, C. B. (2003) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2003.tb01879.x · Cited by 94
“Within-group agreement was estimated using (rwg) (James, Demaree, and Wolf, 1993).”
BackgroundMethod
Individual and Crew Level Determinants of Collective Efficacy in Rowing
Magyar, T. M., Feltz, D. L., & Simpson, I. P. (2004) · Journal of Sport and Exercise Psychology · doi:10.1123/jsep.26.1.136 · Cited by 90
The Impact of Participative and Directive Leadership on Teachers' Performance: The Intervening Effects of Job Structuring, Decision Domain, and Leader-Member Exchange
Somech, A., & Wenderow, M. (2006) · Educational Administration Quarterly · doi:10.1177/0013161x06290648 · Cited by 105
“A value of .70 or above is suggested as a “good” amount of within-group interrater agreement (James et al., 1993).”
BackgroundInfluential
Spiritual Leadership and Organizational Performance: An Exploratory Study
Fry, L. W., & Matherly, L. L. (2006) · Cited by 90
Coaches’ and Athletes’ Perceptions of Efficacy-Enhancing Techniques
Vargas‐Tonsing, T. M., Myers, N. D., & Feltz, D. L. (2004) · The Sport Psychologist · doi:10.1123/tsp.18.4.397 · Cited by 90
Is what you see what you get?
Stewart, M. (2020) · Journal of Nuclear Cardiology · doi:10.1007/s12350-020-02212-2 · Cited by 90
Getting ahead, getting along, and getting prosocial: Examining extraversion facets, peer reactions, and leadership emergence.
Hu, J. J., Zhang, Z., Jiang, K., & Chen, W. (2019) · Journal of Applied Psychology · doi:10.1037/apl0000413 · Cited by 97
“Similar to Study 1, we averaged all peers’ ratings to reflect a focal member’s advice seeking by peers and peer liking (advice seeking by peers: median rwg .99; peer liking: median rwg .96; James et al., 1993).”
Result
360 Feedback from Another Angle
Bracken, D. W., Timmreck, C. W., Fleenor, J. W., & Summers, L. (2001) · Human Resource Management · doi:10.1002/hrm.4012 · Cited by 100
Perceptions of safety climate across construction personnel: Associations with injury rates
Marín, L. S., Lipscomb, H. J., Cifuentes, M., & Punnett, L. (2019) · Safety Science · doi:10.1016/j.ssci.2019.05.056 · Cited by 89
The Organizational Readiness for Change scale in adolescent programs: Criterion validity
Saldana, L., Chapman, J. E., Henggeler, S. W., & Rowland, M. D. (2007) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2006.12.029 · Cited by 88
A Double-Edged Sword: The Effects of Challenge and Hindrance Time Pressure on New Product Development Teams
Chong, D., van Eerde, W., Chai, K., & Rutte, C. G. (2010) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2010.2048914 · Cited by 97
“We conducted the aggregation analysis with the average interrater agreement coefficient (rwg ; [75]) and the interclass correlation coefficient (ICC; [86]) because both tests”
Method
A Neurological and Ideological Perspective of Ethical Leadership
Waldman, D. A., Wang, D., Hannah, S. T., & Balthazard, P. A. (2016) · Academy of Management Journal · doi:10.5465/amj.2014.0644 · Cited by 89
The healthy personality from a basic trait perspective.
Bleidorn, W., Hopwood, C. J., Ackerman, R. A., Witt, E. A., Kandler, C., Riemann, R., Samuel, D. B., & Donnellan, M. B. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000231 · Cited by 102
Does transactional leadership count for team innovativeness?
Liu, J., Liu, X., & Zeng, X. (2011) · Journal of Organizational Change Management · doi:10.1108/09534811111132695 · Cited by 89
Long-Term Youth Criminal Outcomes in MST Transport: The Impact of Therapist Adherence and Organizational Climate and Structure
Schoenwald, S. K., Chapman, J. E., Sheidow, A. J., & Carter, R. E. (2009) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374410802575388 · Cited by 94
“Interrater agreement as indexed using r WG (James, Demaree, & Wolf, 1993 ; LeBreton, James, & Lindell, 2005) ranged from medium to high (r WG 59–.”
Method
Roles of safety climate and shift work on perceived injury risk: A multi-level analysis
Huang, Y.-H., Chen, J., DeArmond, S. E., Cigularov, K. P., & Chen, P. Y. (2007) · Accident Analysis & Prevention · doi:10.1016/j.aap.2007.02.006 · Cited by 88
Occupational safety: Application of the job demand–control-support model
Snyder, L. A., Krauss, A. D., Chen, P. Y., Finlinson, S., & Huang, Y.-H. (2008) · Accident Analysis & Prevention · doi:10.1016/j.aap.2008.06.008 · Cited by 90
Differentiation in leader–member exchange: A hierarchical linear modeling approach
Ma, L., & Qu, Q. (2010) · The Leadership Quarterly · doi:10.1016/j.leaqua.2010.07.004 · Cited by 88
When do bad apples not spoil the barrel? Negative relationships in teams, team performance, and buffering mechanisms.
de Jong, J. P., Curșeu, P. L., & Leenders, R. (2014) · Journal of Applied Psychology · doi:10.1037/a0036284 · Cited by 92
Development and validation of an instrument for assessing collective psychological ownership in organizational field settings
Pierce, J. L., Jussila, I., & Li, D. (2017) · Journal of Management & Organization · doi:10.1017/jmo.2016.66 · Cited by 90
Empowering leadership, mindful organizing and safety performance in a nuclear power plant: A multilevel structural equation model
Gracia, F. J., Tomás-Marco, I., Martínez‐Córcoles, M., & Silla, J. M. P. (2019) · Safety Science · doi:10.1016/j.ssci.2019.104542 · Cited by 99
Knowledge sharing, shared leadership and innovative behaviour: a cross-level analysis
Vandavasi, R. K. K., McConville, D. C., Uen, J. F., & Yepuru, P. (2020) · International Journal of Manpower · doi:10.1108/ijm-04-2019-0180 · Cited by 87
An organization's ethical climate, innovation, and performance
Choi, B. K., Moon, H. K., & Ko, W. (2013) · Management Decision · doi:10.1108/md-sep-2011-0334 · Cited by 98
When R&D spending is not enough: The critical role of culture when you really want to innovate
Wang, S., Guidice, R. M., Tansky, J. W., & Wang, Z. (2010) · Human Resource Management · doi:10.1002/hrm.20365 · Cited by 93
Selecting sustainable teams for PPP projects
Kumaraswamy, M. M., & Anvuur, A. M. (2007) · Building and Environment · doi:10.1016/j.buildenv.2007.02.001 · Cited by 87
“’s [60, 61] procedures for the within-group interrater agreement statistic, rwg, were used to compute the within-group interrater agreement coefficients for: (1) the criterion ratings of experts, rwg(I); and experts’ mean ratings of the overall and relational frameworks, rwg(J).”
Method
Antecedents of Severe and Nonsevere Medication Errors
Chang, Y., & Mark, B. A. (2009) · Journal of Nursing Scholarship · doi:10.1111/j.1547-5069.2009.01253.x · Cited by 101
Emotional Intelligence, Communication Competence, and Student Perceptions of Team Social Cohesion
Troth, A. C., Jordan, P. J., & Lawrence, S. A. (2012) · Journal of Psychoeducational Assessment · doi:10.1177/0734282912449447 · Cited by 100
Mechanisms of Change in the ARC Organizational Strategy: Increasing Mental Health Clinicians’ EBP Adoption Through Improved Organizational Culture and Capacity
Williams, N. J., Glisson, C. A., Hemmelgarn, A. L., & Green, P. (2016) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-016-0742-5 · Cited by 93
“Clinicians’ responses to the proficiency culture items were aggregated to the agency level based on a referentshift consensus composition model (Chan 1998) and evidence of adequate within-agency agreement using the rwg(j) index (James et al. 1993).”
Method
A multilevel investigation of factors influencing university students’ behavioral engagement in flipped classrooms
Lai, H., Hsieh, P.-J., Uden, L., & Yang, C.-H. (2021) · Computers & Education · doi:10.1016/j.compedu.2021.104318 · Cited by 103
Ethical leader behavior and leader effectiveness: the role of prototypicality and trust
Kalshoven, K., & Hartog, D. N. D. (2009) · UvA-DARE (University of Amsterdam) · Cited by 86
Measuring Interrater Agreement for Ratings of a Single Target
Lindell, M. K., & Brandt, C. J. (1997) · Applied Psychological Measurement · doi:10.1177/01466216970213006 · Cited by 86
Investigating the relationship between transactive memory and performance in collaborative learning
Michinov, N., & Michinov, E. (2008) · Learning and Instruction · doi:10.1016/j.learninstruc.2008.01.003 · Cited by 86
System Dynamics Understanding in Projects: Information Sharing, Psychological Safety, and Performance Effects
Bendoly, E. (2013) · Production and Operations Management · doi:10.1111/poms.12024 · Cited by 99
“Internal consistency in scoring across the interviewers for this random sample was sufﬁciently high ( r WG = 0.89) (James et al. 1993).”
BackgroundInfluential
The roles of perceived task interdependence and group members' interdependence in the development of collective efficacy in university student group contexts
Alavi, S. B., & McCormick, J. (2007) · British Journal of Educational Psychology · doi:10.1348/000709907x240471 · Cited by 86
Ambidexterity in new ventures: The impact of new product development alliances and transactive memory systems
Dai, Y., Du, K., Byun, G., & Zhu, X. (2017) · Journal of Business Research · doi:10.1016/j.jbusres.2017.02.009 · Cited by 84
Developing a Method for Evaluating Crew Resource Management Skills: A European Perspective
O'Connor, P. E., Hörmann, H.-J., Flin, R. H., Lodge, M., Goeters, K.-M., & The JARTEL Group (2002) · International Journal of Aviation Psychology · doi:10.1207/s15327108ijap1203_5 · Cited by 85
“The inter-rater agreement was measured using an index developed by James, Demaree and Wolf (1984, 1993) called the within group inter-rater reliability measure (rwg; see James et al, 1984, 1993 for more detail).”
Method
Mood scores: Mood and performance in professional cricketers
Totterdell, P. (1999) · British Journal of Psychology · doi:10.1348/000712699161422 · Cited by 85
Group‐level safety climate in the Australian construction industry: within‐group homogeneity and between‐group differences in road construction and maintenance
Lingard, H. C., Cooke, T., & Blismas, N. G. (2009) · Construction Management and Economics · doi:10.1080/01446190902822971 · Cited by 85
Procedural justice, ethical climate and service outcomes in restaurants
Luria, G., & Yagil, D. (2007) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2007.07.022 · Cited by 85
“…per unit (Reichers and Schneider, 1990); and (b) strength, i.e., within-group variability of perceptions, quantified with measures such as intra-class correlation (ICC1, ICC2; James, 1982), within-group correlation (Rwg; James et al., 1984, 1993) and standard deviation (Schneider et al., 2002).”
Method
Managing Conflict in School Teams: The Impact of Task and Goal Interdependence on Conflict Management and Team Effectiveness
Somech, A. (2008) · Educational Administration Quarterly · doi:10.1177/0013161x08318957 · Cited by 87
“A value of .70 or greater is suggested as a “good” amount of within-group interrater agreement (James et al., 1993).”
Background
When and how knowledge sharing benefits team creativity: The importance of cognitive team diversity
Men, C., Fong, P. S. W., Luo, J., Zhong, J., & Huo, W. (2017) · Journal of Management & Organization · doi:10.1017/jmo.2017.47 · Cited by 87
Exploratory and confirmatory factor analysis: guidelines, issues, and alternatives
Hurley-Hanson, A. E., Scandura, T. A., Schriesheim, C. A., Brannick, M. Τ., Seers, A., Vandenberg, R. J., & Williams, L. J. (1997) · Journal of Organizational Behavior · doi:10.1002/(sici)1099-1379(199711)18:6 · Cited by 928
Systematic variation in organizationally-shared cognitive prototypes of effective leadership based on organizational form
Dickson, M. W., Resick, C. J., & Hanges, P. J. (2006) · The Leadership Quarterly · doi:10.1016/j.leaqua.2006.07.005 · Cited by 83
Effect of leadership and communication practices on the safety climate and behaviour of construction workgroups
Lingard, H. C., Zhang, R. P., & Oswald, D. (2019) · Engineering Construction & Architectural Management · doi:10.1108/ecam-01-2018-0015 · Cited by 83
Ethics, Values, and Organizational Justice: Individuals, Organizations, and Beyond
Schminke, M. J., Arnaud, A., & Taylor, R. M. (2014) · Journal of Business Ethics · doi:10.1007/s10551-014-2251-3 · Cited by 94
Self-management competencies in self-managing teams: Their impact on multi-team system productivity
Millikin, J. P., Hom, P. W., & Manz, C. C. (2010) · The Leadership Quarterly · doi:10.1016/j.leaqua.2010.07.001 · Cited by 82
Testing extrinsic and intrinsic motivation as explanatory variables for the safety climate–safety performance relationship among long-haul truck drivers
Zohar, D. M., Huang, Y.-H., Lee, J., & Robertson, M. M. (2015) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2015.01.014 · Cited by 89
Turnover and Knowledge Loss: An Examination of the Differential Impact of Production Manager and Worker Turnover in Service and Manufacturing Firms
Eckardt, R., Skaggs, B. C., & Youndt, M. A. (2014) · Journal of Management Studies · doi:10.1111/joms.12096 · Cited by 82
The magic of collective emotional intelligence in learning groups: No guys needed for the spell!
Curșeu, P. L., Pluut, H., Boroş, S., & Meslec, N. (2014) · British Journal of Psychology · doi:10.1111/bjop.12075 · Cited by 84
“To explore the consistency of within-group individual evaluations of the variables considered in our study,weused the group-size-corrected intraclass correlations (Bliese& Halverson, 1998) and the within-group agreement index (James et al., 1993).”
Method
A CASE FOR PROCEDURAL JUSTICE CLIMATE: DEVELOPMENT AND TEST OF A MULTILEVEL MODEL.
Naumann, S. E., & Bennett, N. (2000) · Academy of Management Journal · doi:10.5465/1556416 · Cited by 708
A longitudinal examination of the relationship between team leadership and performance
Gupta, V. K., Huang, R., & Niranjan, S. (2010) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051809359184 · Cited by 82
Work-Group Design and Autonomy: A Field Study of the Interaction Between Task Interdependence and Group Autonomy
Langfred, C. W. (2000) · Small Group Research · doi:10.1177/104649640003100103 · Cited by 82
“Generally, a within-group interrater agreement coefficient of above .7 is considered acceptable to justify aggregation (James et al., 1993).”
Background
The Impact of Poor Performers on Team Outcomes: An Empirical Examination of Attribution Theory
Taggar, S., & Neubert, M. (2004) · Personnel Psychology · doi:10.1111/j.1744-6570.2004.00011.x · Cited by 86
An Alternative Taxonomy of Project Management Structures: Linking Project Management Structures and Project Success
Lechler, T. G., & Dvir, D. (2010) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2010.2044441 · Cited by 83
Leadership's impact on employee engagement
Papalexandris, N., & Galanaki, E. (2009) · Leadership & Organization Development Journal · doi:10.1108/01437730910961685 · Cited by 102
A Multilevel Examination of Benevolent Leadership and Innovative Behavior in R&D Contexts: A Social Identity Approach
Gumusluoğlu, L., Karakitapoğlu‐Aygün, Z., & Scandura, T. A. (2017) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051817705810 · Cited by 88
“These values were above the .70 cutoff proposed by James, Demaree, and Wolf (1993).”
Background
Build‐to‐order and just‐in‐time as predictors of applied supply chain knowledge and market performance
Christensen, W. J., Germain, R. N., & Birou, L. M. (2004) · Journal of Operations Management · doi:10.1016/j.jom.2004.10.007 · Cited by 82
The role of corporate culture in the process of strategic human resource management: Evidence from Chinese enterprises
Wei, L., Liu, J., Zhang, Y., & Chiu, R. K. (2008) · Human Resource Management · doi:10.1002/hrm.20244 · Cited by 79

Authors: James, L. R., Demaree, R. G., & Wolf, G.
Affiliations: University of Tennessee at Knoxville (US)
Venue: Journal of Applied Psychology
Publisher: American Psychological Association (APA)
Published: 1993-04
Volume: 78
Issue: 2
Pages: 306-309
ISSN: 0021-9010 (ISSN-L)
1939-1854
DOI: 10.1037/0021-9010.78.2.306
OpenAlex: W2022950201
Cited by: 2,221
References: 21
Citations by year: 2026: 54 · 2025: 88 · 2024: 76 · 2023: 85 · 2022: 88 · 2021: 97 · 2020: 95 · 2019: 98 · 2018: 108 · 2017: 82 · 2016: 86 · 2015: 100 · 2014: 111 · 2013: 110 · 2012: 96
FWCI: 12.64
Type: journal article
Language: en
Open access: closed
Locations: Journal of Applied Psychology (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
Keywords: inter-rater reliability, interobserver agreement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

rwg: an assessment of within-group interrater agreement

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks