Cited by 2,221Reliability and Agreement in MeasurementPsychometric Methodologies and TestingTeam Dynamics and Performance

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

rwg: an assessment of within-group interrater agreement

James, Lawrence R., Demaree, Robert G., Wolf, Gerrit·1993·Journal of Applied Psychology

DOI 10.1037/0021-9010.78.2.306

Open this paper in CoChat — free account

Abstract

Schmidt and Hunter (1989) critiqued the within-group interrater reliability statistic (r wg ) described by James, Demaree, and Wolf (1984). Kozlowski and Hattrup (1992) responded to the Schmidt and Hunter critique and argued that r wg is a suitable index of interrater agreement. This article focuses on the interpretation of r wg as a measure of agreement among judges' ratings of a single target. A new derivation of r wg is given that underscores this interpretation

Integrity signals

Registry

Crossref · 10.1037/0021-9010.78.2.306

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,009
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
Self-citation
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Employee and customer perceptions of service in banks: Replication and extension.
Schneider, B., & Bowen, D. E. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.3.423 · Cited by 1,344
Superiors' evaluations and subordinates' perceptions of transformational and transactional leadership.
Hater, J. J., & Bass, B. M. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.4.695 · Cited by 1,136
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 883
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 830

Show all 21 references

Sorted by citations

Items 801–900 of 1,001

Can we win them all? Benefits and costs of structured and flexible innovation– implementations
Drach‐Zahavy, A., Somech, A., Granot, M., & Spitzer, A. (2004) · Journal of Organizational Behavior · doi:10.1002/job.239 · Cited by 53
A LONGITUDINAL EXAMINATION OF CONCOMITANT CHANGES IN TEAM LEADERSHIP AND CUSTOMER SATISFACTION
Walker, A. G., Smither, J. W., & Waldman, D. A. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00122.x · Cited by 56
When shared cognition leads to closed minds: Temporal mental models, team learning, adaptation and performance
Santos, C. M., Passos, A., & Uitdewilligen, S. (2015) · European Management Journal · doi:10.1016/j.emj.2015.11.006 · Cited by 68
Single‐ and Multiple‐Informant Research Designs to Examine the Human Resource Management-Performance Relationship
Bou-Llusar, J., Beltrán‐Martín, I., Roca-Puig, V., & Escrig-Tena, A. (2016) · British Journal of Management · doi:10.1111/1467-8551.12177 · Cited by 48
A Compositional Analysis of the Organizational Climate‐Performance Relation: Public Schools as Organizations
Griffith, J. E. (2006) · Journal of Applied Social Psychology · doi:10.1111/j.0021-9029.2006.00085.x · Cited by 60
Predicting school innovation
Schwabsky, N., Erdoğan, U., & Tschannen‐Moran, M. (2019) · Journal of Educational Administration · doi:10.1108/jea-02-2019-0029 · Cited by 60
Shared Leadership and Team Performance: The Joint Effect of Team Dispositional Composition and Collective Identification
Siangchokyoo, N. M., & Klinger, R. L. (2021) · Group & Organization Management · doi:10.1177/10596011211019928 · Cited by 56
The impact of digital leadership on hidden champions’ competitive advantage: A moderated mediation model of ambidextrous innovation and value co-creation
Cai, Q., Wu, J., Wu, T., Chang, P., & Mardani, A. (2024) · Journal of Business Research · doi:10.1016/j.jbusres.2024.114819 · Cited by 57
How are organisational climate models and patient satisfaction related? A competing value framework approach
Ancarani, A., Mauro, C. D., & Giammanco, M. D. (2009) · Social Science & Medicine · doi:10.1016/j.socscimed.2009.09.033 · Cited by 52
The role of educational psychology in teacher education: expert opinions on what teachers should know about learning, development, and assessment
Lohse-Bossenz, H., Kunina‐Habenicht, O., & Kunter, M. (2013) · European Journal of Psychology of Education · doi:10.1007/s10212-013-0181-6 · Cited by 55
The Role of Leader Boundary Activities in Enhancing Interdisciplinary Team Effectiveness
Benoliel, P. S., & Somech, A. (2014) · Small Group Research · doi:10.1177/1046496414560028 · Cited by 54
Applied customer knowledge in a manufacturing environment: Flexibility for industrial firms
Claycomb, C., Dröge, C., & Germain, R. N. (2005) · Industrial Marketing Management · doi:10.1016/j.indmarman.2004.10.008 · Cited by 52
Service Innovativeness and Innovation Success in Technology-based Knowledge-Intensive Business Services: An Intellectual Capital Approach
Lavado, A. C., Cuevas‐Rodríguez, G., & Cabello‐Medina, C. (2013) · Industry and Innovation · doi:10.1080/13662716.2013.771482 · Cited by 52
“Within-ﬁrm agreement among managers was assessed by the interrater agreement measure, r wg , developed by James et al. (1984, 1993).”
Method
Safety climate, worker health and organizational health performance
Bronkhorst, B., & Vermeeren, B. (2016) · International Journal of Workplace Health Management · doi:10.1108/ijwhm-12-2015-0081 · Cited by 67
Repatriate knowledge transfer: Antecedents and boundary conditions of a dyadic process
Burmeister, A., Lazarova, M. B., & Deller, J. (2018) · Journal of World Business · doi:10.1016/j.jwb.2018.06.004 · Cited by 52
The effectiveness of verbal self‐guidance as a transfer of training intervention: its impact on presentation performance, self efficacy and anxiety
Brown, T. C., & Morrissey, L. (2004) · Innovations in Education and Teaching International · doi:10.1080/14703290410001733302 · Cited by 54
“This was achieved using the average interrater agreement statistic, rwg (James et al., 1993), which assesses agreement among multiple raters.”
Method
Inside Management Teams: Developing a Teamwork Survey Instrument
Senior, B., & Swailes, S. (2006) · British Journal of Management · doi:10.1111/j.1467-8551.2006.00507.x · Cited by 62
Team Emotional Intelligence: Emotional Processes as a Link Between Managers and Workers
Mindeguia, R., Aritzeta, A., Garmendia, A., Martínez‐Moreno, E., Elorza, U., & Soroa, G. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.619999 · Cited by 59
“Additionally, the within-group agreement should be ensured by means of the James intercoder reliability index (James et al., 1993).”
Method
Fostering Resilience in New Venture Teams: The Role of Behavioral and Affective Integration
Chen, Y., & Zhang, Y. (2021) · Group & Organization Management · doi:10.1177/10596011211033164 · Cited by 52
Further Operationalization of Intra-Team Communication in Sports: An Updated Version of the Scale of Effective Communication in Team Sports (SECTS-2)
Sullivan, P., & Short, S. E. (2011) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2010.00722.x · Cited by 51
Patient satisfaction, managers' climate orientation and organizational climate
Ancarani, A., Mauro, C. D., & Giammanco, M. D. (2011) · International Journal of Operations & Production Management · doi:10.1108/01443571111111900 · Cited by 51
“In order to assess the inter-rater reliability for each climate model, average ICC(1), ICC(2), and RWG(j) were calculated to justify the aggregation of the climate measures at the ward level (Bliese, 2000; James et al., 1993).”
Method
Contributing beyond the call of duty: examining the role of culture in fostering citizenship behavior and success in project‐based work
Aronson, Z. H., & Lechler, T. G. (2009) · R and D Management · doi:10.1111/j.1467-9310.2009.00573.x · Cited by 51
Humane Orientation as a New Cultural Dimension of the GLOBE Project
Schlösser, O., Fresé, M., Heintze, A.-M., Al-Najjar, M., Arciszewski, T., Besevegis, E., Bishop, G. D., Bonnes, M., Clegg, C., Drozda‐Senkowska, E., et al. (2012) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022112465671 · Cited by 51
Non-Technical Skills for Surgeons (NOTSS): Critical appraisal of its measurement properties
Jung, J. J., Borkhoff, C. M., Jüni, P., & Grantcharov, T. (2018) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2018.02.021 · Cited by 51
Cross‐level, three‐way interactions among work‐group climate, gender, and frequency of harassment on morale and withdrawal outcomes of sexual harassment
Kath, L. M., Swody, C. A., Magley, V. J., Bunk, J. A., & Gallus, J. A. (2008) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317908x299764 · Cited by 51
The effects of administrative innovation implementation on performance: an organizational learning approach
Naveh, E., Meilich, O., & Marcus, A. A. (2006) · Strategic Organization · doi:10.1177/1476127006066969 · Cited by 51
Responses to Feedback: The Role of Acceptance, Affect, and Creative Behavior
Christensen-Salem, A., Kinicki, A. J., Zhang, Z., & Walumbwa, F. O. (2018) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051818757691 · Cited by 56
The effect of nonfamily managers on decision-making quality in family firm TMTs: The role of intra-TMT power asymmetries
Vandekerkhof, P., Steijvers, T., Hendriks, W., & Voordeckers, W. (2019) · Journal of Family Business Strategy · doi:10.1016/j.jfbs.2019.01.002 · Cited by 50
Microfoundations of sensing capabilities: From managerial cognition to team behavior
Harvey, J. (2022) · Strategic Organization · doi:10.1177/14761270221142959 · Cited by 56
“…it was appropriate to aggregate data on team environmental scanning from individual team members to teams, I computed interrater agreement scores (rwg(j)) to test whether the variance of responses within groups was lower than the variance between groups (James et al., 1993; LeBreton et al., 2003).”
The Influence of Nursing Unit Empowerment and Social Capital on Unit Effectiveness and Nurse Perceptions of Patient Care Quality
Laschinger, H. K. S., Read, E. A., Wilk, P., & Finegan, J. (2014) · JONA The Journal of Nursing Administration · doi:10.1097/nna.0000000000000080 · Cited by 50
Leader Humility, and Subordinates’ Organizational Citizenship Behavior and Withdrawal Behavior: Exploring the Mediating Mechanisms of Subordinates’ Psychological Capital
Qian, X., Zhang, M., & Jiang, Q. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17072544 · Cited by 57
“70 [53,54], showing a satisfactory internal consistency.”
Reducing Causal Ambiguity In Acquisition Integration: Intermediate Goals As Mediators OfIntegration Decisions and Acquisition Performance
Cording, M., Christmann, P., & King, D. R. (2008) · Academy of Management Journal · doi:10.5465/amr.2008.33665279 · Cited by 50
Hierarchical linear modeling and the "unit of analysis" problem: A solution for analyzing responses of intact group members.
Pollack, B. N. (1998) · Group Dynamics Theory Research and Practice · doi:10.1037//1089-2699.2.4.299 · Cited by 50
Safety leadership: A bibliometric literature review and future research directions
Jiang, Z., Zhao, X., Wang, Z., & Herbert, K. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114437 · Cited by 49
The Determinants of Family Owner‐Managers' Affective Organizational Commitment
Memili, E., Zellweger, T. M., & Fang, H. “. (2013) · Family Relations · doi:10.1111/fare.12015 · Cited by 49
The team‐level model of climate for innovation: A two‐level confirmatory factor analysis
Mathisen, G. E., Torsheim, T., & Einarsen, S. V. (2006) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317905x52869 · Cited by 49
The Relationship between Creative Personality Composition, Innovative Team Climate, and Team Innovativeness: An Input — Process — Output Perspective
Mathisen, G. E., Martinsen, Ø. L., & Einarsen, S. V. (2008) · The Journal of Creative Behavior · doi:10.1002/j.2162-6057.2008.tb01078.x · Cited by 49
Moderating influence of organizational justice on the relationship between job insecurity and its outcomes: A multilevel analysis
Sora, B., Caballer, A., Silla, J. M. P., Silla, I., & Gracia, F. J. (2010) · Economic and Industrial Democracy · doi:10.1177/0143831x10365924 · Cited by 49
“AD values lower than .83 (Burke and Dunlap, 2002; Burke et al., 1999) and r wg values higher than .70 (Bliese, 2000; James et al., 1993) indicated within-organization agreement.”
Background
Deep-Level Similarity and Group Social Capital: Associations With Team Functioning
van Emmerik, I. J. H., & Brenninkmeijer, V. (2009) · Small Group Research · doi:10.1177/1046496409346550 · Cited by 49
A Team-Level Study of the Relationship between Knowledge Sharing and Trust in Kenya: Moderating Role of Collaborative Technology
Kipkosgei, F., Kang, S., & Choi, S. B. (2020) · Sustainability · doi:10.3390/su12041615 · Cited by 49
“To analyze the respondents’ individual-level data at the team level, the within group agreement rWG(J) index [47] was used to aggregate the responses.”
Method
Effects of team leader emotional intelligence and team emotional climate on team member job satisfaction: A cross‐level
Liu, X., & Liu, J. (2013) · Nankai Business Review International · doi:10.1108/nbri-07-2013-0023 · Cited by 48
Assessing determinants of departmental innovation
Mohamed, M. A. K. (2002) · Personnel Review · doi:10.1108/00483480210438799 · Cited by 48
Family Businesses Can Out–Compete: As Long as They Are Willing to Question the Chosen Path
Ensley, M. D. (2006) · Entrepreneurship Theory and Practice · doi:10.1111/j.1540-6520.2006.00148.x · Cited by 47
Organizational justice, sickness absence and employee age
Tenhiälä, A., Linna, A., von Bonsdorff, M. E., Pentti, J., Vahtera, J., Mika Kivimaki, B., & Elovainio, M. J. (2013) · Journal of Managerial Psychology · doi:10.1108/jmp-07-2013-0246 · Cited by 48
“70 (James et al., 1993).”
The Adoption of Information Technology by Self-Managing Service Teams
de Jong, A., de Ruyter, K., & Lemmink, J. G. A. M. (2003) · Journal of Service Research · doi:10.1177/1094670503257046 · Cited by 48
“…justification of aggregation was required, which was tested by means of the rWG(j) coefficient, which is an indicator of within-group agreement (James, Demaree, and Wolf 1993), and the intraclass correlation (ICC) coefficient, which involves the ratio of within-group and between-groups…”
Method
Take care what you bring with you: How coaches’ mood and interpersonal behavior affect coaching success.
Ianiro-Dahm, P. M., & Kauffeld, S. (2014) · Consulting psychology journal · doi:10.1037/cpb0000012 · Cited by 48
Coaching competency and trust in coach in sport teams
Kao, S.-F., Hsieh, M.-H., & Lee, P.-L. (2017) · International Journal of Sports Science & Coaching · doi:10.1177/1747954117710508 · Cited by 50
“Following James et al.41 we estimated the withingroup inter-rater agreement index rwg.”
Method
The Study of the Relationships of Teacher’s Creative Teaching, Imagination, and Principal’s Visionary Leadership
Chen, H.-H., & Yuan, Y. (2021) · SAGE Open · doi:10.1177/21582440211029932 · Cited by 48
“In the results, principals’ visionary leadership had a mean rwg of 0.95 (range: 0.72–0.99), which indicates that our integration process was within reasonable limits (James et al., 1993).”
BackgroundInfluential
High-performance work systems, management team flexibility, employee flexibility and service-oriented organizational citizenship behaviors
Liu, N., & Lin, Y. (2019) · The International Journal of Human Resource Management · doi:10.1080/09585192.2019.1651374 · Cited by 53
The effect of nepotism on tolerance to workplace incivility: mediating role of psychological contract violation and moderating role of authentic leadership
Arıcı, H. E., Araslı, H., & Arıcı, N. Ç. (2020) · Leadership & Organization Development Journal · doi:10.1108/lodj-06-2019-0250 · Cited by 48
From shared leadership to proactive customer service performance: a multilevel investigation
Ji, L., Ye, Y., & Deng, X. (2022) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-09-2021-1077 · Cited by 50
Agile innovation: The role of team climate in rapid research and development
Pirola‐Merlo, A. (2010) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317909x480653 · Cited by 55
Happily putting the pieces together: A test of two explanations for the effects of mood on group‐level information processing
Bramesfeld, K. D., & Gasper, K. (2007) · British Journal of Social Psychology · doi:10.1348/000712607x218295 · Cited by 50
Creative Work Environments in Sport Organizations: The Influence of Sexual Orientation Diversity and Commitment to Diversity
Cunningham, G. B. (2011) · Journal of Homosexuality · doi:10.1080/00918369.2011.598413 · Cited by 51
“70 cutoff ( James, Demaree, & Wolf, 1993 ): sexual orientation diversity r wg = .”
The Dark and Bright Sides of Self-Efficacy in Predicting Learning, Innovative and Risky Performances
Salanova, M., Lorente, L., & Martínez, I. M. (2012) · The Spanish Journal of Psychology · doi:10.5209/rev_sjop.2012.v15.n3.39402 · Cited by 53
Linking positive psychological capital with customer value co-creation
Lee, Y., Hsiao, C., & Chen, Y.-C. (2017) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-05-2015-0266 · Cited by 49
Opening the black box
Plimmer, G., Bryson, J., & Teo, S. (2017) · Personnel Review · doi:10.1108/pr-10-2016-0275 · Cited by 47
“Prior to aggregating the individual level data, we computed the rWG index ( James et al., 1993) to check for within-group agreement for each organisation.”
Method
Does group cohesion moderate auditors’ whistleblowing intentions?
Alleyne, P., Haniffa, R., & Hudaib, M. (2019) · Journal of International Accounting Auditing and Taxation · doi:10.1016/j.intaccaudtax.2019.02.004 · Cited by 47
How to empower employees: using training to enhance work units’ collective empowerment
Voegtlin, C., Boehm, S. A., & Bruch, H. (2015) · International Journal of Manpower · doi:10.1108/ijm-10-2012-0158 · Cited by 50
Effects of Group-Level Fit on Group Conflict and Performance
Seong, J. Y., & Choi, J. N. (2014) · Group & Organization Management · doi:10.1177/1059601113517138 · Cited by 51
How to Facilitate Innovative Behavior and Organizational Citizenship Behavior: Evidence From Public Employees in Korea
Cho, Y. J., & Song, H. J. (2020) · Public Personnel Management · doi:10.1177/0091026020977571 · Cited by 56
“Prior to aggregating 2015 survey data, we first tested the within-group agreement (rWG) level of the data (James et al., 1984, 1993) and the interrater agreement (intraclass correlation [ICC1]) level.”
Method
Quality Management and Cooperative Values: Investigation of Multilevel Influences on Workgroup Performance
Kull, T. J., & Narasimhan, R. (2010) · Decision Sciences · doi:10.1111/j.1540-5915.2009.00260.x · Cited by 46
Creativity of student information system projects: From the perspective of network embeddedness
Yang, H., & Cheng, H.-H. (2009) · Computers & Education · doi:10.1016/j.compedu.2009.08.004 · Cited by 46
The impact of positive and negative intraoperative surgeons' leadership behaviors on surgical team performance
Barling, J. I., Akers, A. L., & Beiko, D. T. (2017) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2017.07.006 · Cited by 50
The Arabic Version of the Mobile App Rating Scale: Development and Validation Study
Bardus, M., Awada, N., Ghandour, L. A., Fares, E.-J., Gherbal, T., Al-Zanati, T., & Stoyanov, S. R. (2019) · JMIR mhealth and uhealth · doi:10.2196/16956 · Cited by 46
“…calculate variance, means, and medians used to establish interrater agreement (IRA) according to the three families of indices: James et al’s r WG(J) [47,48] (based on multiple null distributions [49]); Brown and Hauenstein’s a WG(J) [50]; and the adjusted average deviation index A DMJ(adj) [51].”
Method
Corporate Social Responsibility and Employees’ Negative Behaviors under Abusive Supervision: A Multilevel Insight
Mahmood, F., Qadeer, F., Abbas, Z., Muhammadi, Hussain, I., Saleem, M., Hussain, A., & Aman, J. (2020) · Sustainability · doi:10.3390/su12072647 · Cited by 46
Applicants’ Self-Presentational Behavior
Jansen, A., König, C. J., Stadelmann, E. H., & Kleinmann, M. (2012) · Journal of Personnel Psychology · doi:10.1027/1866-5888/a000046 · Cited by 46
AGREEMENT AND RELIABILITY ASSESSMENTS FOR PERFORMANCE OF SENSORY DESCRIPTIVE PANEL
Bi, J. (2003) · Journal of Sensory Studies · doi:10.1111/j.1745-459x.2003.tb00373.x · Cited by 46
Antecedents and the Performance Outcome of Transactive Memory in Daycare Work Groups
Peltokorpi, V., & Manka, M.-L. (2008) · European Psychologist · doi:10.1027/1016-9040.13.2.103 · Cited by 47
Personal echo chambers: Openness-to-experience is linked to higher levels of psychological interest diversity in large-scale behavioral data.
Matz, S. C. (2021) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000324 · Cited by 52
A multilevel test of Allport’s contact conditions
Koschate, M., & van Dick, R. (2011) · Group Processes & Intergroup Relations · doi:10.1177/1368430211399602 · Cited by 45
Building Cohesive Teams—The Role of Leaders’ Bottom-Line Mentality and Behavior
Riisla, K., Wendt, H., Babalola, M. T., & Euwema, M. (2021) · Sustainability · doi:10.3390/su13148047 · Cited by 62
“intra-class correlation ICC(2), which estimates the reliability of mean differences across organizations, and within-group agreement (Rwg), which assesses interrater reliability among judgments by a single group of respondents on a single variable (in this case, followers assessing their leader) [53,54].”
Background
Data Aggregation in Multilevel Research: Best Practice Recommendations and Tools for Moving Forward
LeBreton, J. M., Moeller, A. N., & Wittmer, J. L. S. (2022) · Journal of Business and Psychology · doi:10.1007/s10869-022-09853-9 · Cited by 48
Organizational career growth and high-performance work systems: The roles of job crafting and organizational innovation climate
Miao, R., Yu, J., Bozionelos, N., & Bozionelos, G. (2023) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2023.103879 · Cited by 58
Leveraging technological readiness and green dynamic capability to enhance sustainability performance in manufacturing firms
Ullah, S., Kukreti, M., Sami, A., & Shaukat, M. R. (2024) · Journal of Manufacturing Technology Management · doi:10.1108/jmtm-05-2024-0268 · Cited by 50
Leader Advancement Motive, Political Skill, Leader Behavior, and Effectiveness: A Moderated Mediation Extension of Socioanalytic Theory
Ewen, C., Wihler, A., Frieder, R. E., Blickle, G., Hogan, R., & Ferris, G. R. (2014) · Human Performance · doi:10.1080/08959285.2014.956174 · Cited by 45
Emotional intelligence and depressed mood in adolescence: A multilevel approach
Balluerka, N., Aritzeta, A., Gorostiaga, A., Gartzia, L., & Soroa, G. (2013) · International Journal of Clinical and Health Psychology · doi:10.1016/s1697-2600(13)70014-0 · Cited by 56
Innovative Entrepreneurial Teams: The Give and Take of Trust and Conflict
Khan, M. S., Breitenecker, R. J., Gustafsson, V., & Schwarz, E. J. (2015) · Creativity and Innovation Management · doi:10.1111/caim.12152 · Cited by 48
Employment contract, job insecurity and employees’ affective well-being: The role of self- and collective efficacy
Sora, B., Höge, T., Caballer, A., & Silla, J. M. P. (2018) · Economic and Industrial Democracy · doi:10.1177/0143831x18804659 · Cited by 45
“To determine the within-organizational agreement, rwg(j) (James et al., 1993) and an average deviation index (ADM(J)) (Burke and Dunlap, 2002; Burke et al., 1999) were computed.”
BackgroundMethod
Culture and gender in entrepreneurial teams: the effect on team processes and outcomes
Santos, S. C., & Neumeyer, X. (2021) · Small Business Economics · doi:10.1007/s11187-020-00432-x · Cited by 50
“…the criteria for aggregating lower level measures to higher level constructs such as the index of within-group interrater agreement (Rwg(j)) (James et al. 1993) using the 0.70 and above the threshold; and the intraclass correlation coefficients ICC(1) and ICC(2)which assess the consistency…”
MethodInfluential
Sharing Strategic Decisions: CEO Humility, TMT Decentralization, and Ethical Culture
Cortes-Mejia, S., Cortes, A. F., & Herrmann, P. (2021) · Journal of Business Ethics · doi:10.1007/s10551-021-04766-8 · Cited by 45
Dependent Correlations in Meta-Analysis
Cheung, S. F., & Chan, D. K. ‐. (2008) · Educational and Psychological Measurement · doi:10.1177/0013164408315263 · Cited by 45
Linking Individual Creativity to Organizational Innovation
Litchfield, R. C., Ford, C. M., & Gentry, R. J. (2014) · The Journal of Creative Behavior · doi:10.1002/jocb.65 · Cited by 45
Active learning climate and employee errors: The moderating effects of personality traits
Naveh, E., Katz‐Navon, T., & Stern, Z. (2015) · Journal of Organizational Behavior · doi:10.1002/job.2000 · Cited by 48
A multilevel approach for assessing business strategies on climate change
Paul, A., Lang, J. W. B., & Baumgartner, R. J. (2017) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2017.04.030 · Cited by 45
“The most common of these statistics is the rwg index (James et al., 1993) and several related indices (Bliese, 2000).”
Background
Validation of a management competency assessment tool for health service managers
Howard, P. F., Liang, Z., Leggat, S. G., & Karimi, L. (2018) · Journal of Health Organization and Management · doi:10.1108/jhom-08-2017-0223 · Cited by 44
Assessing the Quality of Mobile Phone Apps for Weight Management: User-Centered Study With Employees From a Lebanese University
Bardus, M., Ali, A., Demachkieh, F., & Hamadeh, G. (2019) · JMIR mHealth and uHealth · doi:10.2196/mhealth.9836 · Cited by 44
“…cut-off points such as for the r WG(J) : no agreement (<.29), weak (.30-.49), moderate (.50-.69), strong (.70-.89), and very strong (>.90) [64,65]; a WG(J) : not acceptable (<.59), weak (.60-.69), moderate (.70-.79), and strong agreement (>.80) [67]; and A DMJ(adj) : agreement above .80 [68].”
Organizational meta‐scripts as a source of high reliability: the case of an army armored brigade
Zohar, D. M., & Luria, G. (2003) · Journal of Organizational Behavior · doi:10.1002/job.216 · Cited by 44
Construct Validity and Interrater Agreement of the Sport Injury Rehabilitation Adherence Scale
Brewer, B. W., Avondoglio, J. B., Cornelius, A. E., Raalte, J. L. V., Brickner, J. C., Petitpas, A. J., Kolt, G. S., Pizzari, T., Schoo, A. M. M., Emery, K., et al. (2002) · Journal of Sport Rehabilitation · doi:10.1123/jsr.11.3.170 · Cited by 44
How Do I Empathize With You? Let Me Count the Ways: Relations Between Facets of Pain-Related Empathy
Issner, J. B., Caño, A., Leonard, M. T., & Williams, A. M. (2012) · Journal of Pain · doi:10.1016/j.jpain.2011.10.009 · Cited by 43
A Framework for Analyzing Levels of AnalysisIssues in Studies of E-Collaboration
Gallivan, M. J., & Benunan-Fich, R. (2005) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2005.843301 · Cited by 41
“We acknowledge that the concepts we discuss below are not novel, but have been discussed by Klein and her colleagues [22], [36], Rousseau and her colleagues [20], [21], [46], and James and his colleagues [41], [42], [47] in years past.”
BackgroundMethod
Aggression-preventive supervisor behavior: Implications for workplace climate and employee outcomes.
Yang, L., & Caughlin, D. E. (2016) · Journal of Occupational Health Psychology · doi:10.1037/a0040148 · Cited by 47
Job insecurity, employee anxiety, and commitment: The moderating role of collective trust in management
Wen, W. Y., Mather, K., & Seifert, R. V. (2018) · Journal of Trust Research · doi:10.1080/21515581.2018.1463229 · Cited by 43
Advancing theory on the multilevel role of leadership in the implementation of evidence-based health care practices
Guerrero, E. G., Frimpong, J. A., Kong, Y., Fenwick, K. M., & Aarons, G. A. (2018) · Health Care Management Review · doi:10.1097/hmr.0000000000000213 · Cited by 51
Team Ethical Cultures Within an Organization: A Differentiation Perspective on Their Existence and Relevance
Cabana, G. C., & Kaptein, M. (2019) · Journal of Business Ethics · doi:10.1007/s10551-019-04376-5 · Cited by 43
Linking high‐performance work systems and employee well‐being: A multilevel examination of the roles of organisation‐based self‐esteem and departmental formalisation
Wang, H., Zhang, Y., & Wan, M. (2021) · Human Resource Management Journal · doi:10.1111/1748-8583.12391 · Cited by 48
Entry into Emergent and Uncertain Product-Markets: The Role of Associative Rhetoric
Lee, G. K., & Paruchuri, S. (2008) · Academy of Management Journal · doi:10.5465/amj.2008.35732614 · Cited by 43
Perceived Academic Quality and Approaches to Studying in Higher Education: Evidence from Danish Students of Occupational Therapy
Richardson, J. T. E. (2010) · Scandinavian Journal of Educational Research · doi:10.1080/00313831003637972 · Cited by 42
Fields of Gold
Schepers, J., de Jong, A., de Ruyter, K., & Wetzels, M. G. M. (2011) · Journal of Service Research · doi:10.1177/1094670511412354 · Cited by 44
“With regard to the rwg(j), which indicates the homogeneity of individual ratings within teams, we found the lowest median value of .88 for supervisor encouragement of virtual team technology; thus, all variables showed high levels of within-group agreement (James, Demaree, and Wolf 1993).”
Background
Increasing HR's strategic participation: The effect of HR service quality and contribution expectations
Uen, J. F., Ahlström, D., Chen, S., & Tseng, P. (2012) · Human Resource Management · doi:10.1002/hrm.20467 · Cited by 42

Authors: James, L. R., Demaree, R. G., & Wolf, G.
Affiliations: University of Tennessee at Knoxville (US)
Venue: Journal of Applied Psychology
Publisher: American Psychological Association (APA)
Published: 1993-04
Volume: 78
Issue: 2
Pages: 306-309
ISSN: 0021-9010 (ISSN-L)
1939-1854
DOI: 10.1037/0021-9010.78.2.306
OpenAlex: W2022950201
Cited by: 2,221
References: 21
Citations by year: 2026: 54 · 2025: 88 · 2024: 76 · 2023: 85 · 2022: 88 · 2021: 97 · 2020: 95 · 2019: 98 · 2018: 108 · 2017: 82 · 2016: 86 · 2015: 100 · 2014: 111 · 2013: 110 · 2012: 96
FWCI: 12.64
Type: journal article
Language: en
Open access: closed
Locations: Journal of Applied Psychology (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
Keywords: inter-rater reliability, interobserver agreement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

rwg: an assessment of within-group interrater agreement

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks