The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
rwg: an assessment of within-group interrater agreement
Abstract
Schmidt and Hunter (1989) critiqued the within-group interrater reliability statistic (r wg ) described by James, Demaree, and Wolf (1984). Kozlowski and Hattrup (1992) responded to the Schmidt and Hunter critique and argued that r wg is a suitable index of interrater agreement. This article focuses on the interpretation of r wg as a measure of agreement among judges' ratings of a single target. A new derivation of r wg is given that underscores this interpretation
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The MPI has been shown to have good construct and discriminant validity, internal consistency, and test-retest reliability (8, 21).”
From Empathic and nonempathic interaction in chronic pain couples. (2008) · cites this paper as [21]
Citations
Cites
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,009
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
Self-citationThe Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Employee and customer perceptions of service in banks: Replication and extension.
Schneider, B., & Bowen, D. E. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.3.423 · Cited by 1,344
Hater, J. J., & Bass, B. M. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.4.695 · Cited by 1,136
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 883
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 830
Cited by
Sorted by citations
Items 601–700 of 1,001
Hon, A. H. Y., & Leung, A. S. M. (2011) · Cornell Hospitality Quarterly · doi:10.1177/1938965511403921 · Cited by 81
“…individual-level data to a firm-level construct, it was necessary to check the firm-level cultures by examining their inter-rater agreement (R wg ; James, Demaree, and Wolf 1993), between-group and within-group agreement (Bliese 2000; Hofmann, Griffin, and Gavin 2000), intraclass correlation…”
BackgroundServant leadership influencing store-level profit: The mediating effect of employee flourishing
Giolito, V. J., Liden, R. C., van Dierendonck, D., & Cheung, G. W. (2021) · Journal of Business Ethics · doi:10.1007/s10551-020-04509-1 · Cited by 87
“Additionally, within-group agreement measures, rwg (James et al. 1993) and awg (Brown and Hauenstein 2005) were found to be at or above the generally accepted 0.”
BackgroundKnowledge management practices and performance: are they truly linked?
Jayasingam, S., Ansari, M. A., Ramayah, T., & Jantan, M. (2012) · Knowledge Management Research & Practice · doi:10.1057/kmrp.2012.5 · Cited by 80
“Prior to aggregating data to reflect a higher level of analysis, a test of agreement must be conducted ( James, 1982; James et al, 1993).”
MethodFrom Self‐Efficacy to Collective Efficacy in Sport: Transitional Methodological Issues
Myers, N. D., & Feltz, D. L. (2007) · Wiley eBooks · doi:10.1002/9781118270011.ch36 · Cited by 79
Lin, Y., Liu, N.-C., & Lin, J.-W. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2021.11.028 · Cited by 99
Cunningham, G. B. (2011) · Sport Management Review · doi:10.1016/j.smr.2010.11.003 · Cited by 74
Team composition, cognition, and effectiveness: Examining mental model similarity and accuracy.
Resick, C. J., Dickson, M. W., Mitchelson, J. K., Allison, L. K., & Clark, M. A. (2010) · Group Dynamics Theory Research and Practice · doi:10.1037/a0018444 · Cited by 92
Williams, N. J., & Glisson, C. A. (2013) · Children and Youth Services Review · doi:10.1016/j.childyouth.2013.09.002 · Cited by 78
“rwg statistic (James et al., 1993), as well as evidence of significant between-system variance.”
BackgroundChoi, J. N. (2009) · Journal of Management Studies · doi:10.1111/j.1467-6486.2009.00851.x · Cited by 81
“Although rwg(j) values (James et al., 1993) have been widely used to compute within-group agreement, they have several limitations such as dependency on scale and sample size and the bias caused by the uniform null distribution assumption (Brown and Hauenstein, 2005).”
Human capital: the link between leadership and organizational learning
Pasamar, S., Díaz-Fernández, M., & de la Rosa-Navarro, M. D. (2019) · European Journal of Management and Business Economics · doi:10.1108/ejmbe-08-2017-0003 · Cited by 69
Braun, M. T., Kozlowski, S. W. J., Brown, T. A., & DeShon, R. P. (2020) · Small Group Research · doi:10.1177/1046496420907157 · Cited by 97
Team coaching and innovation in work teams
Rousseau, V., Aubé, C., & Tremblay, S. (2013) · Leadership & Organization Development Journal · doi:10.1108/lodj-08-2011-0073 · Cited by 91
Lee, L. T., & Sukoco, B. M. (2011) · R and D Management · doi:10.1111/j.1467-9310.2011.00645.x · Cited by 78
“If the expected agreement is not present (rwg40.70 is the suggested value to represent a ‘good’ amount of within-group inter-rater agreement; James et al., 1993), the variable can then be considered an individual unit-level variable (e.”
Creative leaders promote creative organizations
Mathisen, G. E., Einarsen, S. V., & Mykletun, R. J. (2012) · International Journal of Manpower · doi:10.1108/01437721211243741 · Cited by 78
Wegge, J., & Haslam, S. A. (2005) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320500349961 · Cited by 89
“For a detailed discussion about differences see Schmidt and Hunter ( 1989 ) as well as James, Demaree, and Wolf ( 1993 ).”
BackgroundSuárez, M. K. C., Déniz, M. D. L. C. D., & Martín‐Santana, J. D. (2014) · Family Business Review · doi:10.1177/0894486514526754 · Cited by 78
“Thus, we use the multiitem index suggested by James, Demaree, and Wolf (1993) to determine the level of within-group agreement for every item of the measurement scales.”
MethodYee, R. W. Y., Lee, P. K. C., Yeung, A. C. L., & Cheng, T. C. E. (2011) · International Journal of Production Economics · doi:10.1016/j.ijpe.2011.12.012 · Cited by 84
Tucker, S., Ogunfowora, B., & Ehr, D. (2016) · Journal of Applied Psychology · doi:10.1037/apl0000116 · Cited by 82
An expert consensus approach to relating the proposed DSM-5 types and traits.
Samuel, D. B., Lynam, D. R., Widiger, T. A., & Ball, S. A. (2011) · Personality Disorders Theory Research and Treatment · doi:10.1037/a0023787 · Cited by 77
Korek, S., Felfe, J., & Zaepernick-Rothe, U. (2009) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320902996336 · Cited by 77
“As a consequence, followers with meaningful work should reciprocate with extra effort, loyalty, and commitment (Kanter, 1983; Spreitzer, 1995; Wayne, Liden, & Sparrowe, 2000; Wiley, 1999). Meaningful work and knowledge about the significance of the task, as well as knowledge about one’s personal contribution to organi…”
BackgroundEnriching strategic variety in new ventures through external knowledge
Larrañeta, B., Zahra, S. A., & González, J. L. G. (2012) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2011.11.004 · Cited by 79
Organizational diversity, integration and performance
de Ven, A. H. V., Rogers, R. W., Bechara, J. P., & Sun, K. (2007) · Journal of Organizational Behavior · doi:10.1002/job.511 · Cited by 82
Heled, E., Somech, A., & Waters, L. E. (2015) · The Journal of Positive Psychology · doi:10.1080/17439760.2015.1058971 · Cited by 77
Coaching and employee organizational citizenship behaviours: The role of procedural justice climate
Özduran, A., & Tanova, C. (2016) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2016.10.004 · Cited by 88
Team diversity, mood, and team creativity: The role of team knowledge sharing in Chinese R & D teams
Tang, C., & Naumann, S. E. (2015) · Journal of Management & Organization · doi:10.1017/jmo.2015.43 · Cited by 76
Unbundling task conflict and relationship conflict
Huang, J. (2010) · International Journal of Conflict Management · doi:10.1108/10444061011063207 · Cited by 78
“were calculated using the procedure described by James et al. (1984, 1993)”
MethodMcComb, S. A., Green, S. G., & Compton, W. D. (2007) · Journal of Engineering and Technology Management · doi:10.1016/j.jengtecman.2007.09.004 · Cited by 76
LEADERSHIP AND SEX‐SIMILARITY: A COMPARISON IN A MILITARY SETTING
Vecchio, R. P., & Brazil, D. M. (2007) · Personnel Psychology · doi:10.1111/j.1744-6570.2007.00075.x · Cited by 80
Newman, D., & Sin, H.-P. (2009) · Organizational Research Methods · doi:10.1177/1094428106298969 · Cited by 94
Collective efficacy and vigilant problem solving in group decision making: A non-linear model
Tasa, K., & Whyte, G. (2005) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2005.01.002 · Cited by 80
Hon, A. H. Y., & Lü, L. (2016) · Cornell Hospitality Quarterly · doi:10.1177/1938965515624013 · Cited by 80
Nair, L. B., & Gibbert, M. (2016) · Scientometrics · doi:10.1007/s11192-016-1937-y · Cited by 76
“The rWG interrater agreement was 0.83 (James et al. 1993).”
InfluentialDrivers of innovation success in sustainable businesses
Fellnhofer, K. (2017) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2017.08.197 · Cited by 78
Li, T., & Tang, N. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.934831 · Cited by 139
“The Rwg, ICC1, and ICC2 values supported aggregating individual scores to the team level of analysis (inclusive leadership: ICC1 = 0.475, ICC2 = 0.782, Rwg = 0.972; psychological safety: ICC1 = 0.378, ICC2 = 0.706, Rwg= 0.964; James, 1982; James et al., 1993).”
InfluentialSlaughter, S. A., Ang, S., & Boh, W. F. (2007) · Human Resource Management · doi:10.1002/hrm.20169 · Cited by 75
Communication and performance in software development projects
Brodbeck, F. C. (2001) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320042000043 · Cited by 78
The social aspects of safety management: Trust and safety climate
Luria, G. (2010) · Accident Analysis & Prevention · doi:10.1016/j.aap.2010.02.006 · Cited by 75
Influencing knowledge workers: the power of top management
Jayasingam, S., Ansari, M. A., & Jantan, M. (2010) · Industrial Management & Data Systems · doi:10.1108/02635571011008443 · Cited by 75
“Aggregation of data are appropriate if within group inter-rater agreement on the theorized group level can be demonstrated (George and James, 1993; James et al., 1993).”
BackgroundFormality and discretion in successful R&D projects
Naveh, E. (2006) · Journal of Operations Management · doi:10.1016/j.jom.2006.02.004 · Cited by 75
Uneasy lies the head that wears the crown: The link between guilt proneness and leadership.
Schaumberg, R. L., & Flynn, F. J. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0028127 · Cited by 75
“The median rwg rating was .72, suggesting a sufficient level of agreement among raters to justify aggregating across raters’ responses (James et al., 1984, 1993; LeBreton & Senter, 2008).”
BackgroundKoopman, J., Matta, F. K., Scott, B. A., & Conlon, D. E. (2015) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2015.09.001 · Cited by 79
Rodríguez-Sánchez, A. M., Devloo, T., Rico, R., Salanova, M., & Anseel, F. (2016) · Group & Organization Management · doi:10.1177/1059601116636476 · Cited by 89
“…to statistically justify the aggregation of the team members’ survey responses to the team level (i.e., team cohesion, collective task engagement, and perceived team performance), we calculated ICCs (i.e., ICC1 and ICC2) and also within-group interrater agreement (i.e., Rwg; James et al., 1993).”
MethodResultDai, Y., Roundy, P. T., Chok, J. I., Ding, F., & Byun, G. (2016) · Journal of Management Studies · doi:10.1111/joms.12211 · Cited by 76
Psychological and neurological predictors of abusive supervision
Waldman, D. A., Wang, D., Hannah, S. T., Owens, B. P., & Balthazard, P. A. (2018) · Personnel Psychology · doi:10.1111/peps.12262 · Cited by 84
Flood preparedness: thoughts, feelings and intentions of the Dutch public
Terpstra, T. (2009) · doi:10.3990/1.9789036529549 · Cited by 79
Knowledge Integration and Team Effectiveness: A Team Goal Orientation Approach
Mehta, A., & Mehta, N. (2017) · Decision Sciences · doi:10.1111/deci.12280 · Cited by 86
Leader humility and team innovation: The role of team reflexivity and team proactive personality
Leblanc, P., Rousseau, V., & Harvey, J. (2022) · Journal of Organizational Behavior · doi:10.1002/job.2648 · Cited by 79
Transformational Leadership and Leaders’ Mode of Care Reasoning
Simola, S. K., Barling, J. I., & Turner, N. (2011) · Journal of Business Ethics · doi:10.1007/s10551-011-1080-x · Cited by 74
Arthur, W., Bell, S. T., & Edwards, B. (2007) · Organizational Research Methods · doi:10.1177/1094428106287574 · Cited by 74
“Although there are several operationalizations of construct agreement in the literature (e.g., rwg [James et al., 1993] and standard deviation [Klein, Conn, Smith, & Sorra, 2001; Schneider, Salvaggio, & Subirats, 2002]), we chose rwg to assess the agreement of teammates’ ratings because of its…”
MethodHow to (and how not to) assess the integrity of managers.
Kaiser, R. B., & Hogan, R. T. (2010) · Consulting psychology journal · doi:10.1037/a0022265 · Cited by 74
The Contextual Antecedents of Organizational Trust: A Multidimensional Cross-level Analysis
Li, P. P., Bai, Y., & Xi, Y. (2011) · Management and Organization Review · doi:10.1111/j.1740-8784.2011.00219.x · Cited by 72
Converting service encounters into cross-selling opportunities
Yu, T., Patterson, P. G., & de Ruyter, K. (2015) · European Journal of Marketing · doi:10.1108/ejm-10-2013-0549 · Cited by 72
The effect of integrated product development on efficiency and innovation
Naveh, E. (2005) · International Journal of Production Research · doi:10.1080/00207540500031873 · Cited by 72
“Individual responses to the questionnaires were aggregated to the project level (James et al. 1984), by calculating for each project the mean scores of each variable.”
MethodQin, Q., Wen, B., Ling, Q., Zhou, S., & Tong, M. (2014) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-02-2013-0073 · Cited by 77
Self‐managed learning groups in higher education: Students' perceptions of process and outcomes
Lizzio, A., & Wilson, K. L. (2005) · British Journal of Educational Psychology · doi:10.1348/000709905x25355 · Cited by 75
Andrews, W., Gregg, A. P., & Lammers, J. (2014) · British Journal of Social Psychology · doi:10.1111/bjso.12093 · Cited by 82
The Relationship Between Tenure and Outside Director Task Involvement: A Social Identity Perspective
Veltrop, D. B., Molleman, E., Hooghiemstra, R. B. H., & van Ees, H. (2015) · Journal of Management · doi:10.1177/0149206315579510 · Cited by 73
Schwarz, G., Newman, A., Yu, J., & Michaels, V. (2021) · The International Journal of Human Resource Management · doi:10.1080/09585192.2021.1962388 · Cited by 83
“Additionally, we calculated the interrater agreement coefficient (i.e. rwg; James et al., 1993) to analyze whether team members’ ratings of the employee involvement climate were consistent within teams.”
MethodLarrañeta, B., Zahra, S. A., & González, J. L. G. (2013) · Strategic Management Journal · doi:10.1002/smj.2103 · Cited by 72
“We calculated an inter-rater agreement score (rwg) resulting in median inter-rater agreement scores ranging from 0.84 to 0.90, indicating that single informant bias was not a problem (James, Demaree, and Wolf, 1993).”
MethodInfluentialWang, H., Lu, G., & Liu, Y. (2015) · Journal of Business Ethics · doi:10.1007/s10551-015-2916-6 · Cited by 77
“To justify aggregation, we calculated interrater agreement (James et al. 1993) and two intraclass correlations, ICC(1) and ICC(2), respectively (James 1982; Shrout and Fleiss 1979).”
MethodA Configural Approach to Coordinating Expertise in Software Development Teams1
Kudaravalli, S., Faraj, S. A., & Johnson, S. L. (2017) · MIS Quarterly · doi:10.25300/misq/2017/41.1.03 · Cited by 72
Tate, B. W. (2008) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051808318002 · Cited by 71
External Web content and its influence on organizational performance
Meroño‐Cerdán, A. L., & Soto‐Acosta, P. (2007) · European Journal of Information Systems · doi:10.1057/palgrave.ejis.3000656 · Cited by 71
“In contrast, interrater agreement represents inter-changeability among raters, that is the extent to which raters make the same ratings (James et al ., 1993).”
BackgroundExploring the performance effect of HPWS on professional service supply chain management
Fu, N., Flood, P. C., Bosak, J., Morris, T., & O’Regan, P. (2013) · Supply Chain Management An International Journal · doi:10.1108/scm-04-2012-0118 · Cited by 71
Men, Women, and Leadership Centralization in Groups Over Time.
Berdahl, J. L., & Anderson, C. P. (2005) · Group Dynamics Theory Research and Practice · doi:10.1037/1089-2699.9.1.45 · Cited by 71
“Within-group interrater agreement on leader behavior, rwg ( James, Demaree, & Wolf, 1993 ), ranged from .75 to .79 in the 3 weeks of the study, indicating that group members agreed about how much each member led the group’s activities each week.”
BackgroundBalluerka, N., Gorostiaga, A., Alonso‐Arbiol, I., & Aritzeta, A. (2016) · Journal of Adolescence · doi:10.1016/j.adolescence.2016.08.009 · Cited by 71
Curșeu, P. L., Chappin, M. M. H., & Jansen, R. J. G. (2017) · Social Psychology of Education · doi:10.1007/s11218-017-9419-5 · Cited by 71
“The Cronbach’s alpha for the scale was .777 and the within group agreement index (James et al. 1993) ranges from .86 to 1.00 (M = .96, SD = .02) showing substantial within agreement to support the group level aggregation.”
ResultInfluentialEmpirical Validation of the Team Work Engagement Construct
Costa, P., Passos, A., & Bakker, A. B. (2014) · Journal of Personnel Psychology · doi:10.1027/1866-5888/a000102 · Cited by 93
Zimmermann, A., Hill, S. A., Birkinshaw, J., & Jäckel, M. (2019) · Long Range Planning · doi:10.1016/j.lrp.2019.101927 · Cited by 71
The Relationship between Tour Guide Humor and Tourists’ Behavior Intention: A Cross-Level Analysis
Tu, H., Guo, W., Xiao, X., & Yan, M. N. (2019) · Journal of Travel Research · doi:10.1177/0047287519883033 · Cited by 70
“Specifically, drawing on ICC(1), ICC(2), and R wg (Bliese 2000; James, Demaree, and Wolf 1993), we estimated the possibility of aggregating tour guide humor as a group-level variable.”
BackgroundAsghar, M., Gull, N., Xiong, Z., Shu, A., Faraz, N. A., & Pervaiz, K. (2023) · Journal of Hospitality and Tourism Management · doi:10.1016/j.jhtm.2023.07.007 · Cited by 77
Lüdtke, O., Trautwein, U., Kunter, M., & Baumert, J. (2006) · Zeitschrift für Pädagogische Psychologie · doi:10.1024/1010-0652.20.12.85 · Cited by 70
Baumann, B., Kolko, D. J., Collins, K. S., & Herschell, A. D. (2006) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2006.01.002 · Cited by 75
The impact of state affect on job satisfaction
Niklas, C. D., & Dormann, C. (2005) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320500348880 · Cited by 70
“This is further confirmed by the agreement index r wg(J) (within-group interrater reliability), which assesses the absolute similarity of responses within persons (James, Demaree, & Wolf, 1984 ).”
BackgroundAugustsson, H., Richter, A., Hasson, H., & Schwarz, U. V. T. (2017) · The Journal of Applied Behavioral Science · doi:10.1177/0021886317691930 · Cited by 70
Methodological and Substantive Issues in Conducting Multinational and Cross-Cultural Research
Spector, P. E., Liu, C., & Sanchez, J. I. (2015) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-032414-111310 · Cited by 82
Kistruck, G. M., Lount, R. B., Smith, B. R., Bergman, B. J., & Moss, T. W. (2015) · Academy of Management Journal · doi:10.5465/amj.2014.0201 · Cited by 76
Lee, Y., Masuda, A. D., Fu, X., & Reiche, B. S. (2017) · Academy of Management Discoveries · doi:10.5465/amd.2016.0063 · Cited by 76
Drivers of Team Performance: What Do We Know and What Have We Still To Learn?
Stock, R. (2004) · Schmalenbach Business Review · doi:10.1007/bf03396696 · Cited by 71
Zheng, X., Yin, H., & Liu, Y. (2020) · School Effectiveness and School Improvement · doi:10.1080/09243453.2020.1808484 · Cited by 68
“In addition, to justify the aggregation of individual teacher perceptions of PLC into a school-level variable, interrater agreement (rwg[j]) was calculated (James et al., 1993; LeBreton & Senter, 2008).”
MethodLin, Y., Yang, M., Quade, M. J., & Chen, W. (2021) · Human Relations · doi:10.1177/00187267211002917 · Cited by 69
“The within-team agreement index (rwg; James et al., 1993) and intra-class correlation coefficients (ICC[1] & ICC[2]; Bliese, 2000) were calculated to confirm the reliability and validity of the aggregations.”
MethodWould you trust an AI team member? Team trust in human– AI teams
Georganta, E., & Ulfert, A. (2024) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12504 · Cited by 86
Active learning: When is more better? The case of resident physicians’ medical errors.
Katz‐Navon, T., Naveh, E., & Stern, Z. (2009) · Journal of Applied Psychology · doi:10.1037/a0015979 · Cited by 72
Abrantes, A. C. M., Passos, A., Cunha, M. P. E., & Santos, C. M. (2017) · Journal of Business Research · doi:10.1016/j.jbusres.2017.11.005 · Cited by 75
Management by objectives – an effective tool for teamwork?
Antoni, C. H. (2004) · The International Journal of Human Resource Management · doi:10.1080/0958519042000311381 · Cited by 68
Classroom emotional intelligence and its relationship with school performance
Aritzeta, A., Balluerka, N., Gorostiaga, A., Alonso‐Arbiol, I., Oiharbide, M. H., & Gartzia, L. (2015) · European Journal of Education and Psychology · doi:10.1016/j.ejeps.2015.11.001 · Cited by 83
Hebles, M., Trincado-Munoz, F. J., & Ortega, K. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.758438 · Cited by 93
Mañas, M. A., Díaz-Fúnez, P. A., Pecino-Medina, V., López‐Liria, R., Padilla, D., & Aguilar-Parra, J. M. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.02292 · Cited by 79
“From a consensus-based approach, the Average Deviation Index [ADM(J)] (Burke et al., 1999) and Rwg(J) (James et al., 1993) have been used, and it was concluded that there is a consensus in the unit when the ADM(J) was equal to or less than 1 (Burke et al., 1999).”
MethodBrown, T. C., & Latham, G. P. (2002) · Canadian Journal of Behavioural Science/Revue canadienne des sciences du comportement · doi:10.1037/h0087180 · Cited by 67
Knowledge management in teams: empirical integration and development of a scale
Singh, R. M., & Gupta, M. (2014) · Journal of Knowledge Management · doi:10.1108/jkm-11-2013-0450 · Cited by 66
“rwg is a commonly used measure of a within-group agreement, on a scale (James et al., 1993; LeBreton and Senter, 2008).”
BackgroundWisse, B. M., Rus, D., Keller, A. C., & Sleebos, E. (2019) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2019.1635584 · Cited by 77
Hallinger, P., & Lu, J. (2014) · School Leadership and Management · doi:10.1080/13632434.2014.938039 · Cited by 79
“Both were measured by a multi-item scale, so we calculated rwg(j) for each scale (James, Demaree, and Wolf 1993).”
BackgroundMethodInfluentialModeling consensus emergence in groups using longitudinal multilevel methods
Lang, J. W. B., Bliese, P. D., & de Voogt, A. J. (2017) · Personnel Psychology · doi:10.1111/peps.12260 · Cited by 72
Huang, Y.-H., Zohar, D. M., Robertson, M. M., Garabet, A., Murphy, L. A., & Lee, J. (2013) · Accident Analysis & Prevention · doi:10.1016/j.aap.2013.04.030 · Cited by 65
Winkler, T. J., & Wulf, J. (2019) · Journal of Management Information Systems · doi:10.1080/07421222.2019.1599513 · Cited by 82
Comparison between Inter-rater Reliability and Inter-rater Agreement in Performance Assessment
Liao, S. C., Hunt, E. A., & Chen, W. (2010) · Annals of the Academy of Medicine, Singapore · doi:10.47102/annals-acadmedsg.v39n8p613 · Cited by 65
Rafferty, A. E., & Jimmieson, N. L. (2009) · European Journal of Work and Organizational Psychology · doi:10.1080/13594320903007869 · Cited by 66
“This model uses within group (James, Demaree, & Wolf, 1993 ) to assess consensus at the lower level and to justify aggregation to the higher level.”
MethodSteyrer, J., Schiffinger, M., Huber, C., Valentin, A., & Strunk, G. (2012) · Health Care Management Review · doi:10.1097/hmr.0b013e318272935a · Cited by 70
“Although safety climate was ascertained individually, it represents a construct on group and/or organizational level (here, ICU); the same ultimately applies to error rate (Neal & Griffin, 2006), with this study not focusing on the probability of an individual patient encountering a medical error…”
ResultSomech, A., & Khalaili, A. (2014) · Group & Organization Management · doi:10.1177/1059601114525437 · Cited by 67
Barrick, M. R., Dustin, S., Giluk, T. L., Stewart, G. L., Shaffer, J. A., & Swider, B. W. (2011) · Journal of Occupational and Organizational Psychology · doi:10.1111/j.2044-8325.2011.02036.x · Cited by 74
“Both rwg and AD support a high level of agreement (rwg = .88 and AD = .33 on four-point scale, recommended values are rwg > .70 and AD < .67 for four-point scale; Burke & Dunlap, 2002, James et al., 1984, 1993).”
Background
Metadata
- Authors
- James, L. R., Demaree, R. G., & Wolf, G.
- Affiliations
- University of Tennessee at Knoxville (US)
- Venue
- Journal of Applied Psychology
- Published
- 1993-04
- Volume
- 78
- Issue
- 2
- Pages
- 306-309
- ISSN
- 0021-9010 (ISSN-L)
- 1939-1854
- OpenAlex
- W2022950201
- Cited by
- 2,221
- References
- 21
- Citations by year
- 2026: 54 · 2025: 88 · 2024: 76 · 2023: 85 · 2022: 88 · 2021: 97 · 2020: 95 · 2019: 98 · 2018: 108 · 2017: 82 · 2016: 86 · 2015: 100 · 2014: 111 · 2013: 110 · 2012: 96
- FWCI
- 12.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Applied Psychology (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
- Keywords
- inter-rater reliability, interobserver agreement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
