Cited by 4,774Reliability and Agreement in MeasurementPsychometric Methodologies and TestingSurvey Methodology and Nonresponse

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.

Estimating within-group interrater reliability with and without response bias

James, Lawrence R., Demaree, Robert G., Wolf, Gerrit·1984·Journal of Applied Psychology

DOI 10.1037/0021-9010.69.1.85

Open this paper in CoChat — free account

Abstract

Abstract : This article presents methods for assessing agreement among the judgments made by a single group of judges on a single variable in regard to a single target. For example, the group of judges could be editorial consultants, members of an assessment center, or members of a team. The single target could be a manuscript, a lower-level manager, or a team. The variable on which the target is judged could be overall publishability in the case of the manuscript, managerial potential for the lower-level manager, or team cooperativeness for the team. The methods presented are based on new procedures for estimating interrater reliability. For situations such as the above, these procedures are shown to furnish more accurate and interpretable estimates of agreement than estimates provided by procedures commonly used to estimate agreement, consistency, or interrater reliability. In addition, the proposed methods include processes for controlling for the spurious influences of response biases (e.g., positive leniency, social desirability) on estimates of interrater reliability. (Author)

Integrity signals

Registry

Crossref · 10.1037/0021-9010.69.1.85

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1994) · Psychological Review · doi:10.1037/0033-295x.101.2.343 · Cited by 17,559
Reviews: Lord, Frederic M., and Novick, Melvin R. (with contributions by Allan Birnbaum). Statistical Theories of Mental Test Scores. Reading, Mass.: Addison-Wesley, 1968. xii + 568 pp. $14.95.
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Aggregation bias in estimates of perceptual agreement.
James, L. R. (1982) · Journal of Applied Psychology · doi:10.1037/0021-9010.67.2.219 · Cited by 2,139
Self-citation
Assessing similarity between profiles.
Cronbach, L. J., & Gleser, G. C. (1953) · Psychological Bulletin · doi:10.1037/h0057173 · Cited by 1,149
Category judgment: A range-frequency model.
Parducci, A. (1965) · Psychological Review · doi:10.1037/h0022602 · Cited by 1,132
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 830
Advances in factor analysis and structural equation models
Jöreskog, K. G., Sörbom, D., & Magidson, J. (1979) · Medical Entomology and Zoology · Cited by 826
Further Evidence on Response Sets and Test Design
Cronbach, L. J. (1950) · Educational and Psychological Measurement · doi:10.1177/001316445001000101 · Cited by 748

Show all 37 references

Sorted by citations

The job demands-resources model of burnout.
Demerouti, E., Bakker, A. B., Nachreiner, F., & Schaufeli, W. B. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.3.499 · Cited by 13,643
Ethical leadership: A social learning perspective for construct development and testing
Brown, M. E., Treviño, L. K., & Harrison, D. A. (2005) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2005.03.002 · Cited by 5,435
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608
What Makes Teams Work: Group Effectiveness Research from the Shop Floor to the Executive Suite
Cohen, S. G., & Bailey, D. E. (1997) · Journal of Management · doi:10.1177/014920639702300303 · Cited by 3,526
Differences Between Tight and Loose Cultures: A 33-Nation Study
Gelfand, M. J., Raver, J. L., Nishii, L., Leslie, L. M., Lun, J., Lim, B. C., Duan, L., Almaliach, A., Ang, S., Arnadottir, J., et al. (2011) · Science · doi:10.1126/science.1197754 · Cited by 3,252
Predicting unit performance by assessing transformational and transactional leadership.
Bass, B. M., Avolio, B. J., Jung, D. I., & Berson, Y. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.2.207 · Cited by 3,026
The Work Design Questionnaire (WDQ): Developing and validating a comprehensive measure for assessing job design and the nature of work.
Morgeson, F. P., & Humphrey, S. E. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.6.1321 · Cited by 2,994
Linking organizational resources and work engagement to employee performance and customer loyalty: the mediation of service climate.
Salanova, M., Agut, S., & Peiró, J. M. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.6.1217 · Cited by 2,276
rwg: an assessment of within-group interrater agreement
James, L. R., Demaree, R. G., & Wolf, G. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.2.306 · Cited by 2,221
Self-citation
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053
Transformational leadership, creativity, and organizational innovation
Gumusluoglu, L., & Ilsev, A. (2009) · Journal of Business Research · doi:10.1016/j.jbusres.2007.07.032 · Cited by 1,464
Levels issues in theory development, data collection and analysis
Klein, K. J., Dansereau, F., & Hall, R. J. (1994) · The Academy of Management Review · doi:10.2307/258703 · Cited by 1,607
EMPLOYEE ATTRIBUTIONS OF THE “WHY” OF HR PRACTICES: THEIR EFFECTS ON EMPLOYEE ATTITUDES AND BEHAVIORS, AND CUSTOMER SATISFACTION
NISHII, L. H., LEPAK, D. P., & SCHNEIDER, B. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00121.x · Cited by 1,410
Personality, affect, and behavior in groups
George, J. M. (1990) · Journal of Applied Psychology · doi:10.1037/0021-9010.75.2.107 · Cited by 1,372
Leader personality traits and employee voice behavior: Mediating roles of ethical leadership and work group psychological safety.
Walumbwa, F. O., & Schaubroeck, J. (2009) · Journal of Applied Psychology · doi:10.1037/a0015848 · Cited by 1,114
Understanding Conflict in Geographically Distributed Teams: The Moderating Effects of Shared Identity, Shared Context, and Spontaneous Communication
Hinds, P. J., & Mortensen, M. (2005) · Organization Science · doi:10.1287/orsc.1050.0122 · Cited by 1,072
Bias in peer review
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
Servant leadership, procedural justice climate, service climate, employee attitudes, and organizational citizenship behavior: A cross-level investigation
Walumbwa, F. O., Hartnell, C. A., & Oke, A. (2010) · Journal of Applied Psychology · doi:10.1037/a0018867 · Cited by 940
THE IMPACT OF TEAM EMPOWERMENT ON VIRTUAL TEAM PERFORMANCE: THE MODERATING ROLE OF FACE-TO-FACE INTERACTION.
Kirkman, B. L., Rosen, B., Tesluk, P. E., & Gibson, C. B. (2004) · Academy of Management Journal · doi:10.2307/20159571 · Cited by 889
When is educational specialization heterogeneity related to creativity in research and development teams? Transformational leadership as a moderator.
Shin, S. J., & Zhou, J. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.6.1709 · Cited by 845
Social and Cognitive Factors Driving Teamwork in Collaborative Learning Environments: Team Learning Beliefs and Behaviors
Van den Bossche, P., Gijselaers, W. H., Segers, M., & Kirschner, P. A. (2006) · Small Group Research · doi:10.1177/1046496406292938 · Cited by 792
Team Diversity and Information Use
Dahlin, K. B., Weingart, L. R., & Hinds, P. J. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.19573112 · Cited by 784
RESEARCH ON EMPLOYEE CREATIVITY: A CRITICAL REVIEW AND DIRECTIONS FOR FUTURE RESEARCH
Zhou, J., & Shalley, C. E. (2004) · Research in Personnel and Human Resources Management · doi:10.1016/s0742-7301(03)22004-1 · Cited by 770
The cross‐level effects of culture and climate in human service teams
Glisson, C., & James, L. R. (2002) · Journal of Organizational Behavior · doi:10.1002/job.162 · Cited by 758
Self-citation
The Collective Construction of Work Group Moods
Bartel, C. A., & Saavedra, R. (2000) · Administrative Science Quarterly · doi:10.2307/2667070 · Cited by 678
The impact of chief executive officer personality on top management team dynamics: One mechanism by which leadership affects organizational performance
Peterson, R. S., Smith, D. B., Martorana, P. V., & Owens, P. D. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.5.795 · Cited by 654
Group Size, ICC Values, and Group-Level Correlations: A Simulation
Bliese, P. D. (1998) · Organizational Research Methods · doi:10.1177/109442819814001 · Cited by 636
Dynamic nature of trust in virtual teams
Kanawattanachai, P., & Yoo, Y. (2002) · The Journal of Strategic Information Systems · doi:10.1016/s0963-8687(02)00019-7 · Cited by 594
Leading from within: the effects of emotion recognition and personality on transformational leadership behavior
Rubin, R. S., Munz, D. C., & Bommer, W. H. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.18803926 · Cited by 576
Servant leaders inspire servant followers: Antecedents and outcomes for employees and the organization
Hunter, E. M., Neubert, M. J., Perry, S. J., Witt, L. A., Penney, L. M., & Weinberger, E. (2013) · The Leadership Quarterly · doi:10.1016/j.leaqua.2012.12.001 · Cited by 573
The effects of transformational leadership on teacher attitudes and student performance in Singapore.
Koh, W. L., Steers, R. M., & Terborg, J. R. (1995) · Journal of Organizational Behavior · doi:10.1002/job.4030160404 · Cited by 562
Antecedents of team potency and team effectiveness: An examination of goal and process clarity and servant leadership
Hu, J., & Liden, R. C. (2011) · Journal of Applied Psychology · doi:10.1037/a0022465 · Cited by 563
The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership
Aarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014) · Implementation Science · doi:10.1186/1748-5908-9-45 · Cited by 559
Recognizing and Utilizing Expertise in Work Groups: A Status Characteristics Perspective
Bunderson, J. S. (2003) · Administrative Science Quarterly · doi:10.2307/3556637 · Cited by 527
A disagreement about within-group agreement: disentangling issues of consistency versus consensus
Kozlowski, S. W., & Hattrup, K. (1992) · Journal of Applied Psychology · doi:10.1037/0021-9010.77.2.161 · Cited by 514
Leadership and levels of analysis: A state-of-the-science review
Yammarino, F. J., Dionne, S. D., Uk Chun, J., & Dansereau, F. (2005) · The Leadership Quarterly · doi:10.1016/j.leaqua.2005.09.002 · Cited by 501
Abusive supervision and subordinates’ organization deviance
Tepper, B. J., Henle, C. A., Lambert, L. S., Giacalone, R. A., & Duffy, M. K. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.4.721 · Cited by 468
Knowledge management strategy and its link to knowledge creation process
Choi, B., & Lee, H. (2002) · Expert Systems with Applications · doi:10.1016/s0957-4174(02)00038-6 · Cited by 466
Does servant leadership foster creativity and innovation? A multi-level mediation study of identification and prototypicality
Yoshida, D. T., Sendjaya, S., Hirst, G., & Cooper, B. (2014) · Journal of Business Research · doi:10.1016/j.jbusres.2013.08.013 · Cited by 442
PROCEDURAL RATIONALITY IN THE STRATEGIC DECISION‐MAKING PROCESS*
Dean, J. W., & Sharfman, M. P. (1993) · Journal of Management Studies · doi:10.1111/j.1467-6486.1993.tb00317.x · Cited by 412
Leadership clarity and team innovation in health care
West, M. A., Borrill, C. S., Dawson, J. F., Brodbeck, F., Shapiro, D. A., & Haward, B. (2003) · The Leadership Quarterly · doi:10.1016/s1048-9843(03)00044-4 · Cited by 379
Boundary spanners’ identification, intergroup contact, and effective intergroup relations
Richter, A. W., West, M. A., van Dick, R., & Dawson, J. F. (2006) · Academy of Management Journal · doi:10.5465/amj.2006.23478720 · Cited by 376
Evidence of mood linkage in work groups
Totterdell, P., Kellett, S., Teuchmann, K., & Briner, R. B. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.6.1504 · Cited by 368
AN INVESTIGATION OF THE RELATIONSHIPS AMONG LEADER AND FOLLOWER PSYCHOLOGICAL CAPITAL, SERVICE CLIMATE, AND JOB PERFORMANCE
WALUMBWA, F. O., PETERSON, S. J., AVOLIO, B. J., & HARTNELL, C. A. (2010) · Personnel Psychology · doi:10.1111/j.1744-6570.2010.01193.x · Cited by 350
The Innovation Imperative: The Relationships Between Team Climate, Innovation, and Performance in Research and Development Teams
Bain, P. G., Mann, L., & Pirola-Merlo, A. (2001) · Small Group Research · doi:10.1177/104649640103200103 · Cited by 336
Psychometric properties and U.S. National norms of the Evidence-Based Practice Attitude Scale (EBPAS).
Aarons, G. A., Research Network on Youth Mental Health, Glisson, C., Hoagwood, K., Kelleher, K., Landsverk, J., & Cafri, G. (2010) · Psychological Assessment · doi:10.1037/a0019188 · Cited by 316
Inspired to perform: A multilevel investigation of antecedents and consequences of thriving at work
Walumbwa, F. O., Muchiri, M. K., Misati, E., Wu, C., & Meiliani, M. (2018) · Journal of Organizational Behavior · doi:10.1002/job.2216 · Cited by 323
Modeling reciprocal team cohesion–performance relationships, as impacted by shared leadership and members' competence
Mathieu, J. E., Kukenberger, M. R., D'Innocenzo, L., & Reilly, G. (2015) · Journal of Applied Psychology · doi:10.1037/a0038898 · Cited by 316
Are Two Narcissists Better Than One? The Link Between Narcissism, Perceived Creativity, and Creative Performance
Goncalo, J. A., Flynn, F. J., & Kim, S. H. (2010) · Personality and Social Psychology Bulletin · doi:10.1177/0146167210385109 · Cited by 312
The Effects of Cultural Diversity in Virtual Teams Versus Face-to-Face Teams
Staples, D. S., & Zhao, L. (2006) · Group Decision and Negotiation · doi:10.1007/s10726-006-9042-x · Cited by 291
Reflexivity in Teams: A Measure and Correlates
Schippers, M. C., Den Hartog, D. N., & Koopman, P. L. (2007) · Applied Psychology · doi:10.1111/j.1464-0597.2006.00250.x · Cited by 287
How leadership matters: The effects of leaders' alignment on strategy implementation
O'Reilly, C. A., Caldwell, D. F., Chatman, J. A., Lapiz, M., & Self, W. (2010) · The Leadership Quarterly · doi:10.1016/j.leaqua.2009.10.008 · Cited by 269
The meaning and measurement of implementation climate
Weiner, B. J., Belden, C. M., Bergmire, D. M., & Johnston, M. (2011) · Implementation Science · doi:10.1186/1748-5908-6-78 · Cited by 260
The impact of safety organizing, trusted leadership, and care pathways on reported medication errors in hospital nursing units
Vogus, T. J., & Sutcliffe, K. M. (2007) · Medical Care · doi:10.1097/mlr.0b013e318053674f · Cited by 259
Top management teams within multinational corporations: Effects of cultural heterogeneity
Elron, E. (1997) · The Leadership Quarterly · doi:10.1016/s1048-9843(97)90021-7 · Cited by 251
A multidimensional approach to the group cohesion-group performance relationship
Chang, A., & Bordia, P. (2001) · Small Group Research · doi:10.1177/104649640103200401 · Cited by 243
Scale indicators of social exchange relationships: A comparison of relative content validity.
Colquitt, J. A., Baer, M. D., Long, D. M., & Halvorsen-Ganepola, M. D. K. (2014) · Journal of Applied Psychology · doi:10.1037/a0036374 · Cited by 238
Abusive Supervision Climate: A Multiple-Mediation Model of its Impact on Group Outcomes
Priesemuth, M., Schminke, M., Ambrose, M. L., & Folger, R. (2014) · Academy of Management Journal · doi:10.5465/amj.2011.0237 · Cited by 230
Something(s) old and something(s) new: Modeling drivers of global virtual team effectiveness
Maynard, M. T., Mathieu, J. E., Rapp, T. L., & Gilson, L. L. (2012) · Journal of Organizational Behavior · doi:10.1002/job.1772 · Cited by 225
Research Note —A Model of Conflict, Leadership, and Performance in Virtual Teams
Wakefield, R. L., Leidner, D. E., & Garrison, G. (2008) · Information Systems Research · doi:10.1287/isre.1070.0149 · Cited by 220
The role of transformational leadership in enhancing team reflexivity
Schippers, M. C., Den Hartog, D. N., Koopman, P. L., & van Knippenberg, D. (2008) · Human Relations · doi:10.1177/0018726708096639 · Cited by 215
Gratitude versus Entitlement: A Dual Process Model of the Profitability Implications of Customer Prioritization
Wetzel, H. A., Hammerschmidt, M., & Zablah, A. R. (2014) · Journal of Marketing · doi:10.1509/jm.12.0167 · Cited by 216
Visionary communication qualities as mediators of the relationship between narcissism and attributions of leader charisma.
GALVIN, B. M., WALDMAN, D. A., & BALTHAZARD, P. (2010) · Personnel Psychology · doi:10.1111/j.1744-6570.2010.01179.x · Cited by 212
Transformational leadership as a mediator between emotional intelligence and team outcomes
Hur, Y., van den Berg, P. T., & Wilderom, C. P. M. (2011) · The Leadership Quarterly · doi:10.1016/j.leaqua.2011.05.002 · Cited by 209
Building Employee Commitment to Change Across Organizational Levels: The Influence of Hierarchical Distance and Direct Managers' Transformational Leadership
Hill, N. S., Seo, M.-G., Kang, J. H., & Taylor, M. S. (2012) · Organization Science · doi:10.1287/orsc.1110.0662 · Cited by 203
Careless responding in questionnaire measures: Detection, impact, and remedies
Goldammer, P., Annen, H., Stöckli, P. L., & Jonas, K. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101384 · Cited by 200
Does ethical leadership enhance group learning behavior? Examining the mediating influence of group ethical conduct, justice climate, and peer justice
Walumbwa, F. O., Hartnell, C. A., & Misati, E. (2017) · Journal of Business Research · doi:10.1016/j.jbusres.2016.11.013 · Cited by 179
Harming high performers: A social comparison perspective on interpersonal harming in work teams.
Lam, C. K., Van der Vegt, G. S., Walter, F., & Huang, X. (2011) · Journal of Applied Psychology · doi:10.1037/a0021882 · Cited by 176
Examining the influence of servant and entrepreneurial leadership on the work outcomes of employees in social enterprises
Newman, A., Neesham, C., Manville, G., & Tse, H. H. M. (2018) · The International Journal of Human Resource Management · doi:10.1080/09585192.2017.1359792 · Cited by 173
A multi-level study of servant leadership on creativity: The roles of self-efficacy and power distance
Yang, J., Liu, H., & Gu, J. (2017) · Leadership & Organization Development Journal · doi:10.1108/lodj-10-2015-0229 · Cited by 169
Empowering Leadership and Effective Collaboration in Geographically Dispersed Teams
Hill, N. S., & Bartol, K. M. (2016) · Personnel Psychology · doi:10.1111/peps.12108 · Cited by 170
The effect of servant leadership on customer value co-creation: A cross-level analysis of key mediating roles
Hsiao, C., Lee, Y.-H., & Chen, W.-J. (2015) · Tourism Management · doi:10.1016/j.tourman.2015.02.012 · Cited by 166
Motivating Employees to Explore Collaboration Technology in Team Contexts1
Maruping, L. M., & Magni, M. (2015) · MIS Quarterly · doi:10.25300/misq/2015/39.1.01 · Cited by 126
Empathic and nonempathic interaction in chronic pain couples.
Cano, A., Barterian, J. A., & Heller, J. B. (2008) · The Clinical Journal of Pain · doi:10.1097/ajp.0b013e31816753d8 · Cited by 125
How does product program innovativeness affect customer satisfaction? A comparison of goods and services
Stock, R. M. (2011) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-010-0215-4 · Cited by 122
Emotional Intelligence and Prosocial Behaviors in Adolescents
Charbonneau, D., & Nicol, A. A. M. (2002) · Psychological Reports · doi:10.2466/pr0.2002.90.2.361 · Cited by 118
Emergency Room Culture and the Emotional Support Component of Family-Centered Care
Hemmelgarn, A. L., Glisson, C., & Dukes, D. (2001) · Children's Health Care · doi:10.1207/s15326888chc3002_2 · Cited by 112
Developments in the Criterion–related Validation of Selection Procedures: A Critical Review and Recommendations for Practice
VAN IDDEKINGE, C. H., & PLOYHART, R. E. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00133.x · Cited by 101
The Influence of Project Commitment and Team Commitment on the Relationship between Trust and Knowledge Sharing in Project Teams
Buvik, M. P., & Tvedt, S. D. (2017) · Project Management Journal · doi:10.1177/875697281704800202 · Cited by 99
The Impact of Cultural Value Diversity on Multicultural Team Performance☆
KIRKMAN, B., & SHAPIRO, D. (2005) · Advances in International Management · doi:10.1016/s0747-7929(05)18002-0 · Cited by 94
The Organizational Readiness for Change scale in adolescent programs: Criterion validity
Saldana, L., Chapman, J. E., Henggeler, S. W., & Rowland, M. D. (2007) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2006.12.029 · Cited by 88
Work-Group Design and Autonomy: A Field Study of the Interaction Between Task Interdependence and Group Autonomy
Langfred, C. W. (2000) · Small Group Research · doi:10.1177/104649640003100103 · Cited by 82
Servant leadership influencing store-level profit: The mediating effect of employee flourishing
Giolito, V. J., Liden, R. C., van Dierendonck, D., & Cheung, G. W. (2021) · Journal of Business Ethics · doi:10.1007/s10551-020-04509-1 · Cited by 87
Younger supervisors, older subordinates: An organizational-level study of age differences, emotions, and performance
Kunze, F., & Menges, J. I. (2017) · Journal of Organizational Behavior · doi:10.1002/job.2129 · Cited by 77
Comparison between Inter-rater Reliability and Inter-rater Agreement in Performance Assessment
Liao, S. C., Hunt, E. A., & Chen, W. (2010) · Annals of the Academy of Medicine, Singapore · doi:10.47102/annals-acadmedsg.v39n8p613 · Cited by 65
Subjective Distance and Team Collaboration in Distributed Teams
Siebdrat, F., Hoegl, M., & Ernst, H. (2014) · Journal of Product Innovation Management · doi:10.1111/jpim.12122 · Cited by 61
Perceived entitlement to pain-related support and pain catastrophizing: associations with perceived and observed support.
Cano, A., Leong, L., Heller, J. B., & Lutz, J. R. (2009) · Pain · doi:10.1016/j.pain.2009.09.023 · Cited by 60
“Inter-rater agreement was assessed with the rwg(j) statistic, which is calculated from observed and expected variances across coders and items [16].”
BackgroundMethod
Safety organizing, emotional exhaustion, and turnover in hospital nursing units
Vogus, T. J., Cooil, B., Sitterding, M., & Everett, L. Q. (2014) · Medical Care · doi:10.1097/mlr.0000000000000169 · Cited by 50
Reducing perceived social loafing in virtual teams: The effect of team feedback with guided reflexivity
Peñarroja, V., Orengo, V., & Zornoza, A. (2017) · Journal of Applied Social Psychology · doi:10.1111/jasp.12449 · Cited by 47
The effect of service workers' proactive personality on their psychological withdrawal behaviors: a moderating effect of servant leadership
Song, C., & Lee, C. H. (2020) · Leadership & Organization Development Journal · doi:10.1108/lodj-04-2019-0149 · Cited by 43
Facilitating Bodily Insights Using Electromyography-Based Biofeedback during Physical Activity
Karolus, J., Bachmann, F., Kosch, T., Schmidt, A., & Woźniak, P. W. (2021) · Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction · doi:10.1145/3447526.3472027 · Cited by 25
CEO performance management behaviors’ influence on TMT flourishing, job attitudes, and firm performance
Christensen-Salem, A., Kinicki, A., Perrmann-Graham, J., & Walumbwa, F. O. (2023) · Human Relations · doi:10.1177/00187267221119767 · Cited by 16
Exploring the use of regular reflections in student collaboration: a case study in higher education
Schürmann, V., Bodemer, D., & Marquardt, N. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1526487 · Cited by 12
Improvised herding: Mapping biobehavioral mechanisms that underlie group efficacy during improvised social interaction
Greenberg, D. M., Milstein, N., Gilboa, A., Cohen, S., Haimovich, N., Siegman, S., Pinhasi, S., & Gordon, I. (2023) · Psychophysiology · doi:10.1111/psyp.14307 · Cited by 4

Authors: James, L. R., Demaree, R. G., & Wolf, G.
Affiliations: Georgia Institute of Technology (US)
Venue: Journal of Applied Psychology
Publisher: American Psychological Association (APA)
Published: 1984-02
Volume: 69
Issue: 1
Pages: 85-98
ISSN: 0021-9010 (ISSN-L)
1939-1854
DOI: 10.1037/0021-9010.69.1.85
OpenAlex: W1971711030
Cited by: 4,774
References: 37
Citations by year: 2026: 127 · 2025: 164 · 2024: 145 · 2023: 162 · 2022: 173 · 2021: 210 · 2020: 204 · 2019: 231 · 2018: 202 · 2017: 222 · 2016: 207 · 2015: 216 · 2014: 229 · 2013: 227 · 2012: 232
FWCI: 4.17
Type: journal article
Language: en
Open access: closed
Locations: Journal of Applied Psychology (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Keywords: inter-rater reliability, response bias, leniency bias, social desirability bias, judgment consistency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Estimating within-group interrater reliability with and without response bias

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks