Integration and generalization of kappas for multiple raters
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
J. A. Cohen's kappa (1960) for measuring agreement between 2 raters, using a nominal scale, has been extended for use with multiple raters by R. J. Light (1971) and J. L. Fleiss (1971). In the present article, these indices are analyzed and reformulated in terms of agreement statistics based on all
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“with interrater reliability measured using Cohen’s Kappa (κ = 0.63–0.82), indicating moderate to substantial agreement (Conger, 1980).”
From Rethinking pre-training: cognitive load implications for learners with varying prior knowledge (2025) · cites this paper as Conger, 1980
In the citing paper
“Furthermore, (25) provides an alternative proof that Conger's and Fleiss' quadratically weighted kappas converge to each other as the number of raters, 𝑅, tends to infinity (Conger, 1980;Gwet, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Conger, 1980
In the citing paper
“The average pairwise Pearson, Spearman, Kendall's τ b , ϕ, and MCC over the R 2 judge pairs are all equal, and coincide with κ F when all judges share the same marginal MET rate.”
From Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why (2026) · cites this paper as Conger (1980)
In the citing paper
“A generalization of Cohen’s kappa (i.e., Light’s Kappa [59]) was calculated to assess agreement between judges across their allocation of all items to domains.”
From Discriminant content validity of a theoretical domains framework questionnaire for use in implementation research (2014) · cites this paper as [59]
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Light, R. J. (1971) · Psychological Bulletin · doi:10.1037/h0031643 · Cited by 747
Cited by
Items 101–200 of 555
Weber, U., Zubler, V., Pedersen, S. J., Rufibach, K., Lambert, R. G. W., Chan, S. M., Østergaard, M., & Maksymowych, W. P. (2012) · Arthritis Care & Research · doi:10.1002/acr.21893 · Cited by 60
“Cohen’s kappa (12) was used for binary variables and Fleiss’ kappa for ordinal variables (13,14).”
MethodThe utility and accuracy of computed tomography in the diagnosis of occipitocervical dissociation
Gire, J. D., Roberto, R. F., Bobinski, M., Klineberg, E. O., & Durbin-Johnson, B. P. (2013) · The Spine Journal · doi:10.1016/j.spinee.2013.01.023 · Cited by 60
Klein, S., Quaas, A., Quantius, J., Löser, H., Meinel, J., Peifer, M., Wagner, S., Gattenlöhner, S., Wittekindt, C., Doeberitz, M. V. K., et al. (2021) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-20-3596 · Cited by 66
“Light Kappa was used to determine inter-rater reliability as described previously (25).”
MethodKappa Reliabilities for Continuous Behaviors and Events
Conger, A. J. (1985) · Educational and Psychological Measurement · doi:10.1177/0013164485454016 · Cited by 60
Self-citationUsing Crowdsourcing to Evaluate Published Scientific Literature: Methods and Example
Brown, A. W., & Allison, D. B. (2014) · PLoS ONE · doi:10.1371/journal.pone.0100647 · Cited by 54
“Interrater agreement was calculated by Conger’s extension of Cohen’s kappa [7] using the ‘exact’ option of the kappam.”
MethodKandel, P., Brand, E. C., Pelt, J., Ball, C. T., Chen, W., Bouras, E. P., Gómez, V., Raimondo, M., Woodward, T. A., & Wallace, M. B. (2019) · Gut · doi:10.1136/gutjnl-2018-316574 · Cited by 59
Reliability, fairness, objectivity and other inappropriate goals in peer review
Bailar, J. C. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065705 · Cited by 66
Iagnocco, A., Conaghan, P. G., Aegerter, P., Möller, I., Bruyn, G. A. W., Chary-Valckenaere, I., Filippucci, E., Gandjbakhch, F., Lœuille, D., Naredo, E., et al. (2012) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2012.07.003 · Cited by 54
Climate change sensitivity of threatened, and largely unprotected, Amazonian fishes
Frederico, R. G., Olden, J. D., & Zuanon, J. A. S. (2016) · Aquatic Conservation Marine and Freshwater Ecosystems · doi:10.1002/aqc.2658 · Cited by 56
“The level of inter-respondent agreement for species with multiple entries was quantified using Fleiss’ Kappa index (Conger, 1980).”
MethodImpact of different magnification levels on visual caries detection with ICDAS
Neuhaus, K. W., Jost, F., Perrin, P., & Lussi, A. (2015) · Journal of Dentistry · doi:10.1016/j.jdent.2015.09.002 · Cited by 59
Goals and Distractions: Explanations of Early Attrition from Traditional University Freshmen
Rausch, J. L., & Hamilton, M. (2015) · The Qualitative Report · doi:10.46743/2160-3715/2006.1676 · Cited by 52
Acceptability of behavioral staff management techniques
Davis, J. R., Rawana, E. P., & Capponi, D. R. (1989) · Behavioral Interventions · doi:10.1002/bin.2360040104 · Cited by 52
Mathematical Modelling Abilities of Artificial Intelligence Tools: The Case of ChatGPT
Spreitzer, C., Straser, O., Zehetmeier, S., & Maaß, K. (2024) · Education Sciences · doi:10.3390/educsci14070698 · Cited by 57
Prediction of Who Will Be the Next Speaker and When Using Gaze Behavior in Multiparty Meetings
Ishii, R., Otsuka, K., Kumano, S., & Yamato, J. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2757284 · Cited by 69
“Three annotators annotated the gaze behavior; Conger’s kappa coefficient κ [Conger 1980] as an intercoder agreement of the three annotators was .887.”
MethodViggen, E. M., Merciu, I.-A., Løvstakken, L., & Måsøy, S. (2020) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2020.107539 · Cited by 51
“We tried quantify this agreement by a segment-by-segment agreement metric based on Cohen ’ s weighted kappa extended to multiple raters (Cohen, 1968; Conger, 1980).”
MethodQuantitative dysmorphology assessment in Fabry disease
Ries, M., Moore, D. F., Robinson, C. J., Tifft, C. J., Rosenbaum, K. N., Brady, R. O., Schiffmann, R., & Krasnewich, D. M. (2006) · Genetics in Medicine · doi:10.1097/01.gim.0000200950.25118.dd · Cited by 51
Modeling ordinal scale disagreement.
Tanner, M. A., & Young, M. A. (1985) · Psychological Bulletin · doi:10.1037/0033-2909.98.2.408 · Cited by 51
Magit, D. P., Maak, T. G., Trioano, N., Raphael, B. S., Hamouria, Q., Polzhofer, G. K., Drespe, I., Albert, T. J., & Grauer, J. N. (2006) · Spine · doi:10.1097/01.brs.0000232823.82106.0a · Cited by 50
Dimitrov, D. M. (2002) · Educational and Psychological Measurement · doi:10.1177/001316402236878 · Cited by 57
“, Fan & Thompson, 2001), factorial design (Winer, 1971), Cohen’s kappa (Cohen, 1960), extended (g-wise) kappas (Conger, 1980), G theory coefficients (e.”
MethodCicchetti, D. V. (1984) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00026558 · Cited by 53
Collin, S. Y., Johansson, U., Svensson, K., & Ulvenblad, P. (1996) · British Journal of Management · doi:10.1111/j.1467-8551.1996.tb00175.x · Cited by 50
Many-Faceted Rasch Modeling Expert Judgment in Test Development
Zhu, W., Ennis, C. D., & Chen, A. (1998) · Measurement in Physical Education and Exercise Science · doi:10.1207/s15327841mpee0201_2 · Cited by 49
Gödert, H. W., Gamer, M., Rill, H., & Vossel, G. (2005) · Legal and Criminological Psychology · doi:10.1348/135532505x52680 · Cited by 58
“In order to obtain one index of the overall inter-rater agreement for each CBCA criterion, the three pairwise kappa coefficients were averaged (Conger, 1980).4 The mean weighted kappa coefficients are depicted in Table 1.”
MethodKappa statistic for clustered matched-pair data
Zhao, Y., & Zhou, M. (2014) · Statistics in Medicine · doi:10.1002/sim.6113 · Cited by 49
Inequalities Between Kappa and Kappa-Like Statistics for k × k Tables
Warrens, M. J. (2010) · Psychometrika · doi:10.1007/s11336-009-9138-8 · Cited by 49
“, multirater kappa (Fleiss, 1971; Conger, 1980), or fuzzy kappa (Dou, Ren, Wu, Ruan, Chen, Bloyet, & Constans, 2007). However, several authors have identified difficulties or paradoxes with κ’s interpretation (see, e.g., Brennan & Prediger, 1981, Feinstein & Cicchetti, 1990, or Byrt, Bishop & Carlin, 1993 and the refe…”
BackgroundInfluentialHahmann, S., Purves, R. S., & Burghardt, D. (2014) · Journal of Spatial Information Science · doi:10.5311/josis.2014.9.185 · Cited by 57
“generalization of Fleiss’ Kappa [22] for the case of multiple annotators proposed by Conger [14].”
BackgroundJACQUET, D., Boelaert, M., Seaman, J., Rijal, S., Sundar, S., Menten, J., & Magnus, E. (2006) · Tropical Medicine & International Health · doi:10.1111/j.1365-3156.2006.01743.x · Cited by 48
“In addition, the average agreement over the four batches was estimated using Light’s kappa (Conger 1980).”
MethodFactors affecting intercoder reliability: a Monte Carlo experiment
Feng, G. C. (2012) · Quality & Quantity · doi:10.1007/s11135-012-9745-9 · Cited by 51
The Emergency Severity Index Version 4: Reliability in Pediatric Patients
Durani, Y., Brecher, D., Walmsley, D., Attia, M. W., & LOISELLE, J. M. (2009) · Pediatric Emergency Care · doi:10.1097/pec.0b013e3181b0a0c6 · Cited by 47
“J Statistics were calculated using the program R (Vienna, Austria 2005)(16) with analysis based on Light_s J.(17) The 95% CI for the J statistics were calculated using the bootstrap procedure.”
MethodRetractedAutomated signal quality assessment of mobile phone-recorded heart sound signals
Springer, D., Brennan, T. J., Ntusi, N., Abdelrahman, H. Y., Zuhlke, L. J., Mayosi, B. M., Tarassenko, L., & Clifford, G. D. (2016) · Journal of Medical Engineering & Technology · doi:10.1080/03091902.2016.1213902 · Cited by 46
“The level of agreement between annotators was measured using Conger’s j [48] with the use of quadratic weights.”
MethodKrediet, A. C., Moayeri, N., Bleys, R. L. A. W., & Groen, G. J. (2014) · Regional Anesthesia & Pain Medicine · doi:10.1097/aap.0000000000000124 · Cited by 45
“Interrater reliability was assessed using the exact κ by Conger.18 Sensitivity and specificity and positive/ negative predictive values were calculated per observer, not including video clips that were rated “undetermined.””
MethodMüller, R., & Antoni, C. H. (2020) · Group Dynamics Theory Research and Practice · doi:10.1037/gdn0000130 · Cited by 55
WEIGHTED KAPPA FOR MULTIPLE RATERS
Berry, K. J., Johnston, J. E., & Mielke, P. W. (2008) · Perceptual and Motor Skills · doi:10.2466/pms.107.3.837-848 · Cited by 49
Rodby‐Bousquet, E., Ágústsson, A., Jónsdóttir, G., Czuba, T., Johansson, A., & Hägglund, G. (2012) · Clinical Rehabilitation · doi:10.1177/0269215512465423 · Cited by 53
A comprehensive guide to study the agreement and reliability of multi-observer ordinal data
Vanbelle, S., Engelhart, C. H., & Blix, E. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02431-y · Cited by 55
‘Mind Control’ Experiences on the Internet: Implications for the Psychiatric Diagnosis of Delusions
Bell, V., Maiden, C., Muñoz‐Solomando, A., & Reddy, V. (2006) · Psychopathology · doi:10.1159/000090598 · Cited by 44
“Web site content Inter-rater reliability was assessed by calculating Cohen’s kappa for each permutation of paired raters, with an overall kappa for each category calculated as a mean value, as recommended by Conger [27].”
MethodInfluentialTerslev, L., Naredo, E., Keen, H. I., Bruyn, G. A. W., Iagnocco, A., Wakefield, R. J., Conaghan, P. G., Maxwell, L. J., Beaton, D., Boers, M., et al. (2019) · The Journal of Rheumatology · doi:10.3899/jrheum.181158 · Cited by 44
Dichter, M., Schwab, C. G. G., Meyer, G., Bartholomeyczik, S., Dortmann, O., & Halek, M. (2014) · International Psychogeriatrics · doi:10.1017/s1041610214000052 · Cited by 47
“Each nurse was blinded to the ratings of the other raters.”
BackgroundMethodLind, C. M., Forsman, M., & Rose, L. M. (2019) · Ergonomics · doi:10.1080/00140139.2019.1710576 · Cited by 43
“The proportion of agreement (p0) (Portney and Watkins 2009) and linearly weighted kappa (jw) were computed for all possible pairs of observers and then averaged (Light 1971; Conger 1980).”
MethodAlbaroudi, B., Haddad, M., Albaroudi, O., Abdel‐Rahman, M. E., Jarman, R., & Harris, T. (2021) · European Journal of Emergency Medicine · doi:10.1097/mej.0000000000000866 · Cited by 43
Women, Menopause, and (Ms.)Information: Communication About the Climacteric
Buchanan, M. C., Villagran, M. M., & Ragan, S. L. (2002) · Health Communication · doi:10.1207/s15327027hc1401_5 · Cited by 42
“Coders used operational definitions of each theme and an example dialogic excerpt (Table 1) from each category to assess a multiple-rater kappa (Conger, 1980), which was .88 overall.”
MethodA design-independent method for measuring the reliability of psychiatric diagnosis
Uebersax, J. S. (1982) · Journal of Psychiatric Research · doi:10.1016/0022-3956(82)90039-5 · Cited by 53
Reliability from alpha to omega: a tutorialPreprint
Revelle, W., & Condon, D. M. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/2y3w9 · Cited by 45
Lai, C. S., Suter, V. G. A., Katsaros, C., & Bornstein, M. M. (2013) · European Journal of Orthodontics · doi:10.1093/ejo/cjt074 · Cited by 41
Berthomier, C., Muto, V., Schmidt, C., Vandewalle, G., Jaspar, M., Devillers, J., Gaggioni, G., Chellappa, S. L., Meyer, C., Phillips, C. L. M., et al. (2020) · Journal of Sleep Research · doi:10.1111/jsr.12994 · Cited by 44
“In the case of two raters, Conger's kappa and Cohen's kappa are equivalent (Conger, 1980).”
BackgroundGKAPPA: Generalized Kappa Coefficient
Uebersax, J. S. (1981) · Applied Psychological Measurement · doi:10.1177/014662168100500104 · Cited by 40
Müller, R., Schischke, D., Graf, B., & Antoni, C. H. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107438 · Cited by 44
Mental Health Professionals' Determinations of Adolescent Suicide Attempts
Wagner, B. M., Wong, S. A., & Jobes, D. A. (2002) · Suicide and Life-Threatening Behavior · doi:10.1521/suli.32.3.284.22178 · Cited by 34
The Equivalence of Multiple Rater Kappa Statistics and Intraclass Correlation Coefficients
Rae, G. (1988) · Educational and Psychological Measurement · doi:10.1177/0013164488482009 · Cited by 45
“Light and Margolin's (1971) concept of partioning variance for qualitative data, it is possible to establish correspondences between the multiple-rater kappa statistics of Fleiss (1971) and Conger (1980) and the intraclass correlations commonly used for assessing the reliability of quantitatively scaled data (e.g., Sh…”
MethodSimchen, E., Wax, Y., Pevsner, B., Erdal, M., Michel, J., Modan, M., Rubinstein, E., Sacks, T., Shapiro, M., & Tevaglioglo, B. (1988) · Infection Control and Hospital Epidemiology · doi:10.2307/30144175 · Cited by 39
A one stop shop? Perspectives on the value of adaptive learning technologies in K-12 education
Divanji, R. A., Bindman, S. W., Tung, A., Chen, K., Castaneda, L. M., & Scanlon, M. (2023) · Computers and Education Open · doi:10.1016/j.caeo.2023.100157 · Cited by 43
“Reliability was calculated at the parent code level using Light ’ s kappa to account for multiple coders [62], and 0.80 was used as the threshold for acceptable reliability [63].”
MethodInfluentialVerbeek, D. O., van der List, J. P., Moloney, G. B., Wellman, D. S., & Helfet, D. L. (2018) · Journal of Orthopaedic Trauma · doi:10.1097/bot.0000000000001161 · Cited by 38
Justification for the Use of Cohen’s Kappa Statistic in Experimental Studies of NLP and Text Mining
Kolesnyk, A., & Khairova, N. (2022) · Cybernetics and Systems Analysis · doi:10.1007/s10559-022-00460-3 · Cited by 38
“These metrics are either an extension of Cohen’s kappa measure, or use the kappa formula with subsequent introduction of another evaluation model [8], especially where the number of experts is more than three.”
MethodStatistical Methods for the Analysis of Inter-Reader Agreement Among Three or More Readers
Han, K., & Ryu, L. (2024) · Korean Journal of Radiology · doi:10.3348/kjr.2023.0965 · Cited by 38
“Fleiss’s kappa [4] and Conger’s kappa [5] are well-known alternatives to Cohen’s kappa for three or more readers.”
MethodResampling Probability Values for Weighted Kappa with Multiple Raters
Mielke, P. W., Berry, K. J., & Johnston, J. E. (2008) · Psychological Reports · doi:10.2466/pr0.102.2.606-613 · Cited by 37
The Interrater Reliability of the Autism Diagnostic Interview-Revised (ADI-R) in Clinical Settings
Zander, E., Willfors, C., Berggren, S., Coco, C., Holm, A., Jifält, I., Kosieradzki, R., Linder, J., Nordin, V., Olafsdottir, K., et al. (2017) · Psychopathology · doi:10.1159/000474949 · Cited by 42
Kappa coefficients for dichotomous-nominal classifications
Warrens, M. J. (2020) · Advances in Data Analysis and Classification · doi:10.1007/s11634-020-00394-8 · Cited by 38
“Kappa coefficients have been developed for multiple raters (Conger 1980; Warrens 2010), for hierarchical data (Vanbelle et al.”
BackgroundDevelopment and Evaluation of a Muscle Atrophy Scoring System (MASS) for Horses
Herbst, A. C., Johnson, M., Gammons, H., Reedy, S. E., Urschel, K. L., Harris, P. A., & Adams, A. A. (2021) · Journal of Equine Veterinary Science · doi:10.1016/j.jevs.2021.103771 · Cited by 39
Cohen’s linearly weighted kappa is a weighted average
Warrens, M. J. (2011) · Advances in Data Analysis and Classification · doi:10.1007/s11634-011-0094-7 · Cited by 41
“2004), including, multi-rater kappas (Conger 1980; Warrens 2010c), kappas for groups of raters (Vanbelle and Albert 2009a,b) and weighted kappas (Cohen 1968; Vanbelle and Albert 2009c; Warrens 2010d, 2011a,c).”
Does the need for agreement among reviewers inhibit the publication controversial findings?
Armstrong, J. S., & Hubbard, R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065699 · Cited by 41
Lee, J. J., & Tu, Z. N. (1994) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1994.10474647 · Cited by 39
A new objective histological scale for studying human photoaged skin
Kawabata, K., Kobayashi, M., Kusaka-Kikushima, A., Akasaka, E., Mabuchi, T., Fukui, T., Sugiyama, Y., Takekoshi, S., Miyasaka, M., Ozawa, A., et al. (2013) · Skin Research and Technology · doi:10.1111/srt.12099 · Cited by 36
“Light’s weighted Kappa (23) was computed using R software Ver.”
MethodInfluentialWeighted inter-rater agreement measures for ordinal outcomes
Tran, Q. D., Dolgun, A., & Demirhan, H. (2018) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610918.2018.1490428 · Cited by 40
“Many of these measures of are either extensions of Cohen’s kappa and weighted kappa or are formulated as a Cohen’s kappa-like statistic (Fleiss 1971; Light 1971; Conger 1980).”
MethodInter-review agreement of risk-of-bias judgments varied in Cochrane reviews
Könsgen, N., Barčot, O., Heß, S., Puljak, L., Gooßen, K., Rombey, T., & Pieper, D. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.12.016 · Cited by 37
A Comparison of Methods for the Evaluation of Binary Measurement Systems
van Wieringen, W. N., & van den Heuvel, E. R. (2005) · Quality Engineering · doi:10.1080/08982110500225562 · Cited by 36
“Note that we have adopted Conger ( 1980 ) here instead of Fleiss ( 1971 ).”
MethodMeasurement System Analysis for Categorical Measurements: Agreement and Kappa-Type Indices
de Mast, J., & van Wieringen, W. N. (2007) · Journal of Quality Technology · doi:10.1080/00224065.2007.11917688 · Cited by 36
“Both κUnif and κFleiss depend on the distribution (p(1), . . . , p(a)) of true values in the population.”
InfluentialSoubeyrand, M., Ciais, G., Wassermann, V., Kalouche, I., Biau, D. J., Dumontier, C., & Gagey, O. (2011) · Journal of Bone and Joint Surgery - British Volume · doi:10.1302/0301-620x.93b10.26590 · Cited by 36
A critical analysis of inter-coder reliability methods in information systems research
Nili, A., Tate, M., & Barros, A. (2017) · QUT ePrints (Queensland University of Technology) · Cited by 34
Delayed bone age due to a dual effect of FGFR3 mutation in Achondroplasia
Pannier, S., Dambroise, E., Jonquoy, A., Benoist-Lasselin, C., Odent, T., Jaïs, J., Münnich, A., & Legeai‐Mallet, L. (2010) · Bone · doi:10.1016/j.bone.2010.07.020 · Cited by 35
FAGLIONI, P., & Botti, C. (1993) · Cortex · doi:10.1016/s0010-9452(13)80256-0 · Cited by 34
Asymptotic variability of (multilevel) multirater kappa coefficients
Vanbelle, S. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218794733 · Cited by 33
“A total of 30 subjects were classified as suffering mainly of (1) depression, (2) personality disorder, (3) schizophrenia, (4) neurosis or (5) other psychiatric disorder.”
BackgroundMethodUsing the Delphi Method to Evaluate the Appropriateness of Urban Freight Transport Solutions
Karakikes, I. D., & Nathanail, E. (2020) · Smart Cities · doi:10.3390/smartcities3040068 · Cited by 35
Karimov, Z., Allahverdiyev, I., Yağız, Ö., Demir, D., & Almuradova, E. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08423-w · Cited by 35
“The interpretation of the coefficients was carried out by Gwet's probabilistic method according to the Landis and Koch scale [29].”
MethodRinto, E. E., & Cogbill-Seiders, E. (2014) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2014.11.010 · Cited by 33
“At the conclusion of the norming for each rubric, the author calculated the group’s interrater reliability using Fleiss’s kappa and Conger’s kappa, which modify Cohen’s kappa to adjust for agreement between more than two raters (Stemler, 2004; Oakleaf, 2009; Fleiss, 1971; Conger, 1980; Gwet, 2010).”
MethodCohen’s Linearly Weighted Kappa is a Weighted Average of 2×2 Kappas
Warrens, M. J. (2011) · Psychometrika · doi:10.1007/s11336-011-9210-z · Cited by 34
“…Cohen’s κ has led to the development of many extensions (Nelson & Pepe, 2000, p. 479; Kraemer, Periyakoil, & Noda, 2004), including multi-rater kappas (Conger, 1980; Warrens, 2010e), kappas for groups of raters (Vanbelle & Albert 2009a, 2009b), and weighted kappas (Cohen, 1968; Vanbelle & Albert,…”
BackgroundMethodInfluentialAnalyzing Interpersonal Empathy via Collective Impressions
Kumano, S., Otsuka, K., Mikami, D., Matsuda, M., & Yamato, J. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2417561 · Cited by 34
“By considering that a pair of participants are interacting only if at least one of them is looking at the other, the present study excludes averted gaze states when labeling targets.”
BackgroundAn Empirical Comparative Assessment of Inter-Rater Agreement of Binary Outcomes and Multiple Raters
Konstantinidis, M., Le, L. W., & Gao, X. (2022) · Symmetry · doi:10.3390/sym14020262 · Cited by 35
Koitka, S., Demircioğlu, A., Kim, M., Friedrich, C. M., & Nensa, F. (2018) · PLoS ONE · doi:10.1371/journal.pone.0207496 · Cited by 31
Schuster, C., & Smith, D. A. (2005) · Psychometrika · doi:10.1007/s11336-003-1110-4 · Cited by 30
“Conger (1980) then proposed considering all possible g-wise agreements among R raters, where 2 ≤ g ≤ R. In this spirit, let r (1) ...r (g) denote a g-tuple of raters selected without replacement and define the indicator variable”
BackgroundMethodHumphreys, K., & Janson, H. (2000) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3501_4 · Cited by 31
“a Conger’s (1980) “Fleiss exact” coefficient — a modification of a measure originally proposed by Fleiss (1971) — was used to account for agreement among the three coders.”
BackgroundMethodInfluentialGwet, K. L. (2014) · Wiley eBooks · doi:10.1002/9781118445112.stat06882 · Cited by 35
Psychometric evaluation of the Posture and Postural Ability Scale for children with cerebral palsy
Rodby‐Bousquet, E., Persson-Bunke, M., & Czuba, T. (2015) · Clinical Rehabilitation · doi:10.1177/0269215515593612 · Cited by 37
Dichter, M., Schwab, C. G. G., Meyer, G., Bartholomeyczik, S., & Halek, M. (2016) · BMC Geriatrics · doi:10.1186/s12877-016-0296-0 · Cited by 30
“, Conger’s kappa) [27].”
Solitary pulmonary nodule diagnosis on CT
Shah, S. K., McNitt‐Gray, M. F., Zoysa, K. R. D., Sayre, J. W., Kim, H. J., Batra, P., Behrashi, A., Brown, K., Greaser, L. E., Park, J. M., et al. (2005) · Academic Radiology · doi:10.1016/j.acra.2004.12.017 · Cited by 30
Powell, B. D., Cha, Y., Asirvatham, S. J., Cesario, D. A., Cao, M. K., Jones, P. W., Seth, M., Saxon, L. A., & Gilliam, F. R. (2011) · Pacing and Clinical Electrophysiology · doi:10.1111/j.1540-8159.2011.03093.x · Cited by 29
Inter-observer agreement and reliability assessment for observational studies of clinical work
Walter, S. R., Dunsmuir, W. T. M., & Westbrook, J. I. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103317 · Cited by 30
Examiner Agreement and Judicial Consensus in Forensic Mental Health Evaluations
Acklin, M. W., Fuger, K., & Gowensmith, W. N. (2015) · Journal of Forensic Psychology Practice · doi:10.1080/15228932.2015.1051447 · Cited by 31
“Other criticisms of kappa emerged in the 1970s, noting, for example, the limitations of kappa in multiple-rater situations (Conger, 1980).”
BackgroundPersson-Bunke, M., Czuba, T., Hägglund, G., & Rodby‐Bousquet, E. (2015) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-015-0801-1 · Cited by 28
“clinical spinal examination and the scoliometer measurement was calculated using weighted kappa scores [25].”
MethodOn The Krippendorff's Alpha Coefficient
Gwet, K. L. (2011) · Cited by 37
Lind, C. M., Forsman, M., & Rose, L. M. (2017) · International Journal of Occupational Safety and Ergonomics · doi:10.1080/10803548.2017.1364458 · Cited by 28
“, the multi-observer κw equals the arithmetic average of all, N × (N − 1) / 2, pairwise κ) [29,30].”
BackgroundRedMed: Extending drug lexicons for social media applications
Lavertu, A., & Altman, R. B. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103307 · Cited by 23
Effects of ICT Shared Mental Models on Team Processes and Outcomes
Müller, R., & Antoni, C. H. (2021) · Small Group Research · doi:10.1177/1046496421997889 · Cited by 30
“For each team, two agreement indices (Congers Kappa; Conger, 1980) were calculated to analyze the objective similarity among the individual ICT mental models after the instruction (T0) and after task execution (T1).”
MethodZeng, Z., Gao, Y., Li, J., Zhang, G., Sun, S., Wu, Q., Gong, Y. Q., & Xie, C. (2022) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2022.01.004 · Cited by 28
“Fleiss’ kappa was calculated by R ’irr’ package [35].”
Equivalences of weighted kappas for multiple raters
Warrens, M. J. (2011) · Statistical Methodology · doi:10.1016/j.stamet.2011.11.001 · Cited by 27
“Conger [11] has extended these multi-rater kappas to the case of g-agreement.”
BackgroundMethodInfluentialPathologic Diagnostics of HER2 Positivity in Gastroesophageal Adenocarcinoma
Koopman, T., Louwen, M., Hage, M., Smits, M. M., & Imholz, A. L. T. (2015) · American Journal of Clinical Pathology · doi:10.1309/ajcpcx69hgddgycq · Cited by 27
Measurement System Analysis for Bounded Ordinal Data
de Mast, J., & van Wieringen, W. N. (2004) · Quality and Reliability Engineering International · doi:10.1002/qre.653 · Cited by 27
“Instead, we retain R as the domain of the true value Z and define the map LRD : R → D, LRD(Z) = ⌈ a exp(Z) 1 + exp(Z) ⌉ (11)”
Avaliação das estruturas de concordância e discordância nos estudos de confiabilidade
da Silva, E. F., & Pereira, M. G. (1998) · Revista de Saúde Pública · doi:10.1590/s0034-89101998000400012 · Cited by 36
Robustness of κ‐type coefficients for clinical agreement
Vanacore, A., & Pellegrino, M. S. (2022) · Statistics in Medicine · doi:10.1002/sim.9341 · Cited by 30
“The weighted κ-type coefficient for ordinal classifications is thus formulated as: ?̂?w = paw − pa|cw 1 − pa|cw , (15)”
Interobserver relational agreement.
Stine, W. W. (1989) · Psychological Bulletin · doi:10.1037//0033-2909.106.2.341 · Cited by 26
Analysis and modeling of next speaking start timing based on gaze behavior in multi-party meetings
Ishii, R., Otsuka, K., Kumano, S., & Yamato, J. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853685 · Cited by 27
“Conger’s kappa coefficient κ [16] as an inter-coder agreement of three annotators is .”
Background
Metadata
- Authors
- Conger, A. J.
- Affiliations
- Purdue University West Lafayette (US)
- Corresponding author
- Conger, Anthony J.
- Venue
- Psychological Bulletin
- Published
- 1980-09
- Volume
- 88
- Issue
- 2
- Pages
- 322-328
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1978217317
- Cited by
- 545
- References
- 3
- Citations by year
- 2026: 20 · 2025: 24 · 2024: 35 · 2023: 24 · 2022: 23 · 2021: 23 · 2020: 25 · 2019: 25 · 2018: 22 · 2017: 16 · 2016: 18 · 2015: 27 · 2014: 29 · 2013: 15 · 2012: 15
- FWCI
- 7.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- Cohen's kappa, Fleiss' kappa, interobserver agreement, multiple raters, nominal scale
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.