Integration and generalization of kappas for multiple raters
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
J. A. Cohen's kappa (1960) for measuring agreement between 2 raters, using a nominal scale, has been extended for use with multiple raters by R. J. Light (1971) and J. L. Fleiss (1971). In the present article, these indices are analyzed and reformulated in terms of agreement statistics based on all
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“with interrater reliability measured using Cohen’s Kappa (κ = 0.63–0.82), indicating moderate to substantial agreement (Conger, 1980).”
From Rethinking pre-training: cognitive load implications for learners with varying prior knowledge (2025) · cites this paper as Conger, 1980
In the citing paper
“Furthermore, (25) provides an alternative proof that Conger's and Fleiss' quadratically weighted kappas converge to each other as the number of raters, 𝑅, tends to infinity (Conger, 1980;Gwet, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Conger, 1980
In the citing paper
“The average pairwise Pearson, Spearman, Kendall's τ b , ϕ, and MCC over the R 2 judge pairs are all equal, and coincide with κ F when all judges share the same marginal MET rate.”
From Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why (2026) · cites this paper as Conger (1980)
In the citing paper
“A generalization of Cohen’s kappa (i.e., Light’s Kappa [59]) was calculated to assess agreement between judges across their allocation of all items to domains.”
From Discriminant content validity of a theoretical domains framework questionnaire for use in implementation research (2014) · cites this paper as [59]
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Light, R. J. (1971) · Psychological Bulletin · doi:10.1037/h0031643 · Cited by 747
Cited by
Lord, C., Rutter, M., & Le Couteur, A. (1994) · Journal of Autism and Developmental Disorders · doi:10.1007/bf02172145 · Cited by 9,065
“Multirater weighted kappas ( Conger, 1980 ), reported in the following tables were computed across raters, as was mean percentage agreement.”
MethodThe kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167
Lord, C., Rutter, M., Goode, S., Heemsbergen, J., Jordan, H., Mawhood, L., & Schopler, E. (1989) · Journal of Autism and Developmental Disorders · doi:10.1007/bf02211841 · Cited by 2,030
“Although the number of subjects rated by each pair of raters was too small to allow appropriate evaluation of each separate kappa statistic, the above procedure allowed us to check whether the summary weighted kappa accurately reflected the scores of most of the individual pairs, or was the average of very different d…”
ResultPayne, D. L., Lonsway, K. A., & Fitzgerald, L. F. (1999) · Journal of Research in Personality · doi:10.1006/jrpe.1998.2238 · Cited by 1,114
“Light’s (1971) κ for multiple raters, which equals the average of all pairwise Cohen’s (1960) κs (Conger, 1980), was then calculated for each of the six component ratings from the seven best judges; these multiple κs ranged from .27 to .51, with a mean of .37.”
MethodAutism diagnostic interview: A standardized investigator-based instrument
Le Couteur, A., Rutter, M., Lord, C., Rios, P., Robertson, S., Holdgrafer, M., & McLennan, J. (1989) · Journal of Autism and Developmental Disorders · doi:10.1007/bf02212936 · Cited by 1,036
Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research
Hruschka, D. J., Schwartz, D., St.John, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004) · Field Methods · doi:10.1177/1525822x04266540 · Cited by 829
“Extensions of Cohen’s kappa exist that allows for multiple coders, and further research should examine how the number of coders may effect the coding process (Fleiss 1971; Fleiss, Nee, and Landis 1979; Conger 1980; Fleiss 1981:225–34).”
BackgroundBien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002699 · Cited by 734
The Strategic Management of Innovation: A Systematic Review and Paths for Future Research
Keupp, M. M., Palmié, M., & Gassmann, O. (2012) · International Journal of Management Reviews · doi:10.1111/j.1468-2370.2011.00321.x · Cited by 551
“Average Cohen’s kappas of 0.74 and 0.85 suggested strong interrater agreement (Conger 1980).”
InfluentialSaxon, L. A., Hayes, D. L., Gilliam, F. R., Heidenreich, P. A., Day, J., Seth, M., Meyer, T. E., Jones, P. W., & Boehmer, J. P. (2010) · Circulation · doi:10.1161/circulationaha.110.960633 · Cited by 515
Reliability from α to ω: A tutorial.
Revelle, W., & Condon, D. M. (2019) · Psychological Assessment · doi:10.1037/pas0000754 · Cited by 506
“Rediscovered multiple times and given different names (Conger, 1980; Scott, 1955; Hubert, 1977; Zapf, Castell, Morawietz, & Karch, 2016) perhaps the most standard coefficient is known as Cohen’s Kappa (Cohen, 1960, 1968) which adjusts observed proportions of agreement by the expected proportion:”
BackgroundCicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496
Rüschoff, J., Dietel, M., Baretton, G., Arbogast, S., Walch, A., Monges, G., Chenard, M.-P., Penault-Llorca, F., Nagelmeier, I., Schlake, W., et al. (2010) · Virchows Archiv · doi:10.1007/s00428-010-0952-2 · Cited by 486
“Kappa statistics was calculated according to the method of Conger (1980) by package “irr” of program R [33].”
MethodHandbook of multimethod measurement in psychology.
(2006) · American Psychological Association eBooks · doi:10.1037/11383-000 · Cited by 481
Reproducibility of histomorphologic diagnoses with special reference to the kappa statistic
SVANHOLM, H., STARKLINT, H., GUNDERSEN, H. J. G., FABRICIUS, J., BARLEBO, H., & OLSEN, S. (1989) · APMIS · doi:10.1111/j.1699-0463.1989.tb00464.x · Cited by 453
A Broader Phenotype of Autism: The Clinical Spectrum in Twins
Le Couteur, A., Bailey, A., Goode, S., Pickles, A., Gottesman, I., Robertson, S., & Rutter, M. (1996) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.1996.tb01475.x · Cited by 442
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410
Examining the Dimensions of Cancer-Related Information Seeking and Scanning Behavior
Niederdeppe, J., Hornik, R. C., Kelly, B. J., Frosch, D. L., Romantan, A., Stevens, R. S., Barg, F. K., Weiner, J. L., & Schwartz, J. S. (2007) · Health Communication · doi:10.1080/10410230701454189 · Cited by 395
“We averaged kappa scores for each pair of decisions (Conger, 1980), which included six possible pairings.”
MethodHuijg, J. M., Gebhardt, W. A., Crone, M. R., Dusseldorp, E., & Presseau, J. (2014) · Implementation Science · doi:10.1186/1748-5908-9-11 · Cited by 385
“, Light’s Kappa [59]) was calculated to assess agreement between judges across their allocation of all items to domains.”
The Pre-Linguistic Autism Diagnostic Observation Schedule
DiLavore, P. C., Lord, C., & Rutter, M. (1995) · Journal of Autism and Developmental Disorders · doi:10.1007/bf02179373 · Cited by 359
Measuring Agreement for Multinomial Data
Davies, M., & Fleiss, J. L. (1982) · Biometrics · doi:10.2307/2529886 · Cited by 353
“Conger (1980) developed an average measure of pairwise agreement for multiple devices which may be shown to simplify to (5).”
MethodLexical association measures and collocation extraction
Pecina, P. (2010) · Language Resources and Evaluation · doi:10.1007/s10579-009-9101-4 · Cited by 316
“For two annotators the exact Fleiss’ j reduces to the well known Cohen’s j (Conger 1980).”
MethodInfluentialStange, K. C., Zyzanski, S. J., Smith, T. F., Kelly, R., Langa, D. M., Flocke, S. A., & Jaén, C. R. (1998) · Medical Care · doi:10.1097/00005650-199806000-00009 · Cited by 313
A systematic review of the effects of non-conscious regulatory processes in physical activity
Rebar, A. L., Dimmock, J. A., Jackson, B., Rhodes, R. E., Kates, A., Starling, J., & Vandelanotte, C. (2016) · Health Psychology Review · doi:10.1080/17437199.2016.1183505 · Cited by 276
“Models of how the effects of non-conscious regulatory processes on physical activity have been conceptualised. the screening by full text (agreement = 95%, Fleiss’ kappa = 0.89) (Conger, 1980; Landis & Koch, 1977).”
BackgroundInfluentialLamb, M. E., Thompson, R. A., Gardner, W. P., Charnov, E. L., & Estes, D. (1984) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00026522 · Cited by 266
A Generalization of Cohen's Kappa Agreement Measure to Interval Measurement and Multiple Raters
Berry, K. J., & Mielke, P. W. (1988) · Educational and Psychological Measurement · doi:10.1177/0013164488484007 · Cited by 239
Measuring interrater reliability among multiple raters: An example of methods for nominal data
Posner, K. L., Sampson, P. D., Caplan, R. A., Ward, R. J., & Cheney, F. W. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090917 · Cited by 229
Inequalities between multi-rater kappas
Warrens, M. J. (2010) · Advances in Data Analysis and Classification · doi:10.1007/s11634-010-0073-4 · Cited by 226
“Only two of them (Light 1971; Hubert 1977; Conger 1980) are actually generalizations of Cohen’s κ .”
Effects of Instruction in Deriving Word Meaning from Context: A Meta-Analysis
Fukkink, R. G., & de Glopper, K. (1998) · Review of Educational Research · doi:10.3102/00346543068004450 · Cited by 185
“Generalized Cohen's kappa (Conger, 1980; Light, 1971 ) is computed for nominal categories, r/design 2) (Orwin, 1994) for continuous categories.”
MethodHigh Agreement and High Prevalence: The Paradox of Cohen’s Kappa
Zec, S., Soriani, N., Comoretto, R., & Baldi, I. (2017) · The Open Nursing Journal · doi:10.2174/1874434601711010211 · Cited by 169
“The expression [3] is referring to the extension of Cohen’s Kappa to a more general case with more than two evaluators and more than two categories [26].”
MethodCT-GAN: Malicious Tampering of 3D Medical Imagery using Deep Learning
Mirsky, Y., Mahler, T., Shelef, I., & Elovici, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.03597 · Cited by 168
“We verified this agreement by computing Fliess’ kappa [57] (a statistical interrater reliability measure) which produced an excellent kappa of 0.”
MethodSaah, D. S., Johnson, G. W., Ashmall, B., Tondapu, G., Tenneson, K. R., Patterson, M., Poortinga, A., Markert, K. N., Quyen, N. H., Aung, K. S., et al. (2019) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2019.05.004 · Cited by 177
Modeling Agreement among Raters
Tanner, M. A., & Young, M. A. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10477157 · Cited by 161
“that both Light (1971) and Fleiss (1971) approached the problem by considering average pairwise agreement.”
BackgroundInfluentialImplementing a General Framework for Assessing Interrater Agreement in Stata
Klein, D. (2018) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x1801800408 · Cited by 158
“Conger (1980) applies the idea of averaging over chance agreement, defined in (4), to generalize Cohen’s kappa to three or more raters.”
Genetic and morphologic features for melanoma classification
Broekaert, S. M. C., Roy, R., Okamoto, I., van den Oord, J., Bauer, J., Garbe, C., Barnhill, R. L., Busam, K. J., Cochran, A. J., Cook, M. G., et al. (2010) · Pigment Cell & Melanoma Research · doi:10.1111/j.1755-148x.2010.00778.x · Cited by 158
“Light’s Kappa (Conger, 1980) scores were computed for inter-rater agreements for ordinal and categorical variables, using squared weighted Kappa for ordinal variables.”
MethodFroehlich, D. E., Waes, S. V., & Schäfer, H. (2020) · Review of Research in Education · doi:10.3102/0091732x20903311 · Cited by 172
“Exact Kappa (Conger, 1980) was calculated.”
Native and Nonnative Teachers of L2 Pronunciation: Effects on Learner Performance
Levis, J. M., Sonsaat, S., Link, S., & Barriuso, T. A. (2016) · TESOL Quarterly · doi:10.1002/tesq.272 · Cited by 199
Nili, A., Tate, M., Barros, A., & Johnstone, D. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2020.102154 · Cited by 174
A Comparison of Reliability Coefficients for Ordinal Rating Scales
de Raadt, A., Warrens, M. J., Bosker, R. J., & Kiers, H. A. L. (2021) · Journal of Classification · doi:10.1007/s00357-021-09386-5 · Cited by 112
“Multi-rater extensions of unweighted kappa are presented in Light (1971), Hubert (1977), Conger (1980), and Davies and Fleiss (1982).”
BackgroundQuantifying the development of agreement among experts in Delphi studies
Meijering, J. V., Kampen, J. K., & Tobi, H. (2013) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2013.01.003 · Cited by 172
Desistance and Identity Repair: Redemption Narratives as Resistance to Stigma: Table 1
Stone, R. (2015) · The British Journal of Criminology · doi:10.1093/bjc/azv081 · Cited by 150
Birko, S., Dove, E. S., & Özdemir, V. (2015) · PLOS ONE · doi:10.1371/journal.pone.0135162 · Cited by 128
“The different ways of calculating kappa differ mainly in the value of the probability that there is chance agreement, due to different assumptions of distributions of object ratings among all experts [11, 12].”
BackgroundBerrone, P., Ricart, J. E., Duch, A., Bernardo, V., Salvador, J., Piedra-Peña, J., & Planas, M. R. (2019) · Sustainability · doi:10.3390/su11082339 · Cited by 178
“In Table 4, we present Cohen’s kappa [92,93] and its generalization by Conger [94], along with the Brennan and Prediger coefficient [95], Scott/Fleiss’s kappa/pi [96], Gwet’s AC [89,97], and Krippendorff’s alpha coefficient [98] for the case study of the La Chira WWTP.”
Methodde Miranda Azevedo, R., Roest, A. M., Hoen, P. W., & de Jonge, P. (2014) · Psychological Medicine · doi:10.1017/s0033291714000063 · Cited by 138
“60) (Light, 1971; Conger, 1980).”
MethodInfluentialTesting the Difference of Correlated Agreement Coefficients for Statistical Significance
Gwet, K. L. (2015) · Educational and Psychological Measurement · doi:10.1177/0013164415596420 · Cited by 144
“The kappa coefficient was introduced by Cohen (1960), and later generalized to the case of multiple raters by Conger (1980).”
MethodHow Robust Are Multirater Interrater Reliability Indices to Changes in Frequency Distribution?
Quarfoot, D., & Levine, R. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1141708 · Cited by 134
“These indices were Fleiss’s kappa, Conger’s kappa, Krippendorff ’s alpha, the Brennan-Prediger coefficient, and Gwet’s AC2 index (Fleiss 1971; Conger 1980; Brennan and Prediger 1981; Krippendorff 2012; Gwet 2014).”
MethodMistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices
Feng, G. C. (2014) · Methodology · doi:10.1027/1614-2241/a000086 · Cited by 144
A Measure of Agreement for Interval or Nominal Multivariate Observations
Janson, H., & Olsson, U. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971239 · Cited by 130
“…correlation coefficients is often recommended (e.g., McGraw & Wong, 1996), and for one nominal variable (dichotomous or polytomous), extensions of Cohen’s (1960) kappa coefficient (κ) to the several-observer case have been proposed (e.g., see Berry & Mielke, 1988; Conger, 1980; Fleiss, 1971).”
BackgroundStages of successful time-limited counseling: An interactional examination.
Tracey, T. J. G., & Ray, P. B. (1984) · Journal of Counseling Psychology · doi:10.1037/0022-0167.31.1.13 · Cited by 130
Intercoder reliability indices: disuse, misuse, and abuse
Feng, G. C. (2013) · Quality & Quantity · doi:10.1007/s11135-013-9956-8 · Cited by 142
Aickin, M. (1990) · Biometrics · doi:10.2307/2531434 · Cited by 122
Northrup, N. C., Howerth, E. W., Harmon, B. G., Brown, C. A., Carmicheal, K. P., Garcia, A. P., Latimer, K. S., Munday, J. S., Rakich, P. M., Richey, L. J., et al. (2005) · Journal of Veterinary Diagnostic Investigation · doi:10.1177/104063870501700606 · Cited by 111
Conducting Online Expert panels: a feasibility and experimental replicability study
Khodyakov, D., Hempel, S., Rubenstein, L. V., Shekelle, P., Foy, R., Salem-Schatz, S. R., O'Neill, S. M., Danz, M. S., & Dalal, S. R. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-174 · Cited by 116
“Finally, we used four-way kappa to assess agreement between panels, treating the data as ordinal and using a weight matrix comprising the squared deviations between scores [27].”
MethodPeer interactions of autistic children and adolescents
Lord, C., & MaGill-Evans, J. (1995) · Development and Psychopathology · doi:10.1017/s095457940000674x · Cited by 98
Brembilla, G., Dell’Oglio, P., Stabile, A., Damascelli, A., Brunetti, L., Ravelli, S., Cristel, G., Schiani, E., Venturini, E., Grippaldi, D., et al. (2020) · European Radiology · doi:10.1007/s00330-019-06654-2 · Cited by 113
Medical Diagnostic Tests: A Review of Test Anatomy, Phases, and Statistical Treatment of Data
Bolboaca, S. D. (2019) · Computational and Mathematical Methods in Medicine · doi:10.1155/2019/1891569 · Cited by 108
“(i) Nominal or ordinal scale: Cohen’s kappa coefficient (nominal scale, inclusive dichotomial such as positive/negative test result), symbol κ [128], and its derivatives (Fleiss’ generalized kappa [129], Conger’s generalized kappa [130], and weighted kappa (ordinal scale test result)) [131]”
BackgroundJiang, H., Qiang, M., & Lin, P. (2016) · Environmental Impact Assessment Review · doi:10.1016/j.eiar.2016.06.004 · Cited by 106
Issues in the Use of Kappa to Estimate Reliability
Soeken, K. L., & Prescott, P. A. (1986) · Medical Care · doi:10.1097/00005650-198608000-00008 · Cited by 102
Airas, M., & Alku, P. (2006) · Phonetica · doi:10.1159/000091405 · Cited by 117
D’Agostino, M. A., Aegerter, P., Jousse‐Joulin, S., Chary-Valckenaere, I., Lecoq, B., Gaudin, P., Brault, I., Schmitz, J., Dehaut, F.-X., Parc, J. M. L., et al. (2008) · Arthritis Care & Research · doi:10.1002/art.24369 · Cited by 96
Learning how to Differ: Agreement and Reliability Statistics in Psychiatry
David, L. S. (1995) · The Canadian Journal of Psychiatry · doi:10.1177/070674379504000202 · Cited by 97
“The first problem, having more than two judges, has been dealt with by a relatively simple extension of the equation (10,11).”
BackgroundCategory Distinguishability and Observer Agreement
Darroch, J. N., & McCloud, P. (1986) · Australian Journal of Statistics · doi:10.1111/j.1467-842x.1986.tb00709.x · Cited by 99
von Arx, T., Hänni, S., & Jensen, S. S. (2010) · Journal of Endodontics · doi:10.1016/j.joen.2010.03.040 · Cited by 94
Cohen, D. A., Dabbs, D. J., Cooper, K. L., Amin, M., Jones, T. E., Jones, M. W., Chivukula, M., Trucco, G., & Bhargava, R. (2012) · American Journal of Clinical Pathology · doi:10.1309/ajcp6dkrnd5ckvdd · Cited by 92
Zander, E., Willfors, C., Berggren, S., Olsson, N. C., Coco, C., Elmund, A., Moretti, Å. H., Holm, A., Jifält, I., Kosieradzki, R., et al. (2015) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-015-0793-2 · Cited by 96
Kappa Coefficient Calculation Using Multiple Ratings Per Subject: A Special Communication
Haley, S. M., & Osberg, J. S. (1989) · Physical Therapy · doi:10.1093/ptj/69.11.970 · Cited by 91
Gwet, K. L. (2008) · Wiley eBooks · doi:10.1002/9780471462422.eoct631 · Cited by 93
Nominal Scale Agreement Among Observers
Schouten, H. J. A. (1986) · Psychometrika · doi:10.1007/bf02294066 · Cited by 90
Meyer, R., Fofanov, V. Y., Panigrahi, A., Merchant, F. A., Zhang, N., & Pati, D. (2009) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-08-2454 · Cited by 87
Zlobec, I., Steele, R. J., Michel, R. P., Compton, C. C., Lugli, A., & Jass, J. R. (2006) · Modern Pathology · doi:10.1038/modpathol.3800642 · Cited by 87
Hazlehurst, B. L., Sittig, D. F., Stevens, V. J., SMITH, K., Hollis, J. F., VOGT, T. M., Winickoff, J. P., GLASGOW, R., Palen, T. E., & Rigotti, N. A. (2005) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2005.08.007 · Cited by 87
Parent–child interactions among children with juvenile fibromyalgia, arthritis, and healthy controls
Reid, G. J., McGrath, P. J., & Lang, B. A. (2005) · Pain · doi:10.1016/j.pain.2004.10.018 · Cited by 78
“Kappa measurements are extremely sensitive to skewed distributions and do not accurately reflect inter-rater agreement (Bakeman and Gottman, 1997; Conger, 1980; Light, 1971; Zwick, 1988).”
BackgroundVariance Estimation of Nominal-Scale Inter-Rater Reliability with Random Selection of Raters
Gwet, K. L. (2008) · Psychometrika · doi:10.1007/s11336-007-9054-8 · Cited by 91
“Cohen (1960), criticizing Scott’s statistic for its underlying assumption of marginal homogeneity, proposed the well-known kappa statistic. Other authors such as Bennett, Alpert, and Goldstein (1954), Holley and Guilford (1964), or Maxwell (1977) advocated the use of an agreement coefficient similar in its form to Sco…”
BackgroundMethodInfluentialDNA Barcode Sequence Identification Incorporating Taxonomic Hierarchy and within Taxon Variability
Little, D. P. (2011) · PLoS ONE · doi:10.1371/journal.pone.0020552 · Cited by 89
Hammer, H. B., Iagnocco, A., Mathiessen, A., Filippucci, E., Gandjbakhch, F., Kortekaas, M. C., Möller, I., Naredo, E., Wakefield, R. J., Aegerter, P., et al. (2014) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2014-206289 · Cited by 86
Age at the onset of senescence in birds and mammals is predicted by early-life performance
Péron, G., Giménez, O., Charmantier, A., Gaillard, J., & Crochet, P. (2010) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2010.0530 · Cited by 85
“We tested the estimates for among-judge agreement (ordered Cohen’s k ¼ 0.88, between-judges r2’s all .0.90 indicating satisfactory among-judge agreement; Conger 1980).”
MethodInfluentialPre-Service Teachers’ Digital Competence: A Call for Action
Dolezal, D., Motschnig, R., & Ambros, R. (2025) · Education Sciences · doi:10.3390/educsci15020160 · Cited by 86
Measures of agreement between many raters for ordinal classifications
Nelson, K. P., & Edwards, D. (2015) · Statistics in Medicine · doi:10.1002/sim.6546 · Cited by 84
“Existing summary measures for assessing agreement between multiple raters when ordinal classifications are being examined include Fleiss’ kappa [9], Light’s and Conger’s kappa [10, 11], Kraemer’s kappa coefficient [12], and an AC2 statistic [13].”
Development and validation of a radiological diagnosis model for hypersensitivity pneumonitis
Salisbury, M. L., Gross, B. H., Chughtai, A. R., Sayyouh, M., Kazerooni, E. A., Bartholmai, B. J., Xia, M., Murray, S., Myers, J. L., Lagstein, A., et al. (2018) · European Respiratory Journal · doi:10.1183/13993003.00443-2018 · Cited by 78
“features are via the kappa and Light’s kappa, respectively.(13) The bootstrap method was used”
MethodA Generalized Kappa Coefficient
Uebersax, J. S. (1982) · Educational and Psychological Measurement · doi:10.1177/0013164482421018 · Cited by 76
da Cunha, D. T., de Oliveira, A. B. A., Saccol, A. L. D. F., Tondo, E. C., Silva, E. A., Ginani, V. C., Montesano, F. T., de Castro, A. K. F., & Stedefeldt, E. (2014) · Food Research International · doi:10.1016/j.foodres.2014.01.021 · Cited by 96
“The kappa statistic (Conger, 1980) was then used to assess the agreement between each pair of evaluations made in the same establishment.”
MethodPhan, M., Caigny, A. D., & Coussement, K. (2023) · Decision Support Systems · doi:10.1016/j.dss.2023.113940 · Cited by 83
Moons, F., & Vandervieren, E. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02746-8 · Cited by 73
“Wewillcalculate Po c pairwise(Conger,1980).”
Stability and Interpersonal Agreement of the Interview-Based Diagnosis of Autism
Hill, A., Bölte, S., Petrova, G., Beltcheva, D., Tacheva, S., & Poustka, F. (2001) · Psychopathology · doi:10.1159/000049305 · Cited by 74
Batchelor, J., Tan, W., Tour, S. K., Yong, A., Montgomery, A. A., & Thomas, K. S. (2015) · British Journal of Dermatology · doi:10.1111/bjd.14208 · Cited by 72
Non-treatment-related sudden gains in depression: The role of self-evaluation
Kelly, M. A. R., Roberts, J. E., & Bottonari, K. A. (2006) · Behaviour Research and Therapy · doi:10.1016/j.brat.2006.06.008 · Cited by 78
Office capillaroscopy in systemic sclerosis
Baron, M. G., Bell, M., BOOKMAN, A. A. M., Buchignani, M., Dunne, J. V., Hudson, M., Jerome, D., Johnson, S. R., Jones, N., Kamińska, E., et al. (2006) · Clinical Rheumatology · doi:10.1007/s10067-006-0489-6 · Cited by 71
“Kappa coefficients [26] were used to demonstrate the percent agreement after removing the effect of chance.”
MethodCohen’s kappa is a weighted average
Warrens, M. J. (2011) · Statistical Methodology · doi:10.1016/j.stamet.2011.06.002 · Cited by 70
“The popularity of κ has led to the development of many extensions [17,13], including multi-rater kappas [6,32], kappas for groups of raters [21,22] and weighted kappas [5,23,30,33].”
Relapse prevention and smoking cessation
Davis, J., & Glaros, A. G. (1986) · Addictive Behaviors · doi:10.1016/0306-4603(86)90034-1 · Cited by 73
Some views on agreement to be used in content analysis studies
Popping, R. (2009) · Quality & Quantity · doi:10.1007/s11135-009-9258-3 · Cited by 83
Estimating age of mature adults from the degeneration of the sternal end of the clavicle
Falys, C. G., & Prangle, D. (2014) · American Journal of Physical Anthropology · doi:10.1002/ajpa.22639 · Cited by 79
Calretinin immunostaining as an adjunct in the diagnosis of Hirschsprung disease
Holland, S. K., Ramalingam, P., Podolsky, R. H., Reid‐Nicholson, M. D., & Lee, J. R. (2011) · Annals of Diagnostic Pathology · doi:10.1016/j.anndiagpath.2011.02.010 · Cited by 67
A Formal Proof of a Paradox Associated with Cohen’s Kappa
Warrens, M. J. (2010) · Journal of Classification · doi:10.1007/s00357-010-9060-x · Cited by 74
Christiansen, A., Hendricks, O., Kuettel, D., Hørslev‐Petersen, K., Jurik, A. G., Nielsen, S., Rufibach, K., Loft, A. G., Pedersen, S. J., Hermansen, L. T., et al. (2016) · The Journal of Rheumatology · doi:10.3899/jrheum.160079 · Cited by 69
Agreement and Kappa-Type Indices
de Mast, J. (2007) · The American Statistician · doi:10.1198/000313007x192392 · Cited by 67
“Standard accounts of kappa-type indices are Cohen (1960), Fleiss (1971), Conger (1980), and Davies and Fleiss (1982).”
MethodThe Direct and Indirect Effects of Inadmissible Evidence1
Carretta, T. R., & Moreland, R. L. (1983) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.1983.tb01741.x · Cited by 63
Large-Sample Variance of Fleiss Generalized Kappa
Gwet, K. L. (2021) · Educational and Psychological Measurement · doi:10.1177/0013164420973080 · Cited by 64
“Fleiss (1971), Light (1971), Krippendorff (1970), and Conger (1980), among others, proposed various extensions of Cohen’s kappa to multiple raters.”
Structural Divisions of Personality and the Classification of Traits
Mayer, J. D. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.381 · Cited by 61
“The average kappa value ( Conger, 1980, p. 323 ) for all pairs of judges again indicated that there was significant agree-”
BackgroundBottonari, K. A., Roberts, J. E., Kelly, M. A. R., Kashdan, T. B., & Ciesla, J. A. (2006) · Behaviour Research and Therapy · doi:10.1016/j.brat.2006.01.003 · Cited by 70
An improved method of analysis of observer variation between pathologists
Holman, C., JAMES, I. R., Heenan, P. J., Matz, L. R., Blackwell, J. B., Kelsall, G. R. H., SINGH, A., & Seldam, R. E. J. T. (1982) · Histopathology · doi:10.1111/j.1365-2559.1982.tb02752.x · Cited by 63
Metadata
- Authors
- Conger, A. J.
- Affiliations
- Purdue University West Lafayette (US)
- Corresponding author
- Conger, Anthony J.
- Venue
- Psychological Bulletin
- Published
- 1980-09
- Volume
- 88
- Issue
- 2
- Pages
- 322-328
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1978217317
- Cited by
- 545
- References
- 3
- Citations by year
- 2026: 20 · 2025: 24 · 2024: 35 · 2023: 24 · 2022: 23 · 2021: 23 · 2020: 25 · 2019: 25 · 2018: 22 · 2017: 16 · 2016: 18 · 2015: 27 · 2014: 29 · 2013: 15 · 2012: 15
- FWCI
- 7.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- Cohen's kappa, Fleiss' kappa, interobserver agreement, multiple raters, nominal scale
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.