Integration and generalization of kappas for multiple raters
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
J. A. Cohen's kappa (1960) for measuring agreement between 2 raters, using a nominal scale, has been extended for use with multiple raters by R. J. Light (1971) and J. L. Fleiss (1971). In the present article, these indices are analyzed and reformulated in terms of agreement statistics based on all
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“with interrater reliability measured using Cohen’s Kappa (κ = 0.63–0.82), indicating moderate to substantial agreement (Conger, 1980).”
From Rethinking pre-training: cognitive load implications for learners with varying prior knowledge (2025) · cites this paper as Conger, 1980
In the citing paper
“Furthermore, (25) provides an alternative proof that Conger's and Fleiss' quadratically weighted kappas converge to each other as the number of raters, 𝑅, tends to infinity (Conger, 1980;Gwet, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Conger, 1980
In the citing paper
“The average pairwise Pearson, Spearman, Kendall's τ b , ϕ, and MCC over the R 2 judge pairs are all equal, and coincide with κ F when all judges share the same marginal MET rate.”
From Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why (2026) · cites this paper as Conger (1980)
In the citing paper
“A generalization of Cohen’s kappa (i.e., Light’s Kappa [59]) was calculated to assess agreement between judges across their allocation of all items to domains.”
From Discriminant content validity of a theoretical domains framework questionnaire for use in implementation research (2014) · cites this paper as [59]
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Light, R. J. (1971) · Psychological Bulletin · doi:10.1037/h0031643 · Cited by 747
Cited by
Items 301–400 of 555
Weighted kappa as a function of unweighted kappas
Moradzadeh, N., Ganjali, M., & Baghfalaki, T. (2015) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610918.2015.1105975 · Cited by 10
“Kappa coefficients have been developed for situations with multiple raters (Conger, 1980; Warrens, 2010e) and with different groups of raters (Vanbelle and Albert, 2009a, Vanbelle and Albert, 2009b).”
BackgroundFoirest, C., Granger, B., Gatignol, P., Bernat, I., Nguyen, Y., Lamas, G., & Tankéré, F. (2016) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0000000000003217 · Cited by 10
A Method for Functional Task Alignment Analysis of an Arthrocentesis Simulator
Adams, R. A., Gilbert, G. E., Buckley, L. A., Nino‐Fong, R., Fuentealba, I. C., & Little, E. L. (2018) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000313 · Cited by 10
Structure and analyze the football value chain in Brazil
Serrano, R., Lacerda, D. P., Cassel, R. A., Dresch, A., & Morandi, M. I. W. M. (2019) · International Journal of Sports Marketing and Sponsorship · doi:10.1108/ijsms-02-2017-0013 · Cited by 10
Crop size as an index of chick provisioning in the Greater FlamingoPhoenicopterus roseus
Rendón, M. Á., Garrido, A., Guerrero, J. C., Rendón‐Martos, M., & Amat, J. A. (2012) · Ibis · doi:10.1111/j.1474-919x.2012.01218.x · Cited by 10
“The inter-observer agreement for crop profile scores was evaluated using the kappa coeffi- cient (Conger 1980).”
MethodIs quantitation necessary for assessment of sural nerve biopsies?
Pamphlett, R., & Sjarif, A. (2003) · Muscle & Nerve · doi:10.1002/mus.10363 · Cited by 10
“The features assessed were: (1) MF density, a fundamental aspect of sural nerve pathology; (2) the size range of loss of MF, because this can be important for diagnosing conditions such as amyloid neuropathy14; (3) myelin thickness, to look for thinly myelinated axons; and (4) axonal clusters, to provide an indication…”
BackgroundMethodWhat should be done improve reviewing?
Crandall, R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065766 · Cited by 11
A Modification of Kappa for Interobserver Bias
Krauth, J. (1984) · Biometrical Journal · doi:10.1002/bimj.4710260414 · Cited by 10
Measuring Agreement in Diagnostics: A Practical Guide for Researchers
Vanbelle, S., Engelhart, C. H., & Blix, E. (2025) · Statistics in Medicine · doi:10.1002/sim.70299 · Cited by 11
“…observers under the second approach (i.e., weighted mean over all distinct pairs of observers) and are known as kappa-q, kappa-BP [30], or R andolph’s kappa [37] (chance definition 1), Hubert’s kappa [38] or Conger kappa [13, 35] (chance definition 2) and Fleiss kappa [39] (chance definition 3).”
A Sequential Examination of Parent–Child Interactions at Anesthetic Induction
Wright, K. D., Stewart, S. H., Finley, G. A., & Raazi, M. (2014) · Journal of Clinical Psychology in Medical Settings · doi:10.1007/s10880-014-9413-4 · Cited by 9
Towards automated e-counselling system based on counsellors emotion perception
Awuni, D. E. K., & Montero, C. (2018) · Education and Information Technologies : Official Journal of the IFIP technical committee on Education · doi:10.1007/s10639-017-9643-9 · Cited by 11
“Fleiss kappa can only be used for nominal or binary scaleratings (Conger 1980; Fleiss 1971).”
BackgroundMethodInfluentialStone, R. (2019) · Victims & Offenders · doi:10.1080/15564886.2019.1630696 · Cited by 10
Inter- and intra-examiner agreement of three classification systems of gingival recession
Sarlati, F., Moghaddas, O., Shabahangfar, M. R., Safari, S., & Naser, V. (2019) · Journal of Advanced Periodontology & Implant Dentistry · doi:10.15171/japid.2019.001 · Cited by 11
“Level of agreement was assessed according to Landis and Koch in six levels (Table 1).(14,15)”
MethodMeasures of Agreement with Multiple Raters: Fréchet Variances and Inference
Moss, J. (2024) · Psychometrika · doi:10.1007/s11336-023-09945-2 · Cited by 10
“The pair-wise coefficients measure the agreement between pairs of raters (Conger, 1980), while the consensus coefficients measure the simultaneous agreement between all raters.”
BackgroundInter- and intra-observer agreement on Miller’s classification of gingival tissue recessions
Bertl, K., Ruckenbauer, D., Müller-Kern, M., Durstberger, G., Lettner, S., Bruckmann, C., & Ulm, C. (2014) · Odontology · doi:10.1007/s10266-014-0179-9 · Cited by 9
Sklar’s Omega: A Gaussian copula-based framework for assessing agreement
Hughes, J. (2022) · Statistics and Computing · doi:10.1007/s11222-022-10105-2 · Cited by 9
Surface Ki-67 Expression Improves Reproducibility of Dysplasia Diagnosis in Barrett’s Esophagus
Yousaf, H., Hayat, U., Manivel, J. C., Iwamoto, C. A., Peltola, J. C., Hanson, B., Larson, W., Dachel, S., Gravely, A. A., & Mesa, H. A. (2020) · American Journal of Clinical Pathology · doi:10.1093/ajcp/aqaa004 · Cited by 9
Gwet, K. L. (2014) · Wiley eBooks · doi:10.1002/9781118596333.ch19 · Cited by 9
Evaluating Expert Knowledge: Plant Species Responses to Cattle Grazing and Fire
Iglesias, R. M. R., & Kothmann, M. M. (1998) · Journal of Range Management · doi:10.2307/4003420 · Cited by 9
“Kappa statistics are very sensitive to marginal disagreement; given marginal disagreement, Conger (1980) reported negative estimated Km’s for randomly generated data.”
BackgroundInsights into ‘Practice’ Communication: An Interactional Approach
Jenkins, M. F., Mallett, J., O’Neill, C., McFadden, M., & Baird, H. E. (1994) · British Journal of Occupational Therapy · doi:10.1177/030802269405700804 · Cited by 9
A Dataset of Electrical Components for Mesh Segmentation and Computational Geometry Research
Scheffler, B., Bründl, P., Nguyen, H. G., Stoidner, M., & Franke, J. E. (2024) · Scientific Data · doi:10.1038/s41597-024-03155-w · Cited by 9
Measuring Agreement Using Guessing Models and Knowledge Coefficients
Moss, J. (2023) · Psychometrika · doi:10.1007/s11336-023-09919-4 · Cited by 8
Esterman Visual Field Testing Using a Virtual Reality Headset in Glaucoma
Sharma, M., Savatovsky, E. J., Huertas, L., O’Brien, R. C., Grajewski, A. L., & Bitrian, E. (2024) · Ophthalmology Science · doi:10.1016/j.xops.2024.100534 · Cited by 8
“…also calculated using Light ’ s k statistic to observe the agreement between different ophthalmologists on “ pass ” or “ failure ” determinations for SAP versus VRVF. 16 The bias-corrected and accelerated bootstrap was used to obtain the 95% con fi dence intervals (CIs) for Light ’ s k statistic.”
MethodInfluentialIs Artificial Intelligence a Useful Tool for Clinical Practice of Oral and Maxillofacial Surgery?
Işık, G., Gürbüz, İ. A. K., Elgün, F., Kara, R. U., Berber, B., Özgül, S., & Günbay, T. (2024) · Journal of Craniofacial Surgery · doi:10.1097/scs.0000000000010686 · Cited by 8
“17,18 According to Landis and Koch ’ s scale, the agreement co-ef fi cients were interpreted using Gwet ’ s probabilistic benchmarking method.”
Propuesta de una Clasificación española sobre imágenes de pornografía infantil
Pascual, M. D. P. A. C., Framís, A. G.-S., & Garrido, C. I. (2017) · Revista Española de Investigación Criminológica · doi:10.46381/reic.v15i0.103 · Cited by 8
Lopes, C. D. L., Filho, C. A. D. R. C., Silva, T. A. D. L. E., Gonçalves, M. C. K., de Oliveira, R. L., & de Lima, P. R. G. (2014) · Revista Brasileira de Ortopedia (English Edition) · doi:10.1016/j.rboe.2014.10.010 · Cited by 8
Kevane, M., & Koopmann‐Holm, B. (2021) · Behavior Research Methods · doi:10.3758/s13428-020-01499-w · Cited by 8
Fixed-Effects Modeling of Cohen's Kappa for Bivariate Multinomial Data
Yang, J., & Chinchilli, V. M. (2009) · Communication in Statistics- Theory and Methods · doi:10.1080/03610920802618426 · Cited by 8
Psychometric analysis of the Dutch language Facilitative Interpersonal Skills (FIS) video clips
Thiel, S. V., Joosen, M. C. W., Joki, A.-L., van Dam, A., van der Klink, J. J. L., & de Jong, K. (2021) · Research in Psychotherapy Psychopathology Process and Outcome · doi:10.4081/ripppo.2021.513 · Cited by 9
“Looking at the share of scores across all four quadrants per client vignette (see Table 2), we gained a more nuanced insight into the extent of agreement across the raters (see also Conger, 1980).”
BackgroundIs unreliability in peer review harmful?
Roediger, H. L., III (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065936 · Cited by 9
Heltborg, A., Mogensen, C. B., Skjøt‐Arkil, H., Giebner, M., Al-Masri, A., Khatry, U. B., Khatry, S., Heinemeier, I. I. K., Andreasen, J. J., Hariesh, S. S. S., et al. (2024) · Scandinavian Journal of Trauma Resuscitation and Emergency Medicine · doi:10.1186/s13049-024-01242-w · Cited by 7
“Rater reliability was calculated as both kappa (Cohens kappa for pairwise comparison, Conger’s kappa for multiple raters fully crossed design [18]) and percent agreement.”
MethodRusso, T., Quarta, L., Pellegrino, F., Cosenza, M., Camisassa, E., Lavalle, S., Apostolo, G., Zaurito, P., Scuderi, S., Barletta, F. M., et al. (2025) · La radiologia medica · doi:10.1007/s11547-025-02017-8 · Cited by 7
“Inter-reader agreement for PI-RADS score assigned by readers with and without Quantib Prostate was assessed using Conger's kappa coefficient and percent of agreement [26], using reports of original mpMRI scans as reference standard.”
Wang, Q., Xing, X., Zhang, Z., Ji, X., He, S., Yang, Y., Xu, J., Zhao, Q., & Lang, N. (2025) · Insights into Imaging · doi:10.1186/s13244-025-01997-5 · Cited by 7
“The numerical agreement among readers was assessed by the exact Fleiss Kappa, as it assigns greater penalties to larger discrepancies in ordinal ratings [13].”
Validity and Reliability of the eHealth Analysis and Steering Instrument
Henkemans, O. B., Dusseldorp, E., Keijsers, J. F. E. M., Kessens, J. M., Neerincx, M. A., & Otten, W. (2013) · Medicine 2 0 · doi:10.2196/med20.2571 · Cited by 7
“Interrater Reliability As an index of the interrater reliability, a generalized kappa was computed (ie, Light’s kappa) [42].”
Method(Re)Visualizing Rater Agreement: Beyond Single-Parameter Measures
Eubanks, D. (2017) · The Journal of Writing Analytics · doi:10.37514/jwa-j.2017.1.1.10 · Cited by 7
Shortening of psychological tests – assumptions, methods and doubts
Kleka, P., & Paluchowski, W. J. (2017) · Polish Psychological Bulletin · doi:10.1515/ppb-2017-0058 · Cited by 7
“For the examined LCWS scale of the WEQ questionnaire, the study obtained a statistically significant measure of the conformity of the sequences calculated with the use of Fleiss’ kappa (Conger, 1980) and it amounted to .”
MethodStatistical inference for agreement between multiple raters on a binary scale
Vanbelle, S. (2024) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12333 · Cited by 8
Tsai, M. (2011) · Journal of Applied Statistics · doi:10.1080/02664763.2011.578623 · Cited by 7
“Later, Fleiss [12] and Conger [9] extended this statistic to measuring the agreement among multiple ratings.”
BackgroundA note on the kappa statistic for clustered dichotomous data
Zhou, M., & Zhao, Y. (2014) · Statistics in Medicine · doi:10.1002/sim.6098 · Cited by 7
Why is the reliability of peer review so low?
Laming, D. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065882 · Cited by 8
Learning in the context of evolutionary biology: In search of synthesis
Petrovich, S. B., & Gewirtz, J. L. (1984) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00026662 · Cited by 7
Schönemann, P. H. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065973 · Cited by 7
Chen, T., Wang, Y., Wu, T., Chen, C.-R., Cheng, C.-Y., Hsueh, I., Wang, S., & Hsieh, C. (2024) · American Journal of Occupational Therapy · doi:10.5014/ajot.2024.050386 · Cited by 6
“For each pair of raters (i.e., 1 – 2, 2 – 3, and 1 – 3), the weighted k was calculated, and the three pair-wise weighted k values were averaged (Conger, 1980).”
Megahed, F. M., Chen, Y.-J., Jones‐Farmer, L. A., Lee, Y., Wang, J., & Zwetsloot, I. M. (2026) · Journal of Business Analytics · doi:10.1080/2573234x.2026.2652281 · Cited by 7
Leading Together: Strengthening Relational Trust in the Adult School Community
Leis, M., Rimm‐Kaufman, S. E., Paxton, C. L. C., & Sandilos, L. E. (2017) · Journal of School Leadership · doi:10.1177/105268461702700603 · Cited by 11
“A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256. doi:10.1093/her/18.2.237. Elias, M. J., Zins, J. E., Graczyk, P. A., & Weissberg, R. P. (2003). Implementation, sustainability, and scaling up of social-emotional and a…”
BackgroundMethodInfluentialAsai, K., Ohyama, T., Watanabe, M., Moriyama, H., Kujiraoka, M., Watanabe, R., Shigeta, K., Kakizaki, N., Hagiwara, O., & Saida, Y. (2023) · Journal of Hepato-Biliary-Pancreatic Sciences · doi:10.1002/jhbp.1367 · Cited by 6
Chyr, L., Wolff, J. L., Zissimopoulos, J., & Drabo, E. F. (2024) · Alzheimer s & Dementia · doi:10.1002/alz.13758 · Cited by 7
“We calculated the Cohen and Conger’s kappa statistic ( 𝜅 ), 39,40 a measure of the proportion of agreement greater than that expected by chance, used to assess the strength of agreement between the measures(EquationS1).”
BackgroundMethodChatGPT performance on radiation technologist and therapist entry to practice exams
Duggan, R., & Tsuruda, K. M. (2024) · Journal of medical imaging and radiation sciences · doi:10.1016/j.jmir.2024.04.019 · Cited by 7
Henneberg, S., Henriksen, J. H., Christensen, L., Markvart, M., & Rosing, K. (2024) · Health Science Reports · doi:10.1002/hsr2.2278 · Cited by 6
“20,21 The interrater agreement is initially affected by the response option “ Different terminology ” which causes many answers to appear in agreement while they equally likely may represent disagreement.”
BackgroundA Study on Comparison of Generalized Kappa Statistics in Agreement Analysis
Kim, M., Song, K., Nam, C. M., & Jung, I. (2012) · Korean Journal of Applied Statistics · doi:10.5351/kjas.2012.25.5.719 · Cited by 7
“In this paper, we compare the performance of four different generalized kappa statistics proposed by Fleiss (1971), Conger (1980), Randolph (2005), and Gwet (2008a).”
MethodCelani, M. G., Carrai, R., Cantisani, T. A., Scarpino, M., Ercolani, M. V., Lolli, F., Lanzo, G., Costa, P., Lanteri, P., Bignamini, A. A., et al. (2020) · Resuscitation · doi:10.1016/j.resuscitation.2020.07.029 · Cited by 6
Control in Counseling: Intrapersonal Versus Interpersonal Definitions
Tracey, T. J. G. (1986) · Journal of Counseling & Development · doi:10.1002/j.1556-6676.1986.tb01185.x · Cited by 6
A macroscopic and stereological imaging dataset of Pleuronectes platessa ovaries
Sauger, C., Quinquis, J., Kellner, K., Berthelin, C., Lepoittevin, M., Elie, N., & Dubroca, L. (2020) · Scientific Data · doi:10.1038/s41597-020-0505-8 · Cited by 6
Measurement of rater consistency and its application in voice quality assessments
Xie, Z., Gadepalli, C., Jalalinajafabadi, F., Cheetham, B. M. G., & Homer, J. J. (2017) · International Congress on Image and Signal Processing (CISP) · doi:10.1109/cisp-bmei.2017.8302302 · Cited by 5
“Conger [18] generalised the formulation by Light [16] to more than three raters.”
Ståhl, F. L., Almqvist, H., Aspelin, Å., Kolloch, J., Ghalamkari, O., Gontu, V. K., Schäfer, D., van de Haar, P., Engel, K.-J., van Nijnatten, F., et al. (2024) · Clinical Neuroradiology · doi:10.1007/s00062-024-01439-7 · Cited by 5
“Per-region ASPECTS agreement was assessed by the exact Fleiss Kappa [26].”
O’Carroll, J. P. J., Kennedy, R. M., Ren, L., Nash, S., Hartnett, M., & Brown, C. (2017) · Estuarine Coastal and Shelf Science · doi:10.1016/j.ecss.2017.08.005 · Cited by 5
Performance evaluation of three semantic expansions to query PubMed
Massonnaud, C., Lelong, R., Kerdelhué, G., Lejeune, É., Grosjean, J., Griffon, N., & Darmoni, S. J. (2019) · Health Information & Libraries Journal · doi:10.1111/hir.12291 · Cited by 5
Nawaytou, H. M., Springston, C., Lazar, A., Keller, S., Hogan, W. J., Cresalia, N. M., Peyvandi, S., Cocalis, M. W., Moon‐Grady, A. J., & Brook, M. M. (2022) · Journal of the American Society of Echocardiography · doi:10.1016/j.echo.2022.06.014 · Cited by 5
Reproducibility of clinical data II: categorical outcomes
Blackman, N. J. (2004) · Pharmaceutical Statistics · doi:10.1002/pst.105 · Cited by 5
Kim, K. E., Oh, S., Jeoung, J. W., Suh, M. H., Seo, J. H., Kim, M., Park, K. H., Kim, D. M., & Kim, S. H. (2016) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2016.08.017 · Cited by 5
Ouverson, K., Scherber, C., Oldham, E., & Gilbert, S. B. (2021) · Frontiers in Virtual Reality · doi:10.3389/frvir.2021.765881 · Cited by 7
“Utilizing the R package irr (Gamer et al., 2012), IRR was measured as iota ( ι ) or Conger ’ s Fleiss exact kappa coef fi cient ( κ ), depending on the number of categories per code (Conger, 1980; Janson and Olsson, 2004).”
MethodDiagnostic quality assessment for low-dimensional ECG representations
Kovács, P., Böck, C., Tschoellitsch, T., Huemer, M., & Meier, J. M. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106086 · Cited by 5
“(1) can be generalized according to [24, 25].”
Measures of Agreement for Incompletely Ranked Data
Iachan, R. (1984) · Educational and Psychological Measurement · doi:10.1177/0013164484444005 · Cited by 5
Peer review: An unflattering picture
Adams, K. M. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065687 · Cited by 5
Journal availability and the quality of published research
Fletcher, J. M. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065808 · Cited by 5
Conger, A. J. (1984) · Springer eBooks · doi:10.1007/978-1-4899-2283-0_9 · Cited by 5
Self-citationBuilding Computational Resources : The URDU.KON-TB Treebank and the Urdu Parser
Abbas, Q. (2014) · KOPS (University of Konstanz) · Cited by 5
Divanji, R. A., Bindman, S. W., Hoffman, M., & Casteneda, L. (2025) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3713043.3727062 · Cited by 4
“IRR was calculated at the parent code level using Light’s kappa to account for multiple coders [9].”
Laustsen, A. F., Dineen, R. A., Ilginiene, J., Grønbæk, J. K., Sehested, A. M., Schmiegelow, K., Mathiasen, R., Juhler, M., & Avula, S. (2025) · Pediatric Radiology · doi:10.1007/s00247-025-06326-y · Cited by 4
Ng, T. S. C., Liu, M., Robertson, M. S., Könik, A., Cheng, S., Bakht, M. K., Harrington, K., Wolanski, A., Gilbert, L., Preston, M. A., et al. (2025) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-025-07287-y · Cited by 4
“A unique feature of [ 18 F] F-fluciclovine biodistribution is its low urinary excretion at early post-injection time points when elevated tumor uptake is already seen [11].”
Influentialde Margerie‐Mellon, C., Belin, L., Boussouar, S., Khafagy, P., Debray, M., Levand, K. A., Chabi, M.-L., Khalil, A., Benattia, A., Israël‐Biet, D., et al. (2019) · European Respiratory Journal · doi:10.1183/13993003.01388-2019 · Cited by 4
Toward Gaze-assisted Developer Tools
Kuang, P., Söderberg, E., Niehorster, D. C., & Höst, M. (2023) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier58687.2023.00015 · Cited by 4
“We then performed a Light’s Kappa analysis of the inter-rater reliability of multiple raters, and got a Kappa score of 0.86, indicating that the magnitude of agreement is “strong” [15].”
MethodInfluentialHoulind, M. B., Hansen, L., Iversen, E., Rasmussen, H. B., Larsen, J. B., Jørgensen, S., Dalhoff, K., Damkier, P., Walls, A. B., Vermehren, C., et al. (2024) · Basic & Clinical Pharmacology & Toxicology · doi:10.1111/bcpt.13990 · Cited by 4
Ramanarayanan, V., Leong, C. W., Suendermann‐Oeft, D., & Evanini, K. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3136767 · Cited by 4
van Rijn, P. V., Graf, E. A., Arieli-Attali, M., & Song, Y. (2018) · ETS Research Report Series · doi:10.1002/ets2.12199 · Cited by 3
A Monte Carlo–Based Bayesian Approach for Measuring Agreement in a Qualitative Scale
Calle-Alonso, F., & Pérez, C. J. (2014) · Applied Psychological Measurement · doi:10.1177/0146621614554080 · Cited by 4
“For example, Conger (1980) defined a generalized Kappa as an overall agreement measure for three or more raters based on Cohen’s Kappa, Fleiss (1971) proposed a generalization of Scott’s p, and Mielke, Berry, and Johnston (2007) proposed a generalized weighted Kappa, giving more weight to the…”
MethodA general program for the calculation of the kappa coefficient
Strube, M. (1989) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03210592 · Cited by 4
“Over the past three decades, )( has been extended to allow assessment among K > 2 raters (e.g., Conger, 1980; Uebersax, 1982), to allow missing observations (e.g., Conger, 1980; Fleiss, 1971; Uebersax, 1982), to allow nonmutually exclusive response categories (i.e., subjects can be classified into…”
BackgroundLopes, C. D. L., Filho, C. A. D. R. C., Silva, T. A. D. L. E., Gonçalves, M. C. K., de Oliveira, R. L., & de Lima, P. R. G. (2014) · Revista Brasileira de Ortopedia · doi:10.1016/j.rbo.2014.01.022 · Cited by 4
Discovery and proof in attachment research
Grossmann, K. E., & Grossmann, K. (1984) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00026601 · Cited by 4
What do we learn from the Strange Situation?
Chess, S. (1984) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00026546 · Cited by 5
On forecasting validity and finessing reliability
Gilmore, J. B. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065821 · Cited by 4
Programs for the Collection and Analysis of Observational Data
Roberts, W. L. (1993) · Cited by 4
Toward openness and fairness in the review process
Rourke, B. P. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0006595x · Cited by 4
Rhetorical Moves in an Occluded Genre: A Qualitative Analysis of Suicide Notes
Abaalkhail, A. (2015) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2015-10831 · Cited by 12
Faletti, M. V., & Clark, M. C. (1984) · Springer eBooks · doi:10.1007/978-1-4613-2401-0_30 · Cited by 4
Measuring rater bias in diagnostic tests with ordinal ratings
Kim, C., Lin, X., & Nelson, K. P. (2021) · Statistics in Medicine · doi:10.1002/sim.9011 · Cited by 5
Pretreatment MRI versus postoperative pathology in locoregional staging of rectal cancer
Stelzner, S., Kittner, T., Brown, G., Kühn, M., Ruppert, R., Junginger, T., Juchems, M. S., Merkel, S., Fichtner‐Feigl, S., Weßling, J., et al. (2026) · Deutsches Ärzteblatt international · doi:10.3238/arztebl.m2025.0226 · Cited by 3
A rising tide: revisiting the occupational impact of AI in the generative era
Casas, P., Fernández‐Macías, E., Martínez‐Plumed, F., Gómez, E., González-Vázquez, I., & Salotti, S. (2026) · Structural Change and Economic Dynamics · doi:10.1016/j.strueco.2026.07.016 · Cited by 3
Schoonover, A., Eriksson, C. O., Nguyen, T. D., Meckler, G. D., Hansen, M. L., Harrod, T., & Guise, J. (2022) · Journal of the American College of Emergency Physicians Open · doi:10.1002/emp2.12726 · Cited by 4
The Homogeneity Index as a Measure of Interrater Agreement for Ratings on a Nominal Scale
Bove, G. (2023) · Studies in classification, data analysis, and knowledge organization · doi:10.1007/978-3-031-30164-3_2 · Cited by 3
Santos, P. S., Assunção, M. R. S., Reis, E. M. C., Silva, T. C. D., Terra, F. D. S., & Freitas, P. S. (2024) · Contribuciones a las Ciencias Sociales · doi:10.55905/revconv.17n.1-008 · Cited by 4
From tetrachoric to kappa: How to assess reliability on binary scales
Vanbelle, S. (2025) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.70021 · Cited by 4
An implicit enumeration method for an exact test of weighted kappa
Brusco, M. J., Stahl, S., & Steinley, D. (2007) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711007x227058 · Cited by 3
Andrés, A. M., & Hernández, M. Á. (2021) · Journal of Statistical Computation and Simulation · doi:10.1080/00949655.2021.2013485 · Cited by 4
“When many raters (R ≥ 2) exist, several kappa-based coefficients can be employed, such as Fleiss’ kappa [1], Hubert’s kappa [19], and Conger’s kappa [20].”
BackgroundMethodInfluentialKarakatsanis, A., Charalampoudis, P., Pistioli, L., Micco, R. D., Foukakis, T., Valachis, A., the SentiNot Trialists Group, Wärnberg, F., Bagge, R. O., Eriksson, S., et al. (2021) · British journal of surgery · doi:10.1093/bjs/znab149 · Cited by 3
Yozawitz, E. G., Cilio, M. R., Mizrahi, E. M., Moon, J., Moshé, S. L., Nunes, M. L., Plouin, P., Vanhatalo, S., Zuberi, S. M., & Pressler, R. (2023) · Epileptic Disorders · doi:10.1002/epd2.20005 · Cited by 3
“A multi-rater agreement was measured using Fleiss’ exact Kappa statistic [12] to assess the agreement on categorical responses to the five clinical videos among multiple ( n =194) raters beyond chance agreement.”
Díaz-Muñoz, P., & Maíz-Arévalo, C. (2024) · Journal of Language Aggression and Conflict · doi:10.1075/jlac.00116.dia · Cited by 3
Pathways to reading comprehension in Amharic vary by reading profile
de Azevedo, T. A., & Davidson, M. R. (2024) · Reading and Writing · doi:10.1007/s11145-024-10584-x · Cited by 5
“The agreement of the LPA model and the categorization using the 2015 benchmark was measured using Cohen’s (1960, 1968) kappa with its generalization by Conger (1980).”
Measuring Writing as a Representation of Disciplinary Knowledge
Burke, K. A., Ouellette, J. A., Miller, W. E., Leise, C., & Utschig, T. T. (2012) · Cited by 5
Lebani, G. E., & Pianta, E. (2010) · Institutional Research Information System (Università degli Studi di Trento) · Cited by 3
Metadata
- Authors
- Conger, A. J.
- Affiliations
- Purdue University West Lafayette (US)
- Corresponding author
- Conger, Anthony J.
- Venue
- Psychological Bulletin
- Published
- 1980-09
- Volume
- 88
- Issue
- 2
- Pages
- 322-328
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1978217317
- Cited by
- 545
- References
- 3
- Citations by year
- 2026: 20 · 2025: 24 · 2024: 35 · 2023: 24 · 2022: 23 · 2021: 23 · 2020: 25 · 2019: 25 · 2018: 22 · 2017: 16 · 2016: 18 · 2015: 27 · 2014: 29 · 2013: 15 · 2012: 15
- FWCI
- 7.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- Cohen's kappa, Fleiss' kappa, interobserver agreement, multiple raters, nominal scale
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.