Integration and generalization of kappas for multiple raters
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
J. A. Cohen's kappa (1960) for measuring agreement between 2 raters, using a nominal scale, has been extended for use with multiple raters by R. J. Light (1971) and J. L. Fleiss (1971). In the present article, these indices are analyzed and reformulated in terms of agreement statistics based on all
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“with interrater reliability measured using Cohen’s Kappa (κ = 0.63–0.82), indicating moderate to substantial agreement (Conger, 1980).”
From Rethinking pre-training: cognitive load implications for learners with varying prior knowledge (2025) · cites this paper as Conger, 1980
In the citing paper
“Furthermore, (25) provides an alternative proof that Conger's and Fleiss' quadratically weighted kappas converge to each other as the number of raters, 𝑅, tends to infinity (Conger, 1980;Gwet, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Conger, 1980
In the citing paper
“The average pairwise Pearson, Spearman, Kendall's τ b , ϕ, and MCC over the R 2 judge pairs are all equal, and coincide with κ F when all judges share the same marginal MET rate.”
From Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why (2026) · cites this paper as Conger (1980)
In the citing paper
“A generalization of Cohen’s kappa (i.e., Light’s Kappa [59]) was calculated to assess agreement between judges across their allocation of all items to domains.”
From Discriminant content validity of a theoretical domains framework questionnaire for use in implementation research (2014) · cites this paper as [59]
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Light, R. J. (1971) · Psychological Bulletin · doi:10.1037/h0031643 · Cited by 747
Cited by
Items 201–300 of 555
Degenerative findings in lumbar spine MRI: an inter-rater reliability study involving three raters
Doktor, K. K., Jensen, T. S., Christensen, H. W., Fredberg, U., Kindt, M. W., Boyle, E., & Hartvigsen, J. (2020) · Chiropractic & Manual Therapies · doi:10.1186/s12998-020-0297-0 · Cited by 25
Petrič, G., Cugmas, M., Petrič, R., & Atanasova, S. (2023) · Digital Health · doi:10.1177/20552076231155681 · Cited by 28
“Intercoder agreement measures the degree to which different coders respond in the same way to a certain item.(60) In the case of two coders, interrater agreement pertains to the proportion of messages coded with the same response.”
BackgroundThe reliability of clinical decisions based on the cervical vertebrae maturation staging method
Sohrabi, A., Ahari, S. B., Moslemzadeh, S. H., Rafighi, A., & Aghazadeh, Z. (2015) · European Journal of Orthodontics · doi:10.1093/ejo/cjv030 · Cited by 25
“Since the purpose of agreement studies is to investigate the inter-observer agreement of more than two observers, tests that are based on the agreement of only two observers, such as Kendall’s coefficient, can lead to inaccurate results (17, 18).”
ResultObserver error and its impact on ancestry estimation using dental morphology
Pilloud, M. A., Adams, D. M., & Hefner, J. T. (2018) · International Journal of Legal Medicine · doi:10.1007/s00414-018-1985-3 · Cited by 28
“Instead, we also report Conger’s kappa [40] as a modification of Fleiss’ kappa using the Brel^ package, with the option for linear weighting.”
MethodKnol, W., Keijsers, C. J. P. W., Jansen, P. A. F., Belitser, S. V., Schobben, A. F. A. M., Egberts, T. C. G., & van Marum, R. J. (2008) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.2091 · Cited by 25
The predictive validity of peer review: A neglected issue
Bornstein, R. F. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065717 · Cited by 26
Multiple‐rater kappas for binary data: Models and interpretation
Stoyan, D., Pommerening, A., Hummel, M., & Kopp‐Schneider, A. (2018) · Biometrical Journal · doi:10.1002/bimj.201600267 · Cited by 23
“Therefore, Hubert (1977), Conger (1980), and Schouten (1982) suggested a modification of 𝜅𝐹 where the term P 2 + (1 − P)2 is replaced by 𝑝𝑒, 𝑝𝑒 = 2 𝑟(𝑟 − 1) 𝑟∑ 𝑖=1,𝑗>𝑖 (P𝑖P𝑗 + (1 − P𝑖)(1 − P𝑗)).”
BackgroundMethodInfluentialOn the Usefulness of Interrater Reliability Coefficients
ten Hove, D., Jorgensen, T. D., & van der Ark, L. A. (2018) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-319-77249-3_6 · Cited by 29
“Three generalizations of κ were available to assess nominal data with more than two raters: Fleiss’ kappa (κFleiss; Fleiss 1971), Conger’s exact kappa (κExact; Conger 1980), and Light’s kappa (κLight; Light 1971).”
MethodBetter to be in agreement than in bad company: A critical analysis of many kappa-like tests
Silveira, P. S. P., & Siqueira, J. D. O. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01950-0 · Cited by 28
Kim, D. S., Jeong, J., Kim, J., Cho, Y., Park, I., Lee, S.-M., Oh, Y. T., Baek, S., Kang, D., Lee, E., et al. (2023) · Journal of Korean Medical Science · doi:10.3346/jkms.2023.38.e322 · Cited by 23
Bayesian random effects for interrater and test–retest reliability with nested clinical observations
Hsiao, C. K., Chen, P., & Kao, W.-H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.10.015 · Cited by 23
Plant, J. D., Neradelik, M. B., Polissar, N. L., Fadok, V. A., & Scott, B. A. (2014) · Veterinary Dermatology · doi:10.1111/vde.12104 · Cited by 23
“Light’s kappa (the average of all pairwise kappas, also denoted by j in this text) was calculated for the overall pairwise agreement between all four laboratories for each dog.(25) For kappa calculations, only antigens or antigen groups common to each laboratory in the comparison could be included in the analyses.”
MethodClassifying Mobile Applications Using Word Embeddings
Ebrahimi, F., Tushev, M., & Mahmoud, A. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3474827 · Cited by 27
Hidaka, S., Lee, Y., Nakanishi, M., Wakamiya, K., Nakagawa, T., & Kaburagi, T. (2022) · Journal of Voice · doi:10.1016/j.jvoice.2022.10.020 · Cited by 26
McFatrich, M., Brondon, J., Lucas, N. R., Hinds, P. S., Maurer, S. H., Mack, J. W., Freyer, D. R., Jacobs, S. S., Baker, J. N., Mowbray, C., et al. (2019) · Cancer · doi:10.1002/cncr.32525 · Cited by 21
Choi, N., Chengalur-Smith, I. N., & Nevo, S. (2015) · Journal of the Association for Information Systems · doi:10.17705/1jais.00405 · Cited by 33
When nonreliability of reviews indicates solid science
Eckberg, D. L. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065791 · Cited by 24
Referencing practices in physical geography: How well do we cite what we write?
Haussmann, N. S., McIntyre, T., Bumby, A. J., & Loubser, M. J. (2013) · Progress in Physical Geography: Earth and Environment · doi:10.1177/0309133313482135 · Cited by 20
“We also tested the overall agreement between all four raters using an Exact Fleiss’ Kappa statistic (Conger, 1980; Fleiss, 1971), and agreement was assessed according to the classes proposed by Viera and Garrett (2005).”
MethodDeveloping and Applying an Information Literacy Rubric to Student Annotated Bibliographies
Rinto, E. E. (2013) · Evidence Based Library and Information Practice · doi:10.18438/b8559f · Cited by 20
“Fleiss’s kappa and Conger’s kappa modify Cohen’s kappa to allow for agreement between multiple raters (Stemler, 2004; Oakleaf, 2009; Fleiss, 1971; Conger, 1980; Gwet, 2010).”
BackgroundAn exploratory study to investigate the impact of conceptualization in god class detection
Santos, J. A. M., Mendonça, M., & Paradis, C. V. (2013) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/2460999.2461007 · Cited by 22
Rating experiments in forestry: How much agreement is there in tree marking?
Pommerening, A., Ramos, C. P., Kędziora, W., Haufe, J., & Stoyan, D. (2018) · PLoS ONE · doi:10.1371/journal.pone.0194747 · Cited by 21
Drake, A., Perković, A., Reeve, C., Alexander, S. M., Nguyen, V. M., & Dunmall, K. M. (2022) · FACETS · doi:10.1139/facets-2021-0067 · Cited by 20
Ishii, R., Otsuka, K., Kumano, S., Higashinaka, R., & Tomita, J. (2019) · Multimodal Technologies and Interaction · doi:10.3390/mti3040070 · Cited by 30
“We calculated the Conger’s kappa coefficient [40] to evaluate an inter-coder agreement of the three annotators.”
MethodHsu, D., Rath, T. J., Branstetter, B. F., Anzai, Y., Phillips, C. D., Juliano, A. F. T., Mosier, K. M., Bazylewicz, M., Poliashenko, S. M., Kulzer, M. H., et al. (2021) · Radiology Imaging Cancer · doi:10.1148/rycan.2021200131 · Cited by 20
“Interrater agreement for prose descriptions (ie, lexicon responses) was assessed using unweighted Light generalization of Cohen k measure of interrater reliability, which is based on the mean of the pairwise Cohen k measures of agreement between the eight raters on the scoring of the 80 patients (17,18).”
Methodde Mast, J., Akkerhuis, T. S., & Erdmann, T. P. (2013) · Quality Engineering · doi:10.1080/08982112.2013.846062 · Cited by 19
“…statistic is bPA normalized such that ĵ¼1 for perfect agreement and ĵ¼0 for chance ratings: ĵ ¼ bPA P̂Ajchance 1 P̂Ajchance ¼ 0:639: Alternative definitions have been proposed for the case of more than two appraisers and different situations (e.g., Conger 1980; Fleiss 1971; and many others).”
BackgroundMethodKaiser, I., Pfahlberg, A. B., Mathes, S., Uter, W. J. C., Diehl, K., Steeb, T., Heppt, M. V., & Gefeller, O. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12051976 · Cited by 19
“We also reported values of Cohen’s kappa (κ) [26] and Conger’s κ [27] to ensure comparability with other studies.”
MethodResultFinch, L. A., Broach, V. A., Feinberg, J., Al‐Niaimi, A., Abu‐Rustum, N. R., Zhou, Q. C., Iasonos, A. E., & Dennis, S. (2024) · Gynecologic Oncology · doi:10.1016/j.ygyno.2024.07.007 · Cited by 20
Magnitude of Task-Sampling Variability in Performance Assessment: A Meta-Analysis
Huang, C. (2009) · Educational and Psychological Measurement · doi:10.1177/0013164409344550 · Cited by 19
A measure of association for ordered categorical data in population-based studies
Nelson, K. P., & Edwards, D. (2016) · Statistical Methods in Medical Research · doi:10.1177/0962280216643347 · Cited by 19
Patient-Centered Wound Teleconsultation for Cutaneous Wounds: A Feasibility Study
Chen, C.-H., Young, T.-H., Huang, C.-H., Chang, H.-H., Chen, C.-L., Chien, H.-F., Chen, J.-S., Lai, H.-S., & Cheng, N.-C. (2014) · Annals of Plastic Surgery · doi:10.1097/sap.0000000000000031 · Cited by 18
“The Light’s J is an index of interrater agreement between the remote raters on categorical data, and it was used to estimate whether 2 or more health care workers’ diagnostic results are consistent.(12,13) The agreement from more than 2 of 3 remote surgeons would be treated as the conclusion of image interpretation.”
MethodCorrected Zegers-ten Berge Coefficients Are Special Cases of Cohen’s Weighted Kappa
Warrens, M. J. (2014) · Journal of Classification · doi:10.1007/s00357-014-9156-9 · Cited by 18
“If we use the weights in (8) in (7) we obtain the unweighted kappa for multiple raters that was first considered in Hubert (1977, p. 296, 297) and has been independently proposed by Conger (1980).”
BackgroundMethodA Garbage Classification Method Based on a Small Convolution Neural Network
Yang, Z., Xia, Z., Yang, G., & Lv, Y. (2022) · Sustainability · doi:10.3390/su142214735 · Cited by 18
Fabijan, A., Zawadzka-Fabijan, A., Fabijan, R., Zakrzewski, K. M., Nowosławska, E., & Polis, B. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13144013 · Cited by 22
“Fleiss’ kappa statistics measured inter-rater reliability for categorical variables [37–39].”
MethodTemporal Classification of Medical Events
Raghavan, P., Fosler‐Lussier, E., & Lai, A. M. (2012) · North American Chapter of the Association for Computational Linguistics · Cited by 18
Diagnostic performance evaluation of hepatitis B e antigen rapid diagnostic tests in Malawi
Stockdale, A. J., Silungwe, N., Shawa, I. T., Kreuels, B., Gordon, M. A., & Geretti, A. M. (2021) · BMC Infectious Diseases · doi:10.1186/s12879-021-06134-3 · Cited by 19
“Inter-observer agreement was calculated using the Cohen’s kappa [22].”
MethodInterrater Reliability of Needle Electromyographic Findings in Lumbar Radiculopathy
Chouteau, W. L., Annaswamy, T. M., Bierner, S. M., Elliott, A. C., & Figueroa, I. A. (2010) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0b013e3181e29a3b · Cited by 18
“The kappas were also averaged over all muscles to indicate a summarized measure of agreement.(18) The diagnostic impression was analyzed with expanded and combined variables as well (expanded variables: 0 no evidence of radiculopathy, 1 possibility of radiculopathy, 2 clear evidence of radiculopathy; combined variable…”
MethodReflections on the peer review process
Marsh, H. W., & Ball, S. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065912 · Cited by 18
Reliability and Expected Loss: A Unifying Principle
Cooil, B., & Rust, R. T. (1994) · Psychometrika · doi:10.1007/bf02295184 · Cited by 18
“Measures have been proposed that average pairwise or multiple-judge indices of reliability across all judge subgroups of a given size (e.g., Conger, 1980 ).”
BackgroundEstimating Interinterviewer Reliability for Interview Schedules used in Special Education Research
Goodwin, L. D., Sands, D. J., & Kozleski, E. B. (1991) · The Journal of Special Education · doi:10.1177/002246699102500105 · Cited by 19
Analysis of the Variables Influencing Success in Elite Handball with Polar Coordinates
Font, R., Sobrino, G. D., Irurtia, A., Tremps, V., Roca, M. C., Mesas, J. A., & Iglesias, X. (2022) · Sustainability · doi:10.3390/su142315542 · Cited by 17
Weber, U., Maksymowych, W. P., Chan, S. M., Rufibach, K., Pedersen, S. J., Zhao, Z., Zubler, V., Østergaard, M., & Lambert, R. G. W. (2015) · Arthritis Research & Therapy · doi:10.1186/s13075-015-0729-8 · Cited by 20
The effect of an e-learning module on grading variation of (pre)malignant breast lesions
van Dooijeweert, C., Deckers, I. A. G., de Ruiter, E. J., ter Hoeve, N. D., Vreuls, C. P. H., van der Wall, E., & van Diest, P. J. (2020) · Modern Pathology · doi:10.1038/s41379-020-0556-6 · Cited by 17
Glasauer, C. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111548 · Cited by 20
Hierarchical modeling of agreement
Vanbelle, S., Mutsvari, T., Declerck, D., & Lesaffre, E. M. E. H. (2012) · Statistics in Medicine · doi:10.1002/sim.5424 · Cited by 19
“This method easily extends to the case of R > 2 fixed examiners, as using the 2-wise definition of agreement introduced by Conger and David and Fleiss [4, 6].”
MethodInterpersonal control in psychotherapy: A comparison of two definitions
Tracey, T. J. G., & Miars, R. D. (1986) · Journal of Clinical Psychology · doi:10.1002/1097-4679(198607)42:4<585::aid-jclp2270420407>3.0.co;2-l · Cited by 17
Hubert's multi‐rater kappa revisited
Martín Andrés, A., & Álvarez Hernández, M. (2020) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12167 · Cited by 16
An instrument to assess the statistical intensity of medical research papers
Nieminen, P. A., Virtanen, J. I., & Vähänikkilä, H. (2017) · PLoS ONE · doi:10.1371/journal.pone.0186882 · Cited by 17
Modification of an extended total temporomandibular joint replacement (eTMJR) classification system
Higginson, J. A., Panayides, C., Speculand, B., Mercuri, L. G., & Elledge, R. O. C. (2022) · British Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.bjoms.2022.03.011 · Cited by 16
Kashkoush, A. I., Potter, T. O., Petitt, J. C., Hu, S., Hunter, K., & Kelly, M. L. (2022) · Journal of neurosurgery · doi:10.3171/2022.6.jns212921 · Cited by 16
Jiang, L., Chen, X., Miao, S., & Shi, C. (2023) · Decision Support Systems · doi:10.1016/j.dss.2023.114148 · Cited by 18
Schwanck, A. A., & Ponte, E. M. D. (2016) · Plant Pathology · doi:10.1111/ppa.12526 · Cited by 16
“…between quadrat-count and distance-based methods Agreement between and among results by the two spatial methods was studied using two indices on categorical data, unweighted Cohen’s kappa for testing agreement of two methods and the Fleiss’ kappa for multiple methods (Cohen, 1960; Conger, 1980).”
MethodConfusion between reviewer reliability and wise editorial and funding decisions
Kiesler, C. A. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065869 · Cited by 16
Reflections from the peer review mirror
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00066024 · Cited by 16
Toward improved inference for Krippendorff’s Alpha agreement coefficient
Hughes, J. (2024) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2024.106170 · Cited by 16
“Conger (1980) and Davies and Fleiss (1982) likewise generalized κ to the multi-coder setting.”
Elledge, R. O. C., Higginson, J. A., Mercuri, L. G., & Speculand, B. (2020) · British Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.bjoms.2020.10.020 · Cited by 15
Estimating intercoder reliability: a structural equation modeling approach
Feng, G. C. (2014) · Quality & Quantity · doi:10.1007/s11135-014-0034-7 · Cited by 15
“Agreement among all the seven raters was around .51 based on the indices of Krippendorff (2004)’s α, Fleiss (1971)’s π , Light (1971)’s κ and Conger (1980)’s κ .”
MethodJaudi, S., du Montcel, S. T., Fries, N., Nizard, J., Fontaines, V. H. D., & Dommergues, M. (2011) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.8941 · Cited by 15
Himes‐Cornell, A., Maguire, C., Kasperski, S., Hoelting, K. R., & Pollnac, R. Β. (2016) · Ocean & Coastal Management · doi:10.1016/j.ocecoaman.2016.02.004 · Cited by 15
“As with the previous test, if at least one test result was statistically significant then the kappa statistics from each test for that community were averaged to create a single composite kappa (Conger,1980).”
MethodDuchmann, M., Wagner‐Ballon, O., Boyer, T., Cheok, M. H.-C., Fournier, É., Guérin, E., Fenwarth, L., Badaoui, B., Freynet, N., Benayoun, E., et al. (2021) · Leukemia · doi:10.1038/s41375-021-01435-7 · Cited by 17
On Measuring Agreement among Several Judges on the Presence or Absence of a Trait
Rae, G. (1984) · Educational and Psychological Measurement · doi:10.1177/0013164484442007 · Cited by 15
Durability of “Posthypnotic Suggestions” as a Function of Type of Suggestion and Trance Depth
Berrigan, L. P., Kurtz, R. M., Stabile, J. P., & Strube, M. (1991) · International Journal of Clinical and Experimental Hypnosis · doi:10.1080/00207149108409616 · Cited by 16
Can You be More Social? Injecting Politeness and Positivity into Task-Oriented Conversational AgentsPreprint
Wang, Y., Papangelis, A., Wang, R., Feizollahi, Z., Tür, G., & Kraut, R. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.14653 · Cited by 15
“We measured the inter-rater agreement ( kappa ) among the three crowdworkers for each question and both tasks [13, 40, 15].”
MethodRethinking pre-training: cognitive load implications for learners with varying prior knowledge
Gorbunova, A., Kapuza, A., Chen, O., & Costley, J. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1628047 · Cited by 14
A new coefficient of interrater agreement: The challenge of highly unequal category proportions.
van Oest, R. (2019) · Psychological Methods · doi:10.1037/met0000183 · Cited by 14
“Though lacking a natural extension to more than two raters, it is feasible to take the average of kappa values across all pairs of raters (Conger, 1980).”
Identifying Annotator Bias: A new IRT-based method for bias identification
Amidei, J., Piwek, P., & Willis, A. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.421 · Cited by 14
“The variety criterion reaches a high Conger’s κ agreement coefficient of 0.93.”
MethodInfluentialAndrés, A. M., & Hernández, M. Á. (2024) · Advances in Data Analysis and Classification · doi:10.1007/s11634-024-00581-x · Cited by 16
“The extension R-wise κ HR of κ C can be seen in Conger (1980), Shuster and Smith (2005) and Martín Andrés and Álvarez Hernández (2020).”
InfluentialKendall, T. S., Vaughn, C. R., Farrington, C., Gunter, K., McLean, J., Tacata, C., & Arnson, S. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.648543 · Cited by 22
“Overall, a measure of inter-analyst agreement using Fleiss’ Kappa for multiple raters (Conger, 1980) yields a k = 0.39 with significantly better agreement than chance for each of the three categories (-in: k = 0.38, -ing: k = 0.52; DC: k = 0.22).”
MethodInfluentialBusiness Decision Support System based on Sentiment Analysis
Oppong, S. O., Asamoah, D., Oppong, E., & Lamptey, D. (2019) · International Journal of Information Engineering and Electronic Business · doi:10.5815/ijieeb.2019.01.05 · Cited by 14
“Kappa, Krippendoff alpha and among many other measures are used in the computation of inter-rater agreement (IRA).”
BackgroundMethodInfluentialDana, J., Gauvin, S., Zhang, M., Lotero-Robledo, J. G., Cassim, C., Artho, G. P., Bhatnagar, S., Tanguay, S., & Reinhold, C. (2022) · European Radiology · doi:10.1007/s00330-022-09082-x · Cited by 14
“Inter-reader agreement for morphological features and Bosniak categories was calculated using Light’s Kappa [13].”
MethodAnalysis of Nonagreements among Multiple Raters
James, I. R. (1983) · Biometrics · doi:10.2307/2531092 · Cited by 14
“Measures of agreement among the raters, often in terms of indices called 'kappa' statistics, have received considerable attention in the literature in recent years; see, for example, Light (1971), Fleiss (1971), Landis and Koch (1977a, b, c), Hubert (1977), Fleiss, Nee and Landis (1979), Conger (1980), Kraemer (1980),…”
BackgroundASAVACT: Arabic sentiment analysis for vaccine-related COVID-19 tweets using deep learning
Alhumoud, S. O., Wazrah, A. A. A., Alhussain, L., Alrushud, L., Aldosari, A., Altammami, R. N., Almukirsh, N., Alharbi, H., & Alshahrani, W. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1507 · Cited by 13
“…label probability is the total number of each label being labeled by the first language expert divided by the total number of labeled tweets, multiplied by the total number of the same label being labeled by the second language expert divided by the total number of labeled tweets ( Conger, 1980 ).”
BackgroundA COMPARISON OF COHEN'S KAPPA AND AGREEMENT COEFFICIENTS BY CORRADO GINI
Warrens, M. J. (2013) · Leiden Repository (Leiden University) · Cited by 17
O’Fallon, T., Polissar, N. L., Neradilek, M. B., & Murray, T. (2020) · Heliyon · doi:10.1016/j.heliyon.2020.e03472 · Cited by 13
“For paired ratings of each inventory, we used a weighted Cohen's Kappa statistic, and for multiple raters scoring each inventory, we used the weighted Light's Kappa statistic (Conger, 1980).”
MethodCui, Y., Peng, L., Hu, Y., & Lai, H. J. (2021) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/rssc.12497 · Cited by 13
“…approach which separately evaluates the agreement in terms of zero– non- zero classifications (i.e. absence vs. presence) using Iota coefficient (Conger, 1980; Janson & Olsson, 2001) and the agreement in non- zero components via a subcomponent CCC constructed based on the same rationale for…”
MethodLearning to Temporally Order Medical Events in Clinical Text
Raghavan, P., Lai, A. M., & Fosler‐Lussier, E. (2012) · Cited by 13
Awuni, D. E. K., Montero, C., & Sutinen, E. (2016) · International Conference on Advanced Learning Technologies · doi:10.1109/icalt.2016.21 · Cited by 20
Berry, K. J., Johnston, J. E., Mielke, P. W., & Johnston, L. (2018) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.1429 · Cited by 13
“A number of researchers have advocated chance-corrected measures of effect size, including Brennan and Prediger (1981), Cicchetti, Showalter, and Tyrer (1985), Conger (1980), and Krippendorff (1970).”
BackgroundDeterminants of dentists' decisions to initiate dental implant treatment: A judgment analysis
Koele, P., & Hoogstraten, J. (1999) · Journal of Prosthetic Dentistry · doi:10.1016/s0022-3913(99)80017-1 · Cited by 13
A family of multi-rater kappas that can always be increased and decreased by combining categories
Warrens, M. J. (2011) · Statistical Methodology · doi:10.1016/j.stamet.2011.08.008 · Cited by 13
“Cohen’s κ has also been extended to the important case of multiple raters (Hubert, 1977; Conger 1980; Von Eye & Mun, 2006; Mielke, Berry & Johnston, 2007, 2008; Warrens, 2010d).”
BackgroundMethodInfluentialAgreement between raters and groups of raters
Vanbelle, S. (2009) · Cited by 26
Jacobsen, K., Ortner, V. K., Wenande, E. C., Fredman, G. L., Untracht, G. R., Wolswijk, T., Crüts, E. C., Mosterd, K., Nielsen, K., Philipsen, P. A., et al. (2024) · Archives of Dermatological Research · doi:10.1007/s00403-024-03344-y · Cited by 12
“1 ), expressed as the Conger’s kappa (κ) coefficient (a generalization of Cohen’s kappa for multiple evaluators) [30].”
BackgroundMcKelvey, B. A., Torres-Saavedra, P. A., Li, J., Broeckx, G., Deman, F., Ali, S. M., Andrews, H. S., Arslan, S., Azulay, M., Balasubramanian, S., et al. (2026) · Modern Pathology · doi:10.1016/j.modpat.2025.100944 · Cited by 15
Glennie, A., Noonan, V. K., Fallah, N., Park, S. E., Thorogood, N. P., Cheung, A., Fisher, C. G., Dvorak, M. F., & Street, J. (2014) · Spinal Cord · doi:10.1038/sc.2014.116 · Cited by 12
Non-Expert Correction of Automatically Generated Relation Annotations
Gormley, M. R., Gerber, A., Harper, M. P., & Dredze, M. H. (2010) · North American Chapter of the Association for Computational Linguistics · Cited by 12
Assessing agreement between raters from the point of coefficients and loglinear models
Yılmaz, A. E., Saraçbaşı, T., & Saraçbaşı, T. (2021) · Journal of Data Science · doi:10.6339/jds.201701_15(1).0001 · Cited by 14
Some common errors of experimental design, interpretation and inference in agreement studies
Erdmann, T. P., de Mast, J., & Warrens, M. (2012) · Statistical Methods in Medical Research · doi:10.1177/0962280211433597 · Cited by 14
“If one selects a sample in which the numbers of subjects in the different classes are not representative for the study population, P̂A will be biased, because PA depends on the class prevalences p(l) (per Equation (4)), unless PA(l) is equal for all values of l.”
BackgroundInfluentialREFORMULATION AND GENERALISATION OF THE COHEN AND FLEISS KAPPAS
Xie, Z., Gadepalli, C., & Cheetham, B. M. G. (2017) · LIFE International Journal of Health and Life-Sciences · doi:10.20319/lijhls.2017.33.115 · Cited by 12
Measures of Agreement Versus Measures of Prediction Accuracy
Haberman, S. J. (2019) · ETS Research Report Series · doi:10.1002/ets2.12258 · Cited by 12
“Common practice has often involved probabilities of exact agreement or exact or adjacent agreement (Goodman & Kruskal, 1954) or measures from the kappa family (Cicchetti & Allison, 1971; Cohen, 1960, 1968; Conger, 1980; Fleiss & Cohen, 1973; Fleiss, Cohen, & Everitt, 1969); however, related but somewhat different meas…”
BackgroundMethodInfluentialHargens, L. L. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065857 · Cited by 12
Lind, C. M. (2017) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · doi:10.13140/rg.2.2.20231.91044 · Cited by 12
Nissen, T. P. H., Nørgaard, T. L., Schielke, K. C., Vestergaard, P., Nikontovic, A., Dawidowicz, M., Grauslund, J., Vorum, H., & Aasbjerg, K. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13071128 · Cited by 13
“For the intergrader variability correlation between the three reference graders’ gradings, the variability was calculated using Conger’s Exact Kappa [27] method for multiple graders.”
MethodInfluentialLee, W., Lin, J., & Bao, S. S. (2020) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2020.103025 · Cited by 11
Obuchowski, N. A. (2001) · Statistics in Medicine · doi:10.1002/sim.929 · Cited by 11
“Conger [21] showed that Light’s [17] summary measure of agreement for studies involving multiple readers of a single test or multiple tests from a single reader reduces to the average of the pairwise kappa statistics.”
BackgroundDo we really want more “reliable” reviewers?
KRAEMER, H. C. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065870 · Cited by 14
Disagreement among journal reviewers: No cause for undue alarm
Stricker, L. J. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065985 · Cited by 12
Tobacco policy rating form: a tool for evaluating worksite and tribal smoking control policies
Glasgow, R. E., Boles, S. M., Lichtenstein, E. H., & Strycker, L. A. (1996) · Tobacco Control · doi:10.1136/tc.5.4.286 · Cited by 11
The Impact of Dating Apps on the Mental Health of the LGBTIQA+ Population
Perić, L., Rabar, M., & Orehovački, T. (2025) · Multimodal Technologies and Interaction · doi:10.3390/mti9040030 · Cited by 11
“The reliability of results obtained through the closed card sorting method was measured using Conger’s Kappa coefficient of agreement [87].”
Multicolor Imaging for Detection of Retinal Nerve Fiber Layer Defect in Myopic Eyes With Glaucoma
Kim, Y. H., Ahn, J., & Kim, K. E. (2021) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2021.07.022 · Cited by 10
Raghavan, P., Fosler‐Lussier, E., & Lai, A. M. (2012) · North American Chapter of the Association for Computational Linguistics · Cited by 10
Positron Emission Mammography Image Interpretation for Reduced Image Count Levels
MacDonald, L. R., Hippe, D. S., Bender, L. C., Cotter, E. W., Voria, P. R., Hallam, P. S., Wang, C. L., Haseley, D. R., Kelly, M. M., Parikh, J. R., et al. (2015) · Journal of Nuclear Medicine · doi:10.2967/jnumed.115.165787 · Cited by 10
“Light’s k was used to summarize interinterpreter agreement (20).”
Method
Metadata
- Authors
- Conger, A. J.
- Affiliations
- Purdue University West Lafayette (US)
- Corresponding author
- Conger, Anthony J.
- Venue
- Psychological Bulletin
- Published
- 1980-09
- Volume
- 88
- Issue
- 2
- Pages
- 322-328
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1978217317
- Cited by
- 545
- References
- 3
- Citations by year
- 2026: 20 · 2025: 24 · 2024: 35 · 2023: 24 · 2022: 23 · 2021: 23 · 2020: 25 · 2019: 25 · 2018: 22 · 2017: 16 · 2016: 18 · 2015: 27 · 2014: 29 · 2013: 15 · 2012: 15
- FWCI
- 7.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- Cohen's kappa, Fleiss' kappa, interobserver agreement, multiple raters, nominal scale
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.