Integration and generalization of kappas for multiple raters
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
J. A. Cohen's kappa (1960) for measuring agreement between 2 raters, using a nominal scale, has been extended for use with multiple raters by R. J. Light (1971) and J. L. Fleiss (1971). In the present article, these indices are analyzed and reformulated in terms of agreement statistics based on all
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“with interrater reliability measured using Cohen’s Kappa (κ = 0.63–0.82), indicating moderate to substantial agreement (Conger, 1980).”
From Rethinking pre-training: cognitive load implications for learners with varying prior knowledge (2025) · cites this paper as Conger, 1980
In the citing paper
“Furthermore, (25) provides an alternative proof that Conger's and Fleiss' quadratically weighted kappas converge to each other as the number of raters, 𝑅, tends to infinity (Conger, 1980;Gwet, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Conger, 1980
In the citing paper
“The average pairwise Pearson, Spearman, Kendall's τ b , ϕ, and MCC over the R 2 judge pairs are all equal, and coincide with κ F when all judges share the same marginal MET rate.”
From Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why (2026) · cites this paper as Conger (1980)
In the citing paper
“A generalization of Cohen’s kappa (i.e., Light’s Kappa [59]) was calculated to assess agreement between judges across their allocation of all items to domains.”
From Discriminant content validity of a theoretical domains framework questionnaire for use in implementation research (2014) · cites this paper as [59]
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Light, R. J. (1971) · Psychological Bulletin · doi:10.1037/h0031643 · Cited by 747
Cited by
Items 501–555 of 555
Logez, M. (2010) · HAL · Cited by 6
Evaluating the reliability of four-dimensional computed tomography scans of the wrist
Chang, N. B. (2018) · Memorial University Research Repository (Memorial University)
Quality improvement of biomarker assessment in breast cancer
van Dooijeweert, C. (2020) · Utrecht University Repository (Utrecht University) · doi:10.33540/9
Wetter, C. V., Brandt, D., Cambier, E., Lepore, D., Tinton, N., & Tassin, O. (2021) · Journal de Pharmacie Clinique · doi:10.1684/jpc.2020.0452
Accordo assoluto tra valutazioni espresse su scala ordinale
Bove, G., & Marella, D. (2021) · Journal of Educational Cultural and Psychological Studies (ECPS Journal) · doi:10.7358/ecps-2021-023-boma
Cui, Y., Peng, L., Hu, Y., Sudakaran, S., Murali, S., & Lai, H. (2021) · Journal of Cystic Fibrosis · doi:10.1016/s1569-1993(21)01907-x
484: A 20-year longitudinal cohort study of the cystic fibrosis microbiome
Acosta, N., Thornton, C., Somayaji, R., Surrette, M., Rabin, H. R., & Parkins, M. D. (2021) · Journal of Cystic Fibrosis · doi:10.1016/s1569-1993(21)01908-1
Arolt, V. (1997) · Monographien aus dem Gesamtgebiete der Psychiatrie · doi:10.1007/978-3-642-60850-6_7
Berry, K. J., Johnston, J. E., & Mielke, P. W. (2018) · Springer eBooks · doi:10.1007/978-3-319-98926-6_6
Introduction to Data Manipulation Using R
Falissard, B. (2011) · Chapman & Hall/CRC eBooks · doi:10.1201/b11190-12
Popping, R. (2019) · Springer eBooks · doi:10.1007/978-3-030-11671-2_3
Measurement Systems Analysis, Attribute
Borror, C. M. (2014) · Wiley eBooks · doi:10.1002/9781118445112.stat04139
Popping, R. (2019) · Springer eBooks · doi:10.1007/978-3-030-11671-2_4
Excellence bias related to rating scales with summative jury assessment
Corradi, D. (2022) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2022.2112653
“in R to calculate Fleiss-Conger’s exact kappa (Conger 1980) to get an index of inter-rater agreement between the raters (jury members) on categorical data, Cronbach’s alpha and the intraclass correlation (ICC).”
MethodDucher, A., Saliba‐Serre, B., Renaud, L., Foti, B., & Verna, É. (2023) · Bulletins et Mémoires de la Société d anthropologie de Paris · doi:10.4000/bmsap.11815
Domain Specific Analysis of Privacy Practices and Concerns in the Mobile Application Market
Meymand, F. E. (2023) · Louisiana State University · doi:10.31390/gradschool_dissertations.6099 · Cited by 1
Post-Ureteroscopic Lesion Scale to determine ureteral wall damage, not so easy to employ
Polo, R., Nebot, À. C., Caballero-Romeu, J. P., Caballero, P., Galán-Llopis, J. A., Soria, F., de la Cruz, J. E., & Tuells, J. (2023) · Actas Urológicas Españolas (English Edition) · doi:10.1016/j.acuroe.2023.10.003
Using Third-Party Components’ Metadata to Analyze Cross-cutting Concerns
Carvalho, L. P. D. S., Mendes, T. S., Gomes, F., Freire, S., Novais, R. L., & Mendonça, M. (2024) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2024.3086
“In this case, applying Cohen’s Kappa to deal with multiple raters (Conger, 1980; Berry and Mielke Jr, 1988) might mitigate such a threat.”
MethodLyness, S. A. (2024) · Education Sciences · doi:10.3390/educsci14030300
Options, Problems and Guidelines for Measuring Interrater Agreement - a Descriptive Approach
Kočišová, L. (2024) · Czech Sociological Review · doi:10.13060/csr.2024.007
“…dvoch posudzovateľov pri nominálnych či ordinálnych premenných), exaktný koeficient kappa κ Exact (ge-neralizácia koeficientu kappa pre m posudzovateľov; Conger, 1980), kappa κ Light (použiteľný pre viacerých ako dvoch posudzovateľov a kategorické dáta; je prie-merom všetkých možných kombinácií…”
Aesthetic heuristics for design: cognitive and sociocultural determinants
Cadavid, A., & Maya, J. (2024) · Kepes · doi:10.17151/kepes.2024.21.30.8
“This statistic allows evaluating the agreement between more than two evaluators on categorical data to be evaluated (Conger, 1980).”
Yılmaz, A. E. (2025) · Journal of Polytechnic · doi:10.2339/politeknik.1568563
Stockman, T. I., Kowaleski, M. P., Hicks, J. M., Berns, C. N., Saunders, W. B., & McCarthy, R. J. (2025) · Frontiers in Veterinary Science · doi:10.3389/fvets.2025.1577046
Campos, G. S., Chini, L. T., & Freitas, P. S. (2025) · Arquivos de Ciências da Saúde da UNIPAR · doi:10.25110/arqsaude.v29i2.2025-12265
Reliability And Validity of Bedside Oral Exam Tool’s Turkish Version: Methodological Study
Aktan, G. G., Eşer, İ., Baysal, E., Uyar, M., & Yamaç, C. (2025) · İzmir Katip Çelebi Üniversitesi Sağlık Bilimleri Fakültesi Dergisi · doi:10.61399/ikcusbfd.1673441
Ohyama, T. (2025) · Japanese Journal of Biometrics · doi:10.5691/jjb.46.75
D’hondt, L., Buytaert, D., Kellens, P., Snoeckx, A., & Bacher, K. (2025) · European Radiology Experimental · doi:10.1186/s41747-025-00644-4
Heltborg, A., Mogensen, C. B., Skjøt‐Arkil, H., Giebner, M., Al-Masri, A., Khatry, U. B., Khatry, S., Heinemeier, I. I. K., Andreasen, J. J., Hariesh, S. S. S., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26758550
Heltborg, A., Mogensen, C. B., Skjøt‐Arkil, H., Giebner, M., Al-Masri, A., Khatry, U. B., Khatry, S., Heinemeier, I. I. K., Andreasen, J. J., Hariesh, S. S. S., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26517904
Heltborg, A., Mogensen, C. B., Skjøt‐Arkil, H., Giebner, M., Al-Masri, A., Khatry, U. B., Khatry, S., Heinemeier, I. I. K., Andreasen, J. J., Hariesh, S. S. S., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26517853
Heltborg, A., Mogensen, C. B., Skjøt‐Arkil, H., Giebner, M., Al-Masri, A., Khatry, U. B., Khatry, S., Heinemeier, I. I. K., Andreasen, J. J., Hariesh, S. S. S., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26758553
Stockdale, A. J., Silungwe, N., Shawa, I. T., Kreuels, B., Gordon, M. A., & Geretti, A. M. (2021) · Figshare · doi:10.6084/m9.figshare.14691685
時間周波数表現と深層学習を用いた病的音声の声質自動評価に関する研究
Hidaka, S. (2023) · Institutional Repositories DataBase (IRDB)
Torsiello, B., Giammarino, M., Quatto, P., Battini, M., Mattiello, S., Battaglini, L., & Renna, M. (2026) · Animals · doi:10.3390/ani16040546
“However, as already pointed out for two raters [6,9], also in the presence of three raters, the agreement indices belonging to the Kappa statistic (Fleiss’ K [12], Light’s K [15], Hubert’s K [16], and Conger’s K [14]) can be affected by the paradox behaviour [17].”
Rater analysis using Item Response Theory in a clinical context
Kabic, M. (2025) · University of Klagenfurt
Prébot, B. (2020) · theses.fr (ABES) · doi:10.70675/6692230dzcb48z408fz943dz4c456033fdf3 · Cited by 7
Origin and pronostic impact of intra-leukemic heterogeneity in acute myeloid leukemias
Duchmann, M. (2022) · theses.fr (ABES) · doi:10.70675/74f56644z3676z489bz8e12z27fa072401b7
Maturité objective des poissons par l'histologie quantitative
Sauger, C. (2024) · theses.fr (ABES) · doi:10.70675/9fb920d6zb2efz4341z9ab7z446677e0aae4 · Cited by 1
Boluda, N. B., Torralba, E. G., García-Romero, A., García, P. R., Montenegro, S., Cánovas, M. S., Rubio, A. I., Sánchez-Saura, A., Vicente, M. Á., & de la Peña, F. A. (2026) · The Oncologist · doi:10.1093/oncolo/oyag156
“To evaluate interobserver agreement in ECOG-PS assessments, first we used the linearly weighted Fleiss’ Kappa (wFK), 22 which provides a measure of agreement for ordinal data, 23 along with its 95% confidence interval (CI) calculated via bootstrap ( × 1000) resampling.”
Cerqueira, A. K., Paiva, M. S., Azevedo, D. G. S., Santos, D. A., Lima, C., & Carvalho, L. P. D. S. (2026) · Anais Estendidos do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.5753/sbsi_estendido.2026.249091
Hughes, J. (2026) · ASA Discoveries · doi:10.1080/30681618.2026.2656822
Prosperi, M., Rancati, S., Guo, Y., Guo, J. S., Radwan, R. M., Salemi, M., Bian, J., Bhasuran, B., Egelund, E. F., Janelle, J. W., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02771-7
“” , a survey feature that has been suggested to increase participant cognitive load, confusion, and impact survey validity 33,34 .”
Statistical inference for indices of interrater agreement based on a homogeneity index
Bove, G., & Marella, D. (2026) · Statistical Methods & Applications · doi:10.1007/s10260-026-00846-z
“…authors questioned that joint agreement between three (3-agreement type), four (4-agreement type) or more raters could be considered as well (e.g., Conger (1980), Warrens (2012)); (3) they depend on the marginal distributions of the categories of the scale observed for each rater, a source of…”
Quantitative histology for maturity staging : Timing is everything
Sauger, C., Kellner, K., Martin, V., Meleder, A. L., Berthelin, C., Villain-Naud, N., Elie, N., & Dubroca, L. (2026) · Fisheries Research · doi:10.1016/j.fishres.2026.107775
Benchmarking Multimodal Mathematical Reasoning: Prompt Effects, Modality Gaps, and Failure Modes
Görer, G., Osipenko, M., & Knispel, T. (2026) · Metrics · doi:10.3390/metrics3030011
Sensory contribution of emollient cascades in skin creams evaluated by CATA analysis
Gerken, A., Libby, E., Vigilante, P., Safee, R., Szoboszlai, M., & Baki, G. (2026) · International Journal of Cosmetic Science · doi:10.1111/ics.70123
Ugural, N. E., Bilgin, E. C., Hepdurgun, C., Kayahan, S. T., Dadaş, Ö. F., Sertöz, Ö. Ö., Elbi, H., & Maldonado, J. R. (2026) · Social Work in Health Care · doi:10.1080/00981389.2026.2739411
Animal-based indicators for the welfare assessment of dairy goats under extensive farming systems
Torsiello, B. (n.d.) · Institutional Research Information System University of Turin (University of Turin)
da Silva, G. V., Alves, L. K. S., Pivato, G. M., de Oliveira, M. C., Tinoco, I. A. P., Lopez-Soriano, M., Pairis-Garcia, M. D., Luna, S. P. L., & Trindade, P. H. E. (2026) · The Veterinary Journal · doi:10.1016/j.tvjl.2026.106903
Yaowalaorng, J., Lokeskrawee, T., Pruksathorn, N., Lawanaskol, S., Patumanond, J., Wicha, C., Bumrungpagdee, W., Chanlaor, S., & Lakdee, C. (2026) · PLoS ONE · doi:10.1371/journal.pone.0350690
“Interrater agreement was evaluated using Conger’s linear weighted kappa [8,9] and demonstrated excellent agreement across all three expert panels (S2 Table).”
Weighted Hωand New Paradox of κ
Kwon, N., Kim, J.-G., & Park, Y. G. (2009) · Korean Journal of Applied Statistics · doi:10.5351/kjas.2009.22.5.1073
The level of adoption of analytical tools
Barahona, I. (2013) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.5821/dissertation-2117-95272
Using the ADI-R to diagnose autism in preschool children
Lord, C., Storoschuk, S., Rutter, M., & Pickles, A. (1993) · Infant Mental Health Journal · doi:10.1002/1097-0355(199323)14:3<234::aid-imhj2280140308>3.0.co;2-f
Metadata
- Authors
- Conger, A. J.
- Affiliations
- Purdue University West Lafayette (US)
- Corresponding author
- Conger, Anthony J.
- Venue
- Psychological Bulletin
- Published
- 1980-09
- Volume
- 88
- Issue
- 2
- Pages
- 322-328
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1978217317
- Cited by
- 545
- References
- 3
- Citations by year
- 2026: 20 · 2025: 24 · 2024: 35 · 2023: 24 · 2022: 23 · 2021: 23 · 2020: 25 · 2019: 25 · 2018: 22 · 2017: 16 · 2016: 18 · 2015: 27 · 2014: 29 · 2013: 15 · 2012: 15
- FWCI
- 7.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- Cohen's kappa, Fleiss' kappa, interobserver agreement, multiple raters, nominal scale
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.