High Agreement and High Prevalence: The Paradox of Cohen’s Kappa
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index; none of the 20 sources its claims cite is checked for support yet.
Abstract
Background: Cohen's Kappa is the most used agreement statistic in literature. However, under certain conditions, it is affected by a paradox which returns biased estimates of the statistic itself. Objective: The aim of the study is to provide sufficient information which allows the reader to make an informed choice of the correct agreement measure, by underlining some optimal properties of Gwet’s AC1 in comparison to Cohen’s Kappa, using a real data example. Method: During the process of literature review, we have asked a panel of three evaluators to come up with a judgment on the quality of 57 randomized controlled trials assigning a score to each trial using the Jadad scale. The quality was evaluated according to the following dimensions: adopted design, randomization unit, type of primary endpoint. With respect to each of the above described features, the agreement between the three evaluators has been calculated using Cohen’s Kappa statistic and Gwet’s AC1 statistic and, finally, the values have been compared with the observed agreement. Results: The values of the Cohen’s Kappa statistic would lead to believe that the agreement levels for the variables Unit, Design and Primary Endpoints are totally unsatisfactory. The AC1 statistic, on the contrary, shows plausible values which are in line with the respective values of the observed concordance. Conclusion: We conclude that it would always be appropriate to adopt the AC1 statistic, thus bypassing any risk of incurring the paradox and drawing wrong conclusions about the results of agreement analysis.
Overview
CoChat Summary
TL;DR
Cohen’s Kappa can produce misleadingly low agreement estimates when outcome prevalence is high, even when observed agreement is strong. Using a real reproducibility example, the authors argue Gwet’s AC1 avoids this paradox and should be preferred for agreement analysis.
Key findings
- In sensitivity curves, the Kappa paradox becomes evident when prevalence exceeds 60%, across the scenarios considered; AC1 values remained aligned with observed agreement and were not particularly affected by prevalence level.
- In the case study, Cohen’s Kappa suggested that agreement for the variables Unit, Design, and Primary Endpoints was “totally unsatisfactory,” while AC1 gave plausible values consistent with observed concordance.
- The likely cause of the paradox was high classification probabilities for the levels “Individual,” “Parallel,” and “Continuous” for Unit, Design, and Primary Endpoint, respectively.
- For the Jadad variable, where the authors say the paradox was absent, Kappa and AC1 had quite similar values, both consistent with observed concordance.
Methods
- A panel of three evaluators judged the quality of 57 randomized controlled trials using the Jadad scale, which assigns scores from zero to five; trials were dichotomized as good (>3) or poor (<3) quality.
- Evaluators also classified each trial by randomization unit (individual or community), design (parallel, factorial, or crossover), and primary endpoint (binary, continuous, survival, or other).
- Agreement among the three evaluators was calculated using Cohen’s Kappa and Gwet’s AC1, with 95% confidence intervals, and compared with observed agreement.
Limitations
- The paper does not state a formal limitations section.
- Evident from the text: the empirical demonstration relies on a single case study of 57 RCTs and three evaluators, with no simulation or broader validation reported.
- The authors note that for the Design variable, evaluators may not have had sufficient expertise to distinguish less common designs from “Parallel” type, which may have contributed to high classification probability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
0 Valid · 0 Not valid · 13 Not checked · 11 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even in simple cases with only two evaluators and two outcomes, the paradox tends to occur if, at equal sensitivity and specificity of the evaluators, the prevalence of one of the results is above 60%, as seen in Fig. (1) graphs.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > DISCUSSION
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The values of the Cohen’s Kappa statistic would lead to believe that the agreement levels for the variables Unit, Design and Primary Endpoints are totally unsatisfactory.”
- Also stated in: § RESULTS · hedged
“The values of the Cohen’s Kappa statistic would lead to believe that the agreement levels for the variables Unit, Design and Primary Endpoints are totally unsatisfactory.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > Abstract > Results:
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § RESULTS · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The AC1 statistic, on the contrary, shows plausible values which are in line with the respective values of the observed concordance.”
- Also stated in: § RESULTS · plain
“The AC1 statistic, on the contrary, shows plausible values which are in line with the respective values of the observed concordance.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > Abstract > Results:
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § RESULTS · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we can see, in all scenarios considered (hence independent on the observed correlation values, sensitivity and specificity) the paradox begins to be evident for values of prevalence higher than 60%.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > RESULTS
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most likely explanation for the onset of the paradox can be given by high values, shown in Table 2, taken from the levels "Individual", "Parallel" and "Continuous" for variables Unit, Design and Primary Endpoint. These values have led to high probability of classification and hence to paradox affected values of Kappa statistic.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > RESULTS
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
7 more claims
Create a free account to see all 24 claims and every source verdict.
- It may state two separate claims as one. 8
- The quote only partly states this claim. 1
- Probably not a substantial claim. 6
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The 𝜅 statistic is known to severely underestimate the agreement in situations when one of the labels is clearly dominant [59].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [59]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > METHODS > Case Study: Reproducibility of the Evaluation of Clinical Trial Quality
“An RCT is considered of good quality if it gets a score equal to or greater than 3.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > METHODS > Case Study: Reproducibility of the Evaluation of Clinical Trial Quality
“the Jadad score was dichotomized, distinguishing between good (> 3), and poor (<3) quality trial.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > INTRODUCTION > AC1 Statistic
“The variance of the AC1 statistics, indispensable for the construction of confidence intervals, is calculated through the expression (3), following Gwet [28].”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > INTRODUCTION > The Cohen’s Kappa Statistic
“while the expected agreement in case of random assignment is given by: (3)Pe|K=∑K=1K∑r=2R(−1)r∑i1,i2,.......ir∏j=1rpkij”
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 2×
“The Cohen’s Kappa statistic [16] is the most used agreement measure in literature.”
MethodIntraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582 · cited 1×
“Case Study: Reproducibility of the Evaluation of Clinical Trial Quality During the process of literature review [29], we have asked a panel of three evaluators to come up with a judgment on the quality of 57 randomized controlled trials (RCTs), assigning a score to each trial using the Jadad scale [9].”
MethodZhang, Q., Wang, C., Zheng, M., Li, Y., Li, J., Zhang, L., Shang, X., & Yan, C. (2015) · Cerebrovascular Diseases · doi:10.1159/000369778 · Cited by 11,533
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Dong, Y., Hao, L., Shi, Z., Wang, G., Zhang, Z., & Han, C. (2013) · Urologia Internationalis · doi:10.1159/000351405 · Cited by 4,540
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529786 · Cited by 4,332 · cited 1×
“4 or negative are indicators of discordance [27].”
BackgroundAssessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 2×
“As for the latter, the literature is unanimous in considering that low-quality trials, conducted using inadequate methodological approach, are often associated with the over-estimated treatment effects [5, 7].”
BackgroundHigh agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034 · cited 1×
“In fact, this assumption is weakened - sometimes even contradicted - in presence of strong differences in prevalence of possible outcomes [17].”
BackgroundMoher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755 · cited 1×
“As for the latter, the literature is unanimous in considering that low-quality trials, conducted using inadequate methodological approach, are often associated with the over-estimated treatment effects [5, 7].”
BackgroundVerhagen, A. P., de Vet, H. C. W., de Bie, R. A., Kessels, A. G. H., Boers, M., Bouter, L. M., & Knipschild, P. G. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(98)00131-0 · Cited by 2,379
Byrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167 · cited 3×
“Even in the absence of the paradox, as in the example of Jadad score, the AC1 statistics provides absolutely consistent values and overlapping with the Cohen’s Kappa, which confirms the results found in the literature [21, 28].”
MethodResultReliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
High agreement but low kappa: II. Resolving the paradoxes
Cicchetti, D. V., & Feinstein, A. R. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90159-m · Cited by 1,838
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 852
Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review
Olivo, S. A., Macedo, L. G., Gadotti, I. C., Fuentes, J., Stanton, T., & Magee, D. J. (2008) · Physical Therapy · doi:10.2522/ptj.20070147 · Cited by 829
Integration and generalization of kappas for multiple raters
Conger, A. J. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.2.322 · Cited by 545 · cited 1×
“The expression [3] is referring to the extension of Cohen’s Kappa to a more general case with more than two evaluators and more than two categories [26].”
MethodAssessing the Quality of Randomized Controlled Trials:Current Issues and Future Directions
Moher, D., Jadad, A. R., & Tugwell, P. (1996) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300009570 · Cited by 436
Communications Through Limited Response Questioning
Bennett, E. M., Alpert, R., & Goldstein, A. C. (1954) · Public Opinion Quarterly · doi:10.1086/266520 · Cited by 249
Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity
Gwet, K. L. (2002) · Cited by 195
Aickin, M. (1990) · Biometrics · doi:10.2307/2531434 · Cited by 122 · cited 1×
“The Alpha Aickin statistic [24] is another tool that has very similar properties to the AC1 [30].”
MethodArnbak, B., Jensen, T. S., Manniche, C., et al. (2013) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-14-274 · Cited by 41
Grant, A. D., Thavendiranathan, P., Rodriguez, L. L., Kwon, D. H., & Marwick, T. H. (2013) · Journal of the American Society of Echocardiography · doi:10.1016/j.echo.2013.11.016 · Cited by 32
Huellner, M. W., Bürkert, A., Strobel, K., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0085359 · Cited by 30
Fletcher, J. J., Meurer, W. J., Dunne, M., et al. (2013) · Neurocritical Care · doi:10.1007/s12028-013-9941-z · Cited by 3 · cited 1×
“The expression [3] is referring to the extension of Cohen’s Kappa to a more general case with more than two evaluators and more than two categories [26].”
MethodResearch in Nursing and Nutrition: Is Randomized Clinical Trial the Actual Gold Standard?
Baldi, I., Soriani, N., Lorenzoni, G., Azzolina, D., Lago, E. D., Bardi, S. D., Verduci, E., Zanotti, R., & Gregori, D. (2017) · Gastroenterology Nursing · doi:10.1097/sga.0000000000000246 · Cited by 3
Self-citation(2014) · Medical Entomology and Zoology
Cited by
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
“The 𝜅 statistic is known to severely underestimate the agreement in situations when one of the labels is clearly dominant [59].”
BackgroundThe DoCTRINE Guidelines: Defined Criteria To Report INnovations in Education
Blanco, M., Prunuske, J., DiCorcia, M., Learman, L. A., Mutcheson, B., & Huang, G. C. (2022) · Academic Medicine · doi:10.1097/acm.0000000000004634 · Cited by 67
Kanaparthy, N. S., Villuendas-Rey, Y., Bakare, T., Diao, Z., Iscoe, M., Loza, A., Wright, D., Safranek, C., Faustino, I. V., Brackett, A., et al. (2025) · JMIR AI · doi:10.2196/76743 · Cited by 57
“Both metrics offer a reliable, comprehensive view of interrater reliability, aligning with our study’s emphasis on precision and consistency in data extraction [28,29].”
Social Media Posts About Medical Tests With Potential for Overdiagnosis
Nickel, B., Moynihan, R. N., Gram, E. G., Copp, T., Taba, M., Shih, P., Heiss, R., Gao, M., & Zadro, J. R. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.61940 · Cited by 87
Head-to-head evaluation on diagnostic accuracies of six SARS-CoV-2 serological assays
Tan, S. S. Y., Saw, S., Chew, K. L., Chan, Y., Khoo, C., Pajarillaga, A., Wang, W., Tambyah, P. A., Ong, L., Jureen, R., et al. (2020) · Pathology · doi:10.1016/j.pathol.2020.09.007 · Cited by 39
Metadata
- Authors
- Zec, S., Soriani, N., Comoretto, R., & Baldi, I.
- Affiliations
- University of Padua (IT)
- University of Milan (IT)
- Corresponding author
- Baldi, Ileana
- Venue
- The Open Nursing Journal
- Publisher
- Bentham Science Publishers Ltd.
- Published
- 2017-10-31
- Volume
- 11
- Issue
- 1
- Pages
- 211-218
- ISSN
- 1874-4346 (ISSN-L)
- PMC
- PMC5712640
- OpenAlex
- W2767182834
- Cited by
- 169
- References
- 30
- Citations by year
- 2026: 52 · 2025: 28 · 2024: 16 · 2023: 28 · 2022: 14 · 2021: 15 · 2020: 9 · 2019: 3 · 2018: 2
- FWCI
- 3.51
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- The Open Nursing Journal (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- Institutional Research Information System University of Turin (University of Turin) (submittedVersion, other-oa)
- Padua Research Archive (University of Padova) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Health Sciences › Medicine › Surgery › Hemodynamic Monitoring and Therapy
- Keywords
- Cohen's kappa, Gwet's AC1, interobserver agreement, randomized controlled trials, Jadad scale, trial quality assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.