The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index; none of the 20 sources its claims cite is checked for support yet.
High Agreement and High Prevalence: The Paradox of Cohen’s Kappa
Open accessAbstract
Background: Cohen's Kappa is the most used agreement statistic in literature. However, under certain conditions, it is affected by a paradox which returns biased estimates of the statistic itself. Objective: The aim of the study is to provide sufficient information which allows the reader to make an informed choice of the correct agreement measure, by underlining some optimal properties of Gwet’s AC1 in comparison to Cohen’s Kappa, using a real data example. Method: During the process of literature review, we have asked a panel of three evaluators to come up with a judgment on the quality of 57 randomized controlled trials assigning a score to each trial using the Jadad scale. The quality was evaluated according to the following dimensions: adopted design, randomization unit, type of primary endpoint. With respect to each of the above described features, the agreement between the three evaluators has been calculated using Cohen’s Kappa statistic and Gwet’s AC1 statistic and, finally, the values have been compared with the observed agreement. Results: The values of the Cohen’s Kappa statistic would lead to believe that the agreement levels for the variables Unit, Design and Primary Endpoints are totally unsatisfactory. The AC1 statistic, on the contrary, shows plausible values which are in line with the respective values of the observed concordance. Conclusion: We conclude that it would always be appropriate to adopt the AC1 statistic, thus bypassing any risk of incurring the paradox and drawing wrong conclusions about the results of agreement analysis.
Overview
CoChat Summary
TL;DR
Cohen’s Kappa can produce misleadingly low agreement estimates when outcome prevalence is high, even when observed agreement is strong. Using a real reproducibility example, the authors argue Gwet’s AC1 avoids this paradox and should be preferred for agreement analysis.
Key findings
- In sensitivity curves, the Kappa paradox becomes evident when prevalence exceeds 60%, across the scenarios considered; AC1 values remained aligned with observed agreement and were not particularly affected by prevalence level.
- In the case study, Cohen’s Kappa suggested that agreement for the variables Unit, Design, and Primary Endpoints was “totally unsatisfactory,” while AC1 gave plausible values consistent with observed concordance.
- The likely cause of the paradox was high classification probabilities for the levels “Individual,” “Parallel,” and “Continuous” for Unit, Design, and Primary Endpoint, respectively.
- For the Jadad variable, where the authors say the paradox was absent, Kappa and AC1 had quite similar values, both consistent with observed concordance.
Methods
- A panel of three evaluators judged the quality of 57 randomized controlled trials using the Jadad scale, which assigns scores from zero to five; trials were dichotomized as good (>3) or poor (<3) quality.
- Evaluators also classified each trial by randomization unit (individual or community), design (parallel, factorial, or crossover), and primary endpoint (binary, continuous, survival, or other).
- Agreement among the three evaluators was calculated using Cohen’s Kappa and Gwet’s AC1, with 95% confidence intervals, and compared with observed agreement.
Limitations
- The paper does not state a formal limitations section.
- Evident from the text: the empirical demonstration relies on a single case study of 57 RCTs and three evaluators, with no simulation or broader validation reported.
- The authors note that for the Design variable, evaluators may not have had sufficient expertise to distinguish less common designs from “Parallel” type, which may have contributed to high classification probability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
0 Valid · 0 Not valid · 13 Not checked · 11 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even in simple cases with only two evaluators and two outcomes, the paradox tends to occur if, at equal sensitivity and specificity of the evaluators, the prevalence of one of the results is above 60%, as seen in Fig. (1) graphs.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > DISCUSSION
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The values of the Cohen’s Kappa statistic would lead to believe that the agreement levels for the variables Unit, Design and Primary Endpoints are totally unsatisfactory.”
- Also stated in: § RESULTS · hedged
“The values of the Cohen’s Kappa statistic would lead to believe that the agreement levels for the variables Unit, Design and Primary Endpoints are totally unsatisfactory.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > Abstract > Results:
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § RESULTS · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The AC1 statistic, on the contrary, shows plausible values which are in line with the respective values of the observed concordance.”
- Also stated in: § RESULTS · plain
“The AC1 statistic, on the contrary, shows plausible values which are in line with the respective values of the observed concordance.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > Abstract > Results:
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § RESULTS · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we can see, in all scenarios considered (hence independent on the observed correlation values, sensitivity and specificity) the paradox begins to be evident for values of prevalence higher than 60%.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > RESULTS
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most likely explanation for the onset of the paradox can be given by high values, shown in Table 2, taken from the levels "Individual", "Parallel" and "Continuous" for variables Unit, Design and Primary Endpoint. These values have led to high probability of classification and hence to paradox affected values of Kappa statistic.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > RESULTS
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
7 more claims
Create a free account to see all 24 claims and every source verdict.
- It may state two separate claims as one. 8
- The quote only partly states this claim. 1
- Probably not a substantial claim. 6
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The 𝜅 statistic is known to severely underestimate the agreement in situations when one of the labels is clearly dominant [59].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [59]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > METHODS > Case Study: Reproducibility of the Evaluation of Clinical Trial Quality
“An RCT is considered of good quality if it gets a score equal to or greater than 3.”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > METHODS > Case Study: Reproducibility of the Evaluation of Clinical Trial Quality
“the Jadad score was dichotomized, distinguishing between good (> 3), and poor (<3) quality trial.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > INTRODUCTION > AC1 Statistic
“The variance of the AC1 statistics, indispensable for the construction of confidence intervals, is calculated through the expression (3), following Gwet [28].”
§ High Agreement and High Prevalence: The Paradox of Cohen’s Kappa > INTRODUCTION > The Cohen’s Kappa Statistic
“while the expected agreement in case of random assignment is given by: (3)Pe|K=∑K=1K∑r=2R(−1)r∑i1,i2,.......ir∏j=1rpkij”
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 2×
“The Cohen’s Kappa statistic [16] is the most used agreement measure in literature.”
MethodIntraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582 · cited 1×
“Case Study: Reproducibility of the Evaluation of Clinical Trial Quality During the process of literature review [29], we have asked a panel of three evaluators to come up with a judgment on the quality of 57 randomized controlled trials (RCTs), assigning a score to each trial using the Jadad scale [9].”
MethodZhang, Q., Wang, C., Zheng, M., Li, Y., Li, J., Zhang, L., Shang, X., & Yan, C. (2015) · Cerebrovascular Diseases · doi:10.1159/000369778 · Cited by 11,533
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Dong, Y., Hao, L., Shi, Z., Wang, G., Zhang, Z., & Han, C. (2013) · Urologia Internationalis · doi:10.1159/000351405 · Cited by 4,540
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529786 · Cited by 4,332 · cited 1×
“4 or negative are indicators of discordance [27].”
BackgroundAssessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 2×
“As for the latter, the literature is unanimous in considering that low-quality trials, conducted using inadequate methodological approach, are often associated with the over-estimated treatment effects [5, 7].”
BackgroundHigh agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034 · cited 1×
“In fact, this assumption is weakened - sometimes even contradicted - in presence of strong differences in prevalence of possible outcomes [17].”
Background
Research works printed but not linked: 1
- Gwet K. Kappa statistic is not satisfactory for assessing the extent of agreement between raters. Stat Method Inter-rater Reliab Assessm 2002; 1 (6) : 1-6.
Cited by
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
“The 𝜅 statistic is known to severely underestimate the agreement in situations when one of the labels is clearly dominant [59].”
BackgroundThe DoCTRINE Guidelines: Defined Criteria To Report INnovations in Education
Blanco, M., Prunuske, J., DiCorcia, M., Learman, L. A., Mutcheson, B., & Huang, G. C. (2022) · Academic Medicine · doi:10.1097/acm.0000000000004634 · Cited by 67
Social Media Posts About Medical Tests With Potential for Overdiagnosis
Nickel, B., Moynihan, R. N., Gram, E. G., Copp, T., Taba, M., Shih, P., Heiss, R., Gao, M., & Zadro, J. R. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.61940 · Cited by 86
Kanaparthy, N. S., Villuendas-Rey, Y., Bakare, T., Diao, Z., Iscoe, M., Loza, A., Wright, D., Safranek, C., Faustino, I. V., Brackett, A., et al. (2025) · JMIR AI · doi:10.2196/76743 · Cited by 57
“Both metrics offer a reliable, comprehensive view of interrater reliability, aligning with our study’s emphasis on precision and consistency in data extraction [28,29].”
Head-to-head evaluation on diagnostic accuracies of six SARS-CoV-2 serological assays
Tan, S. S. Y., Saw, S., Chew, K. L., Huak, C. Y., Khoo, C., Pajarillaga, A., Wang, W., Tambyah, P. A., Ong, L., Jureen, R., et al. (2020) · Pathology · doi:10.1016/j.pathol.2020.09.007 · Cited by 38
Metadata
- Authors
- Zec, S., Soriani, N., Comoretto, R., & Baldi, I.
- Affiliations
- University of Padua (IT)
- University of Milan (IT)
- Corresponding author
- Baldi, Ileana
- Venue
- The Open Nursing Journal
- Publisher
- Bentham Science Publishers Ltd.
- Published
- 2017-10-31
- Volume
- 11
- Issue
- 1
- Pages
- 211-218
- ISSN
- 1874-4346 (ISSN-L)
- PMC
- PMC5712640
- OpenAlex
- W2767182834
- Cited by
- 169
- References
- 30
- Citations by year
- 2026: 52 · 2025: 28 · 2024: 16 · 2023: 28 · 2022: 14 · 2021: 15 · 2020: 9 · 2019: 3 · 2018: 2
- FWCI
- 3.51
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- The Open Nursing Journal (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- Institutional Research Information System University of Turin (University of Turin) (submittedVersion, other-oa)
- Padua Research Archive (University of Padova) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Health Sciences › Medicine › Surgery › Hemodynamic Monitoring and Therapy
- Keywords
- Cohen's kappa, Gwet's AC1, interobserver agreement, randomized controlled trials, Jadad scale, trial quality assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: High Agreement and High Prevalence: The Paradox of Cohen’s Kappa. https://app.cochat.ai/papers/10.2174/1874434601711010211
- The paper
- Zec, S., Soriani, N., Comoretto, R., & Baldi, I. (2017). High Agreement and High Prevalence: The Paradox of Cohen’s Kappa. The Open Nursing Journal. https://doi.org/10.2174/1874434601711010211
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.