The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index; 11 of the 25 sources its claims cite were checked for support.
A comparison of cohen's Kappa and gwet's AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples
Open accessAbstract
BACKGROUND: Rater agreement is important in clinical research, and Cohen's Kappa is a widely used method for assessing inter-rater reliability; however, there are well documented statistical problems associated with the measure. In order to assess its utility, we evaluated it against Gwet's AC1 and compared the results. METHODS: This study was carried out across 67 patients (56% males) aged 18 to 67, with a mean SD of 44.13 ± 12.68 years. Nine raters (7 psychiatrists, a psychiatry resident and a social worker) participated as interviewers, either for the first or the second interviews, which were held 4 to 6 weeks apart. The interviews were held in order to establish a personality disorder (PD) diagnosis using DSM-IV criteria. Cohen's Kappa and Gwet's AC1 were used and the level of agreement between raters was assessed in terms of a simple categorical diagnosis (i.e., the presence or absence of a disorder). Data were also compared with a previous analysis in order to evaluate the effects of trait prevalence. RESULTS: Gwet's AC1 was shown to have higher inter-rater reliability coefficients for all the PD criteria, ranging from .752 to 1.000, whereas Cohen's Kappa ranged from 0 to 1.00. Cohen's Kappa values were high and close to the percentage of agreement when the prevalence was high, whereas Gwet's AC1 values appeared not to change much with a change in prevalence, but remained close to the percentage of agreement. For example a Schizoid sample revealed a mean Cohen's Kappa of .726 and a Gwet's AC1of .853 , which fell within the different level of agreement according to criteria developed by Landis and Koch, and Altman and Fleiss. CONCLUSIONS: Based on the different formulae used to calculate the level of chance-corrected agreement, Gwet's AC1 was shown to provide a more stable inter-rater reliability coefficient than Cohen's Kappa. It was also found to be less affected by prevalence and marginal probability than that of Cohen's Kappa, and therefore should be considered for use with inter-rater reliability analysis.
Overview
CoChat Summary
TL;DR
This reliability study compared Cohen’s Kappa and Gwet’s AC1 for DSM-IV personality disorder diagnoses using Thai SCID-II interviews and found AC1 to be more stable and less affected by prevalence and marginal probability. The authors conclude Gwet’s AC1 should be considered for inter-rater reliability analyses, or used alongside Cohen’s Kappa.
Key findings
- Gwet’s AC1 produced higher inter-rater reliability coefficients than Cohen’s Kappa across all PD criteria: AC1 ranged from .752 to 1.000 (mean ± SD = .953 ± .071), while Cohen’s Kappa ranged from 0 to 1.000 (mean ± SD = .821 ± .299); overall agreement ranged from 84% to 100% (mean ± SD = 96.58 ± 4.99).
- AC1 was less affected by trait prevalence and marginal probability. When prevalence was high, Cohen’s Kappa was high and close to percentage agreement; AC1 did not change much with prevalence and remained close to percentage agreement. For Schizoid PD, the mean Cohen’s Kappa was .726 and Gwet’s AC1 was .853, crossing Landis and Koch agreement categories.
- In zero-marginal/100% agreement cases, Cohen’s Kappa gave 0, whereas Gwet’s AC1 gave .858 for Avoidant PD and .890 for Dependent, Passive-Aggressive, and Paranoid PDs in the TW-SR and NW-SR pairs.
- For Antisocial and Histrionic PDs in the VU-MN pair, both had 5.2% prevalence; Antisocial had Cohen’s Kappa .441 (SE .330) vs. AC1 .870 (SE .095) with 89% agreement, and Histrionic had Cohen’s Kappa .641 (SE .326) vs. AC1 .938 (SE .063) with 94% agreement.
Methods
- Reliability study of 67 patients recruited from inpatient and outpatient departments of Maharaj Nakorn Chiang Mai Hospital. Nine raters (7 psychiatrists, 1 social worker, 1 psychiatry resident) formed 8 rater pairs; diagnoses used the Thai version of SCID-II for DSM-IV personality disorders, with interviews 4 to 6 weeks apart. The abstract reports 56% males and mean age 44.13 ± 12.68 years, while the Methods section reports 55% female and mean age 44.07 ± 13.09 years.
- Cohen’s Kappa and Gwet’s AC1 were calculated using AgreeStat version 2011.3. Only rater pairs 1–4 were analyzed for the 2×2 analysis because the remaining pairs had insufficient cell size; analyses compared coefficients by PD criterion, percentage agreement, prevalence, and chance agreement probability.
Limitations
- The paper does not include a dedicated limitations section. It states that only 4 of 8 rater pairs were analyzed because the remaining pairs had insufficient cell size, and it notes that some criteria had unavoidably low prevalence rates and small cell counts.
- Evident from the text: the sample comes from a single hospital in Thailand, uses the Thai SCID-II, and relies mostly on psychiatrist raters, which may limit generalizability. The demographic reporting also differs between the abstract and Methods section.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 2 Not valid · 4 Not checked · 30 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Most recently published articles that have assessed inter-rater reliability have used Cohen’s Kappa exclusively [19-26]”
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Discussion
What the sources say
- Not checked — no full text of the sourcecites [19] Hartling L, Bond K, Santaguida PL, Viswanathan M, Dryden DM: Testing a tool for the classification of study designs in systematic reviews of interventions and exposures showed moderate reliability and low accuracy. J Clin Epidemiol. 2011, 64: 861-871. 10.1016/j.jclinepi.2011.01.010. source
- Partly supportedcites [20] Hernaez R, Lazo M, Bonekamp S, Kamel I, Brancati FL, Guallar E, Clark JM: Diagnostic accuracy and reliability of ultrasonography for the detection of fatty liver: a meta-analysis. Hepatology. 2011, 54: 1082-1090. sourceExperimental — accuracy not yet measured
The Results section, under “Systematic Review of the Reliability of Ultrasonography” (page unavailable), reports that 17 of 22 ultrasound-reliability studies used kappa statistics, including 14 reporting interrater kappa values. This supports kappa being common, but the passage does not identify the statistic specifically as Cohen’s kappa, establish that those studies used it exclusively, or limit the finding to recently published articles. (LLM, conf=0.85)
- Not checked — no full text of the sourcecites [21] Sheehan DV, Sheehan KH, Shytle RD, Janavs J, Bannon Y, Rogers JE, Milo KM, Stock SL, Wilkinson B: Reliability and validity of the Mini International Neuropsychiatric Interview for Children and Adolescents (MINI-KID). J Clin Psychiatry. 2010, 71: 313-326. 10.4088/JCP.09m05305whi. source
- Not checked — no full text of the sourcecites [22] Ingenhoven TJ, Duivenvoorden HJ, Brogtrop J, Lindenborn A, van den Brink W, Passchier J: Interrater reliability for Kernberg's structural interview for assessing personality organization. J Pers Disord. 2009, 23: 528-534. 10.1521/pedi.2009.23.5.528. source
- Supportedcites [23] Øiesvold T, Nivison M, Hansen V, Sørgaard KW, Østensen L, Skre I: Classification of bipolar disorder in psychiatric hospital. A prospective cohort study. BMC Psychiatry. 2012, 12: 13- source
The Methods > Statistics section states that the study used Cohen’s kappa to estimate agreement between expert and clinical diagnoses. The Results report that agreement using kappa; the other statistical tests listed were for assessing sample bias, not inter-rater agreement. This supports citing the paper as an example of a study that assessed agreement using Cohen’s kappa exclusively. (LLM, conf=0.92)
- Disputedcites [24] Clement S, Brohan E, Jeffery D, Henderson C, Hatch SL, Thornicroft G: Development and psychometric properties the Barriers to Access to Care Evaluation scale (BACE) related to people with mental ill health. BMC Psychiatry. 2012, 12: 36-10.1186/1471-244X-12-36. sourceExperimental — accuracy not yet measured
The Methods > Analysis and Results > Reliability sections describe test–retest reliability, not inter-rater reliability. They report weighted kappa for individual items and Lin’s concordance statistic for the treatment stigma subscale, contrary to presenting this paper as one that used Cohen’s kappa exclusively. The copy does not establish what most other recently published articles used. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
- Disputedcites [25] McCoul ED, Smith TL, Mace JC, Anand VK, Senior BA, Hwang PH, Stankiewicz JA, Tabaee A: Interrater agreement of nasal endoscopy in patients with a prior history of endoscopic sinus surgery. Int Forum Allergy Rhinol. 2012, 2: 453-459. 10.1002/alr.21058. sourceExperimental — accuracy not yet measured
The Methods (Statistical Analysis; page unavailable) state that this study assessed inter-rater agreement with Fleiss’ kappa, supplemented by Spearman rank correlations and prevalence- and bias-adjusted kappa. Thus, this cited article is a counterexample to the claim that the cited inter-rater reliability articles used Cohen’s kappa exclusively. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
- Not checked — no full text of the sourcecites [26] Ansari NN, Naghdi S, Forogh B, Hasson S, Atashband M, Lashgari E: Development of the Persian version of the Modified Modified Ashworth Scale: translation, adaptation, and examination of interrater and intrarater reliability in patients with poststroke elbow flexor spasticity. Disabil Rehabil. 2012, 34: 1843-1847. 10.3109/09638288.2012.665133. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“In such cases, some investigators have reported good percentage agreement accompanied by an undesirable Cohen’s Kappa [14]”
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Discussion
What the sources say
- Supportedcites [14] Day FC, Schriger DL, Annals Of Emergency Medicine Journal Club: A consideration of the measurement and reporting of interrater reliability: answers to the July 2009 Journal Club questions. Ann Emerg Med. 2009, 54: 843-853. 10.1016/j.annemergmed.2009.07.013. source
In Q2.b, the paper calculates 83.2% observed agreement but a Cohen’s kappa of 0.07. Q5.b explains that kappa is especially likely to diverge from high percentage agreement when a cell has a small count and the marginal totals are skewed, as can occur with rare answers. The supplied passages do not give page numbers. (LLM, conf=0.91)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“During our research of PubMed (up to February 2013), we found only 2 published articles that mention using Gwet’s AC1 method as part of a study [28,29].”
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Discussion
What the sources say
- Supportedcites [28] Petzold A, Altintas A, Andreoni L, Bartos A, Berthele A, Blankenstein MA, Buee L, Castellazzi M, Cepok S, Comabella M: Neurofilament ELISA validation. J Immunol Methods. 2010, 352: 23-31. 10.1016/j.jim.2009.09.014. source
The citation identifies this paper as one of the articles using Gwet’s AC1, not as evidence for the citing authors’ PubMed search or its count of two. In §2.4 (Data analysis and strategy for validation), the paper says it calculated AC1 coefficients; Table 2 also reports AC1 results. (LLM, conf=0.93)
- Not checked — no full text of the sourcecites [29] Yusuff KB, Tayo F: Frequency, types and severity of medication use-related problems among medical outpatients in Nigeria. Int J Clin Pharm. 2011, 33: 558-564. 10.1007/s11096-011-9508-z. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When the prevalence rate was higher, so were Cohen’s Kappa and the level of agreement”
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Results > The effect of trait prevalence
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Cohen’s Kappa values were high and close to the percentage of agreement when the prevalence was high”
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Abstract > Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
13 more claims
Create a free account to see all 39 claims and every source verdict.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 7
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“To address this issue we also compute Gwet's AC1 which is robust in such situations [58].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [58]
In the citing paper
“Gwet's AC1 metric provides a more stable inter-rater reliability coefficient than Cohens Kappa and is less affected by prevalence and marginal probability than Cohens Kappa (Wongpakaran et al., 2013).”
From Assessing student errors in experimentation using artificial intelligence and large language models: A comparative study with human raters (2023) · cites this paper as Wongpakaran et al., 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Abstract > Methods
“aged 18 to 67, with a mean SD of 44.13 ± 12.68 years”
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Methods > Subjects
“the mean age was 44.07 ± 13.09 years (18 to 67)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Results > Chance agreement probability
“the raters had 100% agreement) as found for the Avoidant, Dependent, Passive-Aggressive and Paranoid PDs in the TW-SR and NW-SR pairs. Cohen’s Kappa gave a ‘0’ value for them all”
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Results
“| Avoidant | No (N) | 9 | 0 | 100 | No (N) | 7 | 1 | 88 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Abstract > Methods
“This study was carried out across 67 patients (56% males)”
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Methods > Subjects
“A total of 67 subjects were recruited from the inpatient and outpatient departments of Maharaj Nakorn Chiang Mai Hospital, part of the Faculty of Medicine at Chiang Mai University. Slightly over half (55%) of the subjects were female”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Discussion
“in this study Schizoid PD had a high percentage of agreement (88% - 100%) among 4 pairs of raters”
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Results
“| Schizoid | N | 13 | 2 | 84 | N | 15 | 0 | 100 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Results
“None of the PDs showed a 100 percent agreement among the 4 pairs of raters.”
§ A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples > Results > The effect of trait prevalence
“| Schizotypal | 10.00 | 1.000 | 1.000 | 100 | | | 5.26 | 1.000 | 1.000 | 100 | | | 0.0 | 1.000 | 1.000 | 100 | | | 0.0 | 1.000 | 1.000 | 100 |”
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 1×
“Because this assessment results in dichotomous outcomes, Cohen’s Kappa [2,3] is commonly used to assess the reliability of raters.”
MethodWeighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · cited 1×
“Because this assessment results in dichotomous outcomes, Cohen’s Kappa [2,3] is commonly used to assess the reliability of raters.”
MethodComputing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167 · cited 1×
“Gwet has also proved the validity of the multiplerater version of the AC1 and the Fleiss’ Kappa statistics, using a Monte-Carlo simulation approach with various estimators [10].”
MethodSelf-citationHigh agreement but low kappa: II. Resolving the paradoxes
Cicchetti, D. V., & Feinstein, A. R. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90159-m · Cited by 1,838 · cited 2×
“This problem has been referred to as the “Kappa paradox” by Feinstein and Cicchetti [7], who stated, “in one paradox, a high value of the observed agreement (Po) can be drastically lowered by a substantial imbalance in the table's marginal totals either vertically or horizontally.”
BackgroundSheehan, D. V., Sheehan, K. H., Shytle, R. D., Janavs, J., Bannon, Y., Rogers, J. E., Milo, K. M., Stock, S. L., & Wilkinson, B. (2010) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.09m05305whi · Cited by 1,730 · cited 1×
“Most recently published articles that have assessed inter-rater reliability have used Cohen’s Kappa exclusively [19-26], and a recent review of the current methods used for inter-rater reliability does not even mention AC1 [27].”
BackgroundHernaez, R., Lazo, M., Bonekamp, S., Kamel, I., Brancati, F. L., Guallar, E., & Clark, J. M. (2011) · Hepatology · doi:10.1002/hep.24452 · Cited by 1,585 · cited 1×
“Most recently published articles that have assessed inter-rater reliability have used Cohen’s Kappa exclusively [19-26], and a recent review of the current methods used for inter-rater reliability does not even mention AC1 [27].”
BackgroundBiostatistics 104: correlational analysis.
Chan, Y. H. (2003) · PubMed · Cited by 1,223
Lobbestael, J., Leurgans, M., & Arntz, A. (2011) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.693 · Cited by 1,124
Interrater agreement and interrater reliability: Key concepts, approaches, and applications
Gisev, N., Bell, J. S., & Chen, T. F. (2013) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2012.04.004 · Cited by 924
The Kappa Statistic: A Second Look
Eugenio, B. D., & Glass, M. (2004) · Computational Linguistics · doi:10.1162/089120104773633402 · Cited by 489 · cited 3×
“Di Eugenio B, Glass M: The Kappa Statistic: A Second Look.”
BackgroundInfluential
Research works printed but not linked: 2
- First MB, Gibbon M, Spitzer RL, Williams JBW, Benjamin LS: Structured Clinical Interview for DSM-IV Axis II Personality Disorder (SCID-II). 1997, Washington, DC: merican Psychiatric Press
- Kittirattanapaiboon P, Khamwongpin M: The Validity of the Mini International Neuropsychiatric Interview (M.I.N.I.)-ThaiVersion. Journal of Mental Health of Thailand. 2005, 13: 126-136.
Cited by
Mishra, S. (2020) · Educational Research Review · doi:10.1016/j.edurev.2019.100307 · Cited by 688
“The latter is similar to Cohen's (1960) kappa but produces more stable results (Wongpakaran, Wongpakaran, Weddng, & Gwet, 2013).”
Margulis, A. V., Pladevall, M., Riera-Guardia, N., Varas-Lorenzo, C., Hazell, L., Berkman, N., Viswanathan, M., & Perez-Gutthann, S. (2014) · Clinical Epidemiology · doi:10.2147/clep.s66677 · Cited by 441
International Journal of Behavioral Medicine
(2013) · International Journal of Behavioral Medicine · doi:10.1007/12529.1532-7558 · Cited by 424
The Safety of Inpatient Health Care
Bates, D. W., Levine, D. M., Salmasian, H., Syrowatka, A., Shahian, D. M., Lipsitz, S., Zebrowski, J. P., Myers, L. C., Logan, M. S., Roy, C. G., et al. (2023) · New England Journal of Medicine · doi:10.1056/nejmsa2206117 · Cited by 411
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
“Percent agreement and Gwet ’ s AC 1 were calculated to evaluate interrater reliability 56 .”
Metadata
- Authors
- Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L.
- Affiliations
- Chiang Mai University (TH)
- Alliant University (US)
- Corresponding author
- Wongpakaran, Nahathai
- Venue
- BMC Medical Research Methodology
- Published
- 2013-04-29
- Volume
- 13
- Issue
- 1
- Pages
- 61-61
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC3643869
- OpenAlex
- W1996445490
- Semantic Scholar
- 8891481c3fa28d95acc9f2c50187ce682a8130e7
- Cited by
- 1,041
- References
- 27
- Citations by year
- 2026: 150 · 2025: 129 · 2024: 113 · 2023: 120 · 2022: 101 · 2021: 127 · 2020: 97 · 2019: 57 · 2018: 51 · 2017: 29 · 2016: 25 · 2015: 19 · 2014: 18 · 2013: 5
- FWCI
- 14.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/2.0 (tdm)
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Chiang Mai University
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Psychology › Clinical Psychology › Personality Disorders and Psychopathology
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Cohen's kappa, Gwet's AC1, inter-rater reliability, personality disorders, diagnostic agreement, chance-corrected agreement, schizoid personality disorder, psychiatric diagnosis, marginal probabilities
- MeSH terms
- Computer Communication Networks, Computer Graphics, Humans, Meta-Analysis as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A comparison of cohen's Kappa and gwet's AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. https://app.cochat.ai/papers/10.1186/1471-2288-13-61
- The paper
- Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013). A comparison of cohen's Kappa and gwet's AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology. https://doi.org/10.1186/1471-2288-13-61
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.