Putting the Kappa Statistic to Use
Open accessThe DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Abstract
Inter‐rater assessments of agreement are an essential criterion in the subjective evaluation of product quality. When assessments among raters demonstrate evidence of a lack of agreement (partial or total), there is a need to identify the source of disagreement. The objective being the reduction or mitigation of the influence different raters have on the assessment and the achievement of consistency among raters. The less influence that raters have on the assessment, the more confident one is in making critical to quality decisions. However, situations do exist in which user perceptions can be unreliable (not repeatable) and demonstrate poor correlation with engineered specifications. Quality management teams must be aware of this. When such situations exist, it is advisable to revisit the voice of the process as a reliable function of specification. Copyright © 2011 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Research works printed but not linked: 3
- 10.1086/266577 · DOI · Paper page
- 10.1177/001316446002000104 · DOI · Paper page
- 10.2307/2529310 · DOI · Paper page
Cited by
An Examination of the Use of Large Language Models to Aid Analysis of Textual Data
(2024) · doi:10.1177/16094069241231168 · Cited by 227
Content validation and usability of a chatbot of guidelines for wound dressing
da Silva Lima Roque, G., Roque de Souza, R., Araújo do Nascimento, J. W., de Campos Filho, A. S., de Melo Queiroz, S. R., & Ramos Vieira Santos, I. C. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104473 · Cited by 31
Choice-75: A Dataset on Decision Branching in Script Learning
Hou, Z. J., Zhang, L., & Callison-Burch, C. (2023) · arXiv · doi:10.48550/arxiv.2309.11737 · Cited by 5
Comparison of cultural preferences and cultural practices in website design in Pakistan
Nizamani, S., Nizamani, S., Basir, N., Khoumbati, K., Nizamani, S., & Memon, S. (2025) · Universal Access in the Information Society · doi:10.1007/s10209-025-01248-w · Cited by 1
“2 Systematic procedure adopted is indicated by values above 0.75 [ 34].”
Influential
Metadata
- Authors
- Nichols, T. R., Wisner, P. M., Cripe, G., & Gulabchand, L.
- Venue
- The Quality Assurance Journal
- Publisher
- Wiley
- Published
- 2010-07
- Volume
- 13
- Issue
- 3-4
- Pages
- 57-61
- ISSN
- 1087-8378 (ISSN-L)
- 1099-1786
- OpenAlex
- W1587080882
- Cited by
- 111
- References
- 1
- FWCI
- 0.99
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.