The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The kappa was .78, which reflects a moderate agreement between the reviewers [25,26].”
From Barriers to the Use of Mobile Health in Improving Health Outcomes in Developing Countries: Systematic Review (2019) · cites this paper as [25]
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Kendall, M. G. (1957) · Biometrika · doi:10.2307/2333282 · Cited by 6,442
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. (1979) · Springer series in statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,255
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,536
Estimation of the reliability of ratings.
Ebel, R. L. (1951) · Psychometrika · doi:10.1007/bf02288803 · Cited by 976
Unitizing and categorizing problems in coding qualitative data
Guetzkow, H. S. (1950) · Journal of Clinical Psychology · doi:10.1002/1097-4679(195001)6:1 · Cited by 436
Intraclass Correlation and the Analysis of Variance
Trese, T., & Haggard, E. A. (1958) · The American Catholic Sociological Review · doi:10.2307/3708305 · Cited by 401
Flagged first, then by citations
Cited by
Sorted by citations
Items 701–750 of 750
Ranking Computer Vision Service Issues using EmotionPreprint
Curumsing, M. K., Cummaudo, A., Graetsch, U. M., Barnett, S., & Vasa, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03120
“We then collated all rater’s results and calculated Light’s Kappa (Lκ ) [13] to measure the overall agreement between raters to measure the similarity in which independent raters classify emotions to SO posts.”
MethodQuality improvement of biomarker assessment in breast cancer
van Dooijeweert, C. (2020) · doi:10.33540/9
L2 acquisition of nuclear accent
Landblom, S. A. (2020) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Burt, H. (2021) · cIRcle (University of British Columbia) · doi:10.14288/1.0401765
Huberty, C. J., & Olejnik, S. F. (2005) · Wiley series in probability and statistics · doi:10.1002/9780471789475.refs
Kroma, R. B., Highland, K. B., Neilson, G., Costantino, R. C., Scott-Richardson, M., & Curell, A. (2022) · doi:10.2196/preprints.36609
Randomized Designs: Ordinal Data, I
Berry, K. J., Mielke, P. W., & Johnston, J. E. (2016) · Springer eBooks · doi:10.1007/978-3-319-28770-6_5
Berry, K. J., Mielke, P. W., & Johnston, J. E. (2016) · Springer eBooks · doi:10.1007/978-3-319-28770-6_1
Berry, K. J., Johnston, J. E., & Mielke, P. W. (2014) · Springer eBooks · doi:10.1007/978-3-319-02744-9_5
Randomized Block Designs: Nominal Data
Berry, K. J., Mielke, P. W., & Johnston, J. E. (2016) · Springer eBooks · doi:10.1007/978-3-319-28770-6_11
Thompson, J. W. (1992) · Springer eBooks · doi:10.1007/978-1-4899-0688-5_10
(1979) · Elsevier eBooks · doi:10.1016/b978-0-12-293150-5.50022-5
Berry, K. J., Johnston, J. E., & Mielke, P. W. (2018) · Springer eBooks · doi:10.1007/978-3-319-98926-6_6
Unwin, D., & McAleese, R. (1978) · Palgrave Macmillan UK eBooks · doi:10.1007/978-1-349-81526-5_15
Berry, K. J., Johnston, J. E., & Mielke, P. W. (2019) · Springer eBooks · doi:10.1007/978-3-030-20933-9_11
Banerjee, M. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess3164
Popping, R. (2019) · Springer eBooks · doi:10.1007/978-3-030-11671-2_3
Strategische Produktgestaltung
Schmidt-Gallas, D., Herrmann, A. P., & Riesenbeck, H. (2004) · Zeitschrift für Planung & Unternehmenssteuerung · doi:10.1007/bf03401251
Ayers, M., Sabella, D., Ramirez, N., & Arscott, R. A. (2023) · Journal of Clinical Engineering · doi:10.1097/jce.0000000000000603
Haussner, W., Laghezza, M., Melchor, J., Sharma, R., & Daniels, B. (2023) · Journal of EMS Medicine · doi:10.35616/jemsm.2022.00031
“The inter-rater reliability of three individual ESI ratings was evaluated using Light’s kappa [11,12].”
Measures of Nominal Association I
Berry, K. J., & Johnston, J. E. (2023) · Springer eBooks · doi:10.1007/978-3-031-41896-9_11
Measures of Fourfold Association II
Berry, K. J., & Johnston, J. E. (2023) · Springer eBooks · doi:10.1007/978-3-031-41896-9_14
Options, Problems and Guidelines for Measuring Interrater Agreement - a Descriptive Approach
Kočišová, L. (2024) · Czech Sociological Review · doi:10.13060/csr.2024.007
“…koeficient kappa κ Exact (ge-neralizácia koeficientu kappa pre m posudzovateľov; Conger, 1980), kappa κ Light (použiteľný pre viacerých ako dvoch posudzovateľov a kategorické dáta; je prie-merom všetkých možných kombinácií bivariačného koeficientu kappa medzi po-sudzovateľmi; Light, 1971) 23 .”
Conditionals, Support and Connexivity
Rott, H. (2025) · Trends in logic · doi:10.1007/978-3-031-82994-9_7
Yılmaz, A. E. (2025) · Journal of Polytechnic · doi:10.2339/politeknik.1568563
Zlevor, A. M., Versaggi, L. M., Kisting, M. A., Couillard, A. B., Wagner, M. G., Kisting, A. L., Falk, K. L., White, J. K., Kutlu, A. Z., Minesinger, G. M., et al. (2025) · Journal of Vascular and Interventional Radiology · doi:10.1016/j.jvir.2025.06.013
Stockman, T. I., Kowaleski, M. P., Hicks, J. M., Berns, C. N., Saunders, W. B., & McCarthy, R. J. (2025) · Frontiers in Veterinary Science · doi:10.3389/fvets.2025.1577046
Rao, Y., & Li, Q. (2025) · Machine learning · doi:10.1007/978-981-96-8853-1_3
Olvermann, M., Güntner, A. V., & Kauffeld, S. (2025) · Energy Sustainability and Society · doi:10.1186/s13705-025-00544-w
“These can be classified as excellent [62].”
Mobasseri, M., Vahabie, A., Jafari, G., Hatami, J., & Khosrowabadi, R. (2025) · Research Square · doi:10.21203/rs.3.rs-7736226/v1
Budde, H., Williams, G. A., Winkelmann, J., Pfirter, L., & Maier, C. B. (2021) · Figshare · doi:10.6084/m9.figshare.16893030
Güntner, A. V., & Nieberle, K. W. (2026) · Journal of Management Scientific Reports · doi:10.1177/27550311261419848 · Cited by 2
“The single index kappa for leader and follower behaviors was calculated as the average across all double-coding (Light, 1971), rendering a good agreement with κ = 0.68 for leader behaviors and κ = 0.65 for follower behaviors (Sachs, 1999).”
InfluentialDybdahl, K., Bak, J., Ekroll, E., Søreid, M. K., Hansen, D. F., Szejniuk, W. M., Bøgsted, M., & Vesteghem, C. (2026) · Clinical Epidemiology · doi:10.2147/clep.s567066
“13 To evaluate this, data entries from two raters at a time were matched based on the patient ID, clinical manifestation, and date.”
Hypercorrect Moun[thɨn] in Utah English
Stanley, J. A. (2026) · English World-Wide A Journal of Varieties of English · doi:10.1075/eww.25028.sta
Cluster effect expression for binary data
Yepnang, A. M. M. (2020) · doi:10.70675/5e3ed03fzabaaz473bzb89bzdecb583d6809
Smith, G. M. G., Long, A., Wragg, M., & Kent, L. (2026) · Mental Health & Prevention · doi:10.1016/j.mhp.2026.200520
Terslev, L., Schmidt, N. S., Fana, V., İnanç, N., DiRenzo, D., Gordon, R. A., Magni‐Manzoni, S., Mar, D. A., Misaki, K., Miguel, M. I., et al. (2026) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2026.153019
Statistical inference for indices of interrater agreement based on a homogeneity index
Bove, G., & Marella, D. (2026) · Statistical Methods & Applications · doi:10.1007/s10260-026-00846-z
“Moreover, the analysis of IRA respect to each category of the nominal scale on the whole groups of raters and targets (Fleiss 1971; Light 1971), and methods to detect subsets of raters who demonstrate a high level of interobserver agreement were also considered (Landis and Koch 1977a).”
Mental disorder prevalence in Australian youth with harmful sexual behaviours
O’Shannessy, D. M., Shumack, D. C., Maple, E., Holland, M., Legdin, T., & Rynne, J. (2026) · Psychiatry Psychology and Law · doi:10.1080/13218719.2026.2677792
Wretö, L., Fornander, L., Kämmerling, N., Nyman, E., & Moloney, M. (2026) · Acta Orthopaedica · doi:10.2340/17453674.2026.46344
“Cohen’s kappa value was used to determine both the intra-(reproducibility) and inter-rater agreement (reliability), and the inter-rater agreement [ 15 ] was computed as the mean of all observer-pairs [ 15,16 ].”
Biçer, B. K., Kıyak, Y. S., & Çoşkun, Ö. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2734312
Khatami, D., Shastri, R., & Shafei, B. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134498
Petersen, J. E., & Frantz, C. M. (2026) · Sustainability · doi:10.3390/su182010331
Unpacking colorectal TSR: Isolating the sources of discrepancy between human and AI assessment
Dikland, F. A., Montezuma, D., Wiers, J., Oliveira, D., Heijke, D., Fraga, J. B. D. P., Woodburn, J., Ribeiro, L., da Silva, R. S., Machado-Neves, R., et al. (2026) · Precision Pathology · doi:10.1016/j.prpath.2026.100013
Petelinsek, S., Pearson, O., Kelner, R. H., Olsen, B., Champion, N., Mahoney, K., Zhang, P., McIntosh, S. E., & Hughes, P. G. (2026) · Wilderness and Environmental Medicine · doi:10.1177/10806032261498049
A Joint Agreement Measure Between Multiple Raters and One Standard
Um, Y. (2005) · Journal of the Korean Data and Information Science Society
Metadata
- Authors
- Light, R. J.
- Affiliations
- Howard University (US)
- Corresponding author
- Light, Richard J.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 365-377
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1989492805
- Cited by
- 747
- References
- 17
- Citations by year
- 2026: 22 · 2025: 26 · 2024: 19 · 2023: 43 · 2022: 24 · 2021: 46 · 2020: 43 · 2019: 38 · 2018: 39 · 2017: 36 · 2016: 46 · 2015: 31 · 2014: 23 · 2013: 15 · 2012: 11
- FWCI
- 8.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- interobserver agreement, qualitative data, categorical data, agreement measures, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
