The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The kappa was .78, which reflects a moderate agreement between the reviewers [25,26].”
From Barriers to the Use of Mobile Health in Improving Health Outcomes in Developing Countries: Systematic Review (2019) · cites this paper as [25]
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Kendall, M. G. (1957) · Biometrika · doi:10.2307/2333282 · Cited by 6,442
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. (1979) · Springer series in statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,255
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,536
Estimation of the reliability of ratings.
Ebel, R. L. (1951) · Psychometrika · doi:10.1007/bf02288803 · Cited by 976
Unitizing and categorizing problems in coding qualitative data
Guetzkow, H. S. (1950) · Journal of Clinical Psychology · doi:10.1002/1097-4679(195001)6:1 · Cited by 436
Intraclass Correlation and the Analysis of Variance
Trese, T., & Haggard, E. A. (1958) · The American Catholic Sociological Review · doi:10.2307/3708305 · Cited by 401
A NEW VIEW OF INTER‐OBSERVER AGREEMENT1
Burdock, E. I., Fleiss, J. L., & Hardesty, A. S. (1963) · Personnel Psychology · doi:10.1111/j.1744-6570.1963.tb01283.x · Cited by 221
The Statistical Measurement of Agreement
Robinson, W. S. (1957) · American Sociological Review · doi:10.2307/2088760 · Cited by 221
The Robustness of Homogeneity Tests in 2 x N Tables
Lewontin, R. C., & Felsenstein, J. (1965) · Biometrics · doi:10.2307/2528349 · Cited by 194
MOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 156
Estimating the Accuracy of Dichotomous Judgments
Fleiss, J. L. (1965) · Psychometrika · doi:10.1007/bf02289537 · Cited by 58
Roy, S., & Mitra, S. K. (1956) · Biometrika · doi:10.1093/biomet/43.3-4.361 · Cited by 53
A Rapid Non-Parametric Estimate of Multi-Judge Reliability
Cartwright, D. S. (1956) · Psychometrika · doi:10.1007/bf02289084 · Cited by 20
Flagged first, then by citations
Cited by
Sorted by citations
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
Hallgren, K. A. (2012) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.08.1.p023 · Cited by 3,884
“Kappa was computed for each coder pair then averaged to provide a single index of IRR (Light, 1971).”
BackgroundMethodInfluentialRelation of sample size to the stability of component patterns.
Guadagnoli, E., & Velicer, W. F. (1988) · Psychological Bulletin · doi:10.1037/0033-2909.103.2.265 · Cited by 2,880
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167
Reliability of Nominal Data Based on Qualitative Judgments
Perreault, W. D., & Leigh, L. E. (1989) · Journal of Marketing Research · doi:10.2307/3172601 · Cited by 1,575
Metadata
- Authors
- Light, R. J.
- Affiliations
- Howard University (US)
- Corresponding author
- Light, Richard J.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 365-377
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1989492805
- Cited by
- 747
- References
- 17
- Citations by year
- 2026: 22 · 2025: 26 · 2024: 19 · 2023: 43 · 2022: 24 · 2021: 46 · 2020: 43 · 2019: 38 · 2018: 39 · 2017: 36 · 2016: 46 · 2015: 31 · 2014: 23 · 2013: 15 · 2012: 11
- FWCI
- 8.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- interobserver agreement, qualitative data, categorical data, agreement measures, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
