Cited by 750Reliability and Agreement in MeasurementPsychometric Methodologies and TestingSurvey Methodology and Nonresponse

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Measures of response agreement for qualitative data: Some generalizations and alternatives.

Light, Richard J.·1971·Psychological Bulletin

DOI 10.1037/h0031643

Open this paper in CoChat — free account

Integrity signals

Registry

Crossref · 10.1037/h0031643

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Rank Correlation Methods.
Kendall, M. G. (1957) · Biometrika · doi:10.2307/2333282 · Cited by 6,442
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. (1979) · Springer series in statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,255
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,536
Estimation of the reliability of ratings.
Ebel, R. L. (1951) · Psychometrika · doi:10.1007/bf02288803 · Cited by 976
Unitizing and categorizing problems in coding qualitative data
Guetzkow, H. S. (1950) · Journal of Clinical Psychology · doi:10.1002/1097-4679(195001)6:1 · Cited by 436
Intraclass Correlation and the Analysis of Variance
Trese, T., & Haggard, E. A. (1958) · The American Catholic Sociological Review · doi:10.2307/3708305 · Cited by 401

Show all 17 references

Sorted by citations

Items 501–600 of 750

Reliability of the CURL classification for proximal ulna fracture-dislocations of the elbow
Ghori, H., Bagga, R., Tathgar, A., Stone, A., Murphy, R., & Phadnis, J. (2023) · JSES International · doi:10.1016/j.jseint.2023.11.008 · Cited by 5
“23 The 4 observers were compared using a pairwise approach, resulting in 6 different statistical comparisons of interobserver agreement.”
Method
The value of additional electrodes when stereo‐electroencephalography is inconclusive
Ilyas, A., Vilella, L., Restrepo, C. E., Johnson, J. A., Pati, S. B. B., Lacuey, N., Lhatoo, S. D., Thompson, S. A., & Tandon, N. (2024) · Epilepsia · doi:10.1111/epi.17885 · Cited by 5
Vocal complexity in the long calls of Bornean orangutans
Erb, W. M., Ross, W., Kazanecki, H., Setia, T. M., Madhusudhana, S., & Clink, D. J. (2024) · PeerJ · doi:10.7717/peerj.17320 · Cited by 5
“Following this, we report supervised classification accuracy using SVM wherein we randomly divided our full dataset into a 60/40 split over 10 different iterations.”
Method
Monitoring in Youth Sport: A Paradigm Shift
Cook, E. V., & Dorsch, K. D. (2013) · Surveillance & Society · doi:10.24908/ss.v11i4.4751 · Cited by 5
Reproducibility of clinical data II: categorical outcomes
Blackman, N. J. (2004) · Pharmaceutical Statistics · doi:10.1002/pst.105 · Cited by 5
Families of Relational Statistics for 2 × 2 Tables
Heiser, W. J., & Warrens, M. J. (2010) · Interdisciplinary mathematical sciences · doi:10.1142/9789814299152_0003 · Cited by 5
Counselors’ Neuroscience Conceptualizations of Depression
Field, T. A., Beeson, E. T., Luke, C., Ghoston, M. R., & Golubovic, N. (2019) · Journal of Mental Health Counseling · doi:10.17744/mehc.41.3.05 · Cited by 7
“The authors evaluated inter-coder consistency by computing kappa correlation coefficients for multiple raters (McHugh, 2012) using the arithmetic mean of kappa coefficients to compute the corrected coefficient (Light, 1971).”
Method
Critical incidents of outsourcing processes in pharmaceutical supply chain: A mixed-methods approach
Marinkovic, V. D., Heine, I., Georgiev, A. M., & Schmitt, R. H. (2020) · Journal of Industrial Engineering and Management · doi:10.3926/jiem.3085 · Cited by 5
“between two raters, first the agreement between each pair of experts was evaluated and then the algorithmic mean was calculated to represent an overall agreement index (Light, 1971).”
BackgroundMethod
Incidence and Management of Articular Impaction in Geriatric Olecranon Fractures
Wadhwa, H., Goodnough, L. H., Finlay, A. K., DeBaun, M. R., Campbell, S. T., Hecht, G., Lucas, J. F., Bishop, J. A., & Gardner, M. J. (2021) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-20-01293 · Cited by 5
Predicting Success: An Examination of the Predictive Validity of a Measure of Motivational-Developmental Dimensions in College Admissions
Paris, J. H., Beckowski, C. P., & Fiorot, S. (2023) · Research in Higher Education · doi:10.1007/s11162-023-09743-w · Cited by 7
“To test the reliability of the MDS measure, first we computed a Light’s kappa statistic (Light, 1971) to measure interrater reliability between the essay readers as there was not a fixed number of readers for each essay question.”
Method
IIB or not IIB, part 2: assessing inter-rater and intra-rater repeatability of the Kenney–Doig scale in equine endometrial biopsy evaluation
Westendorf, J., Wobeser, B. K., & Epp, T. Y. (2021) · Journal of Veterinary Diagnostic Investigation · doi:10.1177/10406387211062866 · Cited by 5
“The arithmetic means of these kappa values were used to show an average agreement coefficient as described by Light for all 8 pathologists, the inter-institution group, and the intra-institution group.(25) Unweighted Light kappa values were computed using the Light kappa function in R; weighted Light kappa values were…”
Method
Comparison of Prostate Imaging and Reporting Data System V2.0 and V2.1 for Evaluation of Transition Zone Lesions: A 5-Reader 202-Patient Analysis
Kim, N. V., Kim, S., Prabhu, V., Shanbhogue, K., Smereka, P. N., Tong, A. Y., Anthopolos, R., Taneja, S. S., & Rosenkrantz, A. B. (2022) · Journal of Computer Assisted Tomography · doi:10.1097/rct.0000000000001313 · Cited by 5
Accuracy and Interrater Reliability of Point-of-Care Ultrasonography Image Interpretation for Intussusception
Bergmann, K. R., Khant, M., Lammers, S. M., Arroyo, A. C., Pablo Avendaño, U., Chaudoin, L. T., Cohen, S. G., Deanehan, J. K., Kornblith, A. E., Lam, S. H. F., et al. (2022) · Pediatric Emergency Care · doi:10.1097/pec.0000000000002786 · Cited by 5
“In particular, Light’s κ statistics was used to account for multiple raters and a fully crossed design.(15,19) The interpretation of Light’s κ statistics is the average Cohen’s κ statistics of all possible pairwise combinations of raters.”
Method
Spectral-domain Optical Coherence Tomography in Manifest Glaucoma: Its Additive Role in Structural Diagnosis
Kim, K. E., Oh, S., Jeoung, J. W., Suh, M. H., Seo, J. H., Kim, M., Park, K. H., Kim, D. M., & Kim, S. H. (2016) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2016.08.017 · Cited by 5
Potassium Hydroxide: An Alternative Reagent to Perform the Modified Apt Test
Chicaiza, H. P., Hellstrand, K., Lerer, T., Smith, S. R., & Sylvester, F. A. (2014) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2014.05.027 · Cited by 5
A Sequential Examination of Parent–Child Interactions at Anesthetic Induction
Wright, K. D., Stewart, S. H., Finley, G. A., & Raazi, M. (2014) · Journal of Clinical Psychology in Medical Settings · doi:10.1007/s10880-014-9413-4 · Cited by 5
Effects of color in filmed behavior sequences on description and elaboration by liberian schoolboys
Stone, V. (1983) · ECTJ · doi:10.1007/bf02765210 · Cited by 5
Micro-CONGRU: A microcomputer program for measuring levels of overall and partial congruence among multiple observers on nominal scales
Watkins, M. W., & Rush, J. C. (1988) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03202708 · Cited by 5
EFFECTS OF TRAINING ON DIMENSIONAL CLASSIFICATION ABILITIES: ADULT AGE COMPARISONS
Mergler, N. L., & Hoyer, W. J. (1981) · Educational Gerontology · doi:10.1080/0380127810060203 · Cited by 5
A general methodology for the measurement of observer agreement when the data are categorical
Landis, J. R. (1975) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 5
Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater ReliabilityPreprint
Wong, K., Paritosh, P. K., & Aroyo, L. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07393 · Cited by 5
“Since its publication, it has been generalized to accommodate multiple raters (Light, 1971; Berry and Mielke Jr, 1988), and to cover different types of annotation scales: ordinal (Cohen, 1968), interval (Berry and Mielke Jr, 1988; Janson and Olsson, 2001), multivariate (Berry and Mielke Jr, 1988), and any arbitrary di…”
Background
Building Computational Resources : The URDU.KON-TB Treebank and the Urdu Parser
Abbas, Q. (2014) · KOPS (University of Konstanz) · Cited by 5
What’s the concern about persistent chemicals? Insights from a Delphi study
Thiele, K., Tobi, H., & Gabbert, S. (2025) · Environmental Science & Policy · doi:10.1016/j.envsci.2025.104109 · Cited by 4
Inter-rater concordance of basal cell carcinoma subtypes: influences on reporting format and opportunities for further classification modifications
Maher, N. G., Prosser, N. J., Aivazian, K., Colebatch, A. J., Ferguson, P. M., Karim, R. Z., McKenzie, C. A., Rawson, R. V., Lo, S., & Scolyer, R. A. (2023) · Pathology · doi:10.1016/j.pathol.2023.04.003 · Cited by 4
The validity of pathology codes for biopsy-confirmed kidney disease in the Danish National Patobank
Møller, M., Bressendorff, I. O., Borg, R., Dieperink, H. H., Gregersen, J. W., Hansen, H. E., Hommel, K., Hornum, M., Ivarsen, P., Jensen, K. H., et al. (2024) · Clinical Kidney Journal · doi:10.1093/ckj/sfae203 · Cited by 4
“Using Light’s Kappa, the interrater reliability demonstrated a substantial agreement with a Kappa value of 0.65, albeit with some uncertainty ( P -value = .99) [12 , 13 ].”
Influential
Comparison between the ATS/ERS/JRS/ALAT criteria of 2011 and 2018 for Usual Interstitial Pneumonia on HRCT: a cross-sectional study
Wuyts, L. L., Camerlinck, M., Surgeloose, D. D., Vermeiren, L., Ceulemans, D., Clukers, J., & Slabbynck, H. (2021) · British Journal of Radiology · doi:10.1259/bjr.20201159 · Cited by 4
Cultural differences in self and affect through drawings of personal experiences
Yap, S., Ji, L., Chan, Y. P. M., & Zhang, Z. (2021) · Asian Journal Of Social Psychology · doi:10.1111/ajsp.12513 · Cited by 4
Interrater reliability of the DSM-5 and ICD-11 Criterion A for PTSD and complex PTSD in parents of children with autism using the Life Events Checklist
Hinde, K., Hald, G. M., Hallford, D. J., Lange, T., Arendt, M. C. B. B., Pavan, S., & Austin, D. (2025) · BJPsych Open · doi:10.1192/bjo.2024.848 · Cited by 4
“34 To estimate the sampling distribution and to be able to make inferences about the wider population, bootstrapping was used to test the two hypotheses.”
Evaluation of Questions Received by Community Pharmacists
Kirking, D. M., Makwym, C. J., Neterer, P. D., Ascione, F. J., & Colvin, C. L. (1986) · Drug Information Journal · doi:10.1177/009286158602000111 · Cited by 4
Operations strategy and analysis of competitive criteria: a case study of a food business
Lohmann, S., Lacerda, D. P., Camargo, L. F. R., & Dresch, A. (2019) · Gestão & Produção · doi:10.1590/0104-530x2290-19 · Cited by 4
“K Interpretation 0 Poor agreement 0.01 – 0.20 Slight agreement 0.21 – 0.40 Fair agreement 0.41 – 0.60 Moderate agreement 0.61 – 0.80 Substantial agreement 0.81 – 1.00 Almost perfect agreement Source: Light (1971).”
BackgroundInfluential
The effects of emotional stimuli on Word retrieval in people with aphasia
Blackett, D. S., Borod, J. C., Speer, S. R., Pan, X. J., & Harnish, S. M. (2023) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2023.108734 · Cited by 4
Reliability of Agreement between Insulin, Clonidine, and Glucagon Stimulation Tests for the Diagnosis of Growth Hormone Deficiency in Children: A Retrospective Cohort Study
Balwi, R. A., Al-Qahtani, M. H., Alrowished, A. K., Shash, H. A., Al-Atrash, R. S., Alhareth, A. S. A., Aldossary, R., Alahmari, M., Hejazi, Y., AlAmmari, A. M., et al. (2023) · Children · doi:10.3390/children10081381 · Cited by 4
“Statistical inter-rater reliability was computed to evaluate agreement across the three medications (raters), insulin, clonidine, and glucagon, using Cohen’s kappa for the poststimulated peak GH level of the positive and negative test results [18].”
Method
The experience and emergence of attitudinal consensus in conversations
Koudenburg, N., Kutlaca, M., & Kuppens, T. (2023) · European Journal of Social Psychology · doi:10.1002/ejsp.2992 · Cited by 4
Comparative analysis of real issues in open-source machine learning projects
Lai, T. D., Simmons, A., Barnett, S., Schneider, J.-G., & Vasa, R. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10467-3 · Cited by 4
“Theattachedtechnicalreportinthesupplementary materialsdescribesthedetailofthelabelling,adjustmentandhowweapplythetaxonomyto achieveamoderatelevelofagreementusingtheLight’sKappametric(Light1971).”
Fatores de Risco de Desordens Osteomusculares Presentes no Posto de Trabalho de Estudantes de Odontologia
Presoto, C. D., Corrocher, P. A., Campos, J. Á. D. B., & Garcia, P. P. N. S. (2012) · Pesquisa Brasileira em Odontopediatria e Clínica Integrada · doi:10.4034/pboci.2012.124.16 · Cited by 4
Mining Decision-Making Processes in Open Source Software Development
Sharma, P., Savarimuthu, B. T. R., & Stanger, N. (2020) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3383219.3383240 · Cited by 4
“To get an indication of overall agreement among raters, we computed the arithmetic mean of the κ values for each pair of raters, as suggested by the literature [10, 16].”
Method
Measures of Agreement for Incompletely Ranked Data
Iachan, R. (1984) · Educational and Psychological Measurement · doi:10.1177/0013164484444005 · Cited by 4
A Measure of Ordinal Concordance for the Evaluation of University Courses
Marasini, D., Quatto, P., & Ripamonti, E. (2014) · Procedia Economics and Finance · doi:10.1016/s2212-5671(14)00876-4 · Cited by 4
“We define the weights in the ordinal case as: 2,1, 1 1 rkj M kj w r jk (7) which generalizes the absolute error weigths (r = 1) and the square error weights in (7) with r = 2 (Cohen, 1968; Light, 1971; Fleiss, 1971).”
BackgroundMethod
Using computers to construct, evaluate, and apply actuarial systems of classification: A methodology and example
Hale, R. L., & Glassman, S. S. (1986) · Computers in Human Behavior · doi:10.1016/0747-5632(86)90003-8 · Cited by 4
Inter-Interpreter Agreement for ABR TracingsResults from 100 Patients under Otoneurologic Investigation
Rossman, R. N. F., & Cashman, M. Z. (1985) · Scandinavian Audiology · doi:10.3109/01050398509045915 · Cited by 4
Outpatient Chemotherapy Decision-making in Lung Cancer
Hirai, K., Tokoro, A., Naka, N., Ogawara, M., & Kawahara, M. (2005) · Haigan · doi:10.2482/haigan.45.105 · Cited by 4
A collaborative evaluation of a rapid automated bacterial identification system: the Autobac IDX
Stevens, M., Feltham, R. K. A., Schneider, F., Grasmick, C., Schaak, F., & Roos, P. (1984) · European Journal of Clinical Microbiology & Infectious Diseases · doi:10.1007/bf02017362 · Cited by 4
A GERONTOLOGICAL CONTINUING EDUCATION PROGRAM: THE RESIDENT ASSESSMENT INSTRUMENT AS A TEACHING TOOL FOR NURSING HOME PROFESSIONALS
Cipriani, L., Landi, F., Sgadari, A., Zuccalà, G., & Bernabei, R. (1995) · Educational Gerontology · doi:10.1080/0360127950210704 · Cited by 4
Determining the Factors Influential in the Validation of Computer-based Problem Solving Systems
Morehead, L. (2000) · doi:10.15760/etd.1244 · Cited by 4
An Experimental Investigation of Judges’ Ratings of Schizophrenics’ and Non-Schizophrenics’ Paintings
Langevin, R., & Hutchins, L. M. (1973) · Journal of Personality Assessment · doi:10.1080/00223891.1973.10119919 · Cited by 4
Design and evaluation of an automatic procedure for detection of large misregistration of medical images
Rodriguez-Carranza, C. E., & Loew, M. H. (2003) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2003.819297 · Cited by 4
“The proportion of overall agreement [27], the proportion of specific agreement [27], and Cohen’s kappa [27], [28] were computed.”
Method
A Capability-Demand Approach to the Aged in Technological Environments: A Case for Improved Task Analysis
Faletti, M. V., & Clark, M. C. (1984) · Springer eBooks · doi:10.1007/978-1-4613-2401-0_30 · Cited by 4
Functionality of symptoms and interpersonal communication in home video recordings of functional/dissociative versus epileptic seizures
Dissouky, N., Kochs, K., Nass, R. D., Baumgartner, T. R., Rácz, Á., Surges, R., Reuber, M., Witt, J., & Helmstaedter, C. (2026) · Epilepsia · doi:10.1002/epi.70107 · Cited by 3
“30 Each nominal variable was re-coded as a dichotomous variable, and Cohen kappa was calculated separately for each option.”
Development of Masked Speech Detection Thresholds in 2- to 15-year-old Children: Speech-Shaped Noise and Two-Talker Speech Maskers
Bonino, A. Y., Ramsey, M. E., Pancoast, E. S., & Vance, E. A. (2021) · Ear and Hearing · doi:10.1097/aud.0000000000001062 · Cited by 3
“Overall reliability was computed by averaging the kappa statistics across all test sessions (Light 1971).”
Method
The Homogeneity Index as a Measure of Interrater Agreement for Ratings on a Nominal Scale
Bove, G. (2023) · Studies in classification, data analysis, and knowledge organization · doi:10.1007/978-3-031-30164-3_2 · Cited by 3
Interobserver variability in lymph node evaluation with endoscopic ultrasonography in cholangiocarcinoma
de Jong, D. M., Roosterman, D., Bruno, M. J., van Driel, L. M. J. W., & Lammers, W. J. (2025) · Endoscopy International Open · doi:10.1055/a-2577-5449 · Cited by 3
The Effects of Health Information Technology on Quality of Care in Emergency Departments: A Systematic Review
Kruse, C. S., Mileski, M., Herzog, B., Frye, L. M., Spencer, J., & Stevenson, E. A. (2025) · Health Science Reports · doi:10.1002/hsr2.70962 · Cited by 5
“This kappa level illustrates near perfect agreement [16, 17].”
Globalization/Americanization and Negotiating National Dreams: Representations of Culture and Economy in Israeli Advertising
First, A., & Avraham, E. (2007) · Israeli Studies Review · doi:10.3167/isf.2007.220103 · Cited by 3
Accuracy Assessment of Thematic Maps Using Inter-Class Spectral Distances
Knight, J. (2002) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 3
Misunderstood and Unattributed: Revisiting M. H. Doolittle's Measures of Association, With a Note on Bayes’ Theorem
Armistead, T. W. (2015) · The American Statistician · doi:10.1080/00031305.2015.1086686 · Cited by 3
“Cohen proposed κ as a measure of inter-rater chance-corrected agreement–one of many such measures that had been developed for categorical data since the 1940s (e.g., Guttman 1941; Guilford 1950; Goodman and Kruskal 1954, 1959; Scott 1959; Light 1971; O’Connell and Dobson 1984).”
Background
Which symptom to address in psychological treatment for cancer survivors when fear of cancer recurrence, depressive symptoms, and cancer-related fatigue co-occur? Exploring the level of agreement between three systematic approaches to select the focus of treatment
Harnas, S. J., Booij, S. H., Csorba, I., Nieuwkerk, P. T., Knoop, H., & Braamse, A. M. J. (2023) · Journal of Cancer Survivorship · doi:10.1007/s11764-023-01423-z · Cited by 3
“To determine the agreement between the three systematic approaches about treatment advice, we calculated an overall index of agreement according to the Light’s kappa statistic [52, 53].”
Method
Rapid development of individual identification and presence systems for a critically endangered antelope, the Mountain bongo
Sandri, T., Bunge, D., Omengo, F. O., Cain, B., Jones, M., & Harris, W. E. (2023) · African Journal of Ecology · doi:10.1111/aje.13201 · Cited by 3
The Ecology of Dating Preferences Among Asian American Adolescents in Emerging Immigrant Communities
Chan, M., & Kiang, L. (2020) · Advances in immigrant family research · doi:10.1007/978-3-030-50235-5_5 · Cited by 3
Inter-Rater Reliability of Unstructured Text Labeling: Artificially vs. Naturally Intelligent Approaches
Danilov, G., Kosyrkova, A., Shults, M. A., Melchenko, S., Tsukanova, T., Shifrin, M. A., & Aa, P. (2021) · Studies in health technology and informatics · doi:10.3233/shti210132 · Cited by 3
“G. Danilov et al. / Inter-Rater 122 Reliability of Unstructured Text Labeling”
Background
Direct observation of classroom behavior
Emmer, E. T. (1972) · International Review of Education · doi:10.1007/bf01450310 · Cited by 3
Some Advances in Permutation Testing
Samuh, M. H. (2011) · Cited by 3
Imitation in a Naturalistic Setting as a Function of Response Uncertainty
Miller, D. J., & Thelen, M. H. (1986) · The Journal of Genetic Psychology · doi:10.1080/00221325.1986.9914486 · Cited by 3
Modeling Physical Activity in Children by Combining Raw Hip-Worn Accelerometry, 2D Pose Estimation, and Direct Observation
Horak, H., Jermakovs, K., & Haamer, R. E. (2022) · IEEE Access · doi:10.1109/access.2022.3160474 · Cited by 3
An Analysis of Variance Model for Assessing Reliability of Naturalistic Observations
Berk, R. A. (1978) · Perceptual and Motor Skills · doi:10.2466/pms.1978.47.1.271 · Cited by 3
ASSESSING VARIABILITY OF AGREEMENT MEASURES IN REMOTE SENSING USING A BAYESIAN APPROACH
Price, W. J., Shafii, B., Lass, L. W., & Thill, D. C. (1998) · Conference on Applied Statistics in Agriculture · doi:10.4148/2475-7772.1276 · Cited by 3
Distal femoral osteotomy for the treatment of chronic patellofemoral instability improves gait patterns
Rab, P., Vieider, R. P., Fritsch, L., Cotic, M., Imhoff, F. B., Siebenlist, S., Achtnich, A., & Hinz, M. (2025) · Archives of Orthopaedic and Trauma Surgery · doi:10.1007/s00402-025-05788-x · Cited by 2
“Interrater reliability among three observers was assessed using Light’s kappa [47] and interpreted according to Landis and Koch (≤ 0.20: slight agreement, 0.21 to 0.40: fair agreement, 0.41 to 0.60: moderate agreement, 0.61 to 0.80: substantial agreement, and 0.81 to 1.0: almost perfect or perfect…”
Influential
Owner-Perceived Undesirable Behaviours in Young Dogs and Changes with Age
Kinsman, R., Casey, R. A., Tasker, S., Cooper, B., Giragosian, K., Harvey, N. D., Owczarczak‐Garstecka, S. C., Samet, L. E., & Murray, J. (2025) · Animals · doi:10.3390/ani15081163 · Cited by 3
“Gwet’s AC1 for each coder pair were computed then averaged to provide a single index of IRR [63].”
Evaluating long-term retention of calcein marks on Brook Trout otoliths
Schwepe, C., Shields, R., Wolf, S. L., & Dudding, J. (2025) · North American Journal of Fisheries Management · doi:10.1093/najfmt/vqaf059 · Cited by 2
Cracking the code: AI’s role in mapping evidence syntheses to the United Nations sustainable development goals
Pilla, B., Stone, J., & Jordan, Z. (2025) · Archives of Public Health · doi:10.1186/s13690-025-01784-0 · Cited by 3
“For fully crossed designs with three or more coders, Light [41] suggests computing kappa for all coder pairs and then using the arithmetic mean of these estimates to provide an overall index of agreement.”
Understanding when anger becomes productive or destructive in collaborative educational games
Kim, T., & Tissenbaum, M. (2025) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-025-09459-2 · Cited by 3
Reprodutibilidade de uma escala odontológica proposta como indicador de saúde bucal em crianças e adolescentes HIV+/SIDA
Campos, J. Á. D. B., & Loffredo, L. D. C. M. (2010) · Ciência & Saúde Coletiva · doi:10.1590/s1413-81232010000400032 · Cited by 2
Ordinal-Level Variables, I
Berry, K. J., Johnston, J. E., & Mielke, P. W. (2018) · Springer eBooks · doi:10.1007/978-3-319-98926-6_5 · Cited by 2
Intra- and inter-rater reliability of a manual codification system for footwear impressions: first lessons learned from the development of a footwear database for forensic intelligence purposes
Mousseau, V., Tapps, M., Volery, R., & Brazeau, J. (2023) · Canadian Society of Forensic Science Journal · doi:10.1080/00085030.2023.2278911 · Cited by 2
“In addition, since we used a fully crossed design with more than two raters (3), we calculated Light’s Kappa [34,35] which is the mean of the kappas for all pairs of raters, to obtain an overall level of agreement.”
Perceptual Evaluation of the Tracheoesophageal Voice: Italian Validation of the Sunderland Tracheoesophageal Perceptual Scale
D’Alatri, L., Longobardi, Y., Marchese, M. R., Meschini, C., Figliolia, D., Mari, G., Rodolico, D., Parrilla, C., & Galli, J. (2023) · Journal of Voice · doi:10.1016/j.jvoice.2023.11.007 · Cited by 2
A content analysis of memoranda of understanding between Texas Public School Districts and law enforcement agencies
Valerio, L. R., Vaughn, M. A., Ekren, E., & Barnes, V. R. (2024) · Policing A Journal of Policy and Practice · doi:10.1093/police/paae019 · Cited by 2
The Impact of Human Resource Management Quality on Work Life Quality through The Moderating Role of Organizational Justice: The Case Study of the Expatriate Employees
Gdoura, I. A., & Trabelsi, K. (2024) · Global Journal of Human Resource Management · doi:10.37745/gjhrm.2013/vol12n31529 · Cited by 2
Evaluating Inter- and Intraobserver Agreement on Pectus Carinatum Severity and Treatment Outcomes: A Comparison of Subjective and Objective Assessment Methods
van Braak, H., de Beer, S. A., Zwaveling, S., Oomen, M. W., van Heurn, E. L. W., & de Jong, J. R. (2024) · European Journal of Pediatric Surgery · doi:10.1055/a-2466-6407 · Cited by 2
“10 Additionally, the interobserver agreement per observer was based on the weighted arithmetic mean of all observer pairs that included the observer in question.”
Concordance of performance metrics among US trauma centers caring for injured children
Sathya, C., Burd, R. S., Nance, M. L., Karanicolas, P. J., Wales, P. W., Scales, D. C., Xiong, W. G., & Nathens, A. B. (2015) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000000678 · Cited by 2
“Light’s weighted Jwas used to test for overall agreement among all four indicators.(22)All statistical analyseswere performedusing SAS version 9.”
Method
Making Connections Across a Four-Year Project-Based Curriculum: ePortfolios as a Space for Reflection and Integrative Learning
Demetry, C., Quinn, P., & Kmiotek, S. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--33076 · Cited by 2
Weighted conditional kappa
Kvålseth, T. O. (1985) · Bulletin of the Psychonomic Society · doi:10.3758/bf03329866 · Cited by 2
Testing stochastic orders in tails of contingency tables
Ng, C. T., Lim, J., & Hahn, K. S. (2011) · Journal of Applied Statistics · doi:10.1080/02664763.2010.484487 · Cited by 2
“Additionally, the notion of κ is generalized to the cases with two or more ratings or with varying sets of ratings (Fleiss, 1971; Light, 1971; Landis and Koch, 1977a,b; Fleiss and Cuzick, 1979; Lau, 1993).”
Background
ASSESSING ATTITUDINAL CONGRUENCE: A CASE FOR ABSOLUTE (AS WELL AS RELATIVE) INDICES
Sirotnik, K. A. (1981) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1981.tb00853.x · Cited by 2
The use of offensive public relations during a conflict: Hamas's efforts to damage Israel's image during the 2010 flotilla
Mozes-Sadeh, T., & Avraham, E. (2014) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 2
Expert-Rater Agreement and Inter-/Intrarater Reliability for the Furtado–Gallagher Computerized Observational Movement Pattern Assessment System
Furtado, O., & Gallagher, J. D. (2018) · Perceptual and Motor Skills · doi:10.1177/0031512518769205 · Cited by 2
“In Study 2, we compared the interrater agreement of three participants (see Table 2) by computing the weighted k and AC 1 for each pair and then averaged these values to produce an index of reliability among raters (Light, 1971).”
Method
Assessing the utility of the Clinical Behavioral Case Conceptualization categories: A contextual behavioral based formulation model
Novoa-Gómez, M. M., Pulido-Castelblanco, D. P., & Muñoz-Martínez, A. (2020) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2020.08.005 · Cited by 2
A Unified Treatment of Agreement Coefficients and their Asymptotic Results: the Formula of the Weighted Mean of Weighted Ratios
Ogasawara, H. (2020) · Journal of Classification · doi:10.1007/s00357-020-09366-1 · Cited by 2
Automatic Recognition of Emotional Subgroups in Images
Veltmeijer, E. A., Gerritsen, C., & Hindriks, K. V. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/190 · Cited by 2
“For the emotion task, there is full agreement on 81.8% of the faces, for the other 18.2% of faces two out of three annotators agree on the emotion, with an inter-rater reliability (IRR) of κ =0.77 (Light’s kappa [Light, 1971]).”
BackgroundInfluential
3D-validation of a simple tool to measure tibiofemoral axial rotation in tibial plateau fractures
Tseng, H. M., Bartels, H., Noppe, N., Vancleef, S., Herteleer, M., & Hoekstra, H. (2023) · European Radiology · doi:10.1007/s00330-023-09947-9 · Cited by 2
“For agreement of rotation classification, the intra-observer agreement was tested with Cohen’s kappa, and the inter-observer agreement between different observers was tested with Light’s kappa [23], interpreted according to Landis and Koch [24].”
Method
Becoming an employer of choice: Lessons learned from a case study.
Stroh, L. K. (2003) · The Psychologist-Manager Journal · doi:10.1037/h0095919 · Cited by 2
Preferences of end-users for raw and post-edited NMT in a business environment
Girletti, S., Bouillon, P., Bellodi, M., & Ursprung, P. (2019) · Archive ouverte UNIGE (University of Geneva) · Cited by 2
Interrater Reliability for a Two-Interval, Observer-Based Procedure for Measuring Hearing in Young Children
Bonino, A. Y., Wiens, A., Nightengale, E. C., & Vance, E. A. (2020) · American Journal of Audiology · doi:10.1044/2020_aja-20-00022 · Cited by 2
A computer program for assessing conjoint interrater agreement with a correct set of classifications
McDermott, P. A., & Watkins, M. W. (1979) · Behavior Research Methods · doi:10.3758/bf03201398 · Cited by 2
Alternative inference models based on matching for a weighted index of nominal scale response agreement
Hubert, L. J. (1980) · Quality & Quantity · doi:10.1007/bf00145804 · Cited by 2
Response to gallamine: An indicator of diminished neuromuscular function in experimental autoimmune myasthenia gravis
Hinman, C. L., Dambach, G. E., Hudson, R. A., & Rauch, H. C. (1985) · Journal of Neuroscience Research · doi:10.1002/jnr.490140212 · Cited by 2
A Fortran Program for Testing Agreement of Multiple Observers With a Categorical Standard on Nominal Scales
McDermott, P. A., & Watkins, M. W. (1979) · Educational and Psychological Measurement · doi:10.1177/001316447903900321 · Cited by 2
“Light (1971) and Fleiss (1971) later devised variations and extentions of K, such as those represented by the statistic Km, that are appropriate for use in situations involving multiple (i.e., more than two) observers.”
Method
Transfer between conjunctive and disjunctive attribute identification problems.
Sanders, R. E., Beley, W., & Speroff, T. (1977) · Journal of Experimental Psychology Human Learning & Memory · doi:10.1037//0278-7393.3.3.337 · Cited by 2
Discussion: A procedure for evaluating the reliability of a gingival index, and, A comparison of 3 clinical indices for measuring gingivitis
Varma, A. O. (1986) · Journal Of Clinical Periodontology · doi:10.1111/j.1600-051x.1986.tb01481.x · Cited by 2
What Is Meant by Agreement
Brook, R. H. (1976) · New England Journal of Medicine · doi:10.1056/nejm197601152940331 · Cited by 2
The Alignment of Knowledge Strategies
Denford, J. S. (2009) · QSpace (Queen's University Library) · Cited by 2
An evaluation of the reliability and quality of expert and novice forensic case formulations
Ryan, T. J. (2020) · Summit (Simon Fraser University) · Cited by 2

Authors: Light, R. J.
Affiliations: Howard University (US)
Corresponding author: Light, Richard J.
Venue: Psychological Bulletin
Publisher: American Psychological Association (APA)
Published: 1971-11
Volume: 76
Issue: 5
Pages: 365-377
ISSN: 0033-2909 (ISSN-L)
1939-1455
DOI: 10.1037/h0031643
OpenAlex: W1989492805
Cited by: 747
References: 17
Citations by year: 2026: 22 · 2025: 26 · 2024: 19 · 2023: 43 · 2022: 24 · 2021: 46 · 2020: 43 · 2019: 38 · 2018: 39 · 2017: 36 · 2016: 46 · 2015: 31 · 2014: 23 · 2013: 15 · 2012: 11
FWCI: 8.62
Type: journal article
Language: en
Open access: closed
Locations: Psychological Bulletin (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Keywords: interobserver agreement, qualitative data, categorical data, agreement measures, inter-rater reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Measures of response agreement for qualitative data: Some generalizations and alternatives.

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks