Cited by 750Reliability and Agreement in MeasurementPsychometric Methodologies and TestingSurvey Methodology and Nonresponse

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Measures of response agreement for qualitative data: Some generalizations and alternatives.

Light, Richard J.·1971·Psychological Bulletin

DOI 10.1037/h0031643

Open this paper in CoChat — free account

Integrity signals

Registry

Crossref · 10.1037/h0031643

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Rank Correlation Methods.
Kendall, M. G. (1957) · Biometrika · doi:10.2307/2333282 · Cited by 6,442
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. (1979) · Springer series in statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,255
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,536
Estimation of the reliability of ratings.
Ebel, R. L. (1951) · Psychometrika · doi:10.1007/bf02288803 · Cited by 976
Unitizing and categorizing problems in coding qualitative data
Guetzkow, H. S. (1950) · Journal of Clinical Psychology · doi:10.1002/1097-4679(195001)6:1 · Cited by 436
Intraclass Correlation and the Analysis of Variance
Trese, T., & Haggard, E. A. (1958) · The American Catholic Sociological Review · doi:10.2307/3708305 · Cited by 401

Flagged first, then by citations

Show all 17 references

Sorted by citations

Items 401–500 of 750

Inter-rater reliability of the diagnosis of otitis media based on otoscopic images and wideband tympanometry measurements
Sundgaard, J. V., Värendh, M., Nordström, F. U., Kamide, Y., Tanaka, C., Harte, J. M., Paulsen, R. R., Christensen, A. N., Bray, P., & Laugesen, S. (2022) · International Journal of Pediatric Otorhinolaryngology · doi:10.1016/j.ijporl.2021.111034 · Cited by 11
Demographic characteristics, torture experiences, and posttraumatic stress disorder symptoms among asylum seekers and refugees persecuted for same‐sex behaviors
Bird, C., Bowers, G., Piwowarczyk, L. A., & Ng, L. C. (2022) · Journal of Traumatic Stress · doi:10.1002/jts.22827 · Cited by 11
Estimating intercoder reliability: a structural equation modeling approach
Feng, G. C. (2014) · Quality & Quantity · doi:10.1007/s11135-014-0034-7 · Cited by 11
Effortful versus automatic emotional processing in schizophrenia: Insights from a face-vignette task
Patrick, R. E., Rastogi, A., & Christensen, B. K. (2014) · Cognition & Emotion · doi:10.1080/02699931.2014.935297 · Cited by 11
“Participants read the vignettes aloud to ensure each 1Given that responses were made on a nominal (i.e., categorical) scale, inter-rater reliability was obtained by computing the arithmetic mean of all kappa (κ) values (Siegel & Castellan, 1988) for each possible rater pair (see Light, 1971).”
Method
Can electronic medical images replace hard‐copy film? Defining and testing the equivalence of diagnostic tests
Obuchowski, N. A. (2001) · Statistics in Medicine · doi:10.1002/sim.929 · Cited by 11
“Single measures of agreement such as the kappa statistic [15] and the intraclass correlation coe8cient (ICC) [16] and various extensions [17–24] to multiple tests or multiple readers are popular.”
Background
Item Reliability of the Milani-Comparetti Motor Development Screening Test
Stuberg, W. A., White, P. J., Miedaner, J. A., & Dehne, P. R. (1989) · Physical Therapy · doi:10.1093/ptj/69.5.328 · Cited by 11
REFORMULATION AND GENERALISATION OF THE COHEN AND FLEISS KAPPAS
Xie, Z., Gadepalli, C., & Cheetham, B. M. G. (2017) · LIFE International Journal of Health and Life-Sciences · doi:10.20319/lijhls.2017.33.115 · Cited by 11
Risk factors for choroidal neovascularization in young patients: A case-control study
Derosa, J. T., Yannuzzi, L. A., Marmor, M. F., Fotino, M., Sorenson, J. A., & Spaide, R. F. (1995) · Documenta Ophthalmologica · doi:10.1007/bf01204172 · Cited by 11
Consensus Opinion and Observer Accuracy in Electrocardiography with Reference to Coronary Arteriographic Information
Weston, M. J., Bett, J. H. N., & Over, R. (1976) · Australian and New Zealand Journal of Medicine · doi:10.1111/j.1445-5994.1976.tb03029.x · Cited by 11
Measuring Agreement Between Two Raters for Ordinal Response: a Model-based Approach
Basu, S., Basu, A., & Raychaudhuri, A. (1999) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.1111/1467-9884.00192 · Cited by 11
“Several indices have been proposed specifically for the agreement between two raters for ordinal and nominal classifications (Goodman and Kruskal, 1954; Cohen, 1960, 1968; Fleiss, 1971; Light, 1971; Cicchetti, 1972; Uebersax, 1981, 1982).”
Background
Accuracy assessments and areal estimates using two-phase stratified random sampling, cluster plots, and the multivariate composite estimator
Czaplewski, R. L. (2000) · Cited by 11
Assessment and design of industrial manual handling to reduce physical ergonomics hazards – use and development of assessment tools
Lind, C. M. (2017) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · doi:10.13140/rg.2.2.20231.91044 · Cited by 11
Phosphorylated S6 ribosomal protein expression by immunohistochemistry correlates with de novo donor-specific HLA antibodies in lung allograft recipients
Cone, B. D., Zhang, J. Q., Sosa, R. A., Calabrese, F., Reed, E. F., & Fishbein, G. A. (2021) · The Journal of Heart and Lung Transplantation · doi:10.1016/j.healun.2021.06.021 · Cited by 10
Interventions to reduce social isolation and food insecurity in older adults: a systematic review
Sen, K., Kruse, C. S., Mileski, M., & Ramamonjiarivelo, Z. H. (2025) · Frontiers in Nutrition · doi:10.3389/fnut.2025.1607057 · Cited by 11
“A Cohen kappa was calculated from this process ( k = 0.95, high agreement) (35, 36).”
Influential
Multicolor Imaging for Detection of Retinal Nerve Fiber Layer Defect in Myopic Eyes With Glaucoma
Kim, Y. H., Ahn, J., & Kim, K. E. (2021) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2021.07.022 · Cited by 10
Treatment of anxiety symptoms in patients receiving rTMS for treatment resistant depression
Tuinstra, D., Percifield, C., Stilwell, K., Plattner, A., Edwards, E., Sanders, W., & Koval, M. (2021) · Psychiatry Research Communications · doi:10.1016/j.psycom.2021.100014 · Cited by 9
A pragmatic feasibility trial of the Primary Care Intervention for PTSD: A health service delivery model to reduce health disparities for low-income and BIPOC youth
Ng, L. C., Miller, A. N., Bowers, G., Cheng, Y., Brigham, R., Tai, M. H., Smith, A. M., Mueser, K. T., Fortuna, L. R., & Coles, M. S. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104310 · Cited by 10
“Kappa was calculated for each theme to assess inter-rater reliability with strong overall agreement ( κ at light ’ s kappa = .828 (Light, 1971).”
Method
A Binary Integer Program to Maximize the Agreement Between Partitions
Brusco, M. J., & Steinley, D. (2008) · Journal of Classification · doi:10.1007/s00357-008-9013-9 · Cited by 10
A computational algorithm for classifying step and spin turns using pelvic center of mass trajectory and foot position
Golyski, P. R., & Hendershot, B. D. (2017) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2017.01.023 · Cited by 10
“Intra- and inter-rater reliability were assessed using Cohen’s kappa (Cohen, 1960), adjusted for greater than two ratings within a set (Light, 1971) and qualitatively interpreted according to criteria set forth by Fleiss (1986): poor (0.00–0.39), fair (0.40–0.59), good (0.60–0.74), and excellent…”
MethodInfluential
EndoNuke: Nuclei Detection Dataset for Estrogen and Progesterone Stained IHC Endometrium Scans
Naumov, A., Ushakov, E. N., Ivanov, A., Midiber, K. Y., Николаевна, Х. Т., Konyukova, A. K., Vishnyakova, P., Nora, S. A., Михалева, Л. М., Fatkhudinov, T., et al. (2022) · Data · doi:10.3390/data7060075 · Cited by 10
“As advised in [30,31], we used mean kappa to obtain the final agreement measure of the cohort.”
Method
Leveraging mHealth and Wearable Sensors to Manage Alcohol Use Disorders: A Systematic Literature Review
Kruse, C. S., Betancourt, J., Madrid, S., Lindsey, C. W., & Wall, V. (2022) · Healthcare · doi:10.3390/healthcare10091672 · Cited by 13
Enhancing Accuracy in Observational Test Scoring: The Comprehensive System As a Case Example
McGrath, R. E. (2003) · Journal of Personality Assessment · doi:10.1207/s15327752jpa8102_02 · Cited by 10
“Despite the important distinction, the statistics generally considered appropriate for rater-standard reliability are the same as those used for evaluating interrater reliability, although they are used in slightly different ways when there is more than one rater. Light (1971) concluded that the best statistic for est…”
BackgroundMethodInfluential
Individual differences in use of left-right visual cues: A reexamination of mirror-image confusions in preschoolers.
Casey, M. B. (1984) · Developmental Psychology · doi:10.1037//0012-1649.20.4.551 · Cited by 10
Models of Sexual Consent Communication by Film Rating: A Content Analysis
Willis, M., Jozkowski, K. N., Canan, S. N., Rhoads, K. E., & Hunt, M. E. (2020) · Sexuality & Culture · doi:10.1007/s12119-020-09731-7 · Cited by 10
“We calculated Light’s kappa for the variable assessing whether a scene was consensual because the data were nominal and we had more than two coders (Light 1971).”
Method
Children of Divorced Parents: A Perceptual Comparison.
Stangeland, C. S. (1989) · Elementary school guidance and counseling · Cited by 10
The effect of the Holocaust on selected socio‐political attitudes of adult children of survivors
Weinfeld, M., & Sigal, J. J. (1986) · Canadian Review of Sociology/Revue canadienne de sociologie · doi:10.1111/j.1755-618x.1986.tb00404.x · Cited by 10
Support: Exploration of a Basic Dimension of Informal Help and Counseling
Pearson, R. E. (1982) · The Personnel and Guidance Journal · doi:10.1002/j.2164-4918.1982.tb00733.x · Cited by 10
A computer program to compute measures of response agreement for nominal scale data obtained from two judges
Antonak, R. F. (1977) · Behavior Research Methods & Instrumentation · doi:10.3758/bf03214005 · Cited by 10
A Modification of Kappa for Interobserver Bias
Krauth, J. (1984) · Biometrical Journal · doi:10.1002/bimj.4710260414 · Cited by 10
A Computer Program for Measuring Levels of Overall and Partial Congruence Among Multiple Observers on Nominal Scales
Watkins, M. W., & McDermott, P. A. (1979) · Educational and Psychological Measurement · doi:10.1177/001316447903900137 · Cited by 10
“Light (1971) has developed the statistic Km, referred to in this paper as the multiple observer congruence statistic. This statistic represents an extention of K to the case in which many observers categorize subjects and in which the overall conjoint agreement among observers is assessed. Fleiss (1971) has described…”
Background
Control variable use and reporting in operations management: a systematic literature review and revisit
Curado, C., Oliveira, M. F. D. A., Schniederjans, D. G., & Teixeira, E. K. (2023) · Management Review Quarterly · doi:10.1007/s11301-023-00348-2 · Cited by 10
“We then used the arithmetic mean of the pairwise estimates to provide an index of agreement (Light 1971).”
Interpretation of p16 and p53 in the Classification of Squamous Cell Carcinoma of the Vulva—An Interobserver Agreement Study
Jeffus, S. K., Wooldridge, J. T., Hoang, L. N., Parra‐Herran, C., Hamza, M., Lindsey, M., Verret, M., Zoumberos, N., Fogel, B., Wyeth, A., et al. (2024) · The American Journal of Surgical Pathology · doi:10.1097/pas.0000000000002336 · Cited by 9
Reinforcement learning profiles and negative symptoms across chronic and clinical high-risk phases of psychotic illness
Spilka, M. J., Raugh, I. M., Berglund, A. M., Visser, K. F., & Strauss, G. P. (2022) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-022-01528-z · Cited by 10
“Agreement among the three cluster solutions was then assessed using Light’s Kappa [66].”
Method
MDMA. Un modèle pour l’identification et l’annotation des marqueurs discursifs « potentiels » en contexte
Bolly, C., Crible, L., Degand, L., & Uygur, D. (2015) · Discours · doi:10.4000/discours.9009 · Cited by 9
“Toutefois, le choix de l’un ou l’autre score – α de Krippendorff (1980), π de Scott (1955), κ de Fleiss (1971) ou de Light (1971) – dépend de plusieurs facteurs (avec ou sans factorisation de la chance, nombre d’annotateurs, équivalence ou non des catégories annotées) et implique certaines…”
Police Shootings in Black and White: Exploring Newspaper Coverage of Officer-Involved Shootings
Klahm, C. F., Papp, J., & Rubino, L. (2016) · Sociology of crime, law and deviance · doi:10.1108/s1521-613620160000021011 · Cited by 9
Agreement Among Clinician, Patient, and Independent Interviewer Ratings of Adaptive Functioning
Nakash, O., Nagar, M., & Westen, D. (2017) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0000000000000765 · Cited by 9
“Overall agreement among all three raters in Cohen's kappa was calculated as the arithmetic mean between all pairs (Light, 1971).”
Method
The Evolution of Media Reporting of Ambush Marketing
Burton, N., Snyder, K., & McKelvey, S. (2018) · Sport Marketing Quarterly · doi:10.32731/smq.271.032018.02 · Cited by 9
“Given the difficulty in establishing interrater agreement for multiple coders in a multi-item scale study (Light, 1971; Davies & Fleiss, 1982), an adjusted score was calculated combining adjacent scores (Stemler, 2004).”
Testing Patterned Hypotheses in Multi-Way Contingency Tables Using Weighted Kappa and Weighted Chi Square
Ross, D. C. (1977) · Educational and Psychological Measurement · doi:10.1177/001316447703700203 · Cited by 9
Inter-rater reliability of published flow diversion occlusion scales
Mazur, M. D., Taussky, P., Shah, L. M., Winegar, B. A., & Park, M. S. (2016) · Journal of NeuroInterventional Surgery · doi:10.1136/neurintsurg-2015-012193 · Cited by 9
Analyzing Abusive Text Messages to Detect Digital Dating Abuse
Roy, T., McClendon, J. L., & Hodges, L. F. (2018) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi.2018.00039 · Cited by 9
“SPAM Critical Mixed td-idf count td-idf count td-idf count LVC(Uni) [46,1][4,35] [36,12][2,38] [45,3][4,36] [39,9][1,39] [45,3][7,33] [38,10][2,38] LVC(Bi) [46,2][15,25] [28,20][1,39] [45,3][15,25] [24,24][0,40] [46,2][19,21] [28,20][0,40] LVC(Tri) [48,0][34,36] [6,42][0,40] [46,2][36,4] [5,43][0,40] [47,1][38,2] [5,4…”
BackgroundInfluential
Interactive role-play with climate policy simulation can motivate evidence-based climate action
Rooney‐Varga, J. N., Coleman, R. L., Jones, A. P., Kapmeier, F., Newsome, P., Noiseux, K., Patten, B., Rath, K., & Sterman, J. D. (2025) · Communications Earth & Environment · doi:10.1038/s43247-025-02744-w · Cited by 8
“Participants ’ text responses were then coded independently by two researchers, calculating Light ’ s kappa for interrater reliability 62 to ensure accurate interpretation of coded themes, with “ substantial ” agreement found for both actions planned ( Κ = 0.72, N = 406 coded comments) and taken (…”
Influential
Leveraging mHealth for the Treatment and Management of PLHIV
Kruse, C. S., Pacheco, G. J., Rosenthal, N., Kopp, C. J., Omorotionmwan, O., & Cruz, J. E. (2023) · Risk Management and Healthcare Policy · doi:10.2147/rmhp.s403946 · Cited by 8
Patient-reported Joint Count in Juvenile Idiopathic Arthritis: The Reliability of a Manikin Format
Dijkstra, M., Anink, J., van Pelt, P. A., Hazes, J. M. W., & van Suijlekom‐Smit, L. W. A. (2014) · The Journal of Rheumatology · doi:10.3899/jrheum.140073 · Cited by 8
“47 87 55 92 9 (12) 13 (17) Ankle right 0.”
Competing Conversations: An Examination of Competition as Intrateam Interactions
Abraham, E. K., McCusker, M. E., & Foti, R. J. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00970 · Cited by 8
Reliability and Change in Erosion Measurements by High-resolution Peripheral Quantitative Computed Tomography in a Longitudinal Dataset of Rheumatoid Arthritis Patients
Finzel, S., Manske, S. L., Barnabe, C. C. M., Burghardt, A. J., Marotte, H., Scharmga, A. M. J., Hauge, E., Chapurlat, R. D., Engelke, K., Li, X., et al. (2020) · The Journal of Rheumatology · doi:10.3899/jrheum.191391 · Cited by 8
Visual diagnosis of pectus excavatum: An inter-observer and intra-observer agreement analysis
Daemen, J. H. T., de Loos, E. R., Geraedts, T. C. M., Veer, H. G. L. V., Huijstee, P. J. V., Elenbaas, T. W. O., Hulsewé, K. W. E., & Vissers, Y. L. J. (2021) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2021.06.003 · Cited by 8
Procedural Collaboration in Educational Games: Supporting Complex System Understandings in Immersive Whole Class Simulations
Kumar, V., Tissenbaum, M., & Kim, T. (2021) · Communication Studies · doi:10.1080/10510974.2021.2011363 · Cited by 10
Revisiting the Languages of Love: An Empirical Test of the Validity Assumptions Underlying Chapman’s (2015) Five Love Languages Typology
Pett, R. C., Lozano, P. A., & Varga, S. (2022) · Communication Reports · doi:10.1080/08934215.2022.2113549 · Cited by 8
“Because all three authors were involved in the coding, an alternative reliability method involving the arithmetic mean for kappa reliability estimates across all coding pairs was used to provide an overall index of agreement (Hallgren, 2012; Light, 1971).”
Method
Student commitment and school organization in relation to on-task behavior and achievement
Curry, L. (1984) · Contemporary Educational Psychology · doi:10.1016/0361-476x(84)90018-3 · Cited by 8
A Bayesian hierarchical model for multi-level repeated ordinal data: analysis of oral practice examinations in a large anaesthesiology training programme
Tan, M. T., Qu, Y., Mascha, E. J., & Schubert, A. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990815)18:15 · Cited by 26
Fixed-Effects Modeling of Cohen's Kappa for Bivariate Multinomial Data
Yang, J., & Chinchilli, V. M. (2009) · Communication in Statistics- Theory and Methods · doi:10.1080/03610920802618426 · Cited by 8
Reliability of a questionnaire for diagnosing the severity of temporomandibular disorder Confiabilidade de um formulário para diagnóstico da severidade da disfunção temporomandibular
Campos, J., Gonçalves, D., Camparis, C. M., & Jg, S. (2009) · Cited by 8
Hepatocellular Carcinoma Screening With Computed Tomography Using the Arterial Enhancement Fraction With Radiologic-Pathologic Correlation
Huber, A. T., Schuster, F., Ebner, L., Bütikofer, Y., Ott, D., Leidolt, L., Jöres, A. P. W., Montani, M., Heverhagen, J. T., & Christe, A. (2015) · Investigative Radiology · doi:10.1097/rli.0000000000000201 · Cited by 8
New radiological diagnostic criteria: impact on idiopathic pulmonary fibrosis diagnosis
Funke-Chambour, M., Guler, S. A., Geiser, T., Christe, A., Heverhagen, J. T., Pöllinger, A., Huber, A. T., & Ebner, L. (2019) · European Respiratory Journal · doi:10.1183/13993003.00905-2019 · Cited by 8
Inter-observer Reliability and Agreement Study on Early Diagnosis of Diabetic Retinopathy and Diabetic Macular Edema Risk
Arias, M. E., Ortega, C., Garrido, J., Ponte, B., Álvarez, F. R., & Marín, D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-31744-1_33 · Cited by 8
“kappa [29] and Fleiss’ kappa [30].”
Dysfluency and topic orientation in bereaved individuals: Bridging individual and group studies.
Stinson, C. H., Milbrath, C., & Horowitz, M. J. (1995) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.63.1.37 · Cited by 8
A review of reliability procedures for measuring observer agreement
Towstopiat, O. (1984) · Contemporary Educational Psychology · doi:10.1016/0361-476x(84)90039-0 · Cited by 8
Rorschach differences in normal and delinquent white male adolescents: A discriminant function analysis
Curtiss, G., Feczko, M. D., & Marohn, R. C. (1979) · Journal of Youth and Adolescence · doi:10.1007/bf02088656 · Cited by 8
“Interrater reliability was determined by a kappa statistic which yields an index of agreement corrected for concurrence by chance (Cohen, 1960; Light, 1971 ).”
Method
Higher-Order Kappa-Type Statistics for a Dichotomous Attribute in Multiple Ratings
Lau, T.-S. (1993) · Biometrics · doi:10.2307/2532565 · Cited by 8
Interrater Agreement Measures for Nominal and Ordinal Data
Denham, B. E. (2016) · doi:10.1002/9781119407201.ch11 · Cited by 8
Lesion follows function: video-oculography compared with MRI to diagnose internuclear ophthalmoplegia in patients with multiple sclerosis
Omary, R., Bockisch, C. J., Vere-Tyndall, A. D., Pazahr, S., Baráth, K., & Weber, K. P. (2022) · Journal of Neurology · doi:10.1007/s00415-022-11428-w · Cited by 9
“Inter-rater agreement was evaluated with the variant of Cohen’s kappa suggested by Light [11] for three raters(R package IRR (kappam.light)).”
Method
Self-supervised Topic Taxonomy Discovery in the Box Embedding Space
Lu, Y., Chen, H., Mao, P., Rao, Y., Xie, H., Wang, F. L., & Li, Q. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00712 · Cited by 8
“According to the value of Light’s kappa (Light, 2011) ( κ = 0.607), the annotation results of the ten raters have a fairly high degree of agreement.”
BackgroundInfluential
Harmonisation of the diagnostic performances of serological ELISA tests for C. burnetii in ruminants in the absence of a gold standard: Optimal cut-offs and performances reassessment
Rivière, L., Rousset, É., Jourdain, E., Delignette‐Muller, M. L., & Lurier, T. (2025) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2025.106509 · Cited by 9
Inter- and intra-rater reliability of handpump functionality field tests
Smith, D. W., Wind, S., Markov, I. A., Ongom, S. A., & Davis, J. (2023) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2023.161616 · Cited by 7
Simulating and estimating agreement in the presence of multiple raters and covariates
McKenzie, K. A., & Mahnken, J. D. (2023) · Statistics in Medicine · doi:10.1002/sim.9694 · Cited by 7
“28,29 We defined observed agreement ( π o in Equation 1) as the joint probability of agreement among all pairs of raters, and expected agreement ( π e in Equation 1) as the probability of agreement for each pair of raters under the assumption of rater conditional independence.”
Background
Back to the basics: Abstract painting as an index of creativity
Bellaiche, L., Smith, A. P., Barr, N., Christensen, A. P., Williams, C., Ragnhildstveit, A., Schooler, J. W., Beaty, R. E., Chatterjee, A. K., & Seli, P. (2023) · Creativity Research Journal · doi:10.1080/10400419.2023.2243100 · Cited by 7
“To determine initial interrater reliability metrics before implementing MFRM, we performed Light’s (1971) using Fleiss-Cohen’s weighted Kappa (Fleiss, Cohen, & Everitt, 1969).”
Method
Differences between SMT and NMT Output - a Translators’ Point of View
Mutal, J., Volkart, L., Bouillon, P., Girletti, S., & Estrella, P. (2019) · doi:10.26615/issn.2683-0078.2019_009 · Cited by 8
“We computed Light’s Kappa (Light, 1971) for the SMT and NMT evaluation.”
Method
Indicators of Impactful Reflection in Pre-Service Teachers: A Case for Creativity, Honesty and Unfamiliar Experiences
Catalana, S. M. (2020) · International Journal for the Scholarship of Teaching and Learning · doi:10.20429/ijsotl.2020.140114 · Cited by 11
“Percent agreement should not be used as the sole IRR estimate, since it does not account for agreement due to chance (Cohen, 1960).”
Method
Stereotyped whistles in southern resident killer whales
Souhaut, M., & Shields, M. W. (2021) · PeerJ · doi:10.7717/peerj.12085 · Cited by 7
“This overall index of agreement is reported as Light’s kappa (Light, 1971).”
Background
ParAlg: A Paraphasia Algorithm for Multinomial Classification of Picture Naming Errors
Casilio, M., Fergadiotis, G., Salem, A. C., Gale, R. C., McKinney-Bock, K., & Bedrick, S. D. (2023) · Journal of Speech Language and Hearing Research · doi:10.1044/2022_jslhr-22-00255 · Cited by 7
Measures of Agreement with Multiple Raters: Fréchet Variances and Inference
Moss, J. (2024) · Psychometrika · doi:10.1007/s11336-023-09945-2 · Cited by 9
“Well-known examples of coefﬁcients in this class are Cohen’s (1960) kappa and its weighted variant (1968), its multi-rater variant Conger’s kappa (Conger, 1980; Light, 1971), Krippendorff’s (1970) alpha, Scott’s (1955) pi, and Fleiss’ (1971) kappa.”
Background
Effects of Diagnostic Errors in Pattern Differentiation and Acupuncture Prescription: A Single-Blinded, Interrater Agreement Study
Oliveira, I. J. D. A. S., & Ferreira, A. D. S. (2015) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2015/469675 · Cited by 7
“The Light’s κ (kappa) coefficient [26] recommended for studies with fully crossed design in which all cases are classified by multiple raters [27] was calculated, grouped by diagnostic outcome.”
Background
Imitating dysphonic voice: a suitable technique to create speech stimuli for spoken language processing tasks?
Schiller, I., Remacle, A., & Morsomme, D. (2019) · Logopedics Phoniatrics Vocology · doi:10.1080/14015439.2019.1659410 · Cited by 7
Treatment Fidelity in Neuroscience-Informed Cognitive-Behavior Therapy: A Feasibility Study
Field, T. A., Miller, R., Beeson, E. T., & Jones, L. K. (2019) · Journal of Mental Health Counseling · doi:10.17744/mehc.41.4.06 · Cited by 7
Use of a Novel, Portable, LED-Based Capillary Refill Time Simulator within a Disaster Triage Context
Chang, T. P., Santillanes, G., Claudius, I. A., Pham, P. K., Koved, J., Cheyne, J., Gausche‐Hill, M., Kaji, A. H., Srinivasan, S., Donofrio, J. J., et al. (2017) · Prehospital and Disaster Medicine · doi:10.1017/s1049023x17006343 · Cited by 6
“Light’s method to determine a mean Cohen’s Kappa values for every pair of phase one physicians with the five simulated CRTs and five video-based CRTs was used.(17,18) The mean Kappa statistics were compared using a paired t-test.”
Method
A program to evaluate general and conditional agreement among categorical assignments of many raters
McDermott, P. A., & Watkins, M. W. (1979) · Behavior Research Methods · doi:10.3758/bf03205686 · Cited by 7
Facial reactions in preschoolers: A descriptive study
Unzner, L., & Schneider, K. (1990) · Journal of Nonverbal Behavior · doi:10.1007/bf01006577 · Cited by 7
Monoclonal antibody BrE-3 participation in a multivariate prognostic model for infiltrating ductal carcinoma of the breast
Chan, C. M., Baratta, F. S., Ozzello, L., & Ceriani, R. (1994) · Breast Cancer Research and Treatment · doi:10.1007/bf00665966 · Cited by 7
“Light's G statistic was used to compare the observed agreements of each of the three participating raters with the standard set by said expert in this study [25].”
Method
"Put Your Hands Up": How Joint Attention Is Initiated Between Blind Children And Their Sighted Peers
Jones, K. M., Grayson, M., Morrison, C. P., Leonards, U., & Metatla, O. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714005 · Cited by 6
“We calculated Cohen’s Kappa to test for inter-rater reliability and then computed the mean to obtain Light’s kappa [36, 59, 78]: • First author and Coder 2: Cohen’s 𝜅 = .”
Reliability of distal radius fracture classification systems: a CT based study
Sharma, M., Choudhury, S. R., Soundararajan, R., Sheth, R., Sinha, A., & Prakash, M. (2024) · Emergency Radiology · doi:10.1007/s10140-024-02294-2 · Cited by 6
Generalized Agreement Statistics over Fixed Group of Experts
Shah, M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23808-6_13 · Cited by 6
JOURNAL CLUB: Assessment of Interobserver Variability in the Peer Review Process: Should We Agree to Disagree?
Verma, N., Hippe, D. S., & Robinson, J. D. (2016) · American Journal of Roentgenology · doi:10.2214/ajr.16.16121 · Cited by 6
Diagnostic Performances of an Occupational Burnout Detection Method Designed for Healthcare Professionals
Huynh, A. N., Besse, C. S., Médiouni, Z., May, E. E., Shoman, Y., Hansez, I., & Canu, I. G. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182312300 · Cited by 6
“Our design being fully crossed, we first calculated Cohen’s kappa [26] for all rater pairs and then used the arithmetic mean of these estimates to provide an overall index of agreement [27].”
Method
‘Are we laughing at the same?’
Žákovská, I., Maíz-Arévalo, C., & Cao, Y. (2023) · European Journal of Humour Research · doi:10.7592/ejhr.2023.11.1.720 · Cited by 6
Assessing cognitive changes in multiple sclerosis: criteria for a reliable decision
Henry, A., Stefaniak, N., Schmid, F., Kwiatkowski, A., Hautecœur, P., & Lenne, B. (2023) · Journal of Clinical and Experimental Neuropsychology · doi:10.1080/13803395.2023.2232122 · Cited by 6
Team Interaction Dynamics during Collaborative Problem Solving
Wiltshire, T. J., & Fiore, S. M. (2015) · Scholarly Commons (Embry–Riddle Aeronautical University) · Cited by 6
Reliability and Preliminary Outcomes of a Protocol for Selection of Test Accommodations for Higher Education Students With Dysgraphia: A Pilot Study
Rosenberg‐Adler, T., & Weintraub, N. (2020) · American Journal of Occupational Therapy · doi:10.5014/ajot.2020.039826 · Cited by 6
“To estimate IRR, Cohen ’ s k was computed for each pair of experts and then averaged to provide a single index of IRR (Cohen, 1960; Hallgren, 2012; Light, 1971).”
Further evaluating interobserver reliability and accuracy with and without structured visual‐inspection criteria
Cox, A. D., Zonneveld, K. L. M., & Tardi, L. D. (2021) · Behavioral Interventions · doi:10.1002/bin.1793 · Cited by 6
Canadian Triage and Acuity Scale: testing the mental health categories
Brown, A.-M., Clarke, D. E., & Spence, J. M. (2015) · Open Access Emergency Medicine · doi:10.2147/oaem.s74646 · Cited by 6
Structure and analyze the football value chain in Brazil
Serrano, R., Lacerda, D. P., Cassel, R. A., Dresch, A., & Morandi, M. I. W. M. (2019) · International Journal of Sports Marketing and Sponsorship · doi:10.1108/ijsms-02-2017-0013 · Cited by 6
Is there inter-observer variation in the interpretation of SSEPs in comatose cardiac arrest survivors? Further considerations following the Italian multicenter ProNeCa study
Celani, M. G., Carrai, R., Cantisani, T. A., Scarpino, M., Ercolani, M. V., Lolli, F., Lanzo, G., Costa, P., Lanteri, P., Bignamini, A. A., et al. (2020) · Resuscitation · doi:10.1016/j.resuscitation.2020.07.029 · Cited by 6
Computer Studies Measuring the Joint Agreement between Multiple Raters and a Standard
Berry, K. J., & Mielke, P. W. (1997) · Educational and Psychological Measurement · doi:10.1177/0013164497057003013 · Cited by 6
The Development of Sighted People's Understanding of Blindness
Conant, S., & Budoff, M. (1982) · Journal of Visual Impairment & Blindness · doi:10.1177/0145482x8207600302 · Cited by 6
Commitment-detachment and authorial presence in postgraduate academic writing: A comparative study of Turkish native speakers, Turkish speakers of English and English native speakers
Akbaş, E. (2014) · White Rose eTheses Online (University of Leeds, The University of Sheffield, University of York) · doi:10.13140/2.1.2922.6240 · Cited by 6
Observer variation and depressive phenomenology
Verghese, A., Burrows, G. D., Foenander, G., Stevenson, J., & Davies, B. (1977) · Psychological Medicine · doi:10.1017/s0033291700018213 · Cited by 6
Equality of endogamy: Statistical approaches
Rust, P. F., & Seed, P. (1985) · Social Science Research · doi:10.1016/0049-089x(85)90012-2 · Cited by 6
Interpreting Cloud Computer Vision Pain-Points: A Mining Study of Stack OverflowPreprint
Cummaudo, A., Vasa, R., Barnett, S., Grundy, J., & Abdelrazek, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.10130 · Cited by 6
“As we used more than 2 coders and our reliability analysis was conducted under the same random sample of 50 posts, we applied Light’s Kappa [26] to our ratings, which indicates an overall index of agreement.”
Method
Can You be More Social? Injecting Politeness and Positivity into Task-Oriented Conversational AgentsPreprint
Wang, Y., Papangelis, A., Wang, R., Feizollahi, Z., Tür, G., & Kraut, R. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.14653 · Cited by 14
“We measured the inter-rater agreement ( kappa ) among the three crowdworkers for each question and both tasks [13, 40, 15].”
Method
Agreement among human and automated estimates of similarity in a global music samplePreprint
Daikoku, H., Ding, S., Sanne, U. S., Benetos, E., Wood, A. L. C., Shimozono, T., Fujii, S., & Savage, P. E. (2020) · doi:10.31234/osf.io/76fmq · Cited by 6
Measurement of rater consistency and its application in voice quality assessments
Xie, Z., Gadepalli, C., Jalalinajafabadi, F., Cheetham, B. M. G., & Homer, J. J. (2017) · International Congress on Image and Signal Processing (CISP) · doi:10.1109/cisp-bmei.2017.8302302 · Cited by 5
“Different generalisations of equation (1) have been proposed by Light [16] and Hubert [17] for categorical scoring.”
BackgroundInfluential

Authors: Light, R. J.
Affiliations: Howard University (US)
Corresponding author: Light, Richard J.
Venue: Psychological Bulletin
Publisher: American Psychological Association (APA)
Published: 1971-11
Volume: 76
Issue: 5
Pages: 365-377
ISSN: 0033-2909 (ISSN-L)
1939-1455
DOI: 10.1037/h0031643
OpenAlex: W1989492805
Cited by: 747
References: 17
Citations by year: 2026: 22 · 2025: 26 · 2024: 19 · 2023: 43 · 2022: 24 · 2021: 46 · 2020: 43 · 2019: 38 · 2018: 39 · 2017: 36 · 2016: 46 · 2015: 31 · 2014: 23 · 2013: 15 · 2012: 11
FWCI: 8.62
Type: journal article
Language: en
Open access: closed
Locations: Psychological Bulletin (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Keywords: interobserver agreement, qualitative data, categorical data, agreement measures, inter-rater reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Measures of response agreement for qualitative data: Some generalizations and alternatives.

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks