Cited by 750Reliability and Agreement in MeasurementPsychometric Methodologies and TestingSurvey Methodology and Nonresponse

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Measures of response agreement for qualitative data: Some generalizations and alternatives.

Light, Richard J.·1971·Psychological Bulletin

DOI 10.1037/h0031643

Open this paper in CoChat — free account

Integrity signals

Registry

Crossref · 10.1037/h0031643

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Rank Correlation Methods.
Kendall, M. G. (1957) · Biometrika · doi:10.2307/2333282 · Cited by 6,442
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. (1979) · Springer series in statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,255
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,536
Estimation of the reliability of ratings.
Ebel, R. L. (1951) · Psychometrika · doi:10.1007/bf02288803 · Cited by 976
Unitizing and categorizing problems in coding qualitative data
Guetzkow, H. S. (1950) · Journal of Clinical Psychology · doi:10.1002/1097-4679(195001)6:1 · Cited by 436
Intraclass Correlation and the Analysis of Variance
Trese, T., & Haggard, E. A. (1958) · The American Catholic Sociological Review · doi:10.2307/3708305 · Cited by 401

Flagged first, then by citations

Show all 17 references

Sorted by citations

Items 201–300 of 750

Effects of Computerized Decision Support Systems on Practitioner Performance and Patient Outcomes: Systematic Review
Kruse, C. S., & Ehrbar, N. (2020) · JMIR Medical Informatics · doi:10.2196/17283 · Cited by 42
“98, showing almost complete agreement on all reviewed articles [13,14].”
Background
On the quality of neural net classifiers
Egmont-Petersen, M., Talmon, J. L., Brender, J., & McNair, P. (1994) · Artificial Intelligence in Medicine · doi:10.1016/0933-3657(94)90002-7 · Cited by 40
Effect of instructional procedure and pupil locus of control on achievement and attitudes
McMillan, J. H. (1980) · Psychology in the Schools · doi:10.1002/1520-6807(198001)17:1 · Cited by 6
New Interpretations of Cohen’s Kappa
Warrens, M. J. (2014) · Journal of Mathematics · doi:10.1155/2014/203907 · Cited by 42
Weekday and weekend food advertising varies on children's television in the USA but persuasive techniques and unhealthy items still dominate
Vilaro, M. J., Barnett, T. E., Watson, A. M., Merten, J. W., & Mathews, A. (2016) · Public Health · doi:10.1016/j.puhe.2016.10.011 · Cited by 41
Writing a Systematic Review for Publication in a Health-Related Degree Program
Kruse, C. S. (2019) · JMIR Research Protocols · doi:10.2196/15490 · Cited by 40
Evaluation of the Interobserver Agreement in the Number of Mitotic Figures Breast Carcinoma as Simulation of Quality Monitoring in the Japan National Surgical Adjuvant Study of Breast Cancer (NSAS‐BC) Protocol
Tsuda, H., Akiyama, F., Kurosumi, M., Sakamoto, G., Yamashiro, K., Oyama, T., Hasebe, T., Kameyama, K., Hasegawa, T., Umemura, S., et al. (2000) · Japanese Journal of Cancer Research · doi:10.1111/j.1349-7006.2000.tb00966.x · Cited by 40
Slowed articulation rate is a sensitive diagnostic marker for identifying non-fluent primary progressive aphasia
Cordella, C., Dickerson, B. C., Quimby, M., Yunusova, Y., & Green, J. R. (2016) · Aphasiology · doi:10.1080/02687038.2016.1191054 · Cited by 44
“The overall index of inter-rater agreement was judged acceptable at .87, calculated as the arithmetic mean of κw across the three rater pairs, following Light (1971).”
Variance approximations for assessments of classification accuracy
Czaplewski, R. L. (1994) · doi:10.2737/rm-rp-316 · Cited by 40
A Technique for Assessing Mathematical Problem-Solving Ability
Collis, K. F., Romberg, T. A., & Jurdak, M. E. (1986) · Journal for Research in Mathematics Education · doi:10.5951/jresematheduc.17.3.0206 · Cited by 39
The relevance of image quality indices for dose optimization in abdominal multi-detector row CT in children: experimental assessment with pediatric phantoms
Brisse, H. J., Brenot, J., Pierrat, N., Gaboriaud, G., Savignoni, A., Rycke, Y. D., Neuenschwander, S., Aubert, B., & Rosenwald, J. (2009) · Physics in Medicine and Biology · doi:10.1088/0031-9155/54/7/002 · Cited by 38
Exploring potentials of digital nudging for business processes
Bammert, S., König, U., Roeglinger, M., & Wruck, T. (2020) · Business Process Management Journal · doi:10.1108/bpmj-07-2019-0281 · Cited by 40
Resampling Probability Values for Weighted Kappa with Multiple Raters
Mielke, P. W., Berry, K. J., & Johnston, J. E. (2008) · Psychological Reports · doi:10.2466/pr0.102.2.606-613 · Cited by 37
Development and evaluation of RAMP II - a practitioner’s tool for assessing musculoskeletal disorder risk factors in industrial manual handling
Lind, C. M., Forsman, M., & Rose, L. M. (2019) · Ergonomics · doi:10.1080/00140139.2019.1710576 · Cited by 37
“The proportion of agreement (p0) (Portney and Watkins 2009) and linearly weighted kappa (jw) were computed for all possible pairs of observers and then averaged (Light 1971; Conger 1980).”
Method
Development of a tobacco 21 policy assessment tool and state-level analysis in the USA, 2015–2019
Dobbs, P. D., Chadwick, G., Ungar, K. W., Dunlap, C. M., White, K. A., Kelly, M. C., & Cheney, M. K. (2019) · Tobacco Control · doi:10.1136/tobaccocontrol-2019-055102 · Cited by 37
A multi-modal approach towards mining social media data during natural disasters - A case study of Hurricane Irma
Mohanty, S. D., Biggers, B., Sayedahmed, S., Pourebrahim, N., Goldstein, E. B., Bunch, R. L., Chi, G., Sadri, F., McCoy, T., & Cosby, A. G. (2021) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2020.102032 · Cited by 37
Cancer-related fatigue: appraising evidence-based guidelines for screening, assessment and management
Pearson, E. J. M., Morris, M. E., & McKinstry, C. (2016) · Supportive Care in Cancer · doi:10.1007/s00520-016-3228-9 · Cited by 36
“Appraisal (1): NHMRC guideline standards Inter-rater agreement across both guidelines using Light’s kappa [30] was 0.48 with a standard error of 0.09, indicating a moderate agreement between reviewers (0.41 ≤ κ ≥ 0.6) [24] (see Table 3).”
MethodInfluential
Heart-rate adjustment of transcatheter haemodynamics improves the prognostic evaluation of paravalvular regurgitation after transcatheter aortic valve implantation
Jilaihawi, H., Chakravarty, T., Shiota, T., Rafique, A. M., Harada, K., Shibayama, K., Doctor, N., Kashif, M., Nakamura, M., Mirocha, J., et al. (2015) · EuroIntervention · doi:10.4244/eijy14m12_08 · Cited by 36
Comparison of industrial hygienists' exposure evaluations for an epidemiologic study
Stewart, P. A., Carel, R. S., Schairer, C., & Blair, A. E. (2000) · Scandinavian Journal of Work Environment & Health · doi:10.5271/sjweh.509 · Cited by 35
“The hypothesis of random agreement of the group of industrial hygienists relative to the standard was tested by calculating the G statistic (8).”
Method
Correlation between preoperative computed tomography and operative findings in functional endoscopic sinus surgery
Jiannetto, D. F., & Pratt, M. F. (1995) · The Laryngoscope · doi:10.1288/00005537-199509000-00010 · Cited by 36
Observational Methods
Hartmann, D. P., & Wood, D. D. (1982) · Springer eBooks · doi:10.1007/978-1-4615-7275-6_5 · Cited by 36
Evaluation of interobserver agreement in scoring immunohistochemical results of HER‐2/neu (c‐erbB‐2) expression detected by HercepTest, Nichirei polyclonal antibody, CB11 and TAB250 in breast carcinoma
Tsuda, H., Sasano, H., Akiyama, F., Kurosumi, M., Hasegawa, T., OSAMURA, R. Y., & Sakamoto, G. (2002) · Pathology International · doi:10.1046/j.1440-1827.2002.01327.x · Cited by 36
Observer variation in assessment of jejunal biopsy specimens. A comparison between subjective criteria and morphometric measurement.
Corazza, G. R., Bonvicini, F., Frazzoni, M., Gatto, M. R., & Gasbarrini, A. (1982) · PubMed · Cited by 36
Developmental analysis of the conditional reasoning abilities of primary-grade children.
Kodroff, J. K., & Roberge, J. J. (1975) · Developmental Psychology · doi:10.1037/h0076116 · Cited by 36
Classification of mental disorder in primary care
Jenkins, R., Smeeton, N., & Shepherd, M. (1988) · Psychological Medicine Monograph Supplement · doi:10.1017/s0264180100001995 · Cited by 36
Analyzing the Effect of Telemedicine on Domains of Quality Through Facilitators and Barriers to Adoption: Systematic Review
Kruse, C. S., Molina-Nava, A., Kapoor, Y., Anerobi, C., & Maddukuri, H. (2022) · Journal of Medical Internet Research · doi:10.2196/43601 · Cited by 42
“A Cohen κ statistic was also calculated to measure agreement between reviewers [25,26].”
Method
Inter-laboratory evaluation of a duplex RT-PCR method using crude extracts for the simultaneous detection of Prune dwarf virus and Prunus necrotic ringspot virus
Massart, S., Brostaux, Y., Barbarossa, L., César, V. R., Cieślińska, M., Dutrecq, O., Fonseca, F., Guillem, R., Laviña, A., Olmos, A., et al. (2008) · European Journal of Plant Pathology · doi:10.1007/s10658-008-9322-1 · Cited by 35
“Repeatability was evaluated by averaging the Kappa agreement between repetitions in each laboratory, and the reproducibility by averaging those values between all pairs of repetitions whatever their origin (Light 1971).”
Method
Development of the Addiction Dimensions for Assessment and Personalised Treatment (ADAPT)
Marsden, J., Eastwood, B., Ali, R. L., Burkinshaw, P., Chohan, G., Copello, A. G., Burn, D., Kelleher, M., Mitcheson, L., Taylor, S., et al. (2014) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2014.03.018 · Cited by 35
Reliability from alpha to omega: a tutorialPreprint
Revelle, W., & Condon, D. M. (2018) · doi:10.31234/osf.io/2y3w9 · Cited by 33
Artificial Intelligence Radiographic Analysis Tool for Total Knee Arthroplasty
Bonnin, M. P., Müller-Fouarge, F., Estienne, T., Bekadar, S., Pouchy, C., & Selmi, T. A. S. (2023) · The Journal of Arthroplasty · doi:10.1016/j.arth.2023.02.053 · Cited by 35
Interobserver variation in the diagnosis of adenoma malignum (minimal deviation adenocarcinoma) of the uterine cervix
Tsuda, H., Mikami, Y., Kaku, T., Akiyama, F., Hasegawa, T., Okada, S., Hayashi, I., & Kasamatsu, T. (2003) · Pathology International · doi:10.1046/j.1440-1827.2003.01497.x · Cited by 34
Assessment of Angiographic Retinal Capillary Nonperfusion in Central Retinal Vein Occlusion
Welch, J. C., & Augsburger, J. J. (1987) · American Journal of Ophthalmology · doi:10.1016/s0002-9394(14)74390-9 · Cited by 34
Agreement between an isolated rater and a group of raters
Vanbelle, S., & Albert, A. (2009) · Statistica Neerlandica · doi:10.1111/j.1467-9574.2008.00412.x · Cited by 33
“Light (1971) developed a statistic 54 for comparing the joint agreement of several raters with a gold standard. This 55 statistic is a mixture of the proportions of concordant pairs obtained between each 56 of the rater in the group and the gold standard (the isolated rater). His method 57 leads to tedious calculation…”
Influential
Understanding women's hesitancy to undergo less frequent cervical cancer screening
Gerend, M. A., Shepherd, M. A., Kaltz, E. A., Davis, W. J., & Shepherd, J. E. (2016) · Preventive Medicine · doi:10.1016/j.ypmed.2016.11.028 · Cited by 36
An Evaluation of the Multi-engine MT Architecture
Hogan, C. J., & Frederking, R. (1998) · Lecture notes in computer science · doi:10.1007/3-540-49478-2_11 · Cited by 33
“Agreement between human subjects is a well-investigated area in Psychology [ 18 ], Sociology [17], Medicine [1,6] as well as Translation [15].”
Background
Three driven approaches to combinational creativity: Problem-, similarity- and inspiration-driven
Han, J., Park, D., Shi, F., Chen, L., Hua, M., & Childs, P. (2017) · Proceedings of the Institution of Mechanical Engineers Part C Journal of Mechanical Engineering Science · doi:10.1177/0954406217750189 · Cited by 36
Weighted inter-rater agreement measures for ordinal outcomes
Tran, Q. D., Dolgun, A., & Demirhan, H. (2018) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610918.2018.1490428 · Cited by 35
“Many of these measures of are either extensions of Cohen’s kappa and weighted kappa or are formulated as a Cohen’s kappa-like statistic (Fleiss 1971; Light 1971; Conger 1980).”
Method
If this title is funny, will you cite me? Citation impacts of humour and other features of article titles in ecology and evolution
Heard, S. B., Cull, C. A., & White, E. R. (2023) · FACETS · doi:10.1139/facets-2022-0079 · Cited by 32
Muscle ultrasound in classic infantile and adult Pompe disease: A useful screening tool in adults but not in infants
Vill, K., Schessl, J., Teusch, V. I., Schroeder, A. S., Blaschek, A., Schoser, B. G. H., & Müller‐Felber, W. (2014) · Neuromuscular Disorders · doi:10.1016/j.nmd.2014.09.016 · Cited by 32
“[23] Light R.”
Method
Detection of glaucoma progression by perimetry and optic disc photography at different stages of the disease: results from the Early Manifest Glaucoma Trial
Öhnell, H. M., Heijl, A., Anderson, H., & Bengtsson, B. (2016) · Acta Ophthalmologica · doi:10.1111/aos.13290 · Cited by 32
“(Light 1971; Byrt et al. 1993; Hallgren 2012).”
Interobserver variability in grading Perthes' disease
Simmons, E., Graham, H. K., & Szalai, J. (1990) · Journal of Bone and Joint Surgery - British Volume · doi:10.1302/0301-620x.72b2.2312555 · Cited by 32
Field Reliability of Comprehensive System Scoring in an Adolescent Inpatient Sample
McGrath, R. E., Pogge, D. L., Stokes, J., Cragnolino, A., Zaccario, M. L., Hayman, J., Piacentini, T., & Wayland-Smith, D. (2005) · Assessment · doi:10.1177/1073191104273384 · Cited by 31
“Kappa is the appropriate reliability statistic for Rorschach codes, whether the goal is to evaluate reliability between two or more raters or the reliability of one or more raters with the correct scoring (see Light, 1971).”
Background
Large observer variation of clinical assessment of dyspnoeic wheezing children
Bekhof, J., Reimink, R., Bartels, I.-M., Eggink, H., & Brand, P. L. P. (2015) · Archives of Disease in Childhood · doi:10.1136/archdischild-2014-307143 · Cited by 31
The classification of osteonecrosis in patients with cancer: validation of a new radiological classification system
Niinimäki, T., Niinimäki, J. L., Halonen, J., Hänninen, P., Harila‐Saari, A. H., & Niinimäki, R. (2015) · Clinical Radiology · doi:10.1016/j.crad.2015.08.011 · Cited by 31
Development and Evaluation of a Muscle Atrophy Scoring System (MASS) for Horses
Herbst, A. C., Johnson, M., Gammons, H., Reedy, S. E., Urschel, K. L., Harris, P. A., & Adams, A. A. (2021) · Journal of Equine Veterinary Science · doi:10.1016/j.jevs.2021.103771 · Cited by 37
Justification for the Use of Cohen’s Kappa Statistic in Experimental Studies of NLP and Text Mining
Kolesnyk, A., & Khairova, N. (2022) · Cybernetics and Systems Analysis · doi:10.1007/s10559-022-00460-3 · Cited by 31
“For example, Fleiss’s kappa [9], Light’s and Conger’s kappa [10, 11], and Mielke’s kappa [12] are such coefficients.”
Background
Measuring Verbal Psychotherapeutic Techniques—A Systematic Review of Intervention Characteristics and Measures
Gumz, A., Treese, B., Marx, C., Strauss, B. M., & Wendt, H. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01705 · Cited by 34
“…of verbal techniques in order to establish its general applicability, i.e., Intraclass Coefficient Correlations (ICC; Shrout and Fleiss, 1979), (weighted) kappa-values (Cohen, 1960, 1968; Fleiss, 1971; Light, 1971), Finn’s r (Finn, 1970, 1972), or Pearson’s product-moment correlation coefficient.”
BackgroundMethod
Statistical Methods for the Analysis of Inter-Reader Agreement Among Three or More Readers
Han, K., & Ryu, L. (2024) · Korean Journal of Radiology · doi:10.3348/kjr.2023.0965 · Cited by 37
“Light’s kappa [6] represents the average Cohen’s kappa value calculated for all two-reader combinations.”
Background
Intensive Care Unit Nurses’ Beliefs About Delirium Assessment and Management
Oosterhouse, K. J., Vincent, C., Foreman, M. D., Gruss, V. A., Corte, C. M., & Berger, B. E. (2016) · AACN Advanced Critical Care · doi:10.4037/aacnacc2016535 · Cited by 29
Heuristics to Evaluate Interactive Systems for Children with Autism Spectrum Disorder (ASD)
Khowaja, K., & Salim, S. S. (2015) · PLoS ONE · doi:10.1371/journal.pone.0132187 · Cited by 32
“There are various common methods that can be used to carry out IRR analysis of nomial data; these include Cohen ’ s Kappa [33], Light ’ s Kappa [34] and Davies and Fleiss ’ s Kappa (DF Kappa) by [35].”
Method
The Efficacy and Limitations of Repeated Slide Conferences for Improving Interobserver Agreement When Judging Nuclear Atypia of Breast Cancer
Tsuda, H., Akiyama, F., Kurosumi, M., Sakamoto, G. K., Watanabe, T., & Section, J. N. S. A. S. O. B. C. (.-B. P. (1999) · Japanese Journal of Clinical Oncology · doi:10.1093/jjco/29.2.68 · Cited by 29
Observer Variation in Assessment of Jejunal Biopsy Specimens
Corazza, G. R., Bonvicini, F., Frazzoni, M., Gatto, M., & Gasbarrini, A. (1982) · Gastroenterology · doi:10.1016/s0016-5085(82)80131-5 · Cited by 29
Impacts of authentic assessment on the development of graduate attributes
Karunanayaka, S. P., & Naidu, S. (2021) · Distance Education · doi:10.1080/01587919.2021.1920206 · Cited by 31
“First, independent coding was done by the two coders, and the IRR was calculated for a random selection of assignments.”
BackgroundInfluential
Dermatoscopic and clinical features of congenital or congenital-type nail matrix nevi: A multicenter prospective cohort study by the International Dermoscopy Society
Pham, F., Boespflug, A. M., Duru, G., Phan, A., Poulalhon, N., Weiler, L., Tanaka, M., Lallas, A., Ogata, D., Davaine, A.-C., et al. (2022) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2022.01.028 · Cited by 28
An Empirical Comparative Assessment of Inter-Rater Agreement of Binary Outcomes and Multiple Raters
Konstantinidis, M., Le, L. W., & Gao, X. (2022) · Symmetry · doi:10.3390/sym14020262 · Cited by 32
Development and evaluation of RAMP I – a practitioner’s tool for screening of musculoskeletal disorder risk factors in manual handling
Lind, C. M., Forsman, M., & Rose, L. M. (2017) · International Journal of Occupational Safety and Ergonomics · doi:10.1080/10803548.2017.1364458 · Cited by 28
“, the multi-observer κw equals the arithmetic average of all, N × (N − 1) / 2, pairwise κ) [29,30].”
Background
Time to reconstitution: purified Fab antivenom vs. unpurified IgG antivenom
Hill, R. E., Bogdan, G., & Dart, R. C. (2001) · Toxicon · doi:10.1016/s0041-0101(00)00198-7 · Cited by 28
Reliability and Validity of Chest Radiograph Surveillance Programs
Lawson, C. C., LeMasters, M. K., LeMasters, G. K., Simpson, S. R., Rice, C. H., & Lockey, J. E. (2001) · CHEST Journal · doi:10.1378/chest.120.1.64 · Cited by 28
International assessment of inter‐ and intrarater reliability of the International Frontal Sinus Anatomy Classification system
Villarreal, R., Wrobel, B., Macias‐Valle, L., Davis, G. E., Prihoda, T. J., Luong, A., McMains, K. C., Weitzel, E. K., Yao, W. C., Brunworth, J., et al. (2018) · International Forum of Allergy & Rhinology · doi:10.1002/alr.22200 · Cited by 28
A simple program to train case managers in community elderly care
Landi, F., Sgadari, A., Cipriani, L., Castellani, M., Pahor, M., Carbonin, P., & Bernabei, R. (1996) · Aging Clinical and Experimental Research · doi:10.1007/bf03339679 · Cited by 28
Therapist interventions using the Psychodynamic Interventions Rating Scale (PIRS) in dynamic therapy, psychoanalysis and CBT
Banon, E., Perry, J. C., Semeniuk, T., Bond, M., de Roten, Y., Hersoug, A. G., & Despland, J.-N. (2013) · Psychotherapy Research · doi:10.1080/10503307.2012.745955 · Cited by 27
“Inter-rater reliabilities for the PIRS were calculated with Light’s kappa (k-value coefficient of agreement; Light, 1971) for each individual intervention or category and for the measure as a whole.”
Method
Specific agreement on ordinal and multiple nominal outcomes can be calculated for more than two raters
de Vet, H. C. W., Mullender, M. G., & Eekhout, I. E. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.024 · Cited by 27
Managers’ Restorative Versus Punitive Responses to Employee Wrongdoing: A Qualitative Investigation
Neale, N. R., Butterfield, K. D., Goodstein, J. D., & Tripp, T. M. (2018) · Journal of Business Ethics · doi:10.1007/s10551-018-3935-x · Cited by 30
“Step into the Real Texas”: Associating and claiming state narrative in advertising and tourism brochures
Avraham, E., & Daugherty, D. (2012) · Tourism Management · doi:10.1016/j.tourman.2011.12.022 · Cited by 27
Equivalences of weighted kappas for multiple raters
Warrens, M. J. (2011) · Statistical Methodology · doi:10.1016/j.stamet.2011.11.001 · Cited by 27
“Light and Hubert [28,22] have formulated generalizations of Cohen’s κ for multiple raters based on 2-agreement and m-agreement.”
Background
On population‐based measures of agreement for binary classifications
Nelson, K. P., & Edwards, D. (2008) · Canadian Journal of Statistics · doi:10.1002/cjs.5550360306 · Cited by 30
“Cohen's kappa has since been extended to provide a summary measure of agreement for classifications made on ordinal scales (Cohen 1968) for more than two raters (Fleiss 1971; Light 1971; Kraemer 1980), and there are various other extensions (Landis & Koch 1977a, 1977b; Barlow 1996; Klar, Lipsitz & Ibrahim 2000), where…”
Background
Unpacking the authenticity gap in corporate social responsibility: lessons learned from Levi’s ‘Go Forth Braddock’ campaign
Samuel, A. J., Taylor, D., White, G. R. T., & Norris, M. (2017) · Journal of Brand Management · doi:10.1057/s41262-017-0067-z · Cited by 27
“Some argue that rigor in interpretive study may be assured through positivistic measures such as Cohen’s kappa (McHugh, 2012; Light, 1971) while others claim it is achieved through adherence to declared-in-advance process (Whittemore et al, 2001; Gronhaug and Olson, 1999).”
Conditional tau coefficient for assessment of producer's accuracy of classified remotely sensed data
Næsset, E. (1996) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/0924-2716(69)00007-4 · Cited by 27
The Value of the Peripheral Blood Smear in Anemic Inpatients
Jen, P. (1983) · Archives of Internal Medicine · doi:10.1001/archinte.1983.00350060042005 · Cited by 27
Interobserver Variability in the Recognition of Hypoxic–Ischemic Brain Injury on Computed Tomography Soon After Out-of-Hospital Cardiac Arrest
Caraganis, A., Mulder, M., Kempainen, R. R., Brown, R. Z., Oswood, M. C., Hoffman, B., & Prekker, M. E. (2020) · Neurocritical Care · doi:10.1007/s12028-019-00900-7 · Cited by 26
Diagnostic Agreement of Conventional and Inverted Scanned Panoramic Radiographs in the Detection of the Mandibular Canal and the Mental Foramen
Sakakura, C. E., Loffredo, L. D. C. M., & Scaf, G. (2004) · Journal of Oral Implantology · doi:10.1563/1548-1336(2004)030 · Cited by 26
Interobserver relational agreement.
Stine, W. W. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.106.2.341 · Cited by 26
Avaliação das estruturas de concordância e discordância nos estudos de confiabilidade
da Silva, E. F., & Pereira, M. G. (1998) · Revista de Saúde Pública · doi:10.1590/s0034-89101998000400012 · Cited by 26
Concordance in judgments among c‐erbB‐2 (HER2/neu) overexpression detected by two immunohistochemical tests and gene amplification detected by Southern blot hybridization in breast carcinoma
Tsuda, H., Tani, Y., Hasegawa, T., & Fukutomi, T. (2001) · Pathology International · doi:10.1046/j.1440-1827.2001.01163.x · Cited by 26
Elaboration and Dyselaboration: Measures of Expression and Defense in Discourse
Horowitz, M. J., Milbrath, C., Reidbord, S. P., & Stinson, C. H. (1993) · Psychotherapy Research · doi:10.1080/10503309312331333849 · Cited by 26
The Novel OMERACT Ultrasound Scoring System for Salivary Gland Changes in Patients With Sjögren Syndrome Is Associated With MRI and Salivary Flow Rates
İnanç, N., Jousse‐Joulin, S., Abacar, K., Çimşit, Ç., Çimşit, C., D’Agostino, M. A., Naredo, E., Hočevar, A., Finzel, S., Pineda, C., et al. (2023) · The Journal of Rheumatology · doi:10.3899/jrheum.2023-0202 · Cited by 25
“Interrater reliability of the ultrasonographers was analyzed using Light κ. 30,31 Since the assessments were on an ordinal scale, weighted κ calculations (Fleiss-Cohen weights) were used for calculating Light κ. Weighted κ coefficients were calculated for each pair of assessments among 9 raters (9…”
Method
Efficacy of the Use of the Calgary Family Intervention Model in Bedside Nursing Education: A Systematic Review
Mileski, M., McClay, R., Heinemann, K., & Dray, G. (2022) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s370053 · Cited by 25
The Use of Health Information Exchange to Augment Patient Handoff in Long-Term Care: A Systematic Review
Kruse, C. S., Marquez, G., Nelson, D., & Palomares, O. (2018) · Applied Clinical Informatics · doi:10.1055/s-0038-1670651 · Cited by 25
A qualitative study of decision-making by breast cancer patients about telling their children about their illness
Yoshida, S., Otani, H., Hirai, K., Ogata, A., Mera, A., Okada, S., & Oshima, A. (2009) · Supportive Care in Cancer · doi:10.1007/s00520-009-0682-7 · Cited by 25
“The concordance rate and kappa coefficient [9] of the determinations of the categories were used as reliability indicators.”
Evaluation of PAS‐diastase and carcinoembryonic antigen staining in the differential diagnosis of malignant mesothelioma and papillary serous carcinoma of the ovary
Silcocks, P., Herbert, A., & Wright, D. H. (1986) · The Journal of Pathology · doi:10.1002/path.1711490208 · Cited by 25
EVALUATION OF TASK AND JOB SKILL LINKAGE JUDGMENTS USED TO DEVELOP TEST SPECIFICATIONS
Hughes, G. L., & Prien, E. P. (1989) · Personnel Psychology · doi:10.1111/j.1744-6570.1989.tb00658.x · Cited by 25
Analysis of Prescribers’ Notes in Electronic Prescriptions in Ambulatory Practice
Dhavle, A. A., Yang, Y., Rupp, M. T., Singh, H., Ward-Charlerie, S., & Ruiz, J. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.7786 · Cited by 25
Examiner Agreement and Judicial Consensus in Forensic Mental Health Evaluations
Acklin, M. W., Fuger, K., & Gowensmith, W. N. (2015) · Journal of Forensic Psychology Practice · doi:10.1080/15228932.2015.1051447 · Cited by 25
Inconsistency in the analysis of morphological deformities in chironomidae (Insecta: Diptera) larvae
Salmelin, J., Vuori, K.-M., & Hämäläinen, H. (2015) · Environmental Toxicology and Chemistry · doi:10.1002/etc.3010 · Cited by 24
Methodological Considerations for Content Analysis of Sexual Consent Communication in Mainstream Films
Jozkowski, K. N., Canan, S. N., Rhoads, K. E., & Hunt, M. (2016) · Sexualization Media & Society · doi:10.1177/2374623816679184 · Cited by 24
“We determined that calculating Light’s k (Light, 1971) to measure ICR for all other variables, with the exception of the consent/refusal cues variable, was most appropriate because data were nominal, and we had more than two coders coding each film.”
Method
Antecedents of graduate student satisfaction through e-portfolio: content analysis
Shea, T. P., & Parayitam, S. (2019) · Education + Training · doi:10.1108/et-04-2019-0064 · Cited by 28
Implementing evidence-based practice in policing family violence:The reliability, validity and feasibility of a risk assessment instrument for prioritising police response
Spivak, B., Mcewan, T., Luebbers, S., & Ogloff, J. (2020) · Policing and Society: An International Journal of Research and Policy · doi:10.1080/10439463.2020.1757668 · Cited by 22
Evaluating the relationship between health information technology and safer-prescribing in the long-term care setting: A systematic review
Kruse, C. S., Mileski, M., Syal, R., MacNeil, L., Chabarria, E., & Basch, C. (2020) · Technology and Health Care · doi:10.3233/thc-202196 · Cited by 23
“A Kappa statistic of 0.83, indicating strong agreement, was calculated following the consensus meeting [22,23].”
MethodInfluential
Two-Computer Pair Programming: Exploring a Feedback Intervention to improve Collaborative Talk in Elementary Students
Zakaria, Z., Vandenberg, J. L., Tsan, J., Boulden, D. C., Lynch, C. F., Boyer, K. E., & Wiebe, E. (2021) · Computer Science Education · doi:10.1080/08993408.2021.1877987 · Cited by 23
“Also, kappa was computed for each coder pair then averaged to provide a single index of interrater reliability (Light, 1971).”
Method
Ultrasonography of Inflammatory and Structural Lesions in Hand Osteoarthritis: An Outcome Measures in Rheumatology Agreement and Reliability Study
Mathiessen, A., Hammer, H. B., Terslev, L., Kortekaas, M. C., D’Agostino, M. A., Haugen, I. K., Bruyn, G. A. W., Filippou, G., Filippucci, E., Kloppenburg, M., et al. (2021) · Arthritis Care & Research · doi:10.1002/acr.24734 · Cited by 23
Do physicians do as they say? The case of mammography
Saver, B. G. (1997) · Archives of Family Medicine · doi:10.1001/archfami.6.6.543 · Cited by 23
MDMA. Identification et annotation des marqueurs discursifs “potentiels” en contexte
Bolly, C., Crible, L., Degand, L., & Uygur, D. (2015) · Digital Access to Libraries · Cited by 22
Spiritual Media Experiences, Trait Transcendence, and Enjoyment of Popular Films
Janicke‐Bowles, S. H., & Ramasubramanian, S. (2017) · Journal of Media and Religion · doi:10.1080/15348423.2017.1311122 · Cited by 22
“Cohen’s kappa, computed for each pair of coders and averaged among the three coders (Light, 1971) indicated substantial inter-coder reliability (.”
MethodInfluential
Interpreting cloud computer vision pain-points
Cummaudo, A., Vasa, R., Barnett, S., Grundy, J., & Abdelrazek, M. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380404 · Cited by 22
“As we used more than 2 coders and our reliability analysis was conducted under the same random sample of 50 posts, we applied Light’s Kappa [26] to our ratings, which indicates an overall index of agreement.”
Method
Prediction of three outcome states from pediatric intensive care
Ruttimann, U. E., Pollack, M. M., & Fiser, D. H. (1996) · Critical Care Medicine · doi:10.1097/00003246-199601000-00014 · Cited by 22
Automatic detection of sounds and murmurs in patients with lonescu-Shiley aortic bioprostheses
Baranek, H., Lee, H. C., Cloutier, G., & Durand, L. (1989) · Medical & Biological Engineering & Computing · doi:10.1007/bf02441460 · Cited by 22
Accuracy and precision of the autobac system for rapid identification of Gram-negative bacilli: a collaborative evaluation
Barry, A. L., GAVAN, T. L., Smith, P. B., Matsen, J. M., Morello, J. A., & Sielaff, B. (1982) · Journal of Clinical Microbiology · doi:10.1128/jcm.15.6.1111-1119.1982 · Cited by 22
“This allowed comparison of all possible pairs of data and expresses the proportion of data pairs that were in agreement (4, 7).”
BackgroundMethod
Adverse Childhood Experiences: A 20‐Year Content Analysis of American Counseling Association and American School Counselor Association Journals
Zyromski, B., Baker, E., Betters-Bubon, J., Dollarhide, C. T., & Antonides, J. (2020) · Journal of Counseling & Development · doi:10.1002/jcad.12338 · Cited by 14
Quantifying Interrater Agreement and Reliability Between Thoracic Pathologists: Paradoxical Behavior of Cohen’s Kappa in the Presence of a High Prevalence of the Histopathologic Feature in Lung Cancer
Tan, K. S., Yeh, Y., Adusumilli, P. S., & Travis, W. D. (2023) · JTO Clinical and Research Reports · doi:10.1016/j.jtocrr.2023.100618 · Cited by 27
Transitional probability predicts native and non‐native use of formulaic sequences
Appel, R., & Trofimovich, P. (2015) · International Journal of Applied Linguistics · doi:10.1111/ijal.12100 · Cited by 21

Authors: Light, R. J.
Affiliations: Howard University (US)
Corresponding author: Light, Richard J.
Venue: Psychological Bulletin
Publisher: American Psychological Association (APA)
Published: 1971-11
Volume: 76
Issue: 5
Pages: 365-377
ISSN: 0033-2909 (ISSN-L)
1939-1455
DOI: 10.1037/h0031643
OpenAlex: W1989492805
Cited by: 747
References: 17
Citations by year: 2026: 22 · 2025: 26 · 2024: 19 · 2023: 43 · 2022: 24 · 2021: 46 · 2020: 43 · 2019: 38 · 2018: 39 · 2017: 36 · 2016: 46 · 2015: 31 · 2014: 23 · 2013: 15 · 2012: 11
FWCI: 8.62
Type: journal article
Language: en
Open access: closed
Locations: Psychological Bulletin (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Keywords: interobserver agreement, qualitative data, categorical data, agreement measures, inter-rater reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Measures of response agreement for qualitative data: Some generalizations and alternatives.

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks