Cited by 750Reliability and Agreement in MeasurementPsychometric Methodologies and TestingSurvey Methodology and Nonresponse

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Measures of response agreement for qualitative data: Some generalizations and alternatives.

Light, Richard J.·1971·Psychological Bulletin

DOI 10.1037/h0031643

Open this paper in CoChat — free account

Integrity signals

Registry

Crossref · 10.1037/h0031643

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Rank Correlation Methods.
Kendall, M. G. (1957) · Biometrika · doi:10.2307/2333282 · Cited by 6,442
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. (1979) · Springer series in statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,255
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,536
Estimation of the reliability of ratings.
Ebel, R. L. (1951) · Psychometrika · doi:10.1007/bf02288803 · Cited by 976
Unitizing and categorizing problems in coding qualitative data
Guetzkow, H. S. (1950) · Journal of Clinical Psychology · doi:10.1002/1097-4679(195001)6:1 · Cited by 436
Intraclass Correlation and the Analysis of Variance
Trese, T., & Haggard, E. A. (1958) · The American Catholic Sociological Review · doi:10.2307/3708305 · Cited by 401

Flagged first, then by citations

Show all 17 references

Sorted by citations

Items 301–400 of 750

Conditional Relative Odds Ratio and Comparison of Accuracy of Diagnostic Tests Based on 2×2 Tables
Suzuki, S. (2006) · Journal of Epidemiology · doi:10.2188/jea.16.145 · Cited by 21
How the Integration of Pathology in the Gross Anatomy Laboratory Affects Medical Students
Rae, G., Cork, J. R., Karpinski, A. C., McGoey, R. R., & Swartz, W. (2016) · Teaching and Learning in Medicine · doi:10.1080/10401334.2016.1194761 · Cited by 21
Quality and Variability of Patient Directions in Electronic Prescriptions in the Ambulatory Care Setting
Yang, Y., Ward-Charlerie, S., Dhavle, A. A., Rupp, M. T., & Green, J. (2018) · Journal of Managed Care & Specialty Pharmacy · doi:10.18553/jmcp.2018.17404 · Cited by 21
Predictors of correspondence between self-reported substance use and urinalysis screening among a racially diverse cohort of young men who have sex with men and transgender women
Li, D. H., Janulis, P. F., & Mustanski, B. S. (2018) · Addictive Behaviors · doi:10.1016/j.addbeh.2018.08.004 · Cited by 21
Identity and treatment adherence in predominantly ethnic minority teens and young adults with type 1 diabetes
COMMISSARIAT, P. V., Laffel, L. M., & Gonzalez, J. S. (2019) · Pediatric Diabetes · doi:10.1111/pedi.12932 · Cited by 21
Viewpoint Analysis for Maturity Classification of Sweet Peppers
Harel, B., van Essen, R., Parmet, Y., & Edan, Y. (2020) · Sensors · doi:10.3390/s20133783 · Cited by 21
Impact of observers' experience and training on reliability of observations for a manual handling task
Denis, D., Lortie, M., & Bruxelles, M. (2002) · Ergonomics · doi:10.1080/00140130210136044 · Cited by 21
“The most frequent answer becomes the reference, re¯ecting the internal consistency of the group (Light 1971).”
Background
Assessing Agreement in Multispectral Images of Yellow Starthistle (Centaurea solstitialis) with Ground Truth Data Using a Bayesian Methodology1
Lass, L. W., Shafii, B., Price, W. J., & Thill, D. C. (2000) · Weed Technology · doi:10.1614/0890-037x(2000)014[0539:aaimio]2.0.co;2 · Cited by 21
Reliability estimates for assessing meal timing derived from longitudinal repeated 24-hour dietary recalls
Peterson, L., Lee, H., Huybrechts, I., Biessy, C., Neuhouser, M. L., Haaland, B. A., Krick, B., Gunter, M. J., Schulze, M. B., Jannasch, F., et al. (2023) · American Journal of Clinical Nutrition · doi:10.1016/j.ajcnut.2023.02.026 · Cited by 22
The Fluid Nature of Representation: Transformations in the Representation of Arabs in Israeli Television News
First, A. (2002) · Howard Journal of Communications · doi:10.1080/10646170290089944 · Cited by 20
Post-traumatic subtalar osteoarthritis: which grading system should we use?
Keizer, R.-J. O. D. M., Backes, M., Dingemans, S. A., Goslings, J. C., & Schepers, T. (2016) · International Orthopaedics · doi:10.1007/s00264-016-3236-x · Cited by 20
Development and Validation of a Rubric to Evaluate Diabetes SOAP Note Writing in APPE
Andrus, M. R., McDonough, S. L. K., Kelley, K. W., Stamm, P. L., McCoy, E. K., Lisenby, K. M., Whitley, H. P., Slater, N. A., Carroll, D. G., Hester, E. K., et al. (2018) · American Journal of Pharmaceutical Education · doi:10.5688/ajpe6725 · Cited by 20
“Arguably, the use of Light’s procedure to adjust kappa for multiple raters, which was used in this study, may actually have deflated inter-rater agreement estimates by excluding some pair comparisons from the analysis.(14) Another limitation is the generalizability of the rubric.”
Method
Expressive and Defensive Behavior during Discourse on Unresolved Topics: A Single Case Study of Pathological Grief
Horowitz, M. J., Milbrath, C., Jordan, D. S., Stinson, C. H., Ewert, M., Redington, D. J., Fridhandler, B., Reidbord, S. P., & Hartley, D. (1994) · Journal of Personality · doi:10.1111/j.1467-6494.1994.tb00308.x · Cited by 20
Research synthesis reviews: an illustrated critique of “hidden” judgments, choices, and compromises
Nurius, P., & Yeaton, W. (1987) · Clinical Psychology Review · doi:10.1016/0272-7358(87)90014-6 · Cited by 24
Stability of the Type A structured interview and related questionnaires in a 10-year follow-up of an adult cohort of twins
Carmelli, D., Rosenman, R. H., & Chesney, M. A. (1987) · Journal of Behavioral Medicine · doi:10.1007/bf00846148 · Cited by 20
“Using the conditional measures of agreement KA and /s ( Light, 1971 ), we can localize more precisely the stability of the SI ratings in this study.”
Method
On the Usefulness of Interrater Reliability Coefficients
ten Hove, D., Jorgensen, T. D., & van der Ark, L. A. (2018) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-319-77249-3_6 · Cited by 19
“Three generalizations of κ were available to assess nominal data with more than two raters: Fleiss’ kappa (κFleiss; Fleiss 1971), Conger’s exact kappa (κExact; Conger 1980), and Light’s kappa (κLight; Light 1971).”
Method
Evaluating audio-visual falls prevention messages with community-dwelling older people using a World Café forum approach
de Jong, L. D., Francis‐Coad, J., Wortham, C., Haines, T. P., Skelton, D. A., Weselman, T., & Hill, A. (2019) · BMC Geriatrics · doi:10.1186/s12877-019-1344-3 · Cited by 19
Analyzing the Effectiveness of mHealth to Manage Diabetes Mellitus Among Adults Over 50: A Systematic Literature Review
Kruse, C. S., Mileski, M., Heinemann, K., Huynh, H., Leafblad, A., & Moreno, E. (2023) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s392693 · Cited by 21
“99, near perfect agreement).(27,28) The initial search yielded 113,698 results.”
Background
The Scope, Trends, and Challenges of Neurosurgical Research in Nigeria: A Bibliometric Review
Ukachukwu, A.-E. K., Ogundeji, O. D., Abu-Bonsrah, N. A., Still, M. E., Trillo-Ordonez, Y., Oboh, E. N., Nischal, S. A., Deng, D. D., Ugorji, C., Seas, A., et al. (2023) · World Neurosurgery · doi:10.1016/j.wneu.2023.10.147 · Cited by 19
Consistência interna e reprodutibilidade da versão em português do critério de diagnóstico na pesquisa para desordens temporomandibulares (RDC/TMD - Eixo II)
Campos, J., Carrascosa, A., Loffredo, L., & Faria, J. B. (2007) · Brazilian Journal of Physical Therapy · doi:10.1590/s1413-35552007000600006 · Cited by 19
Articular Gap and Step-off Revisited: 3D Quantification of Operative Reduction for Posterior Malleolar Fragments
Keizer, R.-J. O. D. M., Meijer, D. T., van der Gronde, B. A. T. D., Teunis, T., Stufkens, S. A. S., Kerkhoffs, G. M. M. J., Goslings, J. C., & Doornberg, J. N. (2016) · Journal of Orthopaedic Trauma · doi:10.1097/bot.0000000000000676 · Cited by 19
“To compute intrarater reliability, we used Lights’ kappa strategy.(34) With this technique, we computed a (square) weighted kappa for both observers’ sessions separately, yielding 3 different intrarater weighted kappa’s for both 3D-displacement and gap surface.”
Method
Reliability of OMERACT ultrasound elementary lesions in gout: results from a multicenter exercise
Cazenave, T., Martiré, M. V., Reginato, A. M., Gutiérrez, M., Waimann, C. A., Pineda, C., Rosa, J., Ruta, S., Sedano-Santiago, O., Bertoli, A. M., et al. (2018) · Rheumatology International · doi:10.1007/s00296-018-4220-0 · Cited by 19
I Don't See Race (or Conflict): Strategic Descriptions of Ambiguous Negative Intergroup Contexts
Karmali, F., Kawakami, K., Vaccarino, E., Williams, A., Phills, C. E., & Friesen, J. P. (2019) · Journal of Social Issues · doi:10.1111/josi.12353 · Cited by 19
The creation and application of a large-scale corpus-based academic multi-word unit list
Rogers, J., Müller, A., Daulton, F., Dickinson, P., Florescu, C., Reid, G. A., & Stoeckel, T. (2021) · English for Specific Purposes · doi:10.1016/j.esp.2021.01.001 · Cited by 18
Including Omission Mistakes in the Calculation of Cohen's Kappa and an Analysis of the Coefficient's Paradox Features
Simon, P. (2006) · Educational and Psychological Measurement · doi:10.1177/0013164405285548 · Cited by 19
“If one has a standard protocol, there is no symmetric measurement anymore, but an asymmetric one, which leads to a necessary modification of the coefficient (cf. Fleiss, 1981; Kvalseth, 1991; Light, 1971).”
Background
Development of a Teen-Informed Coding Tool to Measure the Power of Food Advertisements
Bowman, D. D., Minaker, L. M., Simpson, B. J. K., & Gilliland, J. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16214258 · Cited by 20
“910 [66], and where there was disagreement, a third coder from the authors acted as the decision maker.”
BackgroundMethodInfluential
Interrater Reliability of NI-RADS on Posttreatment PET/Contrast-enhanced CT Scans in Head and Neck Squamous Cell Carcinoma
Hsu, D., Rath, T. J., Branstetter, B. F., Anzai, Y., Phillips, C. D., Juliano, A. F. T., Mosier, K. M., Bazylewicz, M., Poliashenko, S. M., Kulzer, M. H., et al. (2021) · Radiology Imaging Cancer · doi:10.1148/rycan.2021200131 · Cited by 19
“Interrater agreement for prose descriptions (ie, lexicon responses) was assessed using unweighted Light generalization of Cohen k measure of interrater reliability, which is based on the mean of the pairwise Cohen k measures of agreement between the eight raters on the scoring of the 80 patients (17,18).”
Method
The content of management work: Formation of task and job skill composite classifications
Schippmann, J. S., Prien, E. P., & Hughes, G. L. (1991) · Journal of Business and Psychology · doi:10.1007/bf01017706 · Cited by 19
“The resulting 21 dimension task and and 22 dimension job skill classifications were evaluated in terms of internal consistency using the kappa statistic (Fleiss, Nee, & Landis, 1979) and with reference to a standard or "correct solution" using the statistic G ( Light, 1971 ).”
Method
Leveraging Telehealth for the Management of Breast Cancer: A Systematic Review
Kruse, C. S., Pacheco, G. J., Vargas, B., Lozano, N., Castro, S., & Gattu, M. (2022) · Healthcare · doi:10.3390/healthcare10102015 · Cited by 18
Interlaboratory evaluation of two Reverse‐transcriptase Polymeric Chain Reaction‐based methods for detection of four fruit tree viruses
Massart, S., Brostaux, Y., Barbarossa, L., Batlle, A., César, V. R., Dutrecq, O., Fonseca, F., Guillem, R., Komorowska, B., Olmos, A., et al. (2009) · Annals of Applied Biology · doi:10.1111/j.1744-7348.2008.00281.x · Cited by 19
“Repeatability was evaluated by averaging the kappa coefficient between repetitions in each laboratory, and the reproducibility by averaging those coefficients between all pairs of repetitions whatever their origin (Light, 1971).”
Method
Applicability of the iterative technique for cardiac resynchronization therapy optimization: full-disclosure, 50-sequential-patient dataset of transmitral Doppler traces, with implications for future research design and guidelines
Jones, S. H., Shun-Shin, M. J., Cole, G. D., Sau, A., March, K., Williams, S., Kyriacou, A., Hughes, A. D., Mayet, J., Frenneaux, M. P., et al. (2013) · EP Europace · doi:10.1093/europace/eut257 · Cited by 18
A measure of association for ordered categorical data in population-based studies
Nelson, K. P., & Edwards, D. (2016) · Statistical Methods in Medical Research · doi:10.1177/0962280216643347 · Cited by 18
Estimating Interinterviewer Reliability for Interview Schedules used in Special Education Research
Goodwin, L. D., Sands, D. J., & Kozleski, E. B. (1991) · The Journal of Special Education · doi:10.1177/002246699102500105 · Cited by 18
Immediate and Short-term Effects of Straw Phonation in Air or Water on Vocal Fold Vibration and Supraglottic Activity of Adult Patients with Voice Disorders Visualized with Strobovideolaryngoscopy: A Pilot Study
Meerschman, I., Lierde, K. M. V., D’haeseleer, E., Alnouri, G., Burdett, J., Palmer, J., Rose, B., Doucette, P., Paknezhad, H., Ross, J., et al. (2021) · Journal of Voice · doi:10.1016/j.jvoice.2021.09.017 · Cited by 17
“For the nominal data, Cohen’s kappa (K) was run for all rater pairs followed by computing the arithmetic mean of these estimates [47,49].”
Method
Exploring narratives of psychological input in the acute inpatient setting
Donaghay‐Spire, E. G., McGowan, J. F., Griffiths, K., & Barazzone, N. (2015) · Psychology and Psychotherapy Theory Research and Practice · doi:10.1111/papt.12081 · Cited by 17
“Inter-rater reliability analysis was conducted, whereby SPSS was used to calculate the Kappa statistic to determine consistency among raters (Light, 1971).”
Method
Evaluating the implementation of RxNorm in ambulatory electronic prescriptions
Dhavle, A. A., Ward-Charlerie, S., Rupp, M. T., Kilbourne, J., Amin, V. P., & Ruiz, J. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv131 · Cited by 17
The effect of an e-learning module on grading variation of (pre)malignant breast lesions
van Dooijeweert, C., Deckers, I. A. G., de Ruiter, E. J., ter Hoeve, N. D., Vreuls, C. P. H., van der Wall, E., & van Diest, P. J. (2020) · Modern Pathology · doi:10.1038/s41379-020-0556-6 · Cited by 17
A color fusion model based on Markowitz portfolio optimization for optic disc segmentation in retinal images
Escorcia‐Gutierrez, J., Torrents‐Barrena, J., Gamarra, M., Romero‐Aroca, P., Valls, A., & Puig, D. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.114697 · Cited by 17
Do pathologists have extrasensory perception?
Bull, A. D., Cross, S. S., James, D. S., & Silcocks, P. B. S. (1991) · BMJ · doi:10.1136/bmj.303.6817.1604 · Cited by 17
Measurement quality issues in dyadic models of relationships
Iacobucci, D., Neelamegham, R., & Hopkins, N. J. (1999) · Social Networks · doi:10.1016/s0378-8733(99)00010-6 · Cited by 17
Clinical effectiveness of fluorescence, digital images and ICDAS for detecting occlusal caries
Presoto, C. D., Trevisan, T. C., de Andrade, M. C., Dantas, A. A. R., Campos, J. Á. D. B., & de Oliveira, O. B. (2017) · Revista de Odontologia da UNESP · doi:10.1590/1807-2577.12416 · Cited by 17
Repetitive Activity in the Play of Children With Mental Retardation
Lender, W. L., Goodman, J. F., & Linn, M. I. (1998) · Journal of Early Intervention · doi:10.1177/105381519802100405 · Cited by 17
The Internal and External Optimality of Decisions Based on Tests
Mellenbergh, G. J., & van der Linden, W. J. (1979) · Applied Psychological Measurement · doi:10.1177/014662167900300214 · Cited by 17
“Light (1971); and Spitzer, Cohen, Fleiss, and Endicott (1967).”
Prevalence of Elbow Joint Arthritis and Enthesitis in Rheumatoid Arthritis
Schäfer, V. S., Recker, F., Vossen, D., Geffken, I., Matuschek, E., & HARTUNG, W. M. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9051590 · Cited by 16
Inter-reader Agreement for Prostate Cancer Detection Using Micro-ultrasound: A Multi-institutional Study
Zhou, S., Choi, M. H., Vesal, S., Kinnaird, A. S., Brisbane, W. G., Giovanni Lughezzani, A., Maffei, D., Fasulo, V., Albers, P., Zhang, L., et al. (2024) · European Urology Open Science · doi:10.1016/j.euros.2024.06.017 · Cited by 16
“Inter-reader agreement was measured using Light’s j , which is a variation of Cohen’s j appropriate for fully crossed studies with multiple readers [18–21].”
Method
Degenerative Cervical Myelopathy: A 7-Letter Coding System That Supports Decision-Making for the Surgical Approach
Papavero, L., Schmeiser, G., Kothe, R., Boszczyk, B. M., Heese, O., Kawaguchi, Y., MacDowall, A., Olerud, C., Paidakakos, N., Panagiotou, A., et al. (2019) · Neurospine · doi:10.14245/ns.1938010.005 · Cited by 16
Reliability of the Oswestry-Bristol Classification for trochlear dysplasia
Konrads, C., Gonser, C. E., & Ahmad, S. S. (2020) · Bone & Joint Open · doi:10.1302/2046-3758.17.bjo-2020-0048.r1 · Cited by 16
A novel approach to falls classification in Parkinson’s disease: development of the Fall-Related Activity Classification (FRAC)
Ross, A., Yarnall, A. J., Rochester, L., & Lord, S. (2016) · Physiotherapy · doi:10.1016/j.physio.2016.08.002 · Cited by 16
Radiographic classifications in Perthes disease
Huhnstock, S., Svenningsen, S., Merckoll, E., Catterall, A., Terjesen, T., & Wiig, O. (2017) · Acta Orthopaedica · doi:10.1080/17453674.2017.1340040 · Cited by 16
Reliability testing of the HEARTSMAP psychosocial assessment tool for multidisciplinary use and in diverse emergency settings
Virk, P., Stenstrom, R., & Doan, Q. H. (2018) · Paediatrics & Child Health · doi:10.1093/pch/pxy017 · Cited by 16
“In accordance with Light (1971), kappa values were computed for all evaluator pairs, the overall mean of these pairs was then used as our index of agreement (20).”
Method
THE CONCEPTUAL DISTANCES BETWEEN IDEAS IN COMBINATIONAL CREATIVITY
Han, J., Shi, F., Park, D., Chen, L., & Childs, P. (2018) · Proceedings of the ... International Design Conference/Design ... · doi:10.21278/idc.2018.0264 · Cited by 16
“used to measure the overall inter-rater agreement of a category (Light, 1971).”
Method
Application of a human resource value model: A field study
Ogan, P. (1976) · Accounting Organizations and Society · doi:10.1016/0361-3682(76)90023-4 · Cited by 16
Altered plasma antioxidant status in subjects with Alzheimer's disease and vascular dementia
Sinclair, A. J., Bayer, A. J., Johnston, J., Warner, C. A., & Maxwell, S. R. J. (1998) · International Journal of Geriatric Psychiatry · doi:10.1002/(sici)1099-1166(1998120)13:12 · Cited by 147
Factor Structure of Play Creativity: A New Instrument to Assess Movement Creativity
Richard, V., Aubertin, P., Yang, Y. Y., & Kriellaars, D. J. (2020) · Creativity Research Journal · doi:10.1080/10400419.2020.1821567 · Cited by 15
Effect of drying on the reproducibility of DIAGNOdent to detect caries-like lesions
Pinelli, C., Loffredo, L. D. C. M., & Serra, M. C. (2010) · Brazilian Dental Journal · doi:10.1590/s0103-64402010000500005 · Cited by 15
Pain drawings predict outcome of surgical treatment for degenerative disc disease in the cervical spine
MacDowall, A., Robinson, Y., Skeppholm, M., & Olerud, C. (2017) · Upsala Journal of Medical Sciences · doi:10.1080/03009734.2017.1340372 · Cited by 15
“Light’s j (kappa) (18) was computed for the same three observers as above.”
Method
Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability
Wong, K., Paritosh, P. K., & Aroyo, L. M. (2021) · doi:10.18653/v1/2021.acl-long.548 · Cited by 29
“Since its publication, it has been generalized to accommodate multiple raters (Light, 1971; Berry and Mielke Jr, 1988), and to cover different types of annotation scales: ordinal (Cohen, 1968), interval (Berry and Mielke Jr, 1988; Janson and Olsson, 2001), multivariate (Berry and Mielke Jr, 1988), and any arbitrary di…”
Background
Reliability Estimates of Some Survey Data on Family Planning
Mukherjee, B. N. (1975) · Population Studies · doi:10.1080/00324728.1975.10410190 · Cited by 15
The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial
Meijering, J. V., & Tobi, H. (2018) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2017.11.010 · Cited by 16
Using appropriate Kappa statistic in evaluating inter-rater reliability. Short communication on “Groundwater vulnerability and contamination risk mapping of semi-arid Totko river basin, India using GIS-based DRASTIC model and AHP techniques”
Li, M., Gao, Q., & Yu, T. (2023) · Chemosphere · doi:10.1016/j.chemosphere.2023.138565 · Cited by 14
Decreased activation in left prefrontal cortex during role-play: An fNIRS study of the psychodrama sociocognitive model
Lim, M., Carollo, A., Bizzego, A., Chen, S. A., & Esposito, G. (2023) · The Arts in Psychotherapy · doi:10.1016/j.aip.2023.102098 · Cited by 14
Residency Application Selection Committee Discriminatory Ability in Identifying Artificial Intelligence-Generated Personal Statements
Koleilat, I., Bongu, A., Chang, S., Nieman, D. R., Priolo, S., & Patel, N. M. (2024) · Journal of surgical education · doi:10.1016/j.jsurg.2024.02.009 · Cited by 14
Internal Consistency and Inter-Rater Reliability of the Questions About Behavioral Function (QABF) Rating Scale When Used by Teachers and Paraprofessionals
May, M. E., Sheng, Y., Chitiyo, M., Brandt, R. C., & Howe, A. P. (2014) · Education and Treatment of Children · doi:10.1353/etc.2014.0013 · Cited by 14
“59 raters were used per student in the current study (e.g., Light, 1971).”
Method
Inter‐ and intrarater clinician agreement on joint motion patterns during gait in children with cerebral palsy
Nieuwenhuys, A., Papageorgiou, E., Molenaers, G., Monari, D., Laet, T. D., & Desloovere, K. (2017) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.13404 · Cited by 14
Impact of Interdisciplinary Team Care for Children With 22q11.2 Deletion Syndrome
Hickey, S. E., Kellogg, B. C., O’Brien, M., Hall, C., Kirschner, R. E., Santoro, S. L., Leonard, H., & Baylis, A. L. (2020) · The Cleft Palate-Craniofacial Journal · doi:10.1177/1055665620947985 · Cited by 14
“Overall IRR was calculated by combining the ratings and averaging Cohen k for all 3 pairs of raters (Hallgren, 2012; Light, 1971; McHugh, 2012).”
Method
Health Information Technology and Doctor Shopping: A Systematic Review
Kruse, C. S., Kindred, B., Brar, S., Gutiérrez, G., & Cormier, K. (2020) · Healthcare · doi:10.3390/healthcare8030306 · Cited by 14
“95), which indicates near perfect agreement [23,24].”
Method
The form of a ‘half-baked’ creative idea: Empirical explorations into the structure of ill-defined mental representations
Scotney, V. S., Schwartz, J., Carbert, N., Saab, A. M. P., & Gabora, L. M. (2020) · Acta Psychologica · doi:10.1016/j.actpsy.2019.102981 · Cited by 14
“An average of the kappa coefficient for each rater pair of the remaining five judges was calculated (Hallgren, 2012; Light, 1971) and showed fair agreement between the five judges for both ratings of story comprehension (κ = 0.38), and ratings of the incomplete solutions (κ = 0.34; Landis & Koch,…”
BackgroundMethodInfluential
Analysis of Nonagreements among Multiple Raters
James, I. R. (1983) · Biometrics · doi:10.2307/2531092 · Cited by 14
Measuring endogamy
Strauss, D. (1977) · Social Science Research · doi:10.1016/0049-089x(77)90010-2 · Cited by 14
Imagining an Engineer: On GAN-Based Data Augmentation Perpetuating BiasesPreprint
Jain, N., Manikonda, L., Hernandez, A. O., Sengupta, S., & Subbarao Kambhampati, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03751 · Cited by 14
Materials Matter: An Exploration of Text Complexity and Its Effects on Middle School Readers' Comprehension Processing
Dahl, A. C., Carlson, S. E., Renken, M. D., McCarthy, K. S., & Reynolds, E. (2021) · Language Speech and Hearing Services in Schools · doi:10.1044/2021_lshss-20-00117 · Cited by 13
Defining Posttraumatic Sepsis for Population-Level Research
Stern, K., Qiu, Q., Weykamp, M. B., O’Keefe, G. E., & Brakenridge, S. C. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.51445 · Cited by 13
Annotation Agreement of Emotions in Text: The Influence of Counsellors' Emotional State on their Emotion Perception
Kolog, E. A., Montero, C. S., & Sutinen, E. (2016) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2016.21 · Cited by 13
Popular Conceptions of Mindfulness: Awareness and Emotional Control
Hitchcock, P. F., Martin, L., Fischer, L., Marando-Blanck, S., & Herbert, J. D. (2016) · Mindfulness · doi:10.1007/s12671-016-0533-9 · Cited by 15
Development of a Child Abuse Checklist to Evaluate Prehospital Provider Performance
Alphonso, A. L., Auerbach, M. A., Bechtel, K. A., Bilodeau, K. S., Gawel, M., Koziel, J. R., Whitfill, T. M., & Tiyyagura, G. K. (2016) · Prehospital Emergency Care · doi:10.1080/10903127.2016.1229824 · Cited by 13
“Solicits details about actions of caregiver prior to EMS arrival 5/22 (23) 4/11 (36) 1/11 (9) 0.”
Background
Analysis of medication therapy discontinuation orders in new electronic prescriptions and opportunities for implementing CancelRx
Yang, Y., Ward-Charlerie, S., Kashyap, N., deMayo, R., Agresta, T. P., & Green, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy100 · Cited by 14
“Their inter-rater reliability was assessed by calculating the Cohen’s Kappa (j) coefficient.(23) RESULTS”
BackgroundMethod
MRI Bone Marrow Edema Signal Intensity
Sims, K., Kountouris, A., Stegeman, J. R., Rotstein, A. H., Beakley, D., Saw, A. E., & Cook, J. L. (2019) · Spine · doi:10.1097/brs.0000000000003277 · Cited by 13
“Inter-rater agreement of the presence of MRI BMO was calculated as the mean of rater pairs (R0R1, R0-R2, R1-R2).(10)”
Method
Evaluating the Accuracy of Ground-Based Hemlock Dwarf Mistletoe Rating: A Case Study Using the Wind River Canopy Crane
Shaw, D. C., Freeman, E. A., & Mathiasen, R. L. (2000) · Western Journal of Applied Forestry · doi:10.1093/wjaf/15.1.8 · Cited by 13
Norms, reliability, and item analysis of the hopelessness scale in general psychiatric, forensic psychiatric, and college populations
Durham, T. W. (1982) · Journal of Clinical Psychology · doi:10.1002/1097-4679(198207)38:3 · Cited by 78
Signs of Defensive Control Locate Conflicted Topics in Discourse
Horowitz, M. J. (1995) · Archives of General Psychiatry · doi:10.1001/archpsyc.1995.03950240058011 · Cited by 13
Agreement between raters and groups of raters
Vanbelle, S. (2009) · Cited by 13
Corrected Zegers-ten Berge Coefficients Are Special Cases of Cohen’s Weighted Kappa
Warrens, M. J. (2014) · Journal of Classification · doi:10.1007/s00357-014-9156-9 · Cited by 18
“If we use the weights in (8) in (9) we obtain the unweighted kappa for multiple raters proposed in Light (1971).”
Method
Deep Semantic Segmentation of Angiogenesis Images
Ibragimov, A., Сенотрусова, С. А., Markova, K. L., Karpulevich, E., Ivanov, A., Tyshchuk, E. V., Владимировна, Г. П., Игоревна, С. О., Sirotskaya, A., Kovaleva, A., et al. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24021102 · Cited by 12
“As advised in [68,70], we used the mean kappa to obtain a ﬁnal measure for 3 or more participants.”
Critical incidents of quality orientation in lower and middle management
Heine, I., Schmitt, R. H., & Beaujean, P. (2016) · The TQM Journal · doi:10.1108/tqm-11-2015-0137 · Cited by 12
Business Decision Support System based on Sentiment Analysis
Oppong, S. O., Asamoah, D., Oppong, E., & Lamptey, D. (2019) · International Journal of Information Engineering and Electronic Business · doi:10.5815/ijieeb.2019.01.05 · Cited by 12
“Most noble of these researchers is Light [29].”
BackgroundInfluential
Inter‐rater reliability of the BIOCHIP indirect immunofluorescence dermatology mosaic in bullous pemphigoid and pemphigus patients
Yang, A., Xuan, R., Melbourne, W., Hashimoto, T., Uzun, S., Daneshpazhooh, M., Yamagami, J., Zenzo, G. D., Mascaró, J. M., Mahmoudi, H., et al. (2019) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.15817 · Cited by 12
Stigma toward Wuhan people during the COVID-19 epidemic: an exploratory study based on social media
Di, Y., Li, A., Li, H., Wu, P., Yang, S., Zhu, M., Zhu, T., & Liu, X. (2021) · BMC Public Health · doi:10.1186/s12889-021-12001-2 · Cited by 12
Effects of an ego-depletion intervention upon online learning
Greene, J. A., Duke, R. F., Freed, R., Dragnić-Cindrić, D., & Cartiff, B. M. (2021) · Computers & Education · doi:10.1016/j.compedu.2021.104362 · Cited by 14
Analysis of National Drug Code Identifiers in Ambulatory E-Prescribing
Dhavle, A. A., Ward-Charlerie, S., Rupp, M. T., Amin, V. P., & Ruiz, J. (2015) · Journal of Managed Care & Specialty Pharmacy · doi:10.18553/jmcp.2015.21.11.1025 · Cited by 12
Computer-aided diagnosis of renal obstruction: utility of log-linear modeling versus standard ROC and kappa analysis
Manatunga, A. K., Binongo, J., & Taylor, A. T. (2011) · EJNMMI Research · doi:10.1186/2191-219x-1-5 · Cited by 12
“Moreover, kappa-type statistics [9,15,16] can be heavily influenced by the distribution of disease in the population as well as by differences or similarities among raters [17].”
Background
College students’ reasons for using different e-cigarette products: A mixed methods analysis
Luzius, A., Dobbs, P. D., & Jozkowski, K. N. (2019) · Journal of American College Health · doi:10.1080/07448481.2019.1618313 · Cited by 11
Similarity measures for 2 × 2 tables
Warrens, M. J. (2018) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-172291 · Cited by 12
“The similarity measures κ (0) and κ (1) are also considered in Peirce [77] and Light [66].”
BackgroundInfluential
Assessing agreement between raters from the point of coefficients and loglinear models
Yılmaz, A. E., Saraçbaşı, T., & Saraçbaşı, T. (2021) · Journal of Data Science · doi:10.6339/jds.201701_15(1).0001 · Cited by 12
Insight in nonpsychotic disorders: A new model of insight and a systematic review of measures
Halaj, A., & Huppert, J. D. (2021) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/00048674211025722 · Cited by 13
“Kappa was computed for each coder pair and then averaged to provide a single index of IRR (Light, 1971).”
MethodInfluential
An evaluation of the acquisition of sexual information through a sex education class∗
Monge, R. H., Dusek, J. B., & Lawless, J. (1977) · The Journal of Sex Research · doi:10.1080/00224497709550973 · Cited by 12
Deriving consensus sepsis clusters via goal-directed subgroup identification in multi-omics study
Zhang, Z., Chen, L., Shen, H., Wang, J., Yang, J., Yang, S., Zhang, W., Jiang, X., Wu, X., Meng, X., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-65271-4 · Cited by 13
“To quantify the agreement of cluster membership across different omics datasets, we utilized Light ’ s Kappa statistic 59 .”
Evaluation of direct restorations using the revised FDI criteria: results from a reliability study
Mesinger, S., Heck, K., Crispin, A., Frankenberger, R., Cadenaro, M., Burgess, J., Peschke, A., Heintze, S. D., Loomans, B. A. C., Opdam, N. J. M., et al. (2022) · Clinical Oral Investigations · doi:10.1007/s00784-022-04771-9 · Cited by 11
“Later, the data was exported into an Excel spreadsheet (Excel 2016, Microsoft, Redmond, WA, USA) and checked for plausibility before analysis.”
Method
A note on interrater agreement
Hirji, K. F., & Rosove, M. H. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090713 · Cited by 11
The role of consumer perspectives in estimating population need for substance use services: a scoping review
Hyshka, E., Karekezi, K., Tan, B., Slater, L., Jahrig, J., & Wild, T. C. (2017) · BMC Health Services Research · doi:10.1186/s12913-017-2153-z · Cited by 13
“To ensure consistency in the screening process, all raters assessed and monitored inter-rater agreement on triaging decisions using Light’s Kappa coefficient [54, 55].”
Method
Resident perceptions of the impacts of large-scale sugarcane production on ecosystem services in two regions of Brazil
de Oliveira, C. O. F., Walter, A. C. D. S., Guarenghi, M. M., & Favero, C. (2017) · Biomass and Bioenergy · doi:10.1016/j.biombioe.2017.08.029 · Cited by 11

Authors: Light, R. J.
Affiliations: Howard University (US)
Corresponding author: Light, Richard J.
Venue: Psychological Bulletin
Publisher: American Psychological Association (APA)
Published: 1971-11
Volume: 76
Issue: 5
Pages: 365-377
ISSN: 0033-2909 (ISSN-L)
1939-1455
DOI: 10.1037/h0031643
OpenAlex: W1989492805
Cited by: 747
References: 17
Citations by year: 2026: 22 · 2025: 26 · 2024: 19 · 2023: 43 · 2022: 24 · 2021: 46 · 2020: 43 · 2019: 38 · 2018: 39 · 2017: 36 · 2016: 46 · 2015: 31 · 2014: 23 · 2013: 15 · 2012: 11
FWCI: 8.62
Type: journal article
Language: en
Open access: closed
Locations: Psychological Bulletin (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Keywords: interobserver agreement, qualitative data, categorical data, agreement measures, inter-rater reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Measures of response agreement for qualitative data: Some generalizations and alternatives.

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks