The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The kappa was .78, which reflects a moderate agreement between the reviewers [25,26].”
From Barriers to the Use of Mobile Health in Improving Health Outcomes in Developing Countries: Systematic Review (2019) · cites this paper as [25]
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Kendall, M. G. (1957) · Biometrika · doi:10.2307/2333282 · Cited by 6,442
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. (1979) · Springer series in statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,255
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,536
Estimation of the reliability of ratings.
Ebel, R. L. (1951) · Psychometrika · doi:10.1007/bf02288803 · Cited by 976
Unitizing and categorizing problems in coding qualitative data
Guetzkow, H. S. (1950) · Journal of Clinical Psychology · doi:10.1002/1097-4679(195001)6:1 · Cited by 436
Intraclass Correlation and the Analysis of Variance
Trese, T., & Haggard, E. A. (1958) · The American Catholic Sociological Review · doi:10.2307/3708305 · Cited by 401
Flagged first, then by citations
Cited by
Sorted by citations
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
Hallgren, K. A. (2012) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.08.1.p023 · Cited by 3,884
“Kappa was computed for each coder pair then averaged to provide a single index of IRR (Light, 1971).”
BackgroundMethodInfluentialRelation of sample size to the stability of component patterns.
Guadagnoli, E., & Velicer, W. F. (1988) · Psychological Bulletin · doi:10.1037/0033-2909.103.2.265 · Cited by 2,880
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167
Reliability of Nominal Data Based on Qualitative Judgments
Perreault, W. D., & Leigh, L. E. (1989) · Journal of Marketing Research · doi:10.2307/3172601 · Cited by 1,575
Treviño, L. K., Brown, M., & Hartman, L. P. (2003) · Human Relations · doi:10.1177/0018726703056001448 · Cited by 1,372
“We computed inter-rater agreement of 92.1 percent for the ethics officer interview data and 88.7 percent for the executive data, using the P statistic (Light, 1971) which involves dividing the number of cards for which the raters agreed by the total number of cards.”
MethodInfluentialPayne, D. L., Lonsway, K. A., & Fitzgerald, L. F. (1999) · Journal of Research in Personality · doi:10.1006/jrpe.1998.2238 · Cited by 1,114
Autism diagnostic interview: A standardized investigator-based instrument
Le Couteur, A., Rutter, M., Lord, C., Rios, P., Robertson, S., Holdgrafer, M., & McLennan, J. (1989) · Journal of Autism and Developmental Disorders · doi:10.1007/bf02212936 · Cited by 1,036
Beyond kappa: a review of interrater agreement measures
Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999) · Canadian Journal of Statistics · doi:10.2307/3315487 · Cited by 976
Goldman, L., Hashimoto, B. E., Cook, E. F., & Loscalzo, A. (1981) · Circulation · doi:10.1161/01.cir.64.6.1227 · Cited by 974
Weaver, G. R., Treviño, L. K., & Cochran, P. L. (1999) · Academy of Management Journal · doi:10.2307/256975 · Cited by 928
The Handbook of Research Synthesis
Ziegel, E. R., Cooper, H., & Hedges, L. V. (1995) · Technometrics · doi:10.2307/1269651 · Cited by 936
Weaver, G. R., Treviño, L. K., & Cochran, P. L. (1999) · Academy of Management Journal · doi:10.2307/256873 · Cited by 666
The Reliability of Clinical Methods, Data and Judgments
Koran, L. M. (1975) · New England Journal of Medicine · doi:10.1056/nejm197509252931307 · Cited by 654
A Re-analysis of the Reliability of Psychiatric Diagnosis
Spitzer, R. L., & Fleiss, J. L. (1974) · The British Journal of Psychiatry · doi:10.1192/bjp.125.4.341 · Cited by 622
Integration and generalization of kappas for multiple raters
Conger, A. J. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.2.322 · Cited by 545
Kappa coefficients in medical research
KRAEMER, H. C., Periyakoil, V. S., & Noda, A. M. (2002) · Statistics in Medicine · doi:10.1002/sim.1180 · Cited by 547
Reliability from α to ω: A tutorial.
Revelle, W., & Condon, D. M. (2019) · Psychological Assessment · doi:10.1037/pas0000754 · Cited by 569
“For multiple raters, the average κ is known as Light’s κ (Conger, 1980; Light, 1971).”
BackgroundMoral Awareness in Business Organizations: Influences of Issue-Related and Social Context Factors
Butterfield, K. D., Trevin, L. K., & Weaver, G. R. (2000) · Human Relations · doi:10.1177/0018726700537004 · Cited by 508
Applied MANOVA and Discriminant Analysis
Huberty, C. J., & Olejnik, S. F. (2005) · Wiley series in probability and statistics · doi:10.1002/047178947x · Cited by 507
Measuring Agreement between Two Judges on the Presence or Absence of a Trait
Fleiss, J. L. (1975) · Biometrics · doi:10.2307/2529549 · Cited by 506
SPECIES: A Spatial Evaluation of Climate Impact on the Envelope of Species
Pearson, R. G., Dawson, T. P., Berry, P. M., & Harrison, P. A. (2002) · Ecological Modelling · doi:10.1016/s0304-3800(02)00056-x · Cited by 474
“This was achieved through calculation of Cohen’s kappa statistic of similarity (k) (Light, 1971; Monserud and Leemans, 1992) at thresholds increasing in increments of 0.1 until maximum agreement between observed and simulated distributions was identified.”
MethodInfluentialD’Agostino, M. A., Terslev, L., Aegerter, P., Backhaus, M., Balint, P. V., Bruyn, G. A. W., Filippucci, E., Grassi, W., Iagnocco, A., Jousse‐Joulin, S., et al. (2017) · RMD Open · doi:10.1136/rmdopen-2016-000428 · Cited by 438
The Appropriateness Evaluation Protocol
Gertman, P. M., & Restuccia, J. D. (1981) · Medical Care · doi:10.1097/00005650-198108000-00005 · Cited by 403
Examining the Dimensions of Cancer-Related Information Seeking and Scanning Behavior
Niederdeppe, J., Hornik, R., Kelly, B., Frosch, D. L., Romantan, A., Stevens, R. S., Barg, F. K., Weiner, J. L., & Schwartz, J. S. (2007) · Health Communication · doi:10.1080/10410230701454189 · Cited by 396
“To create an overall measure of consistency across the four decisions for any given individual, we applied Light's (1971 ) method for calculating kappa statistics for multiple “raters” (in this case, multiple decisions).”
MethodAkil, M., Pierri, J. N., Whitehead, R. E., Edgar, C. L., Mohila, C., Sampson, A. R., & Lewis, D. A. (1999) · American Journal of Psychiatry · doi:10.1176/ajp.156.10.1580 · Cited by 396
“Assessments of three-way interrater agreement (Light’s kappa estimates [31]) for the four sets of comparisons were as follows: tyrosine hydroxylase-immunoreactive axons in humans, 0.”
MethodMeasuring Agreement for Multinomial Data
Davies, M., & Fleiss, J. L. (1982) · Biometrics · doi:10.2307/2529886 · Cited by 355
A Review and Taxonomy of Choice Architecture Techniques
Münscher, R., Vetter, M., & Scheuerle, T. (2015) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1897 · Cited by 389
“Following Hallgren’s (2012) recommendations for fully crossed designs with nominal data, Cohen’s kappa was computed for each pair of coders and then averaged to provide a single index (Light, 1971).”
MethodSchmiedel, T., Müller, O., & vom Brocke, J. (2019) · Organizational Research Methods · doi:10.1177/1094428118773858 · Cited by 319
“Consolidating the individual coding results, the topic labeling exercise revealed high intercoder reliability with an intercoder agreement of 86% and an average kappa value of 0.86 (Light, 1971; Moore & Benbasat, 1991).”
BackgroundInfluentialMultiPic: A standardized set of 750 drawings with norms for six European languages
Duñabeitia, J. A., Crepaldi, D., Meyer, A. S., New, B., Pliatsikas, C., Smolka, E., & Brysbaert, M. (2017) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2017.1310261 · Cited by 308
“Kappa was computed for each coder pair and then averaged to provide a unitary index of IRR (see Light, 1971).”
Kruse, C., Betancourt, J., Ortiz, S., Valdes Luna, S. M., Bamrah, I. K., & Segovia, N. (2019) · Journal of Medical Internet Research · doi:10.2196/13263 · Cited by 299
“78, which reflects a moderate agreement between the reviewers [25,26].”
BackgroundSymptoms of temporomandibular disorders in the population: an epidemiological study.
Gonçalves, D. A. D. G., Fabbro, A. L. D., Campos, J. Á. D. B., BIGAL, M. E., & Speciali, J. G. (2010) · PubMed · Cited by 295
Ramifications of a Population Model for κ as a Coefficient of Reliability
KRAEMER, H. C. (1979) · Psychometrika · doi:10.1007/bf02296208 · Cited by 295
2 x 2 Kappa Coefficients: Measures of Agreement or Association
Bloch, D. A., & KRAEMER, H. C. (1989) · Biometrics · doi:10.2307/2532052 · Cited by 291
Terslev, L., Naredo, E., Aegerter, P., Wakefield, R. J., Backhaus, M., Balint, P. V., Bruyn, G. A. W., Iagnocco, A., Jousse‐Joulin, S., Schmidt, W., et al. (2017) · RMD Open · doi:10.1136/rmdopen-2016-000427 · Cited by 289
Kruse, C., Fohn, J., Wilson, N., Nunez Patlan, E., Zipp, S., & Mileski, M. (2020) · JMIR Medical Informatics · doi:10.2196/20359 · Cited by 295
“75, representing moderate agreement [24,25].”
Balint, P. V., Terslev, L., Aegerter, P., Bruyn, G. A. W., Chary-Valckenaere, I., Gandjbakhch, F., Iagnocco, A., Jousse‐Joulin, S., Möller, I., Naredo, E., et al. (2018) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2018-213609 · Cited by 281
Tang, L. H., Gönen, M., Hedvat, C. V., Modlin, I. M., & Klimstra, D. S. (2012) · The American Journal of Surgical Pathology · doi:10.1097/pas.0b013e318263207c · Cited by 259
The development and application of a classification scheme of informal helping behaviours.
Gottlieb, B. H. (1978) · Canadian Journal of Behavioural Science/Revue canadienne des sciences du comportement · doi:10.1037/h0081539 · Cited by 243
A Generalization of Cohen's Kappa Agreement Measure to Interval Measurement and Multiple Raters
Berry, K. J., & Mielke, P. W. (1988) · Educational and Psychological Measurement · doi:10.1177/0013164488484007 · Cited by 240
Chrysidis, S., Duftner, C., Dejaco, C., Schäfer, V. S., Ramiro, S., Carrara, G., Scirè, C. A., Hočevar, A., Diamantopoulos, A. P., Iagnocco, A., et al. (2018) · RMD Open · doi:10.1136/rmdopen-2017-000598 · Cited by 250
Measuring interrater reliability among multiple raters: An example of methods for nominal data
Posner, K. L., Sampson, P. D., Caplan, R. A., Ward, R. J., & Cheney, F. W. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090917 · Cited by 229
Inequalities between multi-rater kappas
Warrens, M. J. (2010) · Advances in Data Analysis and Classification · doi:10.1007/s11634-010-0073-4 · Cited by 233
Terslev, L., Naredo, E., Iagnocco, A., Balint, P. V., Wakefield, R. J., Aegerter, P., Aydın, S. Z., Bachta, A., Hammer, H. B., Bruyn, G. A. W., et al. (2013) · Arthritis Care & Research · doi:10.1002/acr.22191 · Cited by 216
The Comprehensive System for the Rorschach: A Critical Examination
Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996) · Psychological Science · doi:10.1111/j.1467-9280.1996.tb00658.x · Cited by 214
“However, percentage of agreement has long been recognized as a potentially inadequate and misleading measure of reliability (Cohen, 1960,1968; Fleiss, 1981; Light, 1971; see also Jensen, 1965), The problem is that percentage of agreement makes no adjustment for agreement by chance and can therefore yield itiflated est…”
BackgroundPUNISHMENT FROM THE MANAGER'S PERSPECTIVE: A GROUNDED INVESTIGATION AND INDUCTIVE MODEL.
Butterfield, K. D., Treviño, L. K., & Ball, G. A. (1996) · Academy of Management Journal · doi:10.2307/257066 · Cited by 211
Headache and Symptoms of Temporomandibular Disorder: An Epidemiological Study
Gonçalves, D. A. D. G., BIGAL, M. E., Jales, L. C. F., Camparis, C. M., & Speciali, J. G. (2009) · Headache The Journal of Head and Face Pain · doi:10.1111/j.1526-4610.2009.01511.x · Cited by 209
“71.(30,33,34) According to the guidelines of the AAOP, a positive answer to at least one of these questions is indicative of possible TMD.”
BackgroundWhat Makes Smartphone Use Meaningful or Meaningless?
Lukoff, K., Yu, C., Kientz, J., & Hiniker, A. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3191754 · Cited by 205
“Light's Kappa, a measure of interrater reliability for fully crossed designs with more than two raters [22,42], was 0.74 (where 0.6-0.8 can be interpreted as indicating substantial agreement [79]).”
BackgroundInfluentialObserver Agreement and Reliabilities of Classroom Observational Measures
Frick, T. W., & Semmel, M. I. (1978) · Review of Educational Research · doi:10.3102/00346543048001157 · Cited by 205
“One advantage of ĸp, ĸ, and πis that they can be tested for significance (see Fleiss, Cohen, & Everitt, 1969; Light, 1971).”
BackgroundStatistical methods for assessing agreement for ordinal data
Jakobsson, U., & Westergren, A. (2005) · Scandinavian Journal of Caring Sciences · doi:10.1111/j.1471-6712.2005.00368.x · Cited by 214
The Reliability of Dichotomous Judgments: Unequal Numbers of Judges per Subject
Fleiss, J. L., & Cuzick, J. M. (1979) · Applied Psychological Measurement · doi:10.1177/014662167900300410 · Cited by 191
“Landis and Koch (1977a) and Light (1971) considered the case of a fixed set of judges, whereas Fleiss (1971) and Landis and Koch (1977b) considered the case of varying sets of judges.”
BackgroundEffects of Instruction in Deriving Word Meaning from Context: A Meta-Analysis
Fukkink, R. G., & de Glopper, K. (1998) · Review of Educational Research · doi:10.3102/00346543068004450 · Cited by 185
“Generalized Cohen's kappa (Conger, 1980; Light, 1971 ) is computed for nominal categories, r/design 2) (Orwin, 1994) for continuous categories.”
MethodFast track algorithm: How to differentiate a “scleroderma pattern” from a “non-scleroderma pattern”
Smith, V. M., Vanhaecke, A., Herrick, A. L., Distler, O., Guerra, M. G., Denton, C. P., Deschepper, E., Foeldvari, I., Gutiérrez, M., HACHULLA, E., et al. (2019) · Autoimmunity Reviews · doi:10.1016/j.autrev.2019.102394 · Cited by 177
Hughes, M. A., & Garrett, D. E. (1990) · Journal of Marketing Research · doi:10.1177/002224379002700206 · Cited by 177
Modeling Agreement Among Raters
Tanner, M. A., & Young, M. A. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10477157 · Cited by 164
Smith, V. M., Beeckman, S., Herrick, A. L., Decuman, S., Deschepper, E., Keyser, F. D., Distler, O., Foeldvari, I., Ingegnoli, F., Müller‐Ladner, U., et al. (2016) · Lara D. Veeken · doi:10.1093/rheumatology/kev441 · Cited by 160
Reliability of Psychiatric Diagnosis
Helzer, J. E. (1977) · Archives of General Psychiatry · doi:10.1001/archpsyc.1977.01770140019001 · Cited by 158
Denburg, S. D., Carbotte, R. M., & Denburg, J. A. (1987) · Journal of Clinical and Experimental Neuropsychology · doi:10.1080/01688638708405054 · Cited by 143
Quantifying the development of agreement among experts in Delphi studies
Meijering, J. V., Kampen, J. K., & Tobi, H. (2013) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2013.01.003 · Cited by 139
Azevedo, R. D. M., Roest, A. M., Hoen, P. W., & de Jonge, P. (2014) · Psychological Medicine · doi:10.1017/s0033291714000063 · Cited by 138
“The agreement on BDI-I symptom assignment to one of the two dimensions across studies was moderate (κ=0.60) (Light, 1971; Conger, 1980).”
MethodInfluentialEvaluation of cutaneous photodamage using a photographic scale
Larnier, C., Ortonne, J. ‐., Alain Venot, A., FAIVRE, B., Béani, J.-C., THOMAS, P., BROWN, T. C., & Sendagorta, E. (1994) · British Journal of Dermatology · doi:10.1111/j.1365-2133.1994.tb02895.x · Cited by 137
Where is the human in human-centered AI? Insights from developer priorities and user experiences
Bingley, W., Curtis, C., Lockey, S., Bialkowski, A., Gillespie, N. A., Haslam, S. A., Ko, R. K. L., Steffens, N. K., Wiles, J., & Worthy, P. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107617 · Cited by 158
Interrater Agreement Measures: Comments on Kappa n , Cohen's Kappa, Scott's π, and Aickin's α
Hsu, L. M., & Field, R. (2003) · Understanding Statistics · doi:10.1207/s15328031us0203_03 · Cited by 133
Landis, J. R., & Koch, G. G. (1975) · Statistica Neerlandica · doi:10.1111/j.1467-9574.1975.tb00254.x · Cited by 135
The Potential of ChatGPT as a Self-Diagnostic Tool in Common Orthopedic Diseases: Exploratory Study
Kuroiwa, T., Sarcon, A. K., Ibara, T., Yamada, E., Yamamoto, A., Tsukamoto, K., & Fujita, K. (2023) · Journal of Medical Internet Research · doi:10.2196/47621 · Cited by 129
“For accuracy, the Fleiss κ coefficient between the categorical variables of the five answers in one rater was calculated and the median of the five values in five raters was regarded as the reproducibility between days [31].”
Role of Ultrasound in the Assessment of Vocal Cord Function in Infants and Children
Friedman, E. M. (1997) · Annals of Otology Rhinology & Laryngology · doi:10.1177/000348949710600304 · Cited by 129
Complexity in Attitudes Toward Abortion Access: Results from Two Studies
Jozkowski, K. N., Crawford, B. L., & Hunt, M. E. (2018) · Sexuality Research and Social Policy · doi:10.1007/s13178-018-0322-4 · Cited by 133
Kruse, C. S., & Heinemann, K. (2021) · Journal of Medical Internet Research · doi:10.2196/31752 · Cited by 130
Mistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices
Feng, G. C. (2014) · Methodology · doi:10.1027/1614-2241/a000086 · Cited by 132
Effects of Augmentative and Alternative Communication on Challenging Behavior: A Meta-Analysis
Walker, V. L., & Snell, M. E. (2013) · Augmentative and Alternative Communication · doi:10.3109/07434618.2013.785020 · Cited by 124
“To account for inclusion ratings due to chance, Cohen ’ s kappa was used to quantify the degree of agreement between raters (Cooper et al., 2009; Light, 1971).”
MethodLanglois, Y., Roederer, G. O., Chan, A., Phillips, D. J., Beach, K. W., Martin, D., Chikos, P. M., & Strandness, D. E. (1983) · Ultrasound in Medicine & Biology · doi:10.1016/0301-5629(83)90109-6 · Cited by 120
A Method for Meta-Analysis of Epidemiological Studies
Einarson, T. R., Leeder, J. S., & Koren, G. (1988) · Annals of Pharmacotherapy · doi:10.1177/106002808802201021 · Cited by 120
Intercoder reliability indices: disuse, misuse, and abuse
Feng, G. C. (2013) · Quality & Quantity · doi:10.1007/s11135-013-9956-8 · Cited by 126
Chance and interrater agreement on manuscripts.
Watkins, M. W. (1979) · American Psychologist · doi:10.1037/0003-066x.34.9.796 · Cited by 116
Assessing the inter-rater agreement for ordinal data through weighted indexes
Marasini, D., Quatto, P., & Ripamonti, E. (2014) · Statistical Methods in Medical Research · doi:10.1177/0962280214529560 · Cited by 122
Silcocks, P. B. S. (1983) · Journal of Clinical Pathology · doi:10.1136/jcp.36.11.1269 · Cited by 115
Kruse, C. S., Lee, K., Watson, J. B., Lobo, L. G., Stoppelmoor, A. G., & Oyibo, S. E. (2019) · Journal of Medical Internet Research · doi:10.2196/13252 · Cited by 120
“85 indicating strong agreement on which articles should be included in the group for analysis [14-16].”
BackgroundA Comparison of Reliability Coefficients for Ordinal Rating Scales
de Raadt, A., Warrens, M. J., Bosker, R., & Kiers, H. A. L. (2021) · Journal of Classification · doi:10.1007/s00357-021-09386-5 · Cited by 146
“Multi-rater extensions of unweighted kappa are presented in Light (1971), Hubert (1977), Conger (1980), and Davies and Fleiss (1982).”
BackgroundHattori, M., & Oaksford, M. (2007) · Cognitive Science · doi:10.1080/03640210701530755 · Cited by 116
“(8) This index has been used as a measure of " response agreement " in psychology (Cohen, 1960, 1968; Fleiss, 1971; Light, 1971), " observer agreement " in clinical medicine (Landis & Koch, 1977), and the " accuracy " of weather forecasting (called the Heidke skill score) especially in America…”
MethodCareer advancement and family balance strategies of executive women
Ezzedeen, S. R., & Ritchey, K. G. (2009) · Gender in Management An International Journal · doi:10.1108/17542410910980388 · Cited by 109
Tsuda, H., Akiyama, F., Kurosumi, M., Sakamoto, G., Watanabe, T., & Section, J. N. S. A. S. O. B. C. (.-B. P. (1998) · Japanese Journal of Clinical Oncology · doi:10.1093/jjco/28.8.486 · Cited by 105
Predicting Sargassum blooms in the Caribbean Sea from MODIS observations
Wang, M., & Hu, C. (2017) · Geophysical Research Letters · doi:10.1002/2017gl072932 · Cited by 111
“Conditional kappa, which can test the individual category agreement [ Coleman , 1966; Light , 1971], was also calculated to help interpret the prediction accuracy.”
MethodStein, J. M., Lintel‐Höping, N., Hammächer, C., Kašaj, A., Tamm, M., & Hanisch, O. (2013) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.12169 · Cited by 104
“Interexaminer reliability was calculated using the method of Light (1971) and resulted in an average kappa of 0.”
MethodEffects of deficient reporting on meta-analysis: A conceptual framework and reanalysis.
Orwin, R. G., & Cordray, D. S. (1985) · Psychological Bulletin · doi:10.1037/0033-2909.97.1.134 · Cited by 102
The role of patient navigators in ambulatory care: overview of systematic reviews
Budde, H., Williams, G. A., Winkelmann, J., Pfirter, L., & Maier, C. B. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-07140-6 · Cited by 111
Watts, F. M., & Finkenstaedt-Quinn, S. A. (2021) · Chemistry Education Research and Practice · doi:10.1039/d1rp00007a · Cited by 115
“Light’s kappa allows for multiple coders, while fuzzy kappa allows for multiple codes to be applied to a single unit of analysis (Light, 1971; Kirilenko and Stepchenkova, 2016).”
An Inductive Model of Collaboration From the Stakeholder’s Perspective
Butterfield, K. D., Reed, R., & Lemak, D. J. (2004) · Business & Society · doi:10.1177/0007650304265956 · Cited by 107
“Interrater agreement was calculated using the relatively conservative P statistic (e.g., Light, 1971).”
MethodFilippou, G., Scirè, C. A., DAMJANOV, N. S., Adinolfi, A., Carrara, G., Picerno, V., Toscano, C., Bruyn, G. A. W., D’Agostino, M. A., Sedie, A. D., et al. (2017) · The Journal of Rheumatology · doi:10.3899/jrheum.161057 · Cited by 98
Examiner alignment and assessment in clinical periodontal research
Hefti, A. F., & Preshaw, P. M. (2012) · Periodontology 2000 · doi:10.1111/j.1600-0757.2011.00436.x · Cited by 102
Replication of Special Education Research
Makel, M. C., Plucker, J. A., Freeman, J., Lombardi, A. R., Simonsen, B. M., & Coyne, M. D. (2016) · Remedial and Special Education · doi:10.1177/0741932516646083 · Cited by 95
“Inter-rater reliability was calculated by taking the average kappa of all paired-combinations of raters (see Light, 1971).”
MethodWebb, J. B., Thomas, E. V., Bonar, A. S., Davies, A., & Etzel, L. (2017) · Body Image · doi:10.1016/j.bodyim.2017.05.003 · Cited by 118
Schizophrenics With Tardive Dyskinesia
Wegner, J. T. (1985) · Archives of General Psychiatry · doi:10.1001/archpsyc.1985.01790320028004 · Cited by 97
Reliability of a Qualitative Video Analysis for Running
Pipkin, A., Kotecki, K., Hetzel, S., & Heiderscheit, B. C. (2016) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2016.6280 · Cited by 96
“Inter-rater reliability was assessed with Kw from 133 Light’s method of Kw for more than 2 raters.(17) 134”
MethodFilippou, G., Scirè, C. A., Adinolfi, A., DAMJANOV, N. S., Carrara, G., Bruyn, G. A. W., Cazenave, T., D’Agostino, M. A., Sedie, A. D., Sabatino, V. D., et al. (2018) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2017-212542 · Cited by 96
Calibration of remotely sensed proportion or area estimates for misclassification error
Czaplewski, R. L., & Catts, G. P. (1992) · Remote Sensing of Environment · doi:10.1016/0034-4257(92)90138-a · Cited by 95
Khawaja, M. Z., Asrress, K., Haran, H., Arri, S., Nadra, I. J., Bolter, K., Wilson, K., Clack, L., Hancock, J., Young, C. P., et al. (2015) · EuroIntervention · doi:10.4244/eijy14m05_09 · Cited by 94
Comparing the Joint Agreement of Several Raters with Another Rater
Williams, G. W. (1976) · Biometrics · doi:10.2307/2529750 · Cited by 94
Bangdiwala, S. I., & Shankar, V. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-97 · Cited by 98
Telemedicine and health policy: A systematic review
Kruse, C. S., Williams, K., Bohls, J., & Shamsi, W. (2021) · Health Policy and Technology · doi:10.1016/j.hlpt.2020.10.006 · Cited by 88
Modest, D. P., Denecke, T., Pratschke, J., Ricard, I., Lang, H., Bemelmans, M. H. A., Becker, T., Rentsch, M., Seehofer, D., Bruns, C. J., et al. (2017) · European Journal of Cancer · doi:10.1016/j.ejca.2017.10.028 · Cited by 91
Metadata
- Authors
- Light, R. J.
- Affiliations
- Howard University (US)
- Corresponding author
- Light, Richard J.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 365-377
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1989492805
- Cited by
- 747
- References
- 17
- Citations by year
- 2026: 22 · 2025: 26 · 2024: 19 · 2023: 43 · 2022: 24 · 2021: 46 · 2020: 43 · 2019: 38 · 2018: 39 · 2017: 36 · 2016: 46 · 2015: 31 · 2014: 23 · 2013: 15 · 2012: 11
- FWCI
- 8.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- interobserver agreement, qualitative data, categorical data, agreement measures, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
