Cited by 750Reliability and Agreement in MeasurementPsychometric Methodologies and TestingSurvey Methodology and Nonresponse

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Measures of response agreement for qualitative data: Some generalizations and alternatives.

Light, Richard J.·1971·Psychological Bulletin

DOI 10.1037/h0031643

Open this paper in CoChat — free account

Integrity signals

Registry

Crossref · 10.1037/h0031643

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Rank Correlation Methods.
Kendall, M. G. (1957) · Biometrika · doi:10.2307/2333282 · Cited by 6,442
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. (1979) · Springer series in statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,255
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,536
Estimation of the reliability of ratings.
Ebel, R. L. (1951) · Psychometrika · doi:10.1007/bf02288803 · Cited by 976
Unitizing and categorizing problems in coding qualitative data
Guetzkow, H. S. (1950) · Journal of Clinical Psychology · doi:10.1002/1097-4679(195001)6:1 · Cited by 436
Intraclass Correlation and the Analysis of Variance
Trese, T., & Haggard, E. A. (1958) · The American Catholic Sociological Review · doi:10.2307/3708305 · Cited by 401

Flagged first, then by citations

Show all 17 references

Sorted by citations

Items 601–700 of 750

Measures of Association
Armistead, T. W. (2017) · Wiley eBooks · doi:10.1002/9781118445112.stat03899.pub2 · Cited by 2
ASPECTS, The Mismeasure of Stroke: A Metrological InvestigationPreprint
Suss, R. A. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/c4tkp · Cited by 2
Diagnosis and treatment of subungual melanoma
Oh, J. S., & Kim, B. J. (2023) · Archives of Hand and Microsurgery · doi:10.12790/ahm.22.0072 · Cited by 1
On the L1-acquisition of the pragmatics of discourse like
Schweinberger, M. (2023) · Functions of Language · doi:10.1075/fol.20025.sch · Cited by 1
Measures of Nominal Association II
Berry, K. J., & Johnston, J. E. (2023) · Springer eBooks · doi:10.1007/978-3-031-41896-9_12 · Cited by 1
Preparation and Use of Physical Agent Modalities Among U.S. Occupational Therapists: A Mixed-Methods, Exploratory-Descriptive Study
Gentry, K., LeSage, T., Hebble, D., Adams, L., & Watts, A. (2024) · The Open Journal of Occupational Therapy · doi:10.15453/2168-6408.2247 · Cited by 1
ORIGINALBEITRÄGE. Einschätzung der verbalen Beziehungsqualität – die Sprachskala des EBQ-Instrumentes / Einschätzung der verbalen Beziehungsqualität – die Sprachskala des EBQ-Instrumentes
Jordan, A.-K., Esterbauer, E., & Schumacher, K. (2024) · Musiktherapeutische Umschau · doi:10.13109/muum.2024.45.3.273 · Cited by 1
Examining Suicide Crisis Syndrome as an Indirect Indicator of Suicide Risk in the Military Suicide Research Consortium’s Common Data Elements
Rogers, M. L., Richards, J. A., Peterkin, D., & GALYNKER, I. I. (2024) · Archives of Suicide Research · doi:10.1080/13811118.2024.2434745 · Cited by 1
Accounting for Patient Characteristics in a Model‐Based Kappa of Agreement Between Two Experts' Ordinal Ratings
Nelson, K. P., & Zhou, T. (2025) · Statistics in Medicine · doi:10.1002/sim.70141 · Cited by 1
“…two experts include the intra-class correlation coefficient defined by Bloch and Kraemer [17], and modified Cohen’s kappa-like statistics by Light [18] and Conger [19]; however, in practice, these measures are vulnerable to the same issuesasCohen’skappaandsimilarlyareunabletoaccommodate patient…”
Teachers' Framework in Assessing Quality of MOOCs
Burcu, E., Viswanathan, N., & Masood, A. (2025) · IEEE Digital Education and MOOCS Conference (DEMOcon) · doi:10.1109/democon65705.2025.11282822 · Cited by 1
“A variant of the Cohen’s kappa statistic, Light’s kappa [46] has been computed for the data collected, since the experiment has been designed to be fully-crossed [47].”
DaRA Dataset: Combining Wearable Sensors, Location Tracking, and Process Knowledge for Enhanced Human Activity and Human Context Recognition in Warehousing
Niemann, F., Rueda, F. M., Kfari, M. K. A., Nair, N. R., Schauten, D., Kretschmer, V., Lüdtke, S., & Kirchheim, A. (2026) · Sensors · doi:10.3390/s26020739 · Cited by 1
“To assess annotation quality as well as the revised datasets, we used Cohen’s κ [114] for exactly two annotators and Light’s κ [115] for more than two annotators, the latter being the mean of all pairwise Cohen’s κ values.”
Early Target Object Prediction in Action Observation
Fanghella, M., D'Asaro, F. A., Quarona, D., Barchiesi, G., Rabuffetti, M., Ferrarin, M., & Sinigaglia, C. (2026) · Quarterly Journal of Experimental Psychology · doi:10.1177/17470218261442536 · Cited by 2
Are assessments in higher education fair for all? Investigating the association between assessment load and type and students’ academic performance: a longitudinal large-scale data set analysis
Tan, Y., Hart, B., & Pickering, J. (2026) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2026.2661360 · Cited by 1
Reducing misclassification of monoclonal gammopathy in canine leishmaniosis: Added value of immunofixation to serum protein electrophoresis
Villanueva‐Saz, S., Fernández, A., Marteles-Aragüés, D., Garza‐Moreno, L., Martínez‐Lostao, L., Harris, R. A., & Moore, A. R. (2026) · Veterinary Parasitology · doi:10.1016/j.vetpar.2026.110843 · Cited by 1
A comparison of multi-way similarity coefficients for binary sequences
Warrens, M. J. (2013) · Leiden Repository (Leiden University) · Cited by 1
Nominal-Level Variables, II
Berry, K. J., Johnston, J. E., & Mielke, P. W. (2018) · Springer eBooks · doi:10.1007/978-3-319-98926-6_4 · Cited by 1
Automatic Assessment of Source Code Highlighting Tasks
Krämer, M., Barkmin, M., Brinda, T., & Tobinski, D. A. (2018) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3279720.3279729 · Cited by 1
“Fleiss et al. [13, p. 618] suggest different solutions to the problem, such as Light’s G statistic [19], which compares the rating of several raters with a “standard”.”
Background
Adaptação Transcultural da The Stigma Scale para o Português Brasileiro: Estudo Piloto
Cardoso, N. D. O., Sanvicente‐Vieira, B., Machado, W. D. L., & Argimon, I. I. D. L. (2020) · Revista Avaliação Psicológica · doi:10.15689/ap.2020.1903.17467.04 · Cited by 1
A simulation study on the association of HRO communication patterns and surgical team performance
Baty, A., Matis, T. I., & Griswold, J. (2022) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzac028 · Cited by 1
Evaluating the impact of a digitally implemented subjective standard on professional rugby union player management decision-making
Ranaweera, J., Weaving, D., Zanin, M., & Roe, G. A. B. (2023) · International Journal of Sports Science & Coaching · doi:10.1177/17479541231188065 · Cited by 1
“Since there were more than two raters and as the study was fully crossed, we used Light ’ s Kappa ( κ ) 21 to evaluate the inter-rater agreement among the ﬁ ve members tasked with rating the events at each of the four time points ( ˆ κ ).”
Method
Symbolic reality bites 2: contemporary representations of women and people of color in top 100 films
Marr, E., & Luchies, L. B. (2023) · Sociological Spectrum · doi:10.1080/02732173.2023.2254431 · Cited by 1
“Cohen’s (1960) kappa was computed for each coder pair, then averaged to provide a single index of IRR for each variable (Hallgren 2012; Light 1971).”
Method
Features of Сohen’s kappa coefficient as a measure of expert opinion agreement
Ybytayeva, G., Mamyrbayev, O. J., Khairova, N., Mukhsina, K., & Жумажанов, Б. (2023) · Bulletin of the National Engineering Academy of the Republic of Kazakhstan · doi:10.47533/2023.1606-146x.26 · Cited by 1
The Use of Facial Recognition in Sociological Research: A Comparison of ClarifAI and Kairos Classifications to Hand-Coded Images
Peoples, C. E., Knudsen, P., & Fuentes, M. (2024) · Socius Sociological Research for a Dynamic World · doi:10.1177/23780231241259659 · Cited by 1
“To find the interrater reliability of our coding scheme, we computed Cohen’s kappa for each coder-pair and report the average score for each (Light 1971).”
Diagnostic performance and reliability of robotic ultrasonography and artificial intelligence-driven synovitis assessment in rheumatoid arthritis: results from the Controlled ARTHUR Trial
Ammitzbøll‐Danielsen, M., Østergaard, M., Tamm, L., & Terslev, L. (2025) · RMD Open · doi:10.1136/rmdopen-2025-006099 · Cited by 1
Randomized Designs: Nominal Data
Berry, K., Mielke, P., & Johnston, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-28770-6_7 · Cited by 1
An Objective Method of Identifying Teachworthy Multi-word Units for Second Language Learners
Rogers, J. (2017) · Conference of the European Society of Phraseology (EUROPHRAS) · doi:10.26615/978-2-9701095-2-5_020 · Cited by 1
“For each of these indices, IRR was computed separately for the three coder pairs then averaged to provide a single IRR index (Light, 1971).”
Method
Reliability in twelve ergonomists’ three-category risk ratings in ten video recorded work tasks.
Forsman, M., Eliasson, K., Rhén, I.-M., Balliu, N., Nyman, T., Lindberg, P., Kjellberg, K., & Palm, P. (2015) · Cited by 1
Measurement of Rater Consistency by Chance-Corrected Agreement Coefficients
Xie, Z., Gadepalli, C., & Cheetham, B. M. G. (2018) · UKSim International Conference on Computer Modelling and Simulation (UKSim) · doi:10.1109/uksim.2018.00015 · Cited by 1
“An alternative generalisation by Light [8] is also identical for two raters but slightly different for more.”
Background
Emotions in Computer Vision Service Q&A
Cummaudo, A., Graetsch, U. M., Curumsing, M. K., Vasa, R., Barnett, S., & Grundy, J. (2021) · IEEE/ACM International Workshop on Emotion Awareness in Software Engineering (SEmotion) · doi:10.1109/semotion52567.2021.00011 · Cited by 1
“Based on suggestions in [14], we calculated Cohen’s Kappa ( C κ ) [15] to measure the average inter-rater agreement between pairs of raters, and then Light’s Kappa ( L κ ) [16] to measure the overall agreement amongst the three raters.”
Method
How to use adjusted degree of distinguishability and inter-rater reliability simultaneously?
Yılmaz, A. E. (2021) · Cumhuriyet Science Journal · doi:10.17776/csj.898192 · Cited by 1
“As the overall agreement coefficient, Light's weighted kappa [16] is used.”
Method
Nominal Scale Agreement
Popping, R. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1793.pub2 · Cited by 1
Measuring Democracy Through Expert Judgments: Lessons from Argentina's Survey of Experts on Provincial PoliticsPreprint
Gervasoni, C. (2010) · SSRN Electronic Journal · Cited by 1
Kappa Coefficient of Agreement
Kvålseth, T. O. (2011) · International Encyclopedia of Statistical Science · doi:10.1007/978-3-642-04898-2_323 · Cited by 1
Reprodutibilidade de um teste microbiológico para estreptococos do grupo mutans
Pinelli, C., Loffredo, L. D. C. M., & Serra, M. C. (2000) · Brazilian Oral Research · doi:10.1590/s1517-74912000000100003 · Cited by 1
Interpersonal tactics and communicative strategies of Anglo-American and Mexican-American mildly mentally retarded and nonretarded students
Rueda, R. S., & Smith, D. C. (1983) · Applied Research in Mental Retardation · doi:10.1016/0270-3092(83)90007-3 · Cited by 1
Accuracy Assessment of Traffic Monitoring Devices Vehicle by Vehicle
Gillmann, R. (2006) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/0361198106194500108 · Cited by 1
Mimetic behavior of rats in flavor aversion learning
Pinnow, M., & Schneider, K. (1994) · Behavioural Processes · doi:10.1016/0376-6357(94)90033-7 · Cited by 1
Improving precision through modelling: an illustration with hierarchical kappa
Basu, S., Raychaudhuri, A., & Basu, A. (1995) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610919508813249 · Cited by 1
Characteristics of the Conditional Reasoning of Educable Retardates
Antonak, R. F., & Roberge, J. J. (1978) · American Educational Research Journal · doi:10.2307/1162645 · Cited by 1
Integrating technology into program assessment implementation and course design: "How can relational, problem-solving models help us reach our goals?"
Reece, G. A. (2003) · IEEE International Professional Communication Conference (ProComm) · doi:10.1109/ipcc.2002.1049101 · Cited by 1
Behavioral treatment for depression: An evaluation of therapeutic components
Turner, R. W., Ward, M. F., & Turner, D. J. (1979) · Journal of Clinical Psychology · doi:10.1002/1097-4679(197901)35:1 · Cited by 35
Measurement of interobserver agreement using a "standard": measure formulation and statistical inferences
Kvålseth, T. O. (2002) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1997.638290 · Cited by 1
Application of Bayesian Methods for Assessing Detection Accuracy in Remote Sensing
Shafii, B., & Price, W. J. (2001) · Cited by 1
COMPARING BINOMIAL BOOTSTRAP AND BAYESIAN ESTIMATION METHODS IN ASSESSING THE AGREEMENT BETWEEN CLASSIFIED IMAGES AND GROUND TRUTH DATA.
Shafii, B., & Price, W. J. (2001) · Conference on Applied Statistics in Agriculture · doi:10.4148/2475-7772.1217 · Cited by 1
“Some of the common measures include conditional kappa, a general index of agreement (Light, 1971) : the omissional error rate, measuring the proportion of pixels incorrectly omitted from a classification: o.”
Background
Psychologie Ergonomique de l'Optimisation des Processus Décisionnels
Raufaste, É. (2003) · HAL · Cited by 1
Identifying Coaching Abuses in Youth Sport: A Human Rights Approach for Policy Development
Cook, E. (2013) · doi:10.82465/5775 · Cited by 1
Reliability and validity of a four-level severity score based triage scalePreprint
Josse, C., Brodeur, S., & Callahan, J.-C. (2019) · doi:10.21203/rs.2.16729/v1 · Cited by 1
“For inter-rater reliability we used the methodology described by Light (9): a quadratic weighted Cohen’s Kappa was calculated between each possible pair of assessors (resulting in fifteen weighted Kappas), then an arithmetic mean of these fifteen Kappas was calculated.”
MethodInfluential
What are the Effects of Computerized Decision Support Systems on Practitioner Performance and Patient OutcomesPreprint
Kruse, C. S., & Ehrbar, N. (2019) · doi:10.21203/rs.2.16890/v1 · Cited by 1
“The kappa score produced was .98 showing almost complete agreement on all reviewed articles(13,14).”
Method
Stigma towards Wuhan people during the COVID-19 epidemic: An exploratory study based on social mediaPreprint
Di, Y., Li, A., Li, H., Wu, P., Yang, S., Zhu, M., Zhu, T., & Liu, X. (2020) · doi:10.31234/osf.io/cq4px · Cited by 1
PECTUS EXCAVATUM
de Loos, E. R. (2022) · doi:10.26481/dis.20220407el · Cited by 1
The Role of Patient Navigators in Ambulatory Care: Overview of Systematic ReviewsPreprint
Budde, H., Williams, G. A., Winkelmann, J., Pfirter, L., & Maier, C. B. (2021) · Research Square · doi:10.21203/rs.3.rs-506357/v1 · Cited by 1
Interrater Agreement
Banerjee, M. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess3164.pub2 · Cited by 1
Nominal Scale Agreement
Popping, R. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1793 · Cited by 1
Kappa Coefficient of Agreement
Kvålseth, T. O. (2025) · International Encyclopedia of Statistical Science · doi:10.1007/978-3-662-69359-9_311 · Cited by 1
Early Target Prediction in Action ObservationPreprint
Fanghella, M., D’Asaro, F. A., Quarona, D., Barchiesi, G., Rabuffetti, M., Ferrarin, M., & Sinigaglia, C. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.28.667200 · Cited by 1
Child hospitalization: the effects on well siblings and parents
Wallinga, C. R. (1984) · doi:10.31274/rtd-180813-6147 · Cited by 1
Quadratically Weighted Agreement Coefficients: Interpretations and Connections
van Oest, R., & Moss, J. (2026) · Psychometrika · doi:10.1017/psy.2026.10085
MATRIX KAPPA: A PROPOSAL FOR A CARD SORT STATISTIC FOR IS SURVEY INSTRUMENT DEVELOPMENT
Denford, J. S. (2010) · International Conference on Information Systems
Unterrichtbeobachtung im Sport
Piéron, M., Ospelt, R., Egger, K., Carlier, G., Andreani, A., Heinilä, L., & Rehbein, E. (1980) · Birkhäuser Basel eBooks · doi:10.1007/978-3-0348-6718-4_6
Reprodutibilidade de medidas antropometricas
Campos, J. Á. D. B., & Loffredo, L. D. C. M. (2005) · Alimentos e Nutrição
Relação entre ceceio e índice oclusal em crianças com fissura labiopalatina
Whitaker, M. E. (2009) · doi:10.11606/t.61.2009.tde-14102009-102315
Methodology for establishing a 'gold standard' (for a medical expert system)
Georgakis, D. C., & Georgakis, C. (2003) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.1992.244926
Three Indexes for Measuring Interanalyst Agreement in Content Analyses
Sjödahl, L. (1974) · Scandinavian Journal of Educational Research · doi:10.1080/0031383740180106
Reducing medication errors: perceived information needs of healthcare practitioners
Warner, B., & Gerrett, D. (2007) · International Journal of Pharmacy Practice · doi:10.1211/ijpp.15.3.0009
A matrix approach to data base exploration: Analysis of classifier results
Goin, J. E., & Fritz, S. L. (1983) · Pattern Recognition · doi:10.1016/0031-3203(83)90028-6
Practical suggestions for increasing the reliability of classroom observational research
Mangano, N. G., Willson, V. L., & Rupley, W. H. (1986) · Reading Research and Instruction · doi:10.1080/19388078609557877
Academic self-concept and teachers' perceptions of normal, mentally retarded, and learning disabled elementary students
Carroll, J. L., Friedrich, D. D., & Hund, J. (1984) · Psychology in the Schools · doi:10.1002/1520-6807(198407)21:3 · Cited by 24
Desempenho clínico da flurescência, imagens digitais ampliadas e ICDAS na detecção de cárie em superfícies oclusais
Presoto, C. D. (2013) · Americanae (AECID Library)
Les orientations de la recherche en éducation dans les constituantes de l’Université du Québec
Papillon, S., Rousseau, R., Tremblay, Y., & Potvin, P. (2009) · Revue des sciences de l éducation · doi:10.7202/900551ar
“La concordance des jugements portés par des juges ou le degré d'accord entre plusieurs juges a fait l'objet de nombreux écrits, dont les principaux sont ceux de Cohen (1960), Maxwell et Pilliner (1968), Fleiss (1971), Light (1971), Fleiss et Cohen (1973), Conger (1980) et Rae (1984).”
Background
The Consistency Degree in the Use of Translation Procedures: A Pilot Study
Chen, Y.-C. (2014) · DOAJ (DOAJ: Directory of Open Access Journals)
A Generalized Anova Model for Estimating the Reliability of Categorical Judgments.
Enger, J. M., & Whitney, D. R. (1975)
Fatores de Risco de Desordens Osteomusculares Presentes no Posto de Trabalho de Estudantes de Odontologia Risk Factors for Musculoskeletal Disorders at the Workplaces of Undergraduate Dental Students
Presoto, C. D., Corrocher, P. A., Duarte, A., & Campos, B. (2012)
Estudos sobre a relação entre disfunção temporomandibular e cefaleia primária. Avaliações populacional e clínica
Araraquara, F. D. O. D. (2009)
Supporting Elementary Education In-Service Teachers' Proficiency in Planning STEM-Centric Lessons
Bowers, S. W. (2015) · VTechWorks (Virginia Tech)
FACIAL REACTIONS IN A DESCRIPTIVE STUDY
Unzner, L., & Schneider, K. (1990)
Reprodutibilidade de um teste microbiolÛgico para estreptococos do grupo mutans Reproducibility of a simplified microbiological test for mutans streptococci
Pinelli, C., de Castro, L., Loffredo, M., & Serra, C. (2000)
Assessment of agreement in measuring orthodontic treatment need with the modified DHC
Baghban, A. A., Nouri, M., & Safavi, M. (2008) · Australasian Orthodontic Journal · doi:10.2478/aoj-2008-0003
Randomized Block Designs: Interval Data
Berry, K., Mielke, P., & Johnston, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-28770-6_9
Permutation P-values for Inter-rater Agreement Measures
Um, Y. (2015) · Journal of the Korea Society of Computer and Information · doi:10.9708/jksci.2015.20.12.169
Completely Randomized Data
Berry, K., Mielke, P., & Johnston, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-28770-6_2
Demystification of commonly used measurements in paediatrics
Bekhof, J. (2014) · University of Groningen research database (University of Groningen / Centre for Information Technology)
Randomized Block Designs: Ordinal Data
Berry, K., Mielke, P., & Johnston, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-28770-6_10
Regression Analysis of Interval Data
Berry, K., Mielke, P., & Johnston, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-28770-6_4
Randomized Designs: Interval Data
Berry, K., Mielke, P., & Johnston, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-28770-6_3
Randomized Block Data
Berry, K., Mielke, P., & Johnston, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-28770-6_8
Randomized Designs: Ordinal Data, II
Berry, K., Mielke, P., & Johnston, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-28770-6_6
Methods for Assessment of Interrater Reliability for Diagnosis and Intervention in Traditional Chinese Medicine Studies
Ferreira, A. D. S., & Oliveira, I. J. D. A. S. (2016) · Springer eBooks · doi:10.1007/978-981-10-2290-6_7
Confiabilidade de um formulário para diagnóstico da severidade da disfunção temporomandibular Reliability of a questionnaire for diagnosing the severity of temporomandibular disorder
Campos, J., Gonçalves, D., Camparis, C. M., & Jg, S. (2009)
Evaluating retinal blood vessels' abnormal tortuosity in digital image fundus
Abdalla, M. (2016) · Lincoln Repository (University of Lincoln) · doi:10.24385/lincoln.24325543
Validation of an Observation and Evaluation Instrument for the Supervision of Middle and Secondary Pre-Service Teachers
Bush, B. (2017) · doi:10.12794/metadc984136
A sequential test for assessing observed agreement between raters
Bersimis, S., Sachlas, A., & Chakraborti, S. (2017) · Biometrical Journal · doi:10.1002/bimj.201600239
“Extensions to the case of more than two raters were proposed in Fleiss (1971), Light (1971), Landis and Koch (1977a, 1977b), Davies and Fleiss (1982), and Kraemer (1980).”
Background
Modeling Association plus Agreement among Multi-Raters for Ordered Categories
Yılmaz, A. E. (2017) · Statistica · doi:10.6092/issn.1973-2201/7241
“For the multi-rater studies, Light (1971), Hubert (1977), and Berry et al. (2007) weighted kappa coefficients can be used.”
Method
Training Educators to Implement Mindfulness-Based Interventions: Evaluating the Effects of In-Service and Coaching on Intervention Fidelity.
Potter, L. M. (2018) · University of Minnesota Digital Conservancy (University of Minnesota)
Visual Emphasis of Lip Protrusion for Pronunciation Learning
Yu, S., Kondo, K., Nakamura, Y., Nakajima, T., Nanjo, H., & Dantsuji, M. (2018) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2017edp7411
“Arithmetic mean of Cohen’s kappa [17], [18] of every pairwise native evaluators on rounded vowel, unrounded vowel, consonant and tone were calculated to evaluate inter-rater reliability (Table 2).”
Method
Lack of photographic documentation undermines assessment of hidradenitis suppurativa phenotypes: reply from the author
Frew, J. W. (2019) · British Journal of Dermatology · doi:10.1111/bjd.17934
Examining Interviewer Effects on the Agricultural Labor Survey: A Mixed-Methods Approach
Biagas, D., Abayomi, E., Rodhouse, J., & Ridolfo, H. E. (2019) · Lincoln (University of Nebraska)
Development of Measurement tools for Leadership Derailment
Lee, J. H. (2017) · THE KOREAN JOURNAL OF COOPERATIVE STUDIES · doi:10.35412/kjcs.2017.35.1.005
PROTOCOL: Stress Management Training and Development Programs for Police Officers and Recruits
Patterson, G. T., & Chung, I. W. (2010) · Campbell Systematic Reviews · doi:10.1002/cl2.67
Quantification of surgical team communication patterns related to high reliability organization principles
Baty, A. (2017) · ThinkTech (Texas Tech University)
A Longitudinal Investigation of the Interactional Process Mechanisms of Leadership Emergence in Dyads
McCusker, M. E. (2018) · VTechWorks (Virginia Tech)

Authors: Light, R. J.
Affiliations: Howard University (US)
Corresponding author: Light, Richard J.
Venue: Psychological Bulletin
Publisher: American Psychological Association (APA)
Published: 1971-11
Volume: 76
Issue: 5
Pages: 365-377
ISSN: 0033-2909 (ISSN-L)
1939-1455
DOI: 10.1037/h0031643
OpenAlex: W1989492805
Cited by: 747
References: 17
Citations by year: 2026: 22 · 2025: 26 · 2024: 19 · 2023: 43 · 2022: 24 · 2021: 46 · 2020: 43 · 2019: 38 · 2018: 39 · 2017: 36 · 2016: 46 · 2015: 31 · 2014: 23 · 2013: 15 · 2012: 11
FWCI: 8.62
Type: journal article
Language: en
Open access: closed
Locations: Psychological Bulletin (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Keywords: interobserver agreement, qualitative data, categorical data, agreement measures, inter-rater reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Measures of response agreement for qualitative data: Some generalizations and alternatives.

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks