The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Interrater reliability and agreement of subjective judgments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 7×
“Fleiss (1971) provides a computational example.”
BackgroundMethodInfluentialFleiss, J. L., & Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300309 · Cited by 3,354
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,299
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. H. (1979) · Springer Series in Statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,244
Boneau, C. A., & Hays, W. L. (1966) · The American Journal of Psychology · doi:10.2307/1420739 · Cited by 1,881
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Estimation of the reliability of ratings.
Ebel, R. L. (1951) · Psychometrika · doi:10.1007/bf02288803 · Cited by 976
Cited by
Is the CVI an acceptable indicator of content validity? Appraisal and recommendations
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Research in Nursing & Health · doi:10.1002/nur.20199 · Cited by 5,273
“Topf (1986), for example, cited a study in which alternative ways of computing inter-rater agreement were explained to college students, who were then asked to compute various indexes with some data.”
BackgroundMethodInfluentialContent Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
rwg: an assessment of within-group interrater agreement
James, L. R., Demaree, R. G., & Wolf, G. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.2.306 · Cited by 2,221
Becker, T. E. (2005) · Organizational Research Methods · doi:10.1177/1094428105278021 · Cited by 2,219
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,570
Simpson, J. A., Rholes, W. S., & Nelligan, J. S. (1992) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.62.3.434 · Cited by 1,321
Social perception and interpersonal behavior: On the self-fulfilling nature of social stereotypes.
Snyder, M., Tanke, E. D., & Berscheid, E. (1977) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.35.9.656 · Cited by 1,310
Perceived Control and the Effects of Crowding and Consumer Choice on the Service Experience
Hui, M. K., & Bateson, J. E. G. (1991) · Journal of Consumer Research · doi:10.1086/209250 · Cited by 1,225
Whither scenic beauty? Visual landscape quality assessment in the 21st century
Daniel, T. C. (2001) · Landscape and Urban Planning · doi:10.1016/s0169-2046(01)00141-4 · Cited by 982
Interrater agreement and interrater reliability: Key concepts, approaches, and applications
Gisev, N., Bell, J. S., & Chen, T. F. (2013) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2012.04.004 · Cited by 924
Dunn, J., Brown, J. R., Slomkowski, C. L., Tesla, C., & Youngblade, L. M. (1991) · Child Development · doi:10.1111/j.1467-8624.1991.tb01610.x · Cited by 884
A prospective longitudinal study of attachment disorganization/disorientation
Carlson, E. A. (1998) · Child Development · doi:10.2307/1132365 · Cited by 819
A Prospective Longitudinal Study of Attachment Disorganization/Disorientation
Carlson, E. A. (1998) · Child Development · doi:10.1111/j.1467-8624.1998.tb06163.x · Cited by 819
Two Quantitative Approaches for Estimating Content Validity
Wynd, C. A., Schmidt, B., & Schaefer, M. A. (2003) · Western Journal of Nursing Research · doi:10.1177/0193945903252998 · Cited by 794
The effects of interpersonal trust on work group performance.
Dirks, K. T. (1999) · Journal of Applied Psychology · doi:10.1037//0021-9010.84.3.445 · Cited by 793
RESEARCH ON EMPLOYEE CREATIVITY: A CRITICAL REVIEW AND DIRECTIONS FOR FUTURE RESEARCH
Zhou, J., & Shalley, C. E. (2004) · Research in Personnel and Human Resources Management · doi:10.1016/s0742-7301(03)22004-1 · Cited by 770
A Healthy Divide: Subgroups as a Stimulus for Team Learning Behavior
Gibson, C., & Vermeulen, F. (2003) · Administrative Science Quarterly · doi:10.2307/3556657 · Cited by 762
Perceptual Evaluation of Voice Quality: Review, Tutorial, and a Framework for Future Research
Kreiman, J., Gerratt, B. R., Kempster, G. B., Erman, A., & Berke, G. S. (1993) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jshr.3601.21 · Cited by 737
Recalculation of the Critical Values for Lawshe’s Content Validity Ratio
Wilson, F. R., Pan, W., & Schumsky, D. A. (2012) · Measurement and Evaluation in Counseling and Development · doi:10.1177/0748175612440286 · Cited by 701
Is Chávez Populist?: Measuring Populist Discourse in Comparative Perspective
Hawkins, K. A. (2009) · Comparative Political Studies · doi:10.1177/0010414009331721 · Cited by 662
Infant–mother attachment: Factors related to its development and changes over time
Egeland, B., & Farber, E. A. (1984) · Child Development · doi:10.2307/1130127 · Cited by 652
Infant-Mother Attachment: Factors Related to Its Development and Changes over Time
Egeland, B. R., & Farber, E. A. (1984) · Child Development · doi:10.1111/j.1467-8624.1984.tb03813.x · Cited by 652
Zhou, J. (1998) · Journal of Applied Psychology · doi:10.1037//0021-9010.83.2.261 · Cited by 645
Straus, S. G., & McGrath, J. E. (1994) · Journal of Applied Psychology · doi:10.1037//0021-9010.79.1.87 · Cited by 638
Implicit leadership theories: Content, structure, and generalizability
Offermann, L. R., Kennedy, J. K., & Wirtz, P. W. (1994) · The Leadership Quarterly · doi:10.1016/1048-9843(94)90005-1 · Cited by 633
Gibson, C. B. (1999) · Academy of Management Journal · doi:10.2307/257089 · Cited by 535
Reflective judgment: Concepts of justification and their relationship to age and education
Kitchener, K. S., & King, P. M. (1981) · Journal of Applied Developmental Psychology · doi:10.1016/0193-3973(81)90032-0 · Cited by 526
A Meta-Analysis of Cohesion and Performance: Effects of Level of Analysis and Task Interdependence
Gully, S. M., Devine, D. J., & Whitney, D. J. (1995) · Small Group Research · doi:10.1177/1046496495264003 · Cited by 522
A REVIEW OF STRUCTURE IN THE SELECTION INTERVIEW
CAMPION, M. A., PALMER, D. K., & CAMPION, J. E. (1997) · Personnel Psychology · doi:10.1111/j.1744-6570.1997.tb00709.x · Cited by 519
A disagreement about within-group agreement: disentangling issues of consistency versus consensus
Kozlowski, S. W., & Hattrup, K. (1992) · Journal of Applied Psychology · doi:10.1037/0021-9010.77.2.161 · Cited by 514
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496
The Impact of Hospital Nursing Characteristics on 30-Day Mortality
Estabrooks, C. A., Midodzi, W. K., Cummings, G. G., Ricker, K. L., & Giovannetti, P. (2005) · Nursing Research · doi:10.1097/00006199-200503000-00002 · Cited by 485
Infant preferences for attractive faces: Rudiments of a stereotype?
Langlois, J. H., Roggman, L. A., Casey, R. J., & Ritter, J. M. (1987) · Developmental Psychology · doi:10.1037//0012-1649.23.3.363 · Cited by 479
The SAGE Dictionary of Statistics
Cramer, D., & Howitt, D. (2004) · SAGE Publications Ltd eBooks · doi:10.4135/9780857020123 · Cited by 475
On Average Deviation Indices for Estimating Interrater Agreement
Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999) · Organizational Research Methods · doi:10.1177/109442819921004 · Cited by 462
Assessing a Multidimensional Measure of Radical Technological Innovation
Green, S. G., Gavin, M. B., & Aiman-Smith, L. (1995) · IEEE Transactions on Engineering Management · doi:10.1109/17.403738 · Cited by 457
Mitchell, S. K. (1979) · Psychological Bulletin · doi:10.1037//0033-2909.86.2.376 · Cited by 445
EFFECTS OF REALISTIC JOB PREVIEWS ON MULTIPLE ORGANIZATIONAL OUTCOMES: A META-ANALYSIS.
Phillips, J. M. (1998) · Academy of Management Journal · doi:10.2307/256964 · Cited by 431
The health benefits of writing about intensely positive experiences
Burton, C. M., & King, L. A. (2004) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00058-8 · Cited by 411
Interrater Reliability and Agreement
Tinsley, H. E. A., & Weiss, D. J. (2000) · Handbook of Applied Multivariate Statistics and Mathematical Modeling · doi:10.1016/b978-012691360-6/50005-7 · Cited by 394
Self-citationObserver-Based Measurement of Facial Expression with the Facial Action Coding System
Cohn, J. F., Ambadar, Z., & Ekman, P. (2007) · Handbook of Emotion Elicitation and Assessment · doi:10.1093/oso/9780195169157.003.0014 · Cited by 393
The effects of psychological therapies under clinically representative conditions: a meta-analysis
Shadish, W. R., Navarro, A. M., Matt, G. E., & Phillips, G. (2000) · Psychological Bulletin · doi:10.1037/0033-2909.126.4.512 · Cited by 381
After-Event Reviews: Drawing Lessons From Successful and Failed Experience.
Ellis, S., & Davidi, I. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.5.857 · Cited by 374
EDWARDS, J. R. (1993) · Personnel Psychology · doi:10.1111/j.1744-6570.1993.tb00889.x · Cited by 369
Perceptions of organizational politics: Theory and research directions
Ferris, G. R., Adams, G., Kolodinsky, R. W., Hochwarter, W. A., & Ammeter, A. P. (2004) · Research in Multi Level Issues · doi:10.1016/s1475-9144(02)01034-2 · Cited by 367
Rating reliability and representation validity in scenic landscape assessments
Palmer, J. F., & Hoffman, R. E. (2001) · Landscape and Urban Planning · doi:10.1016/s0169-2046(01)00133-5 · Cited by 367
A meta‐analysis of bibliotherapy studies
Marrs, R. W. (1995) · American Journal of Community Psychology · doi:10.1007/bf02507018 · Cited by 366
Biemann, T., Cole, M. S., & Voelpel, S. (2012) · The Leadership Quarterly · doi:10.1016/j.leaqua.2011.11.006 · Cited by 361
Brown, J. R., Donelan‐McCall, N., & Dunn, J. (1996) · Child Development · doi:10.1111/j.1467-8624.1996.tb01767.x · Cited by 347
Helpful and nonhelpful events in brief counseling interviews: An empirical taxonomy.
Elliott, R. (1985) · Journal of Counseling Psychology · doi:10.1037//0022-0167.32.3.307 · Cited by 338
Continuities in Emotion Understanding from Three to Six Years
Brown, J. R., & Dunn, J. (1996) · Child Development · doi:10.1111/j.1467-8624.1996.tb01764.x · Cited by 321
Environmental Relationships of the Internal Political Economy of Marketing Channels
Dwyer, F. R., & Welsh, M. A. (1985) · Journal of Marketing Research · doi:10.1177/002224378502200405 · Cited by 320
Smidt, N., van der Windt, D. A., Assendelft, W. J., Mourits, A. J., Devillé, W. L., de Winter, A. F., & Bouter, L. M. (2002) · Archives of Physical Medicine and Rehabilitation · doi:10.1053/apmr.2002.33728 · Cited by 304
Raby, K. L., Roisman, G. I., Fraley, R. C., & Simpson, J. A. (2015) · Child Development · doi:10.1111/cdev.12325 · Cited by 299
STRUCTURED INTERVIEWING: RAISING THE PSYCHOMETRIC PROPERTIES OF THE EMPLOYMENT INTERVIEW
CAMPION, M. A., PURSELL, E. D., & BROWN, B. K. (1988) · Personnel Psychology · doi:10.1111/j.1744-6570.1988.tb00630.x · Cited by 296
An examination of service-related antecedents to retail store performance
Borucki, C. C., & Burke, M. J. (1999) · Journal of Organizational Behavior · doi:10.1002/(sici)1099-1379(199911)20:6<943::aid-job976>3.0.co;2-9 · Cited by 294
What Drives Key Informant Accuracy?
Homburg, C., Klarmann, M., Reimann, M., & Schilke, O. (2012) · Journal of Marketing Research · doi:10.1509/jmr.09.0174 · Cited by 276
Effects of marital conflict on subsequent triadic family interactions and parenting.
Kitzmann, K. M. (2000) · Developmental Psychology · doi:10.1037//0012-1649.36.1.3 · Cited by 272
Adult Attachment, Parent Emotion, and Observed Parenting Behavior: Mediator and Moderator Models
Adam, E. K., Gunnar, M. R., & Tanaka, A. (2004) · Child Development · doi:10.1111/j.1467-8624.2004.00657.x · Cited by 271
JOB SEARCH ACTIVITIES: AN EXAMINATION OF CHANGES OVER TIME
BARBER, A. E., DALY, C. L., GIANNANTONIO, C. M., & PHILLIPS, J. M. (1994) · Personnel Psychology · doi:10.1111/j.1744-6570.1994.tb01574.x · Cited by 269
Infant attractiveness predicts maternal behaviors and attitudes.
Langlois, J. H., Ritter, J. M., Casey, R. J., & Sawin, D. B. (1995) · Developmental Psychology · doi:10.1037//0012-1649.31.3.464 · Cited by 263
Nadort, M., Arntz, A., Smit, J. H., Giesen-Bloo, J., Eikelenboom, M., Spinhoven, P., van Asselt, T., Wensing, M., & van Dyck, R. (2009) · Behaviour Research and Therapy · doi:10.1016/j.brat.2009.07.013 · Cited by 259
A Revised Index of Interrater Agreement for Multi-Item Ratings of a Single Target
Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999) · Applied Psychological Measurement · doi:10.1177/01466219922031257 · Cited by 255
Interdisciplinary approaches to job design: A constructive replication with extensions.
Campion, M. A. (1988) · Journal of Applied Psychology · doi:10.1037/0021-9010.73.3.467 · Cited by 254
A Developmental Investigation of Inattentiveness and Hyperactivity
Carlson, E. A., Jacobvitz, D., & Sroufe, L. A. (1995) · Child Development · doi:10.2307/1131189 · Cited by 195
High Agreement and High Prevalence: The Paradox of Cohen’s Kappa
Zec, S., Soriani, N., Comoretto, R., & Baldi, I. (2017) · The Open Nursing Journal · doi:10.2174/1874434601711010211 · Cited by 169
Similarity relations in recognition
Tulving, E. (1981) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(81)90129-8 · Cited by 165
Minority-Related Stressors and Coping Processes Among African American College Students
Greer, T. M., & Chwalisz, K. (2007) · Journal of College Student Development · doi:10.1353/csd.2007.0037 · Cited by 158
Barber, J., & Critis-Christoph, P. (1996) · Psychotherapy Research · doi:10.1080/10503309612331331608 · Cited by 133
Assumptions behind Intercoder Reliability Indices
Zhao, X., Liu, J. S., & Deng, K. (2013) · Annals of the International Communication Association · doi:10.1080/23808985.2013.11679142 · Cited by 113
Chen, C.-H., Nguyen, B., Klaus, P. “., & Wu, M.-S. (2015) · Journal of Travel & Tourism Marketing · doi:10.1080/10548408.2014.956165 · Cited by 107
Munyewende, P. O., Levin, J., & Rispel, L. C. (2016) · Global Health Action · doi:10.3402/gha.v9.32486 · Cited by 46
“To determine the agreement between each of the three possible pairs of type of respondents, we used the Tinsley and Weiss index (40).”
MethodCorrelates of perceived coercion during psychiatric hospital admission
Cascardi, M., & Poythress, N. G. (1997) · International Journal of Law and Psychiatry · doi:10.1016/s0160-2527(97)00014-9 · Cited by 39
Gullion, D. S., Tschann, J. M., Adamson, E. T., & Coates, T. J. (1988) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.4750080402 · Cited by 31
Bhattacharyya, S. S., Kulkarni, O., & Mishra, A. (2023) · Benchmarking: An International Journal · doi:10.1108/bij-03-2022-0163 · Cited by 23
Autism and epistemology III: child development, behavioral stability, and reliability of measurement
Fisch, G. S. (2012) · American Journal of Medical Genetics Part A · doi:10.1002/ajmg.a.35269 · Cited by 13
Hsu, P.-L., Reis, G., & Monarrez, A. (2017) · Canadian Journal of Science, Mathematics and Technology Education · doi:10.1080/14926156.2017.1343517 · Cited by 13
The Connections Between STEM Learning Outcomes in a STEM Hackathon
Chen, Y., & Teo, T. W. (2026) · Research in Science Education · doi:10.1007/s11165-026-10369-y
Metadata
- Authors
- Tinsley, H. E., & Weiss, D. J.
- Venue
- Journal of Counseling Psychology
- Published
- 1975-07
- Volume
- 22
- Issue
- 4
- Pages
- 358-376
- ISSN
- 0022-0167 (ISSN-L)
- 1939-2168
- OpenAlex
- W2090280719
- Cited by
- 852
- References
- 29
- FWCI
- 24.09
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.