The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 1,401–1,500 of 23,883
Affective traits link to reliable neural markers of incentive anticipation
Wu, C. C., Samanez-Larkin, G. R., Katovich, K., & Knutson, B. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.08.055 · Cited by 176
Validation of a Tool to Measure and Promote Clinical Teamwork
Guise, J., Deering, S. H., Kanki, B. G., Osterweil, P., Li, H., Mori, M., & Lowe, N. K. (2008) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0b013e31816fdd0a · Cited by 176
New Method to Assess Scapular Upward Rotation in Subjects With Shoulder Pathology
Johnson, M. P., McClure, P. W., & Karduna, A. R. (2001) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2001.31.2.81 · Cited by 176
Bagwell, C. L., Bender, S. E., Andreassi, C. L., Kinoshita, T. L., Montarello, S. A., & Muller, J. G. (2005) · Journal of Social and Personal Relationships · doi:10.1177/0265407505050945 · Cited by 200
Some Things Are Better Left Unsaid: How Word of Mouth Influences the Storyteller
Moore, S. G. (2012) · Journal of Consumer Research · doi:10.1086/661891 · Cited by 178
Interrater reliability of sleep stage scoring: a meta-analysis
Lee, Y. J., Lee, J. Y., Cho, J. H., & Choi, J. H. (2021) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.9538 · Cited by 178
User-Guided Segmentation of Multi-modality Medical Imaging Datasets with ITK-SNAP
Yushkevich, P. A., Pashchinskiy, A., Oguz, I., Mohan, S., Schmitt, J. E., Stein, J. M., Zukić, D., Vicory, J., McCormick, M., Yushkevich, N., et al. (2019) · Neuroinformatics · doi:10.1007/s12021-018-9385-x · Cited by 175
Impaired sadness recognition is linked to social interaction deficit in autism
Boraston, Z., Blakemore, S., Chilvers, R., & Skuse, D. (2006) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2006.11.010 · Cited by 186
Kurtz, J. E., & Sherker, J. L. (2003) · Journal of Personality · doi:10.1111/1467-6494.t01-1-00005 · Cited by 175
Martiniello, M. (2009) · Educational Assessment · doi:10.1080/10627190903422906 · Cited by 175
Coppieters, M. W., Stappaerts, K. H., Janssens, K., & Jull, G. A. (2002) · Physiotherapy Research International · doi:10.1002/pri.251 · Cited by 175
Eells, T. D., Lombart, K. G., Kendjelic, E. M., Turner, L., & Lucas, C. P. (2005) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.73.4.579 · Cited by 175
Crafting Social Issues at Work
Sonenshein, S. B. (2006) · Academy of Management Journal · doi:10.5465/amj.2006.23478243 · Cited by 175
Robbins, M. S., Liddle, H. A., Turner, C. W., Dakof, G. A., Alexander, J. F., & Kogan, S. M. (2006) · Journal of Family Psychology · doi:10.1037/0893-3200.20.1.108 · Cited by 175
How Much Does National Culture Constrain Organizational Culture?
Gerhart, B. A. (2008) · Management and Organization Review · doi:10.1111/j.1740-8784.2008.00117.x · Cited by 182
Hughes, M. A., & Garrett, D. E. (1990) · Journal of Marketing Research · doi:10.1177/002224379002700206 · Cited by 177
Validation of a Spanish version of the Revised Fibromyalgia Impact Questionnaire (FIQR)
Salgueiro, M., García-Leiva, J. M., Ballesteros, J., Hidalgo, J., Molina, R., & Calandre, E. P. (2013) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-11-132 · Cited by 171
Effects of social influence on eating in couples, friends and strangers
Salvy, S., Jarrin, D. C., Paluch, R. A., Irfan, N., & Pliner, P. P. (2007) · Appetite · doi:10.1016/j.appet.2006.12.004 · Cited by 174
Semantic Similarity and Relatedness between Clinical Terms: An Experimental Study.
Pakhomov, S., McInnes, B. T., Adam, T. J., Liu, Y., Ted Pedersen, A., & Melton, G. B. (2010) · PubMed Central · Cited by 174
Assessment of Limb Muscle Strength in Critically Ill Patients
Vanpee, G., Hermans, G., Segers, J., & Gosselink, R. (2013) · Critical Care Medicine · doi:10.1097/ccm.0000000000000030 · Cited by 176
Muscle Fatigue during Dynamic Contractions Assessed by New Spectral Indices
Dimitrov, G. V., Arabadzhiev, T. I., Mileva, K. N., Bowtell, J. L., Crichton, N., & Dimitrova, N. A. (2006) · Medicine & Science in Sports & Exercise · doi:10.1249/01.mss.0000233794.31659.6d · Cited by 179
Validity of the Child Behaviour Checklist in a Norwegian sample
Nøvik, T. S. (1999) · European Child & Adolescent Psychiatry · doi:10.1007/s007870050098 · Cited by 174
Mulhern, B. J., Rowen, D., Brazier, J. E., Smith, S. C., Romeo, R., Tait, R., Watchurst, C., Chua, K.-C., Loftus, V., Young, T. A., et al. (2013) · Health Technology Assessment · doi:10.3310/hta17050 · Cited by 178
Traumatic Brain Injury Detection Using Electrophysiological Methods
Rapp, P. E., Keyser, D. O., Albano, A. M., Hernandez, R. S., Gibson, D. B., Zambon, R. A., Hairston, W. D., Hughes, J., Krystal, A. D., & Nichols, A. S. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00011 · Cited by 184
Factors Affecting Reliability of the Biodex Balance System: A Summary of Four Studies
Hinman, M. R. (2000) · Journal of Sport Rehabilitation · doi:10.1123/jsr.9.3.240 · Cited by 174
Análise de correlação em estudos clínicos e experimentais
Miot, H. A. (2018) · Jornal Vascular Brasileiro · doi:10.1590/1677-5449.174118 · Cited by 174
Reduction in aortic aneurysm size: Early results after endovascular graft placement
Matsumura, J. S., Pearce, W. H., McCarthy, W. J., et al. (1997) · Journal of Vascular Surgery · doi:10.1016/s0741-5214(97)70327-6 · Cited by 174
Lund, H., Søndergaard, K. H. E., Zachariassen, T., Christensen, R., Bülow, P., Henriksen, M., Bartels, E. M., Danneskiold‐Samsøe, B., & Bliddal, H. (2005) · Clinical Physiology and Functional Imaging · doi:10.1111/j.1475-097x.2004.00593.x · Cited by 174
Adolescent and Parent Alliance and Treatment Outcome in Multidimensional Family Therapy.
Shelef, K., Diamond, G. M., Diamond, G. S., & Liddle, H. A. (2005) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.73.4.689 · Cited by 173
Persson, A., Brogårdh, C. B., & Sjölund, B. H. (2004) · Journal of Rehabilitation Medicine · doi:10.1080/16501970310015218 · Cited by 173
Comparing Reliability of Perceptual Ratings of Roughness and Acoustic Measures of Jitter
Rabinov, C. R., Kreiman, J. E., Gerratt, B. R., & Bielamowicz, S. A. (1995) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3801.26 · Cited by 172
Psychometric Testing of the Revised 15-item Bakas Caregiving Outcomes Scale
Bakas, T., Champion, V. L., Perkins, S. M., Farran, C. J., & Williams, L. S. (2006) · Nursing Research · doi:10.1097/00006199-200609000-00007 · Cited by 172
Hersoug, A. G., Høglend, P. A., Monsen, J. T., & Havik, O. E. (2001) · PubMed Central · Cited by 173
Gusi, N., Adsuar, J. C., Corzo, H., Cruz, B. D. P., Olivares, P. R., & Parraça, J. A. (2012) · Journal of physiotherapy · doi:10.1016/s1836-9553(12)70089-9 · Cited by 172
Gruber, J., Eidelman, P., Johnson, S. L., Smith, B., & Harvey, A. G. (2011) · Journal of Abnormal Psychology · doi:10.1037/a0023667 · Cited by 181
Risk factors for recidivism among spousal assault and spousal homicide offenders
Grann, M., & Wedin, I. (2002) · Psychology Crime and Law · doi:10.1080/10683160208401806 · Cited by 172
Global assessment of psychosocial functioning in child and adolescent psychiatry
Schorre, B. E. H., & Vandvik, I. H. (2004) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-004-0390-2 · Cited by 172
Foot measurements from three-dimensional scans: A comparison and evaluation of different methods
Witana, C. P., Xiong, S., Zhao, J., et al. (2006) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2006.06.004 · Cited by 172
The role of alcohol and drugs in triggering criminal violence: a case‐crossover study*
Haggård, U., Hallqvist, J., Långström, N., & Möller, J. (2005) · Addiction · doi:10.1111/j.1360-0443.2005.01293.x · Cited by 178
Reassessment of the Craniocervical Junction: Normal Values on CT
Rojas, C. A., Bertozzi, J. C., Martinez, C. R., et al. (2007) · American Journal of Neuroradiology · doi:10.3174/ajnr.a0660 · Cited by 172
The Health of America’s Aging Prison Population
Skarupski, K. A., Gross, A. L., Schrack, J. A., Deal, J. A., & Eber, G. B. (2017) · Epidemiologic Reviews · doi:10.1093/epirev/mxx020 · Cited by 173
Cseh, G. M., & Jeffries, K. K. (2019) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/aca0000220 · Cited by 171
The Duke severity of illness checklist (DUSOI) for measurement of severity and comorbidity
Parkerson, G. R., Broadhead, W. E., Chiu-Kit, J., & Tse, T. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90153-r · Cited by 172
Swann, W. B., & Gill, M. J. (1997) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.73.4.747 · Cited by 172
Beersma, B., & Dreu, C. K. W. D. (2002) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.2001.2964 · Cited by 173
Wallace, C., Liberman, R. P., MacKain, S. J., Blackwell, G., & Eckman, T. A. (1992) · American Journal of Psychiatry · doi:10.1176/ajp.149.5.654 · Cited by 172
Complementarity of interpersonal behaviors in dyadic interactions
Markey, P. M., Funder, D. C., & Ozer, D. J. (2003) · Personality and Social Psychology Bulletin · doi:10.1177/0146167203253474 · Cited by 171
Learning with retrieval-based concept mapping.
Blunt, J. R., & Karpicke, J. D. (2014) · Journal of Educational Psychology · doi:10.1037/a0035934 · Cited by 171
Dillen, L. R. V., Bloom, N. J., Gombatto, S. P., & Susco, T. M. (2008) · Physical Therapy in Sport · doi:10.1016/j.ptsp.2008.01.002 · Cited by 171
The reliability and validity of qualitative scores for the Controlled Oral Word Association Test
ROSS, T., CALHOUN, E., Cox, T., WENNER, C., KONO, W., & Pleasant, M. M. (2007) · Archives of Clinical Neuropsychology · doi:10.1016/j.acn.2007.01.026 · Cited by 171
Odom, C. J., Taylor, A. B., Hurd, C. E., et al. (2001) · Physical Therapy · doi:10.1093/ptj/81.2.799 · Cited by 171
Parker, S. K. (2007) · Human Relations · doi:10.1177/0018726707076684 · Cited by 172
Evaluating the Quality of Life of Long-Term Care Residents With Dementia
Sloane, P. D., Zimmerman, S. I., Williams, C. S., Reed, P. S., Gill, K., & Preisser, J. S. (2005) · The Gerontologist · doi:10.1093/geront/45.suppl_1.37 · Cited by 172
Küçükdeveci, A. A., Tennant, A., Elhan, A. H., & Niyazoglu, H. (2001) · Spine · doi:10.1097/00007632-200112150-00024 · Cited by 170
Predicting Red-Light Running Behavior
Porter, B. E., & England, K. J. (2000) · Journal of Safety Research · doi:10.1016/s0022-4375(99)00024-9 · Cited by 170
Altered Cortical Network Dynamics
Rasetti, R. (2011) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2011.103 · Cited by 173
Validations of the Brazilian version of the Early Childhood Oral Health Impact Scale (ECOHIS)
Martins‐Júnior, P. A., Ramos‐Jorge, J., Paiva, S. M., Marques, L. S., & Ramos‐Jorge, M. L. (2012) · Cadernos de Saúde Pública · doi:10.1590/s0102-311x2012000200015 · Cited by 170
A Bifactor Approach to Modeling the Structure of the Psychopathy Checklist-Revised
Patrick, C. J., Hicks, B. M., Nichol, P. E., & Krueger, R. F. (2007) · Journal of Personality Disorders · doi:10.1521/pedi.2007.21.2.118 · Cited by 170
Russell, M. A., Hill, K., Blackberry, I., Day, L., & Dharmage, S. C. (2008) · Age and Ageing · doi:10.1093/ageing/afn129 · Cited by 177
Reliability of Temporal Summation and Diffuse Noxious Inhibitory Control
Cathcart, S., Winefield, A. H., Rolan, P. E., & Lushington, K. (2009) · Pain Research and Management · doi:10.1155/2009/523098 · Cited by 170
Banff Histopathological Consensus Criteria for Preimplantation Kidney Biopsies
Liapis, H., Gaut, J. P., Klein, C. L., Bagnasco, S. M., Kraus, E. S., Farris, A. B., Honsová, E., Perkowska‐Ptasińska, A., David, D., Goldberg, J. C., et al. (2016) · American Journal of Transplantation · doi:10.1111/ajt.13929 · Cited by 169
Knitza, J., Taşçılar, K., Meßner, E.-M., Meyer, M., Vossen, D., Pulla, A., Bösch, P., Kittler, J., Kleyer, A., Sewerin, P., et al. (2019) · JMIR mhealth and uhealth · doi:10.2196/14991 · Cited by 170
Overend, T. J., Wuori, J. L., Kramer, J. F., et al. (1999) · Journal of Hand Therapy · doi:10.1016/s0894-1130(99)80031-3 · Cited by 170
Wedderburn, L. R., Varsani, H., Li, C. K.-C., et al. (2007) · Arthritis Care & Research · doi:10.1002/art.23012 · Cited by 170
High Agreement and High Prevalence: The Paradox of Cohen’s Kappa
Zec, S., Soriani, N., Comoretto, R., & Baldi, I. (2017) · The Open Nursing Journal · doi:10.2174/1874434601711010211 · Cited by 169
Organizational learning and compensation strategies: Evidence from the Spanish chemical industry
Gómez, P. J., Lorente, J. J. C., & Cabrera, R. V. (2005) · Human Resource Management · doi:10.1002/hrm.20071 · Cited by 168
Wylde, V., Palmer, S., Learmonth, I., & Dieppe, P. A. (2011) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2011.02.009 · Cited by 169
Effectiveness of Pharmacotherapy for Severe Personality Disorders
Ingenhoven, T., Lafay, P., Rinne, T., Passchier, J., & Duivenvoorden, H. J. (2009) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.08r04526gre · Cited by 166
The content and validity of music-genre stereotypes among college students
Rentfrow, P. J., & Gosling, S. D. (2007) · Psychology of Music · doi:10.1177/0305735607070382 · Cited by 169
Goldbeck, T. G., & Davies, G. J. (2000) · Journal of Sport Rehabilitation · doi:10.1123/jsr.9.1.35 · Cited by 169
Lahaye, M. J., Engelen, S. M. E., Kessels, A. G. H., de Bruı̈ne, A. P., MEYENFELDT, M. F. V., van Engelshoven, J. M. A., van de Velde, C. J. �., Beets, G. L., & Beets‐Tan, R. G. H. (2008) · Radiology · doi:10.1148/radiol.2463070221 · Cited by 168
Emotional complexity: Clarifying definitions and cultural correlates.
Grossmann, I., Huynh, A. C., & Ellsworth, P. C. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000084 · Cited by 185
Wright, J. G., & Young, N. L. (1997) · Journal of Bone and Joint Surgery · doi:10.2106/00004623-199707000-00003 · Cited by 167
Reliability of File-Based Retrospective Ratings of Psychopathy With the PCL-R
Grann, M., Långström, N., Tengström, A., & Stålenheim, E. G. (1998) · Journal of Personality Assessment · doi:10.1207/s15327752jpa7003_2 · Cited by 169
Measuring Mood Spectrum: Comparison of interview (SCI-MOODS) and self-report (MOODS-SR) instruments
Dell’Osso, L., Armani, A., Rucci, P., Frank, E., Fagiolini, A. M., Corretti, G., Shear, M. K., Grochocinski, V. J., Maser, J. D., Endicott, J., et al. (2002) · Comprehensive Psychiatry · doi:10.1053/comp.2002.29852 · Cited by 169
Depressive Symptoms in Schizophrenia
Zisook, S., McAdams, L. A., Kuck, J., Harris, M. J., Bailey, A. L., Patterson, T. L., Judd, L. L., & Jeste, D. V. (1999) · American Journal of Psychiatry · doi:10.1176/ajp.156.11.1736 · Cited by 169
Kim, Y., Park, D., Kim, W., et al. (2010) · JACC: Cardiovascular Interventions · doi:10.1016/j.jcin.2010.04.004 · Cited by 168
Measurement and reproducibility of strength and voluntary activation of lower‐limb muscles
Todd, G., Gorman, R. B., & Gandevia, S. C. (2004) · Muscle & Nerve · doi:10.1002/mus.20027 · Cited by 168
Differences in Male and Female Spino-Pelvic Alignment in Asymptomatic Young Adults
Janssen, M. M. A., Drevelle, X., Humbert, L., Wafa Skalli, E., & Castelein, R. M. (2009) · Spine · doi:10.1097/brs.0b013e3181a9fd85 · Cited by 168
Comparison of electrical and magnetic stimulations to assess quadriceps muscle function
Vergès, S., Maffiuletti, N. A., Kerhervé, H. A., Decorte, N., Wuyam, B., & Millet, G. Y. (2008) · Journal of Applied Physiology · doi:10.1152/japplphysiol.01051.2007 · Cited by 167
MASIA-WARNER, C., STORCH, E. A., PINCUS, D. B., KLEIN, R. G., HEIMBERG, R. G., & LIEBOWITZ, M. R. (2003) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/01.chi.0000070249.24125.89 · Cited by 169
Bartels, A. L., Balash, Y., Gurevich, T., Schaafsma, J. D., Hausdorff, J. M., & Giladi, N. (2003) · Journal of Clinical Neuroscience · doi:10.1016/s0967-5868(03)00192-9 · Cited by 167
Ağırdağ, O., Houtte, M. V., & Avermaet, P. V. (2011) · European Sociological Review · doi:10.1093/esr/jcq070 · Cited by 192
Harari, G. M., Müller, S. R., Stachl, C., Wang, R., Wang, W., Bühner, M., Rentfrow, P. J., Campbell, A. T., & Gosling, S. D. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000245 · Cited by 168
Quantitative assessment of head and shoulder posture.
Braun, B. L., & Amundson, L. R. (1989) · PubMed · Cited by 168
Interobserver Reproducibility of Histopathologic Prognostic Variables in Primary Cutaneous Melanomas
Scolyer, R. A., Shaw, H. M., Thompson, J. F., et al. (2003) · The American Journal of Surgical Pathology · doi:10.1097/00000478-200312000-00011 · Cited by 168
Reliability of Pain and Stiffness Assessments in Clinical Manual Lumbar Spine Examination
Maher, C. G., & Adams, R. D. (1994) · Physical Therapy · doi:10.1093/ptj/74.9.801 · Cited by 168
Statistical properties of the rWG(J) index of agreement.
Cohen, A., Doveh, E., & Eick, U. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.3.297 · Cited by 167
Hoemann, C. D., Kandel, R. A., Roberts, S., et al. (2011) · Cartilage · doi:10.1177/1947603510397535 · Cited by 167
Rosenthal, R., Hoffmann, H., Clavien, P., Bucher, H. C., & Dell‐Kuster, S. (2015) · World Journal of Surgery · doi:10.1007/s00268-015-3003-y · Cited by 167
Salyers, M. P., & Mueser, K. T. (2001) · Schizophrenia Research · doi:10.1016/s0920-9964(00)00063-3 · Cited by 167
Dückelmann, A. M., Bamberg, C., Michaelis, S. A. M., de Lange, J. M., Nonnenmacher, A., Dudenhausen, J. W., & Kalache, K. D. (2009) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.7521 · Cited by 169
Generalizability theory: A practical guide to study design, implementation, and interpretation
Briesch, A. M., Swaminathan, H., Welsh, M. E., & Chafouleas, S. M. (2014) · Journal of School Psychology · doi:10.1016/j.jsp.2013.11.008 · Cited by 190
Furr, R. M. (2009) · Journal of Personality Assessment · doi:10.1080/00223890903379134 · Cited by 167
Lievens, F., Buyse, T., & Sackett, P. R. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.3.442 · Cited by 173
Zheng, Y., Lee, T. T., Lai, K. K., et al. (2016) · Scoliosis and Spinal Disorders · doi:10.1186/s13013-016-0074-y · Cited by 167
Disabuse of the drug metaphor: psychotherapy process-outcome correlations.
Stiles, W. B., & Shapiro, D. A. (1994) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.62.5.942 · Cited by 168
Jensen, P. S., Traylor, J. A., Xenakis, S. N., & Davis, H. (1988) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-198807000-00012 · Cited by 170
Borman, W. C., Buck, D. E., Hanson, M. A., Motowidlo, S. J., Stark, S. E., & Drasgow, F. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.5.965 · Cited by 176
The Dissociative Experiences Scale: Further replication and validation.
Frischholz, E. J., Braun, B. G., Sachs, R. G., & Hopkins, L. (1990) · Cited by 167
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
