The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 1,301–1,400 of 23,883
The Elephant (or Donkey) in the Boardroom
Gupta, A., & Wowak, A. J. (2016) · Administrative Science Quarterly · doi:10.1177/0001839216668173 · Cited by 185
Dong, L., Qiao, X., Tian, X., Liu, N., Jin, Y., Si, H., & Wang, C. (2017) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2017.06.011 · Cited by 186
Agreement between patients' and proxies' reports of quality of lifein Alzheimer's disease
Novella, J., Jochum, C., Jolly, D., et al. (2001) · Quality of Life Research · doi:10.1023/a:1012522013817 · Cited by 185
Posture in Patients With Shoulder Overuse Injuries and Healthy Individuals
Greenfield, B. H., Catlin, P. A., Coats, P. W., et al. (1995) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1995.21.5.287 · Cited by 185
Representation of Numerical and Sequential Patterns in Macaque and Human Brains
Wang, L., Uhrig, L., Jarraya, B., & Dehaene, S. (2015) · Current Biology · doi:10.1016/j.cub.2015.06.035 · Cited by 166
Automatic detection of pain intensity
Hammal, Z., & Cohn, J. F. (2012) · Proceedings of the 14th ACM international conference on Multimodal interaction · doi:10.1145/2388676.2388688 · Cited by 185
Group motivational interviewing for adolescents: Change talk and alcohol and marijuana outcomes.
D’Amico, E. J., Houck, J. M., Hunter, S. B., Miles, J. N. V., Osilla, K. C., & Ewing, B. A. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0038155 · Cited by 185
Wakschlag, L. S., Briggs‐Gowan, M. J., Hill, C., Danis, B., LEVENTHAL, B. L., Keenan, K. E., EGGER, H. L., Cicchetti, D. V., Burns, J. L., & Carter, A. S. (2008) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/chi.0b013e31816c5c10 · Cited by 187
Stöber, J. (1999) · Diagnostica · doi:10.1026//0012-1924.45.4.173 · Cited by 184
Determinants of repeated blood donation among new and experienced blood donors
Godin, G., Conner, M. T., Sheeran, P., Bélanger‐Gravel, A., & Germain, M. (2007) · Transfusion · doi:10.1111/j.1537-2995.2007.01331.x · Cited by 185
Accuracy of 2 Activity Monitors in Detecting Steps in People With Stroke and Traumatic Brain Injury
Fulk, G. D., Combs, S. A., Danks, K. A., Nirider, C. D., Raja, B., & Reisman, D. S. (2014) · Physical Therapy · doi:10.2522/ptj.20120525 · Cited by 182
Lee, K. M., Lee, J., Chung, C. Y., Ahn, S., Sung, K. H., Kim, T. W., Lee, H. J., & Park, M. S. (2012) · Clinics in Orthopedic Surgery · doi:10.4055/cios.2012.4.2.149 · Cited by 216
Time and learning efficiency in Internet-based learning: a systematic review and meta-analysis
Cook, D. A., Levinson, A. J., & Garside, S. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9231-x · Cited by 181
Positive and Negative Effects of HIV Infection in Women with Low Socioeconomic Resources
Updegraff, J. A., Taylor, S. E., Kemeny, M. E., & Wyatt, G. E. (2002) · Personality and Social Psychology Bulletin · doi:10.1177/0146167202286009 · Cited by 180
Kellas, J. K. (2005) · Communication Monographs · doi:10.1080/03637750500322453 · Cited by 184
Reliability and Feasibility of Methods to Quantitatively Assess Peripheral Edema
Brodovicz, K. G., McNaughton, K. S., Uemura, N., et al. (2009) · Clinical Medicine & Research · doi:10.3121/cmr.2009.819 · Cited by 183
Measuring femoral and rotational alignment: EOS system versus computed tomography
Folinais, D., Thelen, P., Delin, C., et al. (2013) · Orthopaedics & Traumatology Surgery & Research · doi:10.1016/j.otsr.2012.12.023 · Cited by 183
Reliable quantification of BOLD fMRI cerebrovascular reactivity despite poor breath-hold performance
Bright, M. G., & Murphy, K. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.07.007 · Cited by 183
INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS
Murphy, K. R., & DeShon, R. P. (2000) · Personnel Psychology · doi:10.1111/j.1744-6570.2000.tb02421.x · Cited by 183
Bello-Haas, V. P. M. D., Klassen, L., Sheppard, M. S., & Metcalfe, A. (2010) · Physiotherapy Canada · doi:10.3138/ptc.2009-08 · Cited by 190
Antecedents and Performance Consequences of Helping Behavior in Work Groups
Ng, K., & Dyne, L. V. (2005) · Group & Organization Management · doi:10.1177/1059601104269107 · Cited by 193
Identifying Appropriate Compensation Types for Service Failures
Roschk, H., & Gelbrich, K. (2013) · Journal of Service Research · doi:10.1177/1094670513507486 · Cited by 196
Syed, M. (2010) · Developmental Psychology · doi:10.1037/a0020738 · Cited by 184
Arkachaisri, T., Vilaiyuk, S., Li, S. C., O’Neil, K. M., Pope, E., Higgins, G. C., Punaro, M., Rabinovich, E., Rosenkranz, M. E., KIETZ, D. A., et al. (2009) · The Journal of Rheumatology · doi:10.3899/jrheum.081284 · Cited by 183
Wyatt, M. C., Weidner, J., Pfluger, D. H., & Beck, M. (2016) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-016-5137-0 · Cited by 184
Stronger shared taste for natural aesthetic domains than for artifacts of human culture
Vessel, E. A., Maurer, N., Denker, A. H., & Starr, G. G. (2018) · Cognition · doi:10.1016/j.cognition.2018.06.009 · Cited by 178
Measuring freezing of gait during daily-life: an open-source, wearable sensors approach
Mancini, M., Shah, V. V., Stuart, S., Curtze, C., Horak, F. B., Safarpour, D., & Nutt, J. G. (2021) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-020-00774-3 · Cited by 180
Validation of self reported diagnosis of hypertension in a cohort of university graduates in Spain
Alonso, Á., Beunza, J. J., Delgado‐Rodríguez, M., & Martínez‐González, M. Á. (2005) · BMC Public Health · doi:10.1186/1471-2458-5-94 · Cited by 178
Outcome evaluation in patients with elbow pathology: Issues in instrument development and evaluation
MacDermid, J. C. (2001) · Journal of Hand Therapy · doi:10.1016/s0894-1130(01)80040-5 · Cited by 182
Foo, M., Wong, P. K., & Ong, A. (2004) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2004.04.001 · Cited by 179
Vergnaud, A., Touvier, M., Méjean, C., Kesse‐Guyot, E., Pollet, C., Malon, A., Castetbon, K., & Herçberg, S. (2011) · International Journal of Public Health · doi:10.1007/s00038-011-0257-5 · Cited by 184
Reproducibility of the Oral Glucose Tolerance Test in Overweight Children
Libman, I. M., Barinas‐Mitchell, E., Bartucci, A., Robertson, R. J., & Arslanian, S. A. (2008) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2008-0801 · Cited by 183
Timing of Palatal Surgery and Speech Outcome
Chapman, K. L., Hardin-Jones, M. A., Goldstein, J. A., et al. (2008) · The Cleft Palate-Craniofacial Journal · doi:10.1597/06-244 · Cited by 182
Subsyndromal Symptoms in Bipolar Disorder
Keller, M. B. (1992) · Archives of General Psychiatry · doi:10.1001/archpsyc.1992.01820050035005 · Cited by 182
Picciano, A. M., Rowlands, M. S., & Worrell, T. W. (1993) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1993.18.4.553 · Cited by 182
Franc, B. (2003) · Human Pathology · doi:10.1016/s0046-8177(03)00403-9 · Cited by 182
Kashdan, T. B., & Roberts, J. E. (2004) · Cognitive Therapy and Research · doi:10.1023/b:cotr.0000016934.20981.68 · Cited by 182
Sakzewski, L., Ziviani, J. M., Abbott, D. F., Macdonell, R. A. L., Jackson, G. D., & Boyd, R. N. (2011) · Developmental Medicine & Child Neurology · doi:10.1111/j.1469-8749.2010.03859.x · Cited by 181
Lea, R. D., & Gerhardt, J. J. (1995) · Journal of Bone and Joint Surgery · doi:10.2106/00004623-199505000-00017 · Cited by 181
Parent-Child Interactions in Clinically Anxious Children and Their Siblings
Hudson, J. L., & Rapee, R. M. (2002) · Journal of Clinical Child & Adolescent Psychology · doi:10.1207/s15374424jccp3104_13 · Cited by 197
Tasca, G. A., Ritchie, K., Conrad, G., Balfour, L., Gayton, J., Lybanon, V., & Bissada, H. (2005) · Psychotherapy Research · doi:10.1080/10503300500090928 · Cited by 181
Atrophy in the parahippocampal gyrus as an early biomarker of Alzheimer’s disease
Echávarri, C., Aalten, P., Uylings, H. B. M., Jacobs, H. I. L., Visser, P. J., Gronenschild, E. H. B. M., Verhey, F. R. J., & Burgmans, S. (2010) · Brain Structure and Function · doi:10.1007/s00429-010-0283-8 · Cited by 180
Interrater Reliability of the History and Physical Examination in Patients With Mechanical Neck Pain
Cleland, J. A., Childs, J. D., Fritz, J. M., & Whitman, J. M. (2006) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2006.06.011 · Cited by 182
Wei, Y. S., Samiee, S., & Lee, R. P. (2013) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-013-0337-6 · Cited by 184
Refinement, Reliability, and Validity of the Segmental Assessment of Trunk Control
Butler, P. B., Saavedra, S. L., Sofranac, M., et al. (2010) · Pediatric Physical Therapy · doi:10.1097/pep.0b013e3181e69490 · Cited by 181
McFadyen, A. K., Maclaren, W. M., & Webster, V. S. (2007) · Journal of Interprofessional Care · doi:10.1080/13561820701352531 · Cited by 186
The reliability of network density and composition measures
Marsden, P. V. (1993) · Social Networks · doi:10.1016/0378-8733(93)90014-c · Cited by 181
Radiographic validation of the Manchester scale for the classification of hallux valgus deformity
Menz, H. B., & Munteanu, S. E. (2005) · Rheumatology · doi:10.1093/rheumatology/keh687 · Cited by 178
van der Voort, Gaillard, R., Albert Hofman, D., de Jongste, J. C., & Duijts, L. (2013) · Cited by 181
Trevethan, R. (2016) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-016-0156-6 · Cited by 181
Toward a connectivity gradient-based framework for reproducible biomarker discovery
Hong, S., Xu, T., Nikolaidis, A., Smallwood, J., Margulies, D. S., Bernhardt, B. C., Vogelstein, J. T., & Milham, M. P. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117322 · Cited by 194
A description of physical therapists' knowledge in managing musculoskeletal conditions
Childs, J. D., Whitman, J. M., Sizer, P. S., Pugia, M. L., Flynn, T. W., & Delitto, A. (2005) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-6-32 · Cited by 180
The reliability of ERP components in the auditory oddball paradigm
Segalowitz, S. J., & BARNES, K. L. (1993) · Psychophysiology · doi:10.1111/j.1469-8986.1993.tb02068.x · Cited by 181
Attributional style of lonely and depressed people.
Anderson, C. A., Horowitz, L. M., & French, R. D. S. (1983) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.45.1.127 · Cited by 181
Seeing Through the Eyes of a Rival: Competitor Acumen Based on Rival-Centric Perceptions
Tsai, W., Su, K., & Chen, M.-J. (2011) · Academy of Management Journal · doi:10.5465/amj.2011.64870138 · Cited by 184
Agreement between Self‐reported and Routinely Collected Health‐care Utilization Data among Seniors
Raina, P. S., Torrance-Rynard, V., Wong, M., & Woodward, C. A. (2002) · Health Services Research · doi:10.1111/1475-6773.00047 · Cited by 180
Firm, fair, and caring officer-offender relationships protect against supervision failure.
Kennealy, P., Skeem, J., Manchak, S. M., & Louden, J. E. (2012) · Law and Human Behavior · doi:10.1037/h0093935 · Cited by 189
Differences in Clinical Parameters and Tear Film of Tolerant and Intolerant Contact Lens Wearers
Glasson, M. J., Stapleton, F. J., Keay, L. J., Sweeney, D. F., & Willcox, M. D. P. (2003) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.03-0685 · Cited by 180
Duration of maintained hamstring flexibility after cessation of an acute static stretching protocol.
Depino, G. M., Webright, W. G., & Arnold, B. L. (2000) · PubMed · Cited by 179
Cervical Range of Motion Associations With Subclinical Neck Pain
Lee, H., Nicholson, L. L., & Adams, R. D. (2003) · Spine · doi:10.1097/01.brs.0000103944.10408.ba · Cited by 180
Nielsen, M., Mushin, I., Tomaselli, K. G., & Whiten, A. (2014) · Child Development · doi:10.1111/cdev.12265 · Cited by 179
Intra-individual variability in the response to anodal transcranial direct current stimulation
López-Alonso, V., Fernández‐del‐Olmo, M., Costantini, A., González, J. J., & Cheeran, B. (2015) · Clinical Neurophysiology · doi:10.1016/j.clinph.2015.03.022 · Cited by 179
Videofluorographic Study of Swallowing in Parkinson's Disease
Nagaya, M., Kachi, T., Yamada, T., & Igata, A. (1998) · Dysphagia · doi:10.1007/pl00009562 · Cited by 178
Tousignant, M., Poulin, L., Marchand, S., et al. (2005) · Disability and Rehabilitation · doi:10.1080/09638280400018411 · Cited by 179
Differential aging of the human striatum: a prospective MR imaging study.
Gunning‐Dixon, F. M., Head, D. M., McQuain, J., Acker, J. D., & Raz, N. (1998) · PubMed Central · Cited by 178
Radiographic Identification of the Primary Medial Knee Structures
Wijdicks, C. A., Griffith, C. J., LaPrade, R. F., Johansen, S., Sunderland, A., Arendt, E. A., & Engebretsen, L. (2009) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.h.00909 · Cited by 179
Non-invasive monitoring of chewing and swallowing for objective quantification of ingestive behavior
Sazonov, E. E., Schuckers, S., Lopez‐Meyer, P., Makeyev, O., Sazonova, N. A., Melanson, E. L., & Neuman, M. R. (2008) · Physiological Measurement · doi:10.1088/0967-3334/29/5/001 · Cited by 177
Howe, J., Inness, E., Venturini, A., Williams, J., & Verrier, M. (2006) · Clinical Rehabilitation · doi:10.1177/0269215506072183 · Cited by 183
A guide to the measurement and interpretation of fMRI test-retest reliability
Noble, S., Scheinost, D., & Constable, R. T. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.12.012 · Cited by 185
Child, Parent, and Peer Predictors of Early-Onset Substance Use: A Multisite Longitudinal Study
Kaplow, J. B., Curran, & Dodge, K. A. (2002) · Research on Child and Adolescent Psychopathology · doi:10.1023/a:1015183927979 · Cited by 200
Behavioral Syndromes in Alzheimer’s Disease
Devanand, D. P., Brockington, C. D., Moody, B. J., Brown, R. P., Mayeux, R. P., Endicott, J., & Sackeïm, H. A. (1992) · International Psychogeriatrics · doi:10.1017/s104161029200125x · Cited by 178
Kugaya, A., Akechi, T., Okuyama, T., Nakano, T., Mikami, I., Okamura, H., & Uchitomi, Y. (2000) · Cancer · doi:10.1002/1097-0142(20000615)88:12 · Cited by 309
Glenohumeral Joint Internal and External Rotation Range of Motion in Elite Junior Tennis Players
Ellenbecker, T. S., Roetert, E. P., Piorkowski, P. A., & Schulz, D. A. (1996) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1996.24.6.336 · Cited by 180
Ramirez, J., Berezuk, C., McNeely, A. A., Scott, C. J. M., Gao, F., & Black, S. E. (2014) · Journal of Alzheimer s Disease · doi:10.3233/jad-132528 · Cited by 183
Reliability of physical functioning measures in ambulatory subjects with MS
Paltamaa, J., West, H., Sarasoja, T., et al. (2005) · Physiotherapy Research International · doi:10.1002/pri.30 · Cited by 178
Reliability of error-related brain activity
Olvet, D. M., & Hajcak, G. (2009) · Brain Research · doi:10.1016/j.brainres.2009.05.079 · Cited by 183
Gait Variability in Community-Dwelling Older Adults
Brach, J. S., Berthold, R. V., Craik, R. L., VanSwearingen, J. M., & Newman, A. B. (2001) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2001.49274.x · Cited by 178
Maternal Personality, Parenting Cognitions and Parenting Practices
Bornstein, M. H., Hahn, C.-S., & Haynes, O. (2011) · Developmental Psychology · doi:10.1037/a0023181 · Cited by 176
Köllner, M. G., & Schultheiss, O. C. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00826 · Cited by 162
Measuring Older Adults' Sedentary Time
Gardiner, P. A., Clark, B., Healy, G. N., Eakin, E., WINKLER, E. A. H., & Owen, N. G. (2011) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0b013e31821b94f7 · Cited by 176
Age and gender differences in seven tests of functional mobility
Butler, A. A., Menant, J. C., Tiedemann, A., & Lord, S. R. (2009) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-6-31 · Cited by 178
Radiomics feature reliability assessed by intraclass correlation coefficient: a systematic review
Xue, C., Yuan, J., Lo, G. G., Chang, A. T. Y., Poon, D. M. C., Wong, O. L., Zhou, Y., & Chu, C. W. (2021) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-21-86 · Cited by 194
Calculating sample size for reliability studies
Borg, D. N., Bach, A. J. E., O'Brien, J. L., & Sainani, K. L. (2022) · PM&R · doi:10.1002/pmrj.12850 · Cited by 182
Pintauro, S. J., Nagy, T. R., Duthie, C. M., & Goran, M. (1996) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/63.3.293 · Cited by 178
Hill, C. E., O'Grady, K. E., & Elkin, I. (1992) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.60.1.73 · Cited by 190
Detection of Congestive Heart Failure in Dogs by Doppler Echocardiography
Schober, K. E., Hart, T. M., Stern, J. A., Li, X., Samii, V., Zekas, L. J., Scansen, B. A., & Bonagura, J. D. (2010) · Journal of Veterinary Internal Medicine · doi:10.1111/j.1939-1676.2010.0592.x · Cited by 178
Hatala, R. A., Cook, D. A., Brydges, R., & Hawkins, R. E. (2015) · Advances in Health Sciences Education · doi:10.1007/s10459-015-9593-1 · Cited by 177
Criterion-Related Validity of the Three-Factor Model of Psychopathy
Hall, J. R., Benning, S. D., & Patrick, C. J. (2004) · Assessment · doi:10.1177/1073191103261466 · Cited by 192
Melatonin profiles and sleep characteristics in boys with fragile X syndrome: A preliminary study
Gould, E. L., Loesch, D. Z., Martin, M. J., Hagerman, R. J., Armstrong, S. M., & Huggins, R. (2000) · American Journal of Medical Genetics · doi:10.1002/1096-8628(20001211)95:4 · Cited by 101
An automated method for the extraction of regional data from PET images
Rusjan, P., Mamo, D. C., Ginovart, N., Hussey, D., Vitcu, I., Yasuno, F., Suhara, T., Houle, S., & Kapur, S. (2006) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2006.01.011 · Cited by 177
Community program therapist adherence and competence in motivational enhancement therapy
Martino, S., Ball, S. A., Nich, C., Frankforter, T. L., & Carroll, K. M. (2008) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2008.01.020 · Cited by 179
Verstijnen, I. M., van Leeuwen, C., Goldschmidt, G., Hamel, R., & Hennessey, J. M. (1998) · Acta Psychologica · doi:10.1016/s0001-6918(98)00010-9 · Cited by 177
Development of the responsiveness to child feeding cues scale
Hodges, E. A., Johnson, S. L., Hughes, S. O., Hopkinson, J. M., Butte, N. F., & Fisher, J. O. (2013) · Appetite · doi:10.1016/j.appet.2013.02.010 · Cited by 177
Birmingham, T. B., Hunt, M. A., Jones, I. C., Jenkyn, T. R., & Giffin, J. R. (2007) · Arthritis Care & Research · doi:10.1002/art.22899 · Cited by 176
Occupational complexity and lifetime cognitive abilities
Smart, E. L., Gow, A. J., & Deary, I. J. (2014) · Neurology · doi:10.1212/wnl.0000000000001075 · Cited by 176
Tamber, A.-L., Wilhelmsen, K. T., & Strand, L. I. (2009) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-7-101 · Cited by 176
Reliability of surface EMG during sustained contractions of the quadriceps
Mathur, S., Eng, J. J., & MacIntyre, D. L. (2004) · Journal of Electromyography and Kinesiology · doi:10.1016/j.jelekin.2004.06.003 · Cited by 174
Caudate, putamen, and globus pallidus volume in schizophrenia: A quantitative MRI study
Hokama, H., Shenton, M. E., Nestor, P. G., Kikinis, R., Levitt, J. J., Metcalf, D., Wible, C. G., O'Donnella, B. F., Jolesz, F. A., & McCarley, R. W. (1995) · Psychiatry Research Neuroimaging · doi:10.1016/0925-4927(95)02729-h · Cited by 178
Development of the Cognitive Therapy Adherence and Competence Scale
Barber, J. P., Liese, B. S., & Abrams, M. J. (2003) · Psychotherapy Research · doi:10.1093/ptr/kpg019 · Cited by 176
Mylius, V., Ayache, S. S., Ahdab, R., Farhat, W. H., Zouari, H. G., Belke, M., Brugières, P., Wehrmann, E., Krakow, K., Timmesfeld, N., et al. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.03.061 · Cited by 175
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
