The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 19,501–19,600 of 23,883
Pereira, R. C. M., Filho, C. A. L., de Melo, L. A., et al. (2019) · Journal of Phytopathology · doi:10.1111/jph.12855 · Cited by 2
Duarte, C. S., Klotz, J., Elkington, K. S., Shrout, P. E., CANINO, G. J., Eisenberg, R. E., Ortin‐Peralta, A., Henriquez, M., Corbeil, T., & Bird, H. (2019) · Journal of Child and Family Studies · doi:10.1007/s10826-019-01661-9 · Cited by 2
Self-citationKim, A. G., Mon, R. A., Karmakar, M., et al. (2019) · Fetal Diagnosis and Therapy · doi:10.1159/000504510 · Cited by 2
Latorre, J., Mollà‐Casanova, S., Salinas, B., et al. (2019) · International Conference on Virtual Rehabilitation · doi:10.1109/icvr46560.2019.8994584 · Cited by 2
Sándor, Z., RÁTHONYI, G. K., & Dinya, E. (2020) · Computational and Mathematical Methods in Medicine · doi:10.1155/2020/7312125 · Cited by 2
Hou, W., Chiou, H., & Chang, F. (2020) · American Journal of Occupational Therapy · doi:10.5014/ajot.2019.035725 · Cited by 2
Wegner, M., Iskender, E., Azzarok, A., et al. (2020) · Medicine · doi:10.1097/md.0000000000019701 · Cited by 2
Tarnas, J., Schaffert, N., Elegańczyk-Kot, H., et al. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17103494 · Cited by 2
Seo, S.-J., Kim, Y. K., & Lee, S. (2020) · The American Journal of Psychology · doi:10.5406/amerjpsyc.133.2.0181 · Cited by 2
Preliminary Evaluations of Habituation of Operant Responding for Sensory Stimuli in Humans
Tonkin, S., & Hawk, L. W. (2020) · Behavioural Processes · doi:10.1016/j.beproc.2020.104159 · Cited by 2
Effects of 38 years of wildfires on tree density in the Blue Mountains, Australia
Rahmani, S., & Price, O. F. (2020) · Austral Ecology · doi:10.1111/aec.12952 · Cited by 2
Models of Physical Health and Personality
Heilmayr, D., & Friedman, H. S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108264822.019 · Cited by 2
Construct validity and reliability of the Concussion Knowledge Assessment Tool (CKAT).
Savic, M., Kazemi, M., Lee, A., et al. (2020) · PubMed · Cited by 2
Perić, I., Spasić, M., Novak, D., Ostojić, S. M., & Sekulić, D. (2021) · Diagnostics · doi:10.3390/diagnostics11010146 · Cited by 2
320 Slice CT in Imaging of Congenital Heart Diseases in Infants: A Single-Center Experience
Sreedher, G., Bruckman, D., & Ganapathy, S. S. (2021) · Cureus · doi:10.7759/cureus.13348 · Cited by 2
DEVELOPMENT OF RISK ASSESSMENT METHOD FOR SMALL SIZED HOSPITALS USING AHP: A CASE IN NORTHERN INDIA
Singh, L. P., & Suthar, H. (2021) · International Journal of the Analytic Hierarchy Process · doi:10.13033/ijahp.v13i2.771 · Cited by 2
Development and validation of diagrammatic scales to assess septoriose in tomato
Monteiro, F., Ogoshi, C., Cardoso, D., et al. (2021) · Plant Pathology & Quarantine · doi:10.5943/ppq/11/1/13 · Cited by 2
Shams, A., Hardy, L. L., Vameghi, R., et al. (2022) · Sport Sciences for Health · doi:10.1007/s11332-022-00971-8 · Cited by 2
Exploring the Effects of Growth Mind-Set Usernames in STEM Games
Kao, D. (2019) · Proceedings of the 2019 AERA Annual Meeting · doi:10.3102/1427645 · Cited by 3
de Amorim, J. C., Kishimoto, S. T., Elorza, C. L. C., CAVALETTI, F. A., Marini, R., Silva, C. A., Saad‐Magalhães, C., Fernandes, P. T., Brunner, H. I., & Appenzeller, S. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.945425 · Cited by 2
Bastías, G. F., Bravo, F., Astudillo, C. A., et al. (2022) · Foot & Ankle Orthopaedics · doi:10.1177/24730114221141388 · Cited by 2
Po, Y. (2016) · Rehabilitation practice and science · doi:10.6315/2016.44(2)02 · Cited by 2
Xiong, C., Gao, F., Yan, Y., Luo, J. R., Sung, Y., & Shi, G. (2008) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1209615780 · Cited by 2
Exploring therapeutic action in psychoanalytic psychotherapy : Attachment to therapist and change
Lilliengren, P. (2014) · DiVA (Stockholm University) · Cited by 2
Assessing soldiers' wellness holistically: An evaluation of instruments applicable to primary care
Schafer, M. L. (2012) · UA Campus Repository (The University of Arizona) · Cited by 2
Type 2 diabetes mellitus and the prevalence of age-related cataract in a clinic population.
Machan, C. M. (2012) · UWSpace (University of Waterloo) · Cited by 2
Single-case Experimental Designs in Clinical Settings
Follette, W. C. (2001) · Elsevier eBooks · doi:10.1016/b0-08-043076-7/01331-0 · Cited by 2
Mollov, N. (2012) · e-Publications@Marquette (Marquette University) · Cited by 2
Zanatta, R. G. (2005) · Summit (Simon Fraser University) · Cited by 2
Communication training for paid caregivers of people with traumatic brain injury (TBI)
Behn, N. (2012) · UPT. Syiah Kuala University Library (Syiah Kuala University) · Cited by 2
HIP disability : Patient education, classification and assessment
Klässbo, M. (2003) · Figshare · Cited by 2
Dorio, J. M. (2005) · Digital Commons - University of South Florida (University of South Florida) · Cited by 2
Perceptions and Attitudes Toward Prenatal Cocaine Exposure in Young Children
Leckman, E. B., Mayes, L. C., & Hodgins, H. S. (2001) · Child Psychiatry & Human Development · doi:10.1023/a:1010286406789 · Cited by 2
Mandell, W. (2006) · Digital Commons - PCOM (Philadelphia College of Osteopathic Medicine) · Cited by 2
The competence of physiotherapists to deliver a cognitive behavioural approach for low back pain
Hansen, Z. (2026) · Warwick Research Archive Portal (University of Warwick) · doi:10.82444/warw.33646315 · Cited by 2
The reliability of foot and ankle water volumetry
Balasundaram, J. (2006) · Tuwhera (Auckland University of Technology) · Cited by 2
Enebrink, P. (2005) · Figshare · Cited by 2
Processes of care in paediatric rehabilitation
Siebes, R. C. (2006) · Utrecht University Repository (Utrecht University) · Cited by 2
Risk factors for criminal offending among men with schizophrenia
Eriksson, Å. I. (2006) · Figshare · Cited by 2
Self-efficacy of knee function in patients with an Anterior Cruciate Ligament injury
Thomeé, P. (2007) · Gothenburg University Publications Electronic Archive (Gothenburg University) · Cited by 2
Repeated Measurements, Design and Analysis for
Wiener, L. E., & Koch, G. G. (2017) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat00309.pub2 · Cited by 2
Measuring disease progression in MS: do patients' and physicians' perspectives match?
Kragt, J. J. (2010) · Digital Academic REpository of VU University Amsterdam (Vrije Universiteit Amsterdam) · Cited by 2
Szostek, A. M., & Eggen, B. H. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-03655-2_22 · Cited by 2
Koskinen, S. K. (2011) · Työväentutkimus Vuosikirja · Cited by 2
Maarsingh, E. J. W. (2005) · UvA-DARE (University of Amsterdam) · Cited by 2
Patellar tendinopathy : on evaluation methods and rehabilitation techniques
Frohm, A. (2006) · Figshare · Cited by 2
Allet, L. (2009) · doi:10.26481/dis.20091216la · Cited by 2
Sable, D. N. (2012) · Library and Archives Canada (Government of Canada) · Cited by 2
Sterke, S. (2011) · RePub (Erasmus University Rotterdam) · Cited by 2
DEVELOPMENT, EVALUATION AND APPLICATION OF A PEDIATRIC ULCERATIVE COLITIS ACTIVITY INDEX (PUCAI)
Turner, D. (2008) · Cited by 2
Coenen, P. (2013) · Digital Academic REpository of VU University Amsterdam (Vrije Universiteit Amsterdam) · doi:10.13140/2.1.1978.2082 · Cited by 2
Morphine more fine? Its effects in critically ill newborns
Simons, S. H. P. (2004) · RePub (Erasmus University Rotterdam) · Cited by 2
Elfving, B. (2002) · Figshare · Cited by 2
Long-term consequences of differences in early growth : epidemiological aspects
Euser, A. M. (2009) · Leiden Repository (Leiden University) · Cited by 2
Jorquera, S., & Inés, M. (2013) · MPG.PuRe (Max Planck Society) · Cited by 2
Knowledge Desk Limited Objective Experiment (LOE)
Oonk, H. M., Schermerhorn, J. H., Glaser, D., & Morrison, J. G. (2003) · doi:10.21236/ada413070 · Cited by 2
Robot-Mediated Interviews: a Robotic Intermediary for Facilitating Communication with Children
Wood, L. J. (2015) · University of Hertfordshire Research Archive (University of Hertfordshire) · doi:10.18745/th.16313 · Cited by 2
Disability and Physical Exercise in adults with Myotonic Dystrophy type 1
Kierkegaard, M. (2010) · Figshare · Cited by 2
Messersmith, J. G. (2008) · KU ScholarWorks (The University of Kansas) · Cited by 2
Parietal matters in early Alzheimer's disease : evidence from structural and functional MRI
Jacobs, H. I. L. (2011) · Research Publications (Maastricht University) · doi:10.26481/dis.20110706hj · Cited by 1
The Effect of Concentric and Eccentric Training on the Strengthening of Tibialis Anterior
Reinking, M. F. (1991) · Isokinetics and Exercise Science · doi:10.3233/ies-1991-1404 · Cited by 2
The reliability of the Developmental Profile based on video ratings
Scholte, W. R., Tiemens, B. G., Verheul, R., et al. (2009) · Bulletin of the Menninger Clinic · doi:10.1521/bumc.2009.73.3.226 · Cited by 2
An Evaluation of Aviator Training Ability Requirements Scale Ratings
McAnulty, D. M., & Jones, D. H. (1984) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193128402800415 · Cited by 2
Self Assessment, Realistic Job Previews and Occupational Decisions
Makin, P. J., & Robertson, I. T. (1983) · Personnel Review · doi:10.1108/eb055482 · Cited by 2
Bobko, P., & Donnelly, L. (1988) · Human Performance · doi:10.1207/s15327043hup0103_4 · Cited by 2
The stability of neuropsychological test performance in a group of parenteral drug users
Richards, M., Sano, M. C., Goldstein, S. G., Mindry, D. L., Todak, G. G., & Stern, Y. (1992) · Journal of Substance Abuse Treatment · doi:10.1016/0740-5472(92)90033-k · Cited by 2
Correction: Reproducibility of Frankfort Horizontal Plane on 3D Multi-Planar Reconstructed MR Images
Daboul, A., Schwahn, C., Schaffner, G., et al. (2012) · PLoS ONE · doi:10.1371/annotation/6bc54ac4-3639-447d-92bc-dad0ddb24c1e · Cited by 2
Hedge, J. W., Borman, W. C., Kubisiak, U. C., et al. (2007) · Performance Improvement Journal · doi:10.1002/pfi.123 · Cited by 2
Security of infantile attachment: The person–situation debate revisited
Mills, C. J., & Eiserer, L. A. (1984) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00026650 · Cited by 2
Rouse, S. J. (1998) · British Journal of Occupational Therapy · doi:10.1177/030802269806100209 · Cited by 2
Quality of Life measures in children with epilepsy: Applicability in different populations
Mikati, M. A., & Rahi, A. C. (2008) · Epilepsy & Behavior · doi:10.1016/j.yebeh.2008.01.002 · Cited by 2
Stranger in a strange situation: Comments by a comparative psychologist
Denenberg, V. H. (1984) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0002656x · Cited by 2
USING JUST NOTICEABLE DIFFERENCES TO INTERPRET TEST OF SPOKEN ENGLISH SCORES
Stricker, L. J. (1997) · ETS Research Report Series · doi:10.1002/j.2333-8504.1997.tb01725.x · Cited by 2
Patient Safety in Nursing Homes
Thomas, K. S. (2011) · Digital Commons - University of South Florida (University of South Florida) · Cited by 2
Poole, J. L. (2000) · The Occupational Therapy Journal of Research · doi:10.1177/153944920002000202 · Cited by 2
Maiwald, C. (2008) · Qucosa (Saxon State and University Library Dresden) · Cited by 2
Psychometric Properties of the Scorable Self-Care Evaluation
Tarbell, M. H., Henry, A. D., & Coster, W. J. (2004) · American Journal of Occupational Therapy · doi:10.5014/ajot.58.3.324 · Cited by 2
Solem, M., & Nøvik, T. S. (1995) · Nordic Journal of Psychiatry · doi:10.3109/08039489509026794 · Cited by 2
Validez de las teorías implícitas de personalidad de profesores de Educación General Básica
Castillo, J. L. (1992) · International Journal of Social Psychology Revista de Psicología Social · doi:10.1080/02134748.1992.10821659 · Cited by 2
da Silva, G. A. (2013) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.47.2013.tde-24052013-143728 · Cited by 2
Keshwani, N. (2011) · QSpace (Queen's University Library) · Cited by 2
Statistical Image Analysis for a Confocal Microscopy 2D Section of Cartilage Growth
Al-Awadhi, F. A., Jennison, C., & Hurn, M. A. (2003) · Cited by 2
Association between wrist angle and carpal tunnel syndrome among workers
You, D. (2013) · eScholarship (California Digital Library) · Cited by 2
Test-Re-test Reliability of Sahrmann Lower Abdominal Core Stability Test for DII Baseball Athletes
Dekart, K. Q. (2014) · doi:10.33915/etd.125 · Cited by 2
Ratto, A. B. (2019) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/hp6z-3w69 · Cited by 2
Kyte, T. B. (2008) · OakTrust (Texas A&M University Libraries) · Cited by 2
Generators of mammalian vestibular surface responses to head motion
Gaines, G. C. (2012) · The Scholarship East Carolina University's Institutional Repository (East Carolina University) · Cited by 2
Wu, J.-Y. (2011) · OakTrust (Texas A&M University Libraries) · Cited by 2
Trainingsmaßnahmen zur Verbesserung der Interrater-Reliabilität in einer Alzheimer-Studie
Yekrangi-Hartmann, C., Bernhardt, T., & Baltissen, R. (1999) · Zeitschrift für Gerontopsychologie & -psychiatrie · doi:10.1024//1011-6877.12.3.143 · Cited by 2
Gender role portrayals of modern Disney royalty: Stereotypical or androgynous?
Lopreore, K. M. (2016) · Cited by 2
McCullough, J. (2016) · UWSpace (University of Waterloo) · Cited by 2
Jena, F.-S.-U. (2004) · Cited by 2
Shoulder pain within the ICF framework; patient experiences of functioning and assessment methods
Røe, Y. (2014) · NORA - Norwegian Open Research Archives · Cited by 2
Multimedia-based performance assessment in Dutch vocational education
de Klerk, S. (2015) · doi:10.3990/1.9789036539975 · Cited by 2
Fjelnseth, K. P. (2016) · NORA - Norwegian Open Research Archives · Cited by 2
Holmes, M. G. (2016) · Cited by 2
Catching liars by listening carefully
Bogaard (2017) · doi:10.26481/dis.20170127gb · Cited by 2
D’Silva, C. (2016) · e-scholar@UOIT (University of Ontario Institute of Technology) · Cited by 2
Pushing/Pulling Exertions Disturb Trunk Postural Stability
Lee, H. (2007) · VTechWorks (Virginia Tech) · Cited by 2
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
