The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 21,701–21,800 of 23,883
Reprodutibilidade da medida do comprimento do colo uterino por ultrassonografia
Ferreira, A. E. G. D. M. T. (2019) · doi:10.11606/t.17.2019.tde-17072019-101516
Exhaled breath analysis for diagnosis and phenotyping in obstructive lung diseases
Ibrahim, B. (2011) · Research Explorer (The University of Manchester)
The Perception of Effort, Training Load, and Exercise Performance in the Heat
Borg, D. N. (2019) · Queensland University of Technology · doi:10.5204/thesis.eprints.131876
Assessing Method Agreement for Paired Repeated Binary MeasurementsPreprint
Wang, W., Lin, N., Oberhaus, J., et al. (2018) · arXiv (Cornell University)
Acoustic and Perceptual Correlates of Breathy Vocal Quality
Cleveland, R. A. (1991)
Smartphone picture organization: A hierarchical approach
Lonn, S., Radeva, P. I., & Dimiccoli, M. (2018) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · doi:10.48550/arxiv.1803.05940
Lee, J. Y., Oh, Y., Kim, Y., Shim, J., & Hwang, S. H. (2019) · Journal of the Korean Society of Radiology · doi:10.3348/jksr.2019.80.5.930
Ateminsuffizienz bei Motoneuronerkrankungen: Früh diagnostizieren und zeitnah therapieren
Knoch, J. (2019) · Kompass Pneumologie · doi:10.1159/000502242
Modroño, C., García, F. G., & González, J. J. (2010)
Hall, N. E., Mamrot, J., FRAMPTON, C. M. A., Read, P., Steele, E. J., Bischof, R. J., & Lindley, R. A. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/807073
Intraepidermal Nerve Fiber Density in Postmortem Skin: A Novel Approach
Waheed, W., Taicher, C., Dulaney, E. D., Eckenstein, F. P., Hehir, M., Sprague, J. R., & Tandan, R. (2019) · Journal of Clinical Neuromuscular Disease · doi:10.1097/cnd.0000000000000264
MEASURING THE RELIABILITY OF RANKING IN INFORMATION RETRIEVAL SYSTEMS EVALUATION
Rajagopal, P., & Ravana, S. D. (2019) · Malaysian Journal of Computer Science · doi:10.22452/mjcs.vol32no4.1
Instrument for Forensic Treatment Evaluation
Schuringa, E., Heininga, V. E., Spreen, M., et al. (2018) · PsycTESTS Dataset · doi:10.1037/t73580-000
Heron, S. (2019) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r3-enpc-6412
Lyman, S. L., Koltsov, J. C. B., Hidaka, C., et al. (2019) · Journal of ISAKOS Joint Disorders & Orthopaedic Sports Medicine · doi:10.1136/jisakos-2019-000301
Chen, C., Shih, Y., Hsu, Y., Wen, T.-K., Fang, S., Tang, D.-L., Lee, S., & Tseng, W. I. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/855551
Correa, J. C., Pinto, D., Salas, L. A., Camacho, J. C., Rondón, M., & Quintero, J. (2011) · ISEE Conference Abstracts · doi:10.1289/isee.2011.00873
Kimmel, B. (2019) · Digital Commons-TMC (Texas Medical Center)
Moreno, L., Fernández-Pampillón, A. M., Sarasa Cabezuelo, A., et al. (2019) · Journal of Accessibility and Design for All · doi:10.17411/jacces.v9i2.193
A New In-Skates Balance Error Scoring System for Ice Hockey Players
Robert, A., Moroz, M., Correa, J. A., et al. (2019) · Clinical Journal of Sport Medicine · doi:10.1097/jsm.0000000000000816
Gray, J. S., & Phillips, G. A. (2019) · Assessment Update · doi:10.1002/au.30191
On within-defense variability and defenses in male federal offenders
Sheppard, M. D. (2009) · University Library - University of Saskatchewan (University of Saskatchewan)
Pulmonary rehabilitation and cardiovascular risk in COPD
Aldabayan, Y. S. (2019) · UCL Discovery (University College London)
Normal Hindfoot Alignment Assessed by Weight Bearing CT: Presence of a Constitutional Valgus?
Burssens, A. (2019) · Springer eBooks · doi:10.1007/978-3-030-31949-6_9
One-week test-retest reliability of nine binocular tests and saccades used in concussionPreprint
Long, S., Schuster, T., Steele, R. J., Leclerc, S., & Shrier, I. (2019) · medRxiv · doi:10.1101/19011619
Lee, S. (2013) · Korean Journal of Industrial and Organizational Psychology · doi:10.24230/kjiop.v26i2.177-194
Three‐dimensional neuroanatomy of the intraepidermal nervous system
Tan, Y., Zhang, Y. N., Xia, Y., Lee, B. T. K., Ng, L. G., & Tey, H. L. (2020) · British Journal of Dermatology · doi:10.1111/bjd.18924
Bringing the underground economy to light
Kurti, M. K. (2019) · doi:10.7282/t3-8y7a-s814
Goel, R. (2019) · UEA Digital Repository (University of East Anglia)
Bénony, H., Vatageot, S., Lignier, B., et al. (2020) · L Évolution Psychiatrique · doi:10.1016/j.evopsy.2020.01.003
Expanding the cross-cultural psychological assessment tool box with IQ test short forms
Nuñez, A., Miguel, L. S., Batista, S. G., Etcoff, L. M., Barchard, K. A., Donohue, B. C., & Allen, D. N. (2020) · Applied Neuropsychology Child · doi:10.1080/21622965.2020.1740093
Holderried, F., Herrmann‐Werner, A., Mahling, M., Holderried, M., Rießen, R., Zipfel, S., & Celebi, N. (2020) · PLoS ONE · doi:10.1371/journal.pone.0230522
de la Rosa, O. M. A., & Angulo, L. M. V. (2019) · Psychology Society & Education · doi:10.25115/psye.v11i3.2081
Refinement of a protocol to induce reliable muscle cramps in the abductor hallucis
Akerman, A. P., Walker, R. J., Schollum, J. B. W., et al. (2020) · Physiological Measurement · doi:10.1088/1361-6579/ab8855
DIAGNOSING SECOND LANGUAGE PRONUNCIATION
Isbell, D. R. (2019) · doi:10.13140/rg.2.2.14652.08325
Descriptive Analyses of Interrater Agreement for Ordinal Rating Scales
Bove, G., & Serafini, A. (2020) · Behaviormetrics · doi:10.1007/978-981-15-2700-5_22
Jennifer, J., & Ena, O. T. (2020) · Language Circle Journal of Language and Literature · doi:10.15294/lc.v14i2.23336
Agreement, Modeling of Categorical
Barlow, W. E. (2014) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat05228
Subjective Nuances of an Objective Survey Research and Scale Development
Alcruz, J., Eckardt, P. A., & Eckardt, S. (2021) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-64802-604-120251003
Wrist ROM measurements using smartphone photography: A common mistake in reliability analysis
Sabour, S. (2020) · Hand surgery & rehabilitation · doi:10.1016/j.hansur.2020.05.001
Reproductive outcomes in women with low ovarian reserve
Karunakaran, B. (2019) · University of Birmingham Institutional Research Archive (University of Birmingham)
Guédeney, N., & Fermanian, J. (1998) · European psychiatry (Ed Española) · doi:10.1017/s1134066500002721
Readability and suitability of implantable hearing device information published online in English.
Goddard, M. (2020) · UC Research Repository (University of Canterbury) · doi:10.26021/8953
Inter-rater Reliability of Participatory Evaluation Data - Brachiaria Grasses in NyandaruaPreprint
Mailu, S., & Maro, F. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3274582
Roumengous, T. P. T., Howell, P. A., & Peterson, C. L. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.06.12.148536
The Single-Factor Between-Subjects Analysis of Variance
Sheskin, D. J. (2020) · Chapman & Hall/CRC eBooks · doi:10.1201/9780429186196-28
Validation of the Sinhalese version of the Oswestry Disability Index for low back pain
Gamage, N. N., & Ranasinghe, P. D. (2020) · Sri Lankan Journal of Sports and Exercise Medicine · doi:10.4038/sljsem.v2i1.19
Knapik, D. M., Kolaczko, J. G., Drummond, I., et al. (2020) · Clinical Anatomy · doi:10.1002/ca.23642
Gillen, Z. M., Shoemaker, M. E., McKay, B. D., Bohannon, N. A., Gibson, S. M., & Cramer, J. T. (2020) · The Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000003728
Blake, S. R., Proscia, C., Eleuteri, A., et al. (2020) · European Heart Journal · doi:10.1093/ehjci/ehaa946.1457
What Are Data? Measurements and Errors
Banning, E. B. (2020) · Interdisciplinary Contributions to Archaeology/Interdisciplinary contributions to archaeology · doi:10.1007/978-3-030-47992-3_1
Gupta, S. D. (2020) · The University of Queensland · doi:10.14264/uql.2020.880
Fickling, S. D. (2020) · Summit (Simon Fraser University)
Shaffer, M. A., Kestel, L. L., Wolf, B. R., & Shields, R. K. (2020) · PubMed Central
Bagherzadeh, N., Roshan, N. M., & Jafarian, A. H. (2020) · Indian Journal of Gynecologic Oncology · doi:10.1007/s40944-020-00444-1
Iwashita, M., & Yoshizawa, M. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.08.20.258640
Children Evaluated for Speech and Language Concerns: One Year Outcomes
Davis, T. N. (2016)
Spoelma, T. M. (2018) · UA Campus Repository (The University of Arizona)
Wang, C., Liu, Z., Wu, Y., et al. (2020) · Reproduction in Domestic Animals · doi:10.1111/rda.13823
L’acromioplastie antérieure diminue-t-elle le Critical Shoulder Angle ?
Girard, M., Colombi, R., Azoulay, V., et al. (2020) · Revue de Chirurgie Orthopédique et Traumatologique · doi:10.1016/j.rcot.2020.07.005
IMPLEMENTATION OF INNOVATION AND ETHICS USING VIDEO-BASED LEARNING IN NERVE SURGERY
Yee, A. H. (2020) · Utrecht University Repository (Utrecht University) · doi:10.33540/321
Golja, A., Sapač, K., Žagar, D., et al. (2020) · Acta hydrotechnica · doi:10.15292/acta.hydro.2020.03
Construção e validação de um instrumento de enfermagem para avaliação da dispneia
Sagica, T. D. P., Farias, V. A., Conceição, C. D. M., et al. (2020) · Revista Eletrônica Acervo Saúde · doi:10.25248/reas.e4358.2020
Personality Assessment Methods
Boyle, G. J., & Helmes, E. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108264822.011 · Cited by 1
Schermer, J. A., & Saklofske, D. H. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108264822.014
Social Relations and Social Support
Austin, E. Q., Knack, J. M., Jensen‐Campbell, L. A., & Rex‐Lear, M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108264822.036
The validity of an isiZulu speech reception threshold test for use with adult isiZulu speakers
Panday, S., Kathard, H., & Wilson, W. J. (2020) · South African Journal of Communication Disorders · doi:10.4102/sajcd.v67i1.690
THE VALUE-BASED HEALTH CARE: A PSYCHOLOGICAL PERSPECTIVE IN CANCER CARE
Marzorati, C. (2020) · Archivio Istituzionale della Ricerca (Universita Degli Studi Di Milano) · doi:10.13130/marzorati-chiara_phd2020-01-28
Ankle ligament thickness in acute ankle sprain: An ultrasound-based study
Fenech, M. (2020) · Queensland University of Technology · doi:10.5204/thesis.eprints.206456
Coles, S. M. (2020) · Deep Blue (University of Michigan)
Primerljivost dosežkov na osnovni in višji ravni izpita iz matematike na splošni maturi
Cankar, G., & Žerovnik, J. (2020) · Psihološka obzorja · doi:10.20419/2020.29.523
Deep Facial Expression Recognition with Occlusion Regularization
Pandya, N., Werner, P., & Al-Hamadi, A. K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64559-5_32
Pitch Perfect: The Investigation of Presentation Quality Effect on Product Concept Evaluation
Kwon, J. (2019) · University of Minnesota Digital Conservancy (University of Minnesota)
Banthia, S. (2020) · INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH · doi:10.36106/ijsr/0418671
Wu, Y., Cao, J., Chen, T., et al. (2020) · doi:10.21203/rs.3.rs-17116/v1
Zhang, X. (2017) · Journal of International Crisis and Risk Communication Research
Preuss, U. W., Bender, M., Franz, M., et al. (2021) · Fortschritte der Neurologie · Psychiatrie · doi:10.1055/a-1128-0356
Güler, S., Çapkın, S., & Özmanevra, R. (2021) · Cyprus Journal of Medical Sciences · doi:10.5152/cjms.2020.2355
Hoefnagel, L., Espin, C. A., & Rippe, R. C. A. (2021) · International journal for research in learning disabilities · doi:10.28987/5.1.42 · Cited by 1
Jeong, Y., & Park, H. I. (2017) · Korean Journal of Industrial and Organizational Psychology · doi:10.24230/kjiop.v30i2.147-176
Avaliação multimétodos da tomada de decisão, regulação emocional e personalidade em jovens adultos
Pinto, A. L. D. C. B. (2020) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.59.2020.tde-11122020-154944 · Cited by 1
Oh, H. S., Yang, Y., Jeong, S. Y., et al. (2021) · Healthcare · doi:10.3390/healthcare9020105
Novel clinical and radiological measures of lower limb obesity
Akehurst, H., Crosswell, S., Maska, D., et al. (2021) · European Journal of Orthopaedic Surgery & Traumatology · doi:10.1007/s00590-020-02865-2
Repeatability of the Data from Social Interactions Experiments Depending on Personal Characteristics
Алексеевна, М. Е., Анатольевич, К. И., Савостьянов, А. Н., Bocharov, A. B., & Knyazev, G. G. (2020) · Vestnik NSU Series Information Technologies · doi:10.25205/1818-7900-2020-18-3-34-43
Yeung, S. E. (2014) · Summit (Simon Fraser University)
Sterkele, I., Pfister, P. B., Knols, R. H., et al. (2021) · physioscience · doi:10.1055/a-1201-6872
Polek, E., Balfour, A., Williams, A., & Monger, E. (2021) · PubMed · doi:10.1891/jnm-d-19-00073
Park, S., Lee, K., Lee, C.-S., Kim, K.-T., Kim, D. H., Lee, C. H., & Kim, G.-L. (2021) · Journal of orthopaedic surgery · doi:10.1177/2309499020983038
Yang, V. Y., Eaton, J. S., Hetzel, S., et al. (2021) · Veterinary Ophthalmology · doi:10.1111/vop.12877
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
