The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 3,201–3,300 of 23,883
Albi, A., Pasternak, O., Minati, L., Marizzoni, M., David Bartrés-faz, A., Bargalló, N., Bosch, B., Rossini, P. M., Marra, C., Müller, B. W., et al. (2016) · Human Brain Mapping · doi:10.1002/hbm.23350 · Cited by 90
Resting-state fMRI correlations: From link-wise unreliability to whole brain stability
Pannunzi, M., Hindriks, R., Bettinardi, R. G., Wenger, E., Lisofsky, N., Mårtensson, J., Butler, O., Filevich, E., Becker, M., Lochstet, M., et al. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.006 · Cited by 93
A comparison of computer-assisted and manual wound size measurement.
Thawer, H. A., Houghton, P. E., Woodbury, M. G., et al. (2002) · PubMed · Cited by 90
Age-Related Regional Differences in Cerebellar Vermis Observed In Vivo
Raz, N., Torres, I. J., Spencer, W. D., White, K., & Acker, J. D. (1992) · Archives of Neurology · doi:10.1001/archneur.1992.00530280106030 · Cited by 90
Hening, W. A., Allen, R. P., Thanner, S., et al. (2003) · Sleep Medicine · doi:10.1016/s1389-9457(03)00006-6 · Cited by 90
Heenan, P. J., Matz, L. R., Blackwell, J. B., Kelsall, G. R. H., SINGH, A., Seldam, R. E. J. T., & Holman, C. (1984) · Histopathology · doi:10.1111/j.1365-2559.1984.tb02388.x · Cited by 90
Akers, J. F., Jones, R. M., & Coyl, D. D. (1998) · Journal of Adolescent Research · doi:10.1177/0743554898132005 · Cited by 90
Lenzenweger, M. F., Dworkin, R. H., & Wethington, E. (1991) · Schizophrenia Bulletin · doi:10.1093/schbul/17.3.515 · Cited by 89
Reliability of pain threshold measurement in young adults
Cathcart, S., & Pritchard, D. (2006) · The Journal of Headache and Pain · doi:10.1007/s10194-006-0265-7 · Cited by 110
Weiß, A., Staes, N., Pereboom, Z., Inoue‐Murayama, M., Stevens, J. M. G., & Eens, M. (2015) · Psychological Science · doi:10.1177/0956797615589933 · Cited by 89
Belief into Action Scale: A Brief but Comprehensive Measure of Religious Commitment
Koenig, H. G., Nelson, B., Shaw, S. F., Zaben, F. A., Wang, Z., & Saxena, S. (2015) · Open Journal of Psychiatry · doi:10.4236/ojpsych.2015.51010 · Cited by 93
Levy, P. T., Holland, M. R., Sekarski, T. J., Hamvas, A., & Singh, G. K. (2013) · Journal of the American Society of Echocardiography · doi:10.1016/j.echo.2013.06.005 · Cited by 90
Parental writing support and preschoolers’ early literacy, language, and fine motor skills
Bindman, S. W., Skibbe, L. E., Hindman, A. H., Aram, D., & Morrison, F. J. (2014) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2014.07.002 · Cited by 89
Santoro, G. A., Wieczorek, A. P., Shobeiri, S. A., et al. (2010) · International Urogynecology Journal · doi:10.1007/s00192-010-1233-y · Cited by 89
Inscoe, A. B., Heilbron, N., Ames, A., & Prinstein, M. J. (2011) · Developmental Psychology · doi:10.1037/a0022546 · Cited by 89
The development of personality across the lifespan
Donnellan, M. B., & Robins, R. W. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9780511596544.015 · Cited by 89
Durão, A. R., Morosolli, A. R. C., Pittayapat, P., et al. (2015) · Imaging Science in Dentistry · doi:10.5624/isd.2015.45.4.213 · Cited by 89
Quantitative MRI reveals decelerated fatty infiltration in muscles of active FSHD patients
Janssen, B. H., Voet, N. B. M., Geurts, A. C. H., et al. (2016) · Neurology · doi:10.1212/wnl.0000000000002640 · Cited by 89
Investigating the accuracy of a novel telehealth diagnostic approach for autism spectrum disorder.
Smith, C. J., Rozga, A., Matthews, N. L., Oberleitner, R., Nazneen, N., & Abowd, G. D. (2016) · Psychological Assessment · doi:10.1037/pas0000317 · Cited by 104
Weingärtner, S., Meßner, N. M., Budjan, J., et al. (2016) · Journal of Cardiovascular Magnetic Resonance · doi:10.1186/s12968-016-0302-x · Cited by 89
Feely, M., Seay, K. D., Lanier, P., Auslander, W., & Kohl, P. L. (2017) · Child and Adolescent Social Work Journal · doi:10.1007/s10560-017-0512-6 · Cited by 90
Vocal similarity predicts the relative attraction of musical chords
Bowling, D. L., Purves, D., & Gill, K. Z. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1713206115 · Cited by 89
Like mother, like child? Maternal determinants of children's early social‐emotional development
Behrendt, H. F., Scharke, W., Herpertz‐Dahlmann, B., Konrad, K., & Firk, C. (2019) · Infant Mental Health Journal · doi:10.1002/imhj.21765 · Cited by 96
Fatigue Detection Using Artificial Intelligence Framework
Parekh, V., Shah, D., & Shah, M. (2019) · Augmented Human Research · doi:10.1007/s41133-019-0023-4 · Cited by 92
Yeung, A., Iaboni, A., Rochon, E. A., Lavoie, M., Santiago, C., Yancheva, M., Novikova, J., Xu, M., Robin, J., Kaufman, L. D., et al. (2021) · Alzheimer s Research & Therapy · doi:10.1186/s13195-021-00848-x · Cited by 90
Dimensions of family–school partnerships for autistic children: Context and congruence.
Santiago, R. T., McIntyre, L. L., & Garbacz, S. A. (2021) · School Psychology · doi:10.1037/spq0000473 · Cited by 89
Maillard, P., Lu, H., Arfanakis, K., Gold, B. T., Bauer, C. E., Zachariou, V., Stables, L. A., Wang, D. J. J., Jann, K., Seshadri, S. S., et al. (2022) · Alzheimer s & Dementia Diagnosis Assessment & Disease Monitoring · doi:10.1002/dad2.12261 · Cited by 88
Eckner, J. T., Kutcher, J. S., & Richardson, J. K. (2011) · Journal of Athletic Training · doi:10.4085/1062-6050-46.4.409 · Cited by 89
A Strategy Tripod Perspective on Knowledge Creation Capability
Su, Z., Peng, M. W., & Xie, E. (2015) · British Journal of Management · doi:10.1111/1467-8551.12097 · Cited by 91
Metachromatic leukodystrophy: natural course of cerebral MRI changes in relation to clinical course
Groeschel, S., Kehrer, C., Engel, C., Dali, C. Í., Bley, A. E., Steinfeld, R., Grodd, W., & Krägeloh‐Mann, I. (2011) · Journal of Inherited Metabolic Disease · doi:10.1007/s10545-011-9361-1 · Cited by 89
Testing and Training Anticipation Skills in Softball Fielders
Gabbett, T. J., Rubinoff, M., Thorburn, L., et al. (2007) · International Journal of Sports Science & Coaching · doi:10.1260/174795407780367159 · Cited by 89
Group Efficacy And Group Effectiveness
Pescosolido, A. T. (2003) · Small Group Research · doi:10.1177/1046496402239576 · Cited by 89
Whitman, J. M., Cleland, J. A., Mintken, P. E., Keirns, M., Bieniek, M. L., Albin, S. R., Magel, J. S., & McPoil, T. G. (2008) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2009.2940 · Cited by 89
Cross‐situational coping with peer and family stressors in adolescent offspring of depressed parents
Jaser, S. S., Champion, J. E., Reeslund, K. L., Keller, G., Merchant, M. J., Benson, M., & Compas, B. E. (2007) · Journal of Adolescence · doi:10.1016/j.adolescence.2006.11.010 · Cited by 87
Personality in felids: A review
Gartner, M. C., & Weiß, A. (2013) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2012.11.010 · Cited by 89
von Garnier, K., Köveker, K., Rackwitz, B., Kober, U., Wilke, S., Ewert, T., & Stucki, G. (2009) · Physiotherapy · doi:10.1016/j.physio.2008.10.003 · Cited by 89
Teaching brief motivational interviewing to Year three medical students
Martino, S., Haeseler, F., Belitsky, R., et al. (2007) · Medical Education · doi:10.1111/j.1365-2929.2006.02673.x · Cited by 89
Comparison of Psychometric Properties of Three Mobility Measures for Patients With Stroke
Hsueh, I., Wang, C.-H., Sheu, C., et al. (2003) · Stroke · doi:10.1161/01.str.0000075295.45185.d4 · Cited by 89
Comparison of ICC and CCC for assessing agreement for data without and with replications
Chen, C., & Barnhart, H. X. (2008) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2008.09.026 · Cited by 89
Use of the TAT in the Assessment of DSM-IV Cluster B Personality Disorders
Ackerman, S. J., Clemence, A. J., Weatherill, R. P., & Hilsenroth, M. J. (1999) · Journal of Personality Assessment · doi:10.1207/s15327752jpa7303_9 · Cited by 104
Ng, G. K. C., Lamontagne, M., Adamczyk, A. P., Rahkra, K. S., & Beaulé, P. E. (2014) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-014-3797-1 · Cited by 90
Krosschell, K. J., Maczulski, J. A., Crawford, T. O., Scott, C., & Swoboda, K. J. (2006) · Neuromuscular Disorders · doi:10.1016/j.nmd.2006.03.015 · Cited by 89
Payan, C., Viallet, F., Landwehrmeyer, G. B., Bonnet, A.-M., Borg, M., Durif, F., Lacomblez, L., Bloch, F., Verny, M., Fermanian, J., et al. (2011) · PLoS ONE · doi:10.1371/journal.pone.0022293 · Cited by 88
Effects of Exercise on Hip Range of Motion, Trunk Muscle Performance, and Gait Economy
Godges, J. J., MacRae, P. G., & Engelke, K. A. (1993) · Physical Therapy · doi:10.1093/ptj/73.7.468 · Cited by 89
Longitudinal reliability of tract‐based spatial statistics in diffusion tensor imaging
Madhyastha, T. M., Mérillat, S., Hirsiger, S., Bezzola, L., Liem, F., Grabowski, T. J., & Jäncke, L. (2014) · Human Brain Mapping · doi:10.1002/hbm.22493 · Cited by 94
Measuring Children's Decision-Making Involvement Regarding Chronic Illness Management
Miller, V. A., & Harris, D. (2011) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsr097 · Cited by 89
Brann, P., Coleman, G. J., & Luk, E. (2001) · Australian & New Zealand Journal of Psychiatry · doi:10.1046/j.1440-1614.2001.00890.x · Cited by 89
Reliability of an interview approach to the Functional Independence Measure
Daving, Y., Andrén, E., Nordholm, L. A., & Grimby, G. (2001) · Clinical Rehabilitation · doi:10.1191/026921501669986659 · Cited by 89
HERMANN, K. M., & Reese, C. S. (2001) · Physical Therapy · doi:10.1093/ptj/81.3.903 · Cited by 89
Measuring cognitive insight in schizophrenia and bipolar disorder: a comparative study
Engh, J. A., Friis, S., Birkenaes, A. B., Jónsdóttir, H., Ringen, P. A., Ruud, T., Sundet, K. S., Opjordsmoen, S., & Andreassen, O. A. (2007) · BMC Psychiatry · doi:10.1186/1471-244x-7-71 · Cited by 93
Human Brain: Reliability and Reproducibility of Pulsed Arterial Spin-labeling Perfusion MR Imaging
Jahng, G., Song, E., Zhu, X., Matson, G. B., Weiner, M. W., & Schuff, N. W. (2005) · Radiology · doi:10.1148/radiol.2343031499 · Cited by 89
Mannerkorpi, K., & Hernelid, C. (2005) · Disability and Rehabilitation · doi:10.1080/09638280400009063 · Cited by 88
Design Based on Intra-Class Correlation Coefficients
Lew (2010) · Current Research in Biostatistics · doi:10.3844/amjbsp.2010.1.8 · Cited by 87
Larivière, C., Gagnon, D., & Loisel, P. (2000) · Clinical Biomechanics · doi:10.1016/s0268-0033(00)00006-1 · Cited by 89
Protas, E. J., Stanley, R. K., Jankovic, J. J., & MacNeill, B. (1996) · Physical Therapy · doi:10.1093/ptj/76.1.34 · Cited by 89
Comparison of Retinal Nerve Fiber Layer Measurement Between 2 Spectral Domain OCT Instruments
Tan, B. B., Natividad, M. G., Chua, K., & Yip, L. W. L. (2011) · Journal of Glaucoma · doi:10.1097/ijg.0b013e3182071cdd · Cited by 88
Horowitz, L. M., French, R. D. S., Wallis, K. D., Post, D. L., & Siegelman, E. Y. (1981) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.90.6.575 · Cited by 89
Schudlich, T. D. R., & Cummings, E. M. (2003) · Child Development · doi:10.1046/j.1467-8624.2003.00630.x · Cited by 93
Interjudgmental reliability of reviews for the Journal of Educational Psychology.
Marsh, H. W., & Ball, S. (1981) · Journal of Educational Psychology · doi:10.1037/0022-0663.73.6.872 · Cited by 89
Greenwell, T. N., Funk, C. K., Cottone, P., Richardson, H. N., Chen, S. A., Rice, K. C., Zorrilla, E. P., & Koob, G. F. (2009) · Addiction Biology · doi:10.1111/j.1369-1600.2008.00142.x · Cited by 87
Hogue, A., Liddle, H. A., Rowe, C., Turner, R. M., Dakof, G. A., & LaPann, K. (1998) · Journal of Counseling Psychology · doi:10.1037/0022-0167.45.1.104 · Cited by 89
A twin study of autism symptoms in Sweden
Ronald, A., Larsson, H., Anckarsäter, H., & Lichtenstein, P. (2011) · Molecular Psychiatry · doi:10.1038/mp.2010.82 · Cited by 88
Dolphens, M., Cagnie, B., Coorevits, P., Vanderstraeten, G., Cardon, G., Dʼhooge, R., & Danneels, L. (2012) · Spine · doi:10.1097/brs.0b013e3182408053 · Cited by 88
Therapist responsiveness to child engagement: flexibility within manual‐based CBT for anxious youth
Chu, B. C., & Kendall, P. C. (2009) · Journal of Clinical Psychology · doi:10.1002/jclp.20582 · Cited by 97
Chow, B. J. W., Kass, M., Gagné, O., et al. (2011) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2010.09.072 · Cited by 88
Long‐term clinical outcome of schizophrenia with special reference to gender differences
Opjordsmoen, S. (1991) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1991.tb05545.x · Cited by 88
Entis, J. J., Doerga, P. N., Barrett, L. F., & Dickerson, B. C. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.12.073 · Cited by 91
Common variation near ROBO2 is associated with expressive vocabulary in infancy
Pourcain, B. S., Cents, R. A. M., Whitehouse, A., Haworth, C. M. A., Davis, O. S. P., O’Reilly, P. F., Roulstone, S., Wren, Y. E., Ang, Q. W., Velders, F. P., et al. (2014) · Nature Communications · doi:10.1038/ncomms5831 · Cited by 88
The effect of internal factors and family influence on firms’ adoption of green product innovation
Huang, Y., Yang, M.-L., & Wong, Y. (2016) · Management Research Review · doi:10.1108/mrr-02-2015-0031 · Cited by 94
Chronset: An automated tool for detecting speech onset
Roux, F., Armstrong, B. C., & Carreiras, M. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0830-1 · Cited by 88
Wu, J., Hsiao, Y.-C., & Nian, M.-W. (2018) · Interactive Learning Environments · doi:10.1080/10494820.2018.1515085 · Cited by 89
Computer Vision Analysis for Quantification of Autism Risk Behaviors
Hashemi, J., Dawson, G., Carpenter, K. L. H., Campbell, K. M., Qiu, Q., Espinosa, S., Marsan, S., Baker, J. P., EGGER, H. L., & Sapiro, G. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2868196 · Cited by 91
Non-invasive recording from the human olfactory bulb
Iravani, B., Arshamian, A., Ohla, K., Wilson, D. A., & Lundström, J. N. (2020) · Nature Communications · doi:10.1038/s41467-020-14520-9 · Cited by 94
Pelfort, X., Torres‐Claramunt, R., Sánchez‐Soler, J., et al. (2015) · Orthopaedics & Traumatology Surgery & Research · doi:10.1016/j.otsr.2015.03.016 · Cited by 88
Comparison of Automated Brain Volume Measures obtained with NeuroQuant® and FreeSurfer
Ochs, A. L., Ross, D. E., Zannoni, M. D., Abildskov, T. J., & Bigler, E. D. (2015) · Journal of Neuroimaging · doi:10.1111/jon.12229 · Cited by 88
A Biomechanical Study of Distal Interphalangeal Joint Subluxation After Mallet Fracture Injury
Husain, S. N., Dietz, J. F., Kalainov, D. M., et al. (2008) · The Journal Of Hand Surgery · doi:10.1016/j.jhsa.2007.09.006 · Cited by 88
Attachment loss with postmenopausal age and smoking
Hildebolt, C. F., Pilgram, T. K., Dotson, M., Yokoyama‐Crothers, N., Muckerman, J., Mauser, J., Cohen, S. C., Kardaris, E. E., Vannier, M. W., Hanes, P. J., et al. (1997) · Journal of Periodontal Research · doi:10.1111/j.1600-0765.1997.tb00940.x · Cited by 88
The Effect of Residential Neighborhood on Child Behavior Problems in First Grade
Caughy, M. O., Nettles, S. M., & O'Campo, P. J. (2008) · American Journal of Community Psychology · doi:10.1007/s10464-008-9185-9 · Cited by 88
Zlobec, I., Steele, R. J., Michel, R. P., et al. (2006) · Modern Pathology · doi:10.1038/modpathol.3800642 · Cited by 88
Reliability of sagittal photographic spinal posture assessment in adolescents
Perry, M. C., Smith, A. J., Straker, L. M., Coleman, J. L., & O’Sullivan, P. (2008) · Advances in Physiotherapy · doi:10.1080/14038190701728251 · Cited by 88
Schanche, E., Stiles, T. C., McCullough, L., Svartberg, M., & Nielsen, G. H. (2011) · Psychotherapy · doi:10.1037/a0022012 · Cited by 87
Bilateral Q angle asymmetry and anterior knee pain syndrome
Livingston, L. A., & Mandigo, J. (1999) · Clinical Biomechanics · doi:10.1016/s0268-0033(98)00045-x · Cited by 88
Fehlings, M. G., Furlan, J. C., Massicotte, E. M., et al. (2006) · Spine · doi:10.1097/01.brs.0000224164.43912.e6 · Cited by 88
Interrater Reliability of Posture Observations
Bao, S. S., Howard, N. L., Spielholz, P. O., et al. (2009) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720809340273 · Cited by 88
Risk Factors and Estimation Tool for Death Among Extremely Premature Infants: A National Study
Bader, D., Kugelman, A., Boyko, V., Levitzki, O., Lerner‐Geva, L., Riskin, A., & Reichman, B. L. (2010) · PEDIATRICS · doi:10.1542/peds.2009-1607 · Cited by 88
The Relationship of Lumbar Flexion to Disability in Patients With Low Back Pain
Sullivan, M. S., Shoaf, L. D., & Riddle, D. L. (2000) · Physical Therapy · doi:10.1093/ptj/80.3.240 · Cited by 88
Lawrence, R. L., Braman, J. P., Staker, J. L., et al. (2014) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2014.5556 · Cited by 88
Faber, G. S., Chang, C.-C., Kingma, I., Dennerlein, J. T., & van Dieën, J. H. (2015) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2015.11.042 · Cited by 88
Pham, T. T. T., Moore, S. T., Lewis, S. J. G., Nguyen, D. N., Dutkiewicz, E., Fuglevand, A. J., McEwan, A. L., & Leong, P. H. W. (2017) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2017.2665438 · Cited by 88
Evaluation of different recall periods for the US National Cancer Institute’s PRO-CTCAE
Mendoza, T. R., Dueck, A. C., Bennett, A. V., Mitchell, S. A., Reeve, B. B., Atkinson, T. M., Li, Y., Castro, K. M., Denicoff, A. M., Rogak, L. J., et al. (2017) · Clinical Trials · doi:10.1177/1740774517698645 · Cited by 88
Curtis, J. T., Silberschatz, G., Sampson, H., & Weiss, J. (1994) · Psychotherapy Research · doi:10.1080/10503309412331334032 · Cited by 88
On What It Means to Know Someone: A Matter of Pragmatics.
Gill, M. J., & Swann, W. B. (2004) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.86.3.405 · Cited by 88
Yao, S., Zou, T., Zhu, X., Abela, J. R. Z., Auerbach, R. P., & Tong, X. (2007) · Child Psychiatry and Human Development · doi:10.1007/s10578-006-0039-0 · Cited by 85
Larsson, H., Tegern, M., Monnier, A., Skoglund, J., Helander, C., Persson, E., Malm, C., Broman, L., & Aasa, U. (2015) · PLoS ONE · doi:10.1371/journal.pone.0132185 · Cited by 86
Everyday Decision-Making Ability in Older Persons With Cognitive Impairment
Lai, J. M., Gill, T. M., Cooney, L. M., et al. (2008) · American Journal of Geriatric Psychiatry · doi:10.1097/jgp.0b013e31816c7b54 · Cited by 87
Reliability and validity of lower extremity computed tomography as a screening tool for osteoporosis
Lee, S. Y., Kwon, S.-S., Kim, H.-S., et al. (2015) · Osteoporosis International · doi:10.1007/s00198-014-3013-x · Cited by 87
Facial Expressivity at 4 Months: A Context by Expression Analysis
Bennett, D. S., Bendersky, M., & Lewis, M. (2002) · Culture Studies ↔ Critical Methodologies · doi:10.1207/s15327078in0301_5 · Cited by 93
Köhncke, Y., Laukka, E. J., Brehmer, Y., Kalpouzos, G., Li, T., Fratiglioni, L., Bäckman, L., & Lövdén, M. (2016) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2016.02.013 · Cited by 89
Lassen, P. B., Charlotte, F., Liu, Y., Bédossa, P., Naour, G. L., Tordjman, J., Poitou, C., Bouillot, J., Genser, L., Zucker, J., et al. (2017) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2017-00138 · Cited by 94
Stacchi, C., Lombardi, T., Ottonelli, R., et al. (2018) · Clinical Oral Implants Research · doi:10.1111/clr.13144 · Cited by 87
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
