The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 6,801–6,900 of 23,882
Mitry, P., Wawro, N., Six-Merker, J., Zoller, D., Jourdan, C., Meisinger, C., Thierry, S., Nöthlings, U., Knüppel, S., Boeing, H., et al. (2019) · Frontiers in Nutrition · doi:10.3389/fnut.2019.00145 · Cited by 40
Predicting Exercise With a Personality Facet: Planfulness and Goal Achievement
Ludwig, R. M., Srivastava, S., & Berkman, E. T. (2019) · Psychological Science · doi:10.1177/0956797619868812 · Cited by 40
A Novel Approach for Manual Segmentation of the Amygdala and Hippocampus in Neonate MRI
Hashempour, N., Tuulari, J. J., Merisaari, H., Lidauer, K., Luukkonen, I., Saunavaara, J. P., Parkkola, R. K., Lähdesmäki, T., Lehtola, S. J., Keskinen, M., et al. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.01025 · Cited by 40
Zhang, Y., Jiang, H., Wu, B., Fan, Y., & Ji, Q. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00082 · Cited by 37
Sarcopenia and Back Muscle Degeneration as Risk Factors for Back Pain: A Comparative Study
Kim, W. J., Kim, K. J., Song, D. G., Lee, J. S., Park, K. Y., Lee, J., Chang, S. H., & Choy, W. S. (2020) · Asian Spine Journal · doi:10.31616/asj.2019.0125 · Cited by 40
Thyreau, B., & Taki, Y. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101639 · Cited by 38
A Robotic Framework to Facilitate Sensory Experiences for Children with Autism Spectrum Disorder
Javed, H., Burns, R. B., Jeon, M., Howard, A., & Park, C. H. (2019) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3359613 · Cited by 46
Test-retest reliability of fMRI-measured brain activity during decision making under risk
Korucuoglu, O., Harms, M. P., Astafiev, S. V., Kennedy, J. T., Golosheykin, S., Barch, D. M., & Anokhin, A. P. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116759 · Cited by 39
Test-retest reliability of the computer-assisted DIA-X-5 interview for mental disorders
Hoyer, J., Voß, C., Strehle, J., Venz, J., Pieper, L., Wïttchen, H., Ehrlich, S., & Beesdo‐Baum, K. (2020) · BMC Psychiatry · doi:10.1186/s12888-020-02653-6 · Cited by 43
Garmendia, A., Elorza, U., Aritzeta, A., & Madinabeitia, D. (2020) · Human Resource Management Journal · doi:10.1111/1748-8583.12307 · Cited by 40
The YOUth cohort study: MRI protocol and test-retest reliability in adults
Buimer, E. E. L., Pas, P., Brouwer, R. M., Froeling, M., Hoogduin, H. M., Leemans, A., Luijten, P. R., van Nierop, B. J., Raemaekers, M. A. H., Schnack, H. G., et al. (2020) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2020.100816 · Cited by 38
A novel index of functional connectivity: phase lag based on Wilcoxon signed rank test
Li, X., Wu, Y., Wei, M., Guo, Y., Yu, Z., Wang, H., Li, Z., & Fan, H. (2020) · Cognitive Neurodynamics · doi:10.1007/s11571-020-09646-x · Cited by 41
Dickinson, A., Donovan‐Hall, M., Kheng, S., et al. (2020) · JPO Journal of Prosthetics and Orthotics · doi:10.1097/jpo.0000000000000350 · Cited by 40
Jahanger, Q. K., Louis, J., Pestana, C., & Trejo, D. (2021) · Journal of Information Technology in Construction · doi:10.36680/j.itcon.2021.001 · Cited by 40
Yu, B., & Zadorozhnyy, A. (2021) · ReCALL · doi:10.1017/s0958344021000136 · Cited by 40
Model‐Based Treatment of Rapid Guessing
Deribo, T., Kroehne, U., & Goldhammer, F. (2021) · Journal of Educational Measurement · doi:10.1111/jedm.12290 · Cited by 40
Winkler, C., King, M., Berthe, J., Ferraioli, D., Garuti, A., Grillo, F., Rodriguez‐Canales, J., Ferrando, L., Chopin, N., Ray-Coquard, I. L., et al. (2021) · JCI Insight · doi:10.1172/jci.insight.146098 · Cited by 43
Ren, J., Yuan, Y., & Tao, X. (2021) · European Radiology · doi:10.1007/s00330-021-08310-0 · Cited by 40
Das Instrument zur Kodierung von Diskussionen (IKD)
Schermuly, C. C., Schröder, T., Nachtwei, J., & Schöll, W. (2010) · Zeitschrift für Arbeits- und Organisationspsychologie A&O · doi:10.1026/0932-4089/a000026 · Cited by 39
On the relationship between pain intensity and postural sway in patients with non-specific neck pain
Ruhe, A., Fejer, R., & Walker, B. F. (2013) · Journal of Back and Musculoskeletal Rehabilitation · doi:10.3233/bmr-130399 · Cited by 40
Huntley, C. D., Salmon, P., Fisher, P., Fletcher, I. J., & Young, B. (2012) · Medical Education · doi:10.1111/j.1365-2923.2011.04162.x · Cited by 40
Schneider, G. M., Smith, A. D., Hooper, A., et al. (2010) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-11-29 · Cited by 40
Pelvic Bone Asymmetry in 323 Study Participants Receiving Abdominal CT Scans
Badii, M., Shin, S. S., Torreggiani, W. C., et al. (2003) · Spine · doi:10.1097/01.brs.0000065480.44620.c5 · Cited by 40
Comparison of two in vivo measurements for skin surface topography
Kottner, J., Schario, M., Bartels, N. G., et al. (2012) · Skin Research and Technology · doi:10.1111/srt.12009 · Cited by 40
Eghlileb, A. M., Basra, M. K. A., & Finlay, A. Y. (2009) · Dermatology · doi:10.1159/000209234 · Cited by 40
Spence, G. M., Hocking, R., Wedge, J. H., et al. (2009) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.h.01392 · Cited by 40
Reliability of the Gross Motor Function Measure for Children with Osteogenesis Imperfecta
Ruck, J., Plotkin, H., Hanley, J. A., et al. (2001) · Pediatric Physical Therapy · doi:10.1097/00001577-200104000-00003 · Cited by 40
Southam‐Gerow, M. A., Flannery-Schroeder, E. C., & Kendall, P. C. (2002) · Journal of Anxiety Disorders · doi:10.1016/s0887-6185(02)00223-2 · Cited by 46
Liu, Y., & Salvendy, G. (2009) · Ergonomics · doi:10.1080/00140130802392999 · Cited by 40
Lowit, A., Dobinson, C., Timmins, C., Howell, P., & Kröger, B. J. (2010) · International Journal of Speech-Language Pathology · doi:10.3109/17549507.2010.497559 · Cited by 40
Validation of the Contextual Assessment Inventory for Problem Behavior
Carr, E. G., Ladd, M. V., & Schulte, C. (2008) · Journal of Positive Behavior Interventions · doi:10.1177/1098300707312543 · Cited by 40
Lukowitsky, M. R., & Pincus, A. L. (2011) · Psychoanalytic Psychology · doi:10.1037/a0020849 · Cited by 39
Waite, M., Theodoros, D., Russell, T. G., & Cahill, L. (2012) · Journal of Telemedicine and Telecare · doi:10.1258/jtt.2012.111116 · Cited by 40
Measurement of Functional Activity in Chronic Obstructive Pulmonary Disease
Hill, C. J., Denehy, L., Holland, A. E., & McDonald, C. F. (2008) · Journal of Cardiopulmonary Rehabilitation and Prevention · doi:10.1097/hcr.0b013e31818c3c65 · Cited by 42
Doohan, E.-A. M., Carrère, S., & Riggs, M. L. (2010) · Journal of Family Communication · doi:10.1080/15267430903396401 · Cited by 40
Cruz, B. D. P., Adsuar, J. C., Parraça, J. A., del Pozo-Cruz, J., Moreno, A., & Gusi, N. (2012) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2012.3980 · Cited by 40
Monitoring system for electrical signals in plants in the greenhouse and its applications
Wang, Z., Leng, Q., Huang, L., et al. (2009) · Biosystems Engineering · doi:10.1016/j.biosystemseng.2009.01.013 · Cited by 40
Quantification of the Three-Dimensional Displacement of Normal Facial Movement
Coulson, S. E., Croxson, G. R., & Gilleard, W. (2000) · Annals of Otology Rhinology & Laryngology · doi:10.1177/000348940010900507 · Cited by 40
Ly, L. P., & Handelsman, D. J. (2002) · Clinical and Experimental Pharmacology and Physiology · doi:10.1046/j.1440-1681.2002.03606.x · Cited by 40
Blunted cardiovascular reactivity in dysphoria during reward and punishment anticipation
Franzén, J., & Brinkmann, K. (2014) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2014.11.007 · Cited by 40
MARSH, R., Alexander, G. M., Packard, M. G., Hongtu Zhu, B., & PETERSON, B. (2005) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2004.12.012 · Cited by 43
Iezzi, R., Cotroneo, A. R., Filippone, A., et al. (2008) · Journal of Computer Assisted Tomography · doi:10.1097/rct.0b013e31814b271d · Cited by 40
Boyer, F. C., Morrone, I., Laffont, I., Dizien, O., Étienne, J., & Novella, J. (2006) · Neuromuscular Disorders · doi:10.1016/j.nmd.2005.11.002 · Cited by 40
Fitness in paradise: Quality of forensic reports submitted to the Hawaii judiciary
Robinson, R., & Acklin, M. W. (2010) · International Journal of Law and Psychiatry · doi:10.1016/j.ijlp.2010.03.001 · Cited by 40
Motivational Interviewing as a Supervision Strategy in Probation: A Randomized Effectiveness Trial
Walters, S. T., Vader, A. M., Nguyen, N., Harris, T. R., & Eells, J. (2010) · Journal of Offender Rehabilitation · doi:10.1080/10509674.2010.489455 · Cited by 40
Diamond, L. M., & Cribbet, M. R. (2012) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2012.08.008 · Cited by 43
Khoudri, I., Belayachi, J., Dendane, T., et al. (2012) · BMC Research Notes · doi:10.1186/1756-0500-5-56 · Cited by 40
Facial Growth in Patients with Apert and Crouzon Syndromes Compared to Normal Children
Reitsma, J., Ongkosuwito, E., Buschang, P., & Prahl-Andersen, B. (2012) · The Cleft Palate-Craniofacial Journal · doi:10.1597/10-021 · Cited by 44
Maurer, S. D. (2002) · Personnel Psychology · doi:10.1111/j.1744-6570.2002.tb00112.x · Cited by 40
Atri, A., O’Brien, J. L., Sreenivasan, A., Rastegar, S., Salisbury, S. L., DeLuca, A. N., O’Keefe, K., LaViolette, P. S., Rentz, D. M., Locascio, J. J., et al. (2011) · Archives of Neurology · doi:10.1001/archneurol.2011.94 · Cited by 40
McClellan, H. L., Sakalidis, V. S., Hepworth, A. R., et al. (2010) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2010.08.005 · Cited by 40
Speech motor correlates of treatment-related changes in stuttering severity and speech naturalness
Tasko, S. M., McClean, M. D., & Runyan, C. M. (2006) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2006.04.002 · Cited by 41
Krishnan, K., Mukhtar, S. F., Lingard, J., Houlton, A. E., Walker, E., Jones, T., Sprigg, N., Cala, L., Becker, J., Dineen, R. A., et al. (2015) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp-2014-309845 · Cited by 42
Jiang, L., & Probst, T. M. (2015) · Work & Stress · doi:10.1080/02678373.2015.1032384 · Cited by 46
Eichner, C., Jafari‐Khouzani, K., Cauley, S., Bhat, H., Polaskova, P., Andronesi, O. C., Rapalino, O., Turner, R., Wald, L. L., Stufflebeam, S. M., et al. (2013) · Magnetic Resonance in Medicine · doi:10.1002/mrm.24960 · Cited by 40
Reliability of Rating Visible Landscape Qualities
Palmer, J. F. (2000) · Landscape Journal · doi:10.3368/lj.19.1-2.166 · Cited by 40
Rinderknecht, M. D., Popp, W., Lambercy, O., et al. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00316 · Cited by 40
Competence to Stand Trial: Clinician Reliability and the Role of Offense Severity
Rosenfeld, B. D., & Ritchie, K. (1998) · Journal of Forensic Sciences · doi:10.1520/jfs16102j · Cited by 40
Werner‐Wilson, R. J., Price, S. J., Zimmerman, T. S., et al. (1997) · Journal of Family Psychology · doi:10.1037//0893-3200.11.3.373 · Cited by 40
Infants' Sensitivity to Manipulations of Maternal Touch During Face‐to‐Face Interactions*
Stack, D. M., & LePage, D. E. (1996) · Social Development · doi:10.1111/j.1467-9507.1996.tb00071.x · Cited by 40
Strength Training Can Have Unexpected Effects on the Self-Concept of Children with Cerebral Palsy
Dodd, K., Taylor, N. F., & Graham, H. K. (2004) · Pediatric Physical Therapy · doi:10.1097/01.pep.0000127566.90996.50 · Cited by 40
The structured interview as a tool for predicting premature withdrawal from medical school
POWIS, D. A., Bristow, T., Waring, T. C., et al. (1992) · Australian and New Zealand Journal of Medicine · doi:10.1111/j.1445-5994.1992.tb04872.x · Cited by 40
Wilczek, A., Weinryb, R. M., Barber, J. P., Gustavsson, P. J., & Åsberg, M. (2000) · Psychotherapy Research · doi:10.1093/ptr/10.1.100 · Cited by 40
Shoulder Rotator Torques in Female Tennis and Nontennis Players
Ng, L. R., & Kramer, J. (1991) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1991.13.1.40 · Cited by 40
Validity of the Superiority and Goal Instability Scales as Measures of Defects in the Self
Robbins, S. B. (1989) · Journal of Personality Assessment · doi:10.1207/s15327752jpa5301_13 · Cited by 40
Morgenstern, J., Kuerbis, A., Houser, J., Levak, S., Amrhein, P., Shao, S., & McKay, J. R. (2017) · Psychology of Addictive Behaviors · doi:10.1037/adb0000317 · Cited by 39
Psychosocial predictors of primiparous breastfeeding initiation and duration
Mathews, M. E., Leerkes, E. M., Lovelady, C. A., & Labban, J. D. (2014) · Journal of Human Lactation · doi:10.1177/0890334414537707 · Cited by 39
Evaluating the Quality of Learning Resources: A Learnersourcing Approach
Abdi, S., Khosravi, H., Sadiq, S., & Demartini, G. (2021) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2021.3058644 · Cited by 40
Ronald, A., Happé, F., Dworzynski, K., Bolton, P., & Plomin, R. (2010) · Child Development · doi:10.1111/j.1467-8624.2009.01387.x · Cited by 39
The Reliability of Clinical Diagnoses: State of the Art
Kraemer, H. C. (2014) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032813-153739 · Cited by 39
Zhu, Y., Gao, Y., Guo, C., Qi, M., Xiao, M., Wu, H., Ma, J., Zhong, Q., Ding, H., Zhou, Q., et al. (2022) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2022.771413 · Cited by 41
Hancock, F., Cabral, J., Luppi, A. I., Rosas, F. E., Mediano, P. A. M., Dipasquale, O., & Turkheimer, F. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119433 · Cited by 36
Preschoolers’ executive functions following indoor and outdoor free play
Koepp, A. E., Gershoff, E. T., Castelli, D. M., & Bryan, A. E. (2022) · Trends in Neuroscience and Education · doi:10.1016/j.tine.2022.100182 · Cited by 54
Ori, A. P. S., Lu, A. T.-H., Horvath, S., & Ophoff, R. A. (2022) · Genome biology · doi:10.1186/s13059-022-02793-w · Cited by 42
Nelson, A. E., Liu, F., Lynch, J. A., et al. (2013) · Arthritis Care & Research · doi:10.1002/acr.22094 · Cited by 39
Xu, K., Huang, K.-F., & Xu, E.-M. (2014) · R and D Management · doi:10.1111/radm.12087 · Cited by 37
Calleja, A. G., Poulin, F., Woo, A., Meineri, M., Jedrzkiewicz, S., Vannan, M. A., Rakowski, H., David, T. E., Tsang, W., & Thavendiranathan, P. (2015) · Journal of the American Society of Echocardiography · doi:10.1016/j.echo.2015.04.019 · Cited by 38
Saey, D., Lépine, P.-A., Ribeiro, F. A., Garceau-Bolduc, C., Coats, V., Maltais, F., & Allard, E. (2015) · International Journal of COPD · doi:10.2147/copd.s81806 · Cited by 39
Jambor, I., Merisaari, H., Aronen, H. J., et al. (2013) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.24271 · Cited by 39
Cross-cultural variation of memory colors of familiar objects
Smet, K. A. G., Lin, Y., Nagy, B. V., et al. (2014) · Optics Express · doi:10.1364/oe.22.032308 · Cited by 39
The posterior parahippocampal gyrus is preferentially affected in age-related memory decline
Burgmans, S., van Boxtel, M. P. J., van den Berg, K. E. M., et al. (2009) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2009.09.008 · Cited by 39
The theory-of-mind network in support of action verb comprehension: Evidence from an fMRI study
Lin, N., Bi, Y., Zhao, Y., Luo, C., & Li, X. (2014) · Brain and Language · doi:10.1016/j.bandl.2014.11.004 · Cited by 41
Barchard, K. A. (2011) · Psychological Methods · doi:10.1037/a0023351 · Cited by 39
Bone marrow fat is increased in chronic kidney disease by magnetic resonance spectroscopy
Moorthi, R. N., Fadel, W. F., Eckert, G. J., Ponsler-Sipes, K., Moe, S. M., & Lin, C. (2015) · Osteoporosis International · doi:10.1007/s00198-015-3064-7 · Cited by 38
Reproducibility of a knee and hip proprioception test in healthy older adults
Arvin, M., Hoozemans, M. J. M., Burger, B. J., et al. (2014) · Aging Clinical and Experimental Research · doi:10.1007/s40520-014-0255-6 · Cited by 39
A More Reliable Method to Assess Acetabular Component Position
Tiberi, J. V., Pulos, N. A., Kertzner, M., et al. (2011) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-011-2006-8 · Cited by 39
Tashiro, Y., Matsuda, S., Okazaki, K., et al. (2014) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-014-3248-2 · Cited by 39
Occlusal tooth wear in the general population of Germany: Effects of age, sex, and location of teeth
Schierz, O., Dommel, S., Hirsch, C., et al. (2014) · Journal of Prosthetic Dentistry · doi:10.1016/j.prosdent.2013.12.005 · Cited by 39
Savelsbergh, E. R., de Jong, T., & Ferguson‐Hessler, M. G. M. (2011) · Physical Review Special Topics - Physics Education Research · doi:10.1103/physrevstper.7.010103 · Cited by 39
The intra-rater reliability and agreement of compensatory stepping thresholds of healthy subjects
Crenshaw, J. R., & Kaufman, K. R. (2013) · Gait & Posture · doi:10.1016/j.gaitpost.2013.11.006 · Cited by 37
Edgar, C. J., Blaettler, T., Bugarski‐Kirola, D., Scouiller, S. L., Garibaldi, G. M., & Marder, S. R. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.04.009 · Cited by 40
Data Reliability From an Instrumented Vertical Jump Platform
Caruso, J. F., Daily, J. S., McLagan, J. R., et al. (2010) · The Journal of Strength and Conditioning Research · doi:10.1519/jsc.0b013e3181b66679 · Cited by 39
Reliability and reproducibility of perfusion MRI in cognitively normal subjects
Jiang, L., Kim, M., Chodkowski, B., Donahue, M. J., Pekar, J. J., van Zijl, P. C. M., & Albert, M. (2010) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2010.05.002 · Cited by 39
Accuracy of the glenohumeral subluxation index in nonpathologic shoulders
Jacxsens, M., Tongel, A. V., Willemot, L., et al. (2014) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2014.07.021 · Cited by 39
Youdas, J. W., Haeflinger, K. M., Kreun, M. K., Holloway, A. M., Krämer, C., & Hollman, J. H. (2010) · Physiotherapy Theory and Practice · doi:10.3109/09593980903015292 · Cited by 39
Immediate effects of AAF devices on the characteristics of stuttering: A clinical analysis
Unger, J. P., Glück, C. W., & Cholewa, J. (2012) · Journal of Fluency Disorders · doi:10.1016/j.jfludis.2012.02.001 · Cited by 39
Imputation of response rates from means and standard deviations in schizophrenia
Samara, M. T., Spineli, L. M., Furukawa, T. A., Engel, R. R., Davis, J. M., Salanti, G., & Leucht, S. (2013) · Schizophrenia Research · doi:10.1016/j.schres.2013.10.029 · Cited by 39
Carotid plaque morphometric assessment with three-dimensional ultrasound imaging
AlMuhanna, K., Hossain, M. M., Zhao, L., Fischell, J. M., Kowalewski, G., Dux, M. C., Sikdar, S., & Lal, B. K. (2014) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2014.10.003 · Cited by 40
BDNF val66met polymorphism influences age differences in microstructure of the corpus callosum
Kennedy, K. M. (2009) · Frontiers in Human Neuroscience · doi:10.3389/neuro.09.019.2009 · Cited by 39
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
