The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 3,601–3,700 of 23,883
Fayad, F., Lefèvre‐Colau, M., Macé, Y., Fermanian, J., Mayoux-Benhamou, A., Rören, A., Rannou, F., Roby-Brami, A., Gautheron, V., Revel, M., et al. (2007) · Joint Bone Spine · doi:10.1016/j.jbspin.2007.04.023 · Cited by 79
Simoneau, G. G., Marklin, R. W., & Berman, J. E. (2003) · Physical Therapy · doi:10.1093/ptj/83.9.816 · Cited by 80
Preferences for female body size in Britain and the South Pacific
Swami, V., Knight, D., Tovée, M. J., Davies, P., & Furnham, A. F. (2007) · Body Image · doi:10.1016/j.bodyim.2007.01.002 · Cited by 80
Busse, M. E., Hughes, G., Wiles, C. M., & Rosser, A. (2008) · Journal of Neurology · doi:10.1007/s00415-008-0964-x · Cited by 81
Validated Composite Assessment Scales for the Global Face
Rzany, B., CARRUTHERS, A. J., Carruthers, J., et al. (2012) · Dermatologic Surgery · doi:10.1111/j.1524-4725.2011.02252.x · Cited by 80
Improvement in the Interpersonal Communication Skills of Dental Students
Hottel, T. L., & Hardigan, P. C. (2005) · Journal of Dental Education · doi:10.1002/j.0022-0337.2005.69.2.tb03914.x · Cited by 80
Assessment of pulse rate variability by the method of pulse frequency demodulation
Hayano, J., Barros, A. K., Kamiya, A., Ohte, N., & Yasuma, F. (2005) · BioMedical Engineering OnLine · doi:10.1186/1475-925x-4-62 · Cited by 80
Children’s engagement and competence in personal recollection: Effects of parents’ reminiscing goals
Cleveland, E. S., Reese, E., & Grolnick, W. S. (2006) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2006.09.003 · Cited by 80
Psychophysiological characteristics of narcissism during active and passive coping
Kelsey, R. M., Ornduff, S. R., McCann, C. M., & Reiff, S. (2001) · Psychophysiology · doi:10.1111/1469-8986.3820292 · Cited by 80
An exploration of psychological factors on emoticon usage and implications for judgement accuracy
Wall, H. J., Kaye, L. K., & Malone, S. A. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.03.040 · Cited by 82
Lee, L., Liao, Y.-S., Lu, H.-K., Hsiao, P.-L., Chen, Y.-Y., Ching-Chi, C., & Hsieh, K.-C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0171568 · Cited by 84
The development of a magnetic resonance imaging index for fistulising Crohn's disease
Samaan, M., Puylaert, C. A. J., Levesque, B. G., Zou, G., Stitt, L., Taylor, S. A., Shackelton, L. M., Vandervoort, M. K., Khanna, R., Santillan, C., et al. (2017) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.14190 · Cited by 81
Empowerment, span of control, and safety performance in work teams after workforce reduction.
Hechanova-Alampay, R. M., & Beehr, T. A. (2001) · Journal of Occupational Health Psychology · doi:10.1037//1076-8998.6.4.275 · Cited by 84
Brewer, B. W., Raalte, J. L. V., Petitpas, A. J., Sklar, J. H., Pohlman, M. H., Krushell, R. J., Ditmar, T. D., Daly, J. M., & Weinstock, J. (2000) · Physical Therapy in Sport · doi:10.1054/ptsp.2000.0019 · Cited by 81
Cognitive correlates to social cue perception in schizophrenia
Corrigan, P. W., Green, M. F., & Toomey, R. (1994) · Psychiatry Research · doi:10.1016/0165-1781(94)90105-8 · Cited by 80
Verbal response mode use in contrasting psychotherapies: A within-subjects comparison.
Stiles, W. B., Shapiro, D. A., & Firth‐Cozens, J. (1988) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.56.5.727 · Cited by 80
Roberts, C. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19991015)18:19 · Cited by 81
Klein, D. N., & Shih, J. H. (1998) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.107.2.319 · Cited by 80
Measurement of health outcomes in the clinical setting: applications to physiotherapy
Beattie, P. (2001) · Physiotherapy Theory and Practice · doi:10.1080/095939801317077632 · Cited by 80
Mottet, B., Aptel, F., Romanet, J.-P., Hubanova, R., Pépin, J.-L., & Chiquet, C. (2013) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2013.5297 · Cited by 94
Lim, S. F. W. T., & Srai, J. S. (2018) · International Journal of Operations & Production Management · doi:10.1108/ijopm-12-2016-0733 · Cited by 92
Kurtz, J. E., Lee, P. A., & Sherker, J. L. (1999) · Assessment · doi:10.1177/107319119900600201 · Cited by 79
Phenotypic diversity in white adults with moderate to severe Class III malocclusion
Uribe, L. M. M., Vela, K. C., Kummet, C. M., et al. (2013) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2013.02.019 · Cited by 79
Relationship Between Tightness of the Hip Joint and Elbow Pain in Adolescent Baseball Players
Saito, M., Kenmoku, T., Kameyama, K., et al. (2014) · Orthopaedic Journal of Sports Medicine · doi:10.1177/2325967114532424 · Cited by 79
Category assignment and relatedness in the group ideation process
Baruah, J., & Paulus, P. B. (2011) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2011.04.007 · Cited by 85
The neural basis of conceptualizing the same action at different levels of abstraction
Spunt, R. P., Kemmerer, D. L., & Adolphs, R. (2015) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsv084 · Cited by 76
Individualizing the Tibial Tubercle–Trochlear Groove Distance
Camp, C. L., Heidenreich, M. J., Dahm, D. L., et al. (2015) · The American Journal of Sports Medicine · doi:10.1177/0363546515602483 · Cited by 79
Fetterman, A. K., Bair, J. L., Werth, M., Landkammer, F., & Robinson, M. D. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000067 · Cited by 81
Boiché, J., Gourlan, M., Trouilloud, D. O., & Sarrazin, P. G. (2016) · Journal of Health Psychology · doi:10.1177/1359105316676626 · Cited by 90
Ediz, B., Özçakır, A., & Bilgel, N. (2017) · Cogent Psychology · doi:10.1080/23311908.2017.1283829 · Cited by 79
Meng, J., Peng, W., Tan, P., Liu, W., Cheng, Y., & Bae, A. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.07.039 · Cited by 79
Validation of the SOS-PD scale for assessment of pediatric delirium: a multicenter study
Ista, E., van Beusekom, B. S., van Rosmalen, J., Kneyber, M. C. J., Lemson, J., Brouwers, A., Dieleman, G. C., Dierckx, B., de Hoog, M., Tibboel, D., et al. (2018) · Critical Care · doi:10.1186/s13054-018-2238-z · Cited by 82
CROCUFID: A Cross-Cultural Food Image Database for Research on Food Elicited Affective Responses
Toet, A., Kaneko, D., de Kruijf, I., Ushiama, S., van Schaik, M. G., Brouwer, A.-M., Kallen, V., & van Erp, J. B. F. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00058 · Cited by 83
Development, validation and reliability of the Chrononutrition Profile - Questionnaire
Veronda, A. C., Allison, K. C., Crosby, R. D., & Irish, L. A. (2019) · Chronobiology International · doi:10.1080/07420528.2019.1692349 · Cited by 85
Bhimani, R., Ashkani‐Esfahani, S., Lubberts, B., et al. (2020) · Foot & Ankle International · doi:10.1177/1071100720917682 · Cited by 79
How To Build a Better Robot . . . for Quick-Service Restaurants
Zemke, D., Tang, J., Raab, C., & Kim, J. (2020) · Journal of Hospitality & Tourism Research · doi:10.1177/1096348020946383 · Cited by 75
Multidisciplinary digital competencies of pre-service vocational teachers
Roll, M. J. J., & Ifenthaler, D. (2021) · Empirical research in vocational education and training · doi:10.1186/s40461-021-00112-4 · Cited by 107
Morbée, L., Chen, M., Herregods, N., et al. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109999 · Cited by 79
Lehmann, R., Thiessen, C., Frick, B., Bosse, H. M., Nikendei, C., Hoffmann, G. F., Tönshoff, B., & Huwendiek, S. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4141 · Cited by 82
Turban, D. B., Jones, A. P., & Rozelle, R. M. (1990) · Motivation and Emotion · doi:10.1007/bf00995570 · Cited by 79
Riddle, D. L., Lee, K. T., & Stratford, P. W. (2001) · Medical Care · doi:10.1097/00005650-200108000-00012 · Cited by 80
Illies, J. J., & Reiter‐Palmon, R. (2007) · Journal of Business Ethics · doi:10.1007/s10551-007-9574-2 · Cited by 78
PERFORMANCE APPRAISAL OF BEHAVIOR‐BASED COMPETENCIES: A RELIABLE AND VALID PROCEDURE
Catano, V. M., Darr, W. A., & CAMPBELL, C. A. (2007) · Personnel Psychology · doi:10.1111/j.1744-6570.2007.00070.x · Cited by 79
Meyer, B. M., & Schermuly, C. C. (2011) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2011.560383 · Cited by 79
Hip Abductor and Knee Extensor Muscle Strength of Children with and without Down Syndrome
Mercer, V. S., & Lewis, C. L. (2001) · Pediatric Physical Therapy · doi:10.1097/00001577-200104000-00004 · Cited by 79
Hollen, P. J., Gralla, R. J., Kris, M. G., McCoy, S., Donaldson, G. W., & Moinpour, C. M. (2005) · Quality of Life Research · doi:10.1007/s11136-004-0833-8 · Cited by 79
Negahban, H., Mostafaee, N., Sohani, S. M., Mazaheri, M., Goharpey, S., Salavati, M., Zahednejad, S., Meshkati, Z., & Montazeri, A. (2011) · Disability and Rehabilitation · doi:10.3109/09638288.2011.570409 · Cited by 80
Periodontitis in psoriasis patients. A blinded, case-controlled study
Preus, H. R., Khanifam, P., Kolltveit, K. M., Mørk, C., & Gjermo, P. E. (2010) · Acta Odontologica Scandinavica · doi:10.3109/00016350903583678 · Cited by 79
Hand injuries and cold sensitivity: Reliability and validity of cold sensitivity questionnaires
Carlsson, I., Cederlund, R., Höglund, P., Lundborg, G., & Rósen, B. (2008) · Disability and Rehabilitation · doi:10.1080/09638280701679705 · Cited by 85
Test-retest reliability of knee kinesthesia in healthy adults
Ageberg, E., Flenhagen, J., & Ljung, J. (2007) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-8-57 · Cited by 79
Armitage, M. S., Elkinson, I., Giles, J. W., et al. (2011) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2011.06.020 · Cited by 79
Peeler, J. D., & Anderson, J. E. (2008) · Journal of Athletic Training · doi:10.4085/1062-6050-43.5.470 · Cited by 83
Jenkins, N. D. M., Miller, J. M., Buckner, S. L., Cochrane, K. C., Bergstrom, H. C., Hill, E. C., Smith, C. M., Housh, T. J., & Cramer, J. T. (2015) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2015.01.017 · Cited by 80
Williams, V. S. L., Smith, M. Y., & Fehnel, S. E. (2006) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2005.06.008 · Cited by 79
The Impact of Level of Personality Organization on Treatment Response: A Systematic Review
Koelen, J. A., Luyten, P., Eurelings‐Bontekoe, L., Diguer, L., Vermote, R., Lowyck, B., & Bühring, M. E. F. (2012) · Psychiatry · doi:10.1521/psyc.2012.75.4.355 · Cited by 86
Differential effects of age and history of hypertension on regional brain volumes and iron
Rodrigue, K. M., Haacke, E. M., & Raz, N. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.09.068 · Cited by 80
Alexanderson, H., Broman, L., Tollbäck, A., Josefson, A., Lundberg, I. E., & Opava, C. H. (2006) · Arthritis Care & Research · doi:10.1002/art.21715 · Cited by 81
Butler, A. J., Cazeaux, J., Fidler, A., Jansen, J., Lefkove, N., Gregg, M. J., Hall, C. R., Easley, K. A., Shenvi, N. V., & Wolf, S. L. (2012) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2012/497289 · Cited by 81
Dantas, E. M., Sant'Anna, M. L., Andreão, R. V., Gonçalves, C. P., Morra, E. A., Baldo, M. P., Rodrigues, S. L., & Mill, J. G. (2011) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2011.11.004 · Cited by 91
Creation and Initial Evaluation of a Stool Form Scale for Children
Chumpitazi, B. P., Lane, M. M., Czyzewski, D. I., Weidler, E. M., Swank, P. R., & Shulman, R. J. (2010) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2010.04.040 · Cited by 79
A Comparison of Perceptually-Based Metrics for Objective Evaluation of Geometry Processing
Lavoué, G., & Corsini, M. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2060475 · Cited by 79
Kerns, J. G., & Becker, T. M. (2008) · Schizophrenia Research · doi:10.1016/j.schres.2007.11.005 · Cited by 84
Sandu, A., Rasmussen, I.-A., Lundervold, A., Kreuder, F., Neckelmann, G., Hugdahl, K., & Specht, K. (2007) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2007.10.005 · Cited by 79
Six-minute walk test in adults with cerebral palsy. A study of reliability
Andersson, C., Asztalos, L., & Mattsson, E. (2006) · Clinical Rehabilitation · doi:10.1191/0269215506cr964oa · Cited by 79
The Contact Lens Impact on Quality of Life (CLIQ) Questionnaire: Development and Validation
Pesudovs, K., Garamendi, E., & Elliott, D. B. (2006) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.05-0933 · Cited by 82
Validity and Reliability of the First Distal Metatarsal Articular Angle
Vittetoe, D. A., Saltzman, C. L., Krieg, J. C., et al. (1994) · Foot & Ankle International · doi:10.1177/107110079401501004 · Cited by 79
A configuration of sustainable sourcing and supply management strategies
Akhavan, R. M., & Beckmann, M. (2016) · Journal of Purchasing and Supply Management · doi:10.1016/j.pursup.2016.07.006 · Cited by 92
Socioeconomic status and hippocampal volume in children and young adults
Yu, Q., Daugherty, A. M., Anderson, D. M., Nishimura, M., Brush, D., Hardwick, A., Lacey, W. H., Raz, S., & Ofen, N. (2017) · Developmental Science · doi:10.1111/desc.12561 · Cited by 78
Wixted, J. T., Vul, E., Mickes, L., & Wilson, B. M. (2018) · Cognitive Psychology · doi:10.1016/j.cogpsych.2018.06.001 · Cited by 79
Rogachefsky, R. A., Lipson, S. R., Applegate, B. E., et al. (2001) · Journal of Bone and Joint Surgery · doi:10.2106/00004623-200104000-00005 · Cited by 79
Okugawa, G., Nobuhara, K., Minami, T., Takase, K., Sugimoto, T., Saito, Y., Yoshimura, M., & Kinoshita, T. (2006) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2006.05.014 · Cited by 81
Bird, P., Lassere, M. N., Shnier, R., & Edmonds, J. P. (2003) · Arthritis & Rheumatism · doi:10.1002/art.10820 · Cited by 78
Quantitative assessment of myocardial scar in delayed enhancement magnetic resonance imaging
Setser, R. M., Bexell, D., O’Donnell, T., Stillman, A. E., Lieber, M. L., Schoenhagen, P., & White, R. D. (2003) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.10391 · Cited by 80
Haley, S. M., Raczek, A. E., Coster, W. J., Dumas, H. M., & Fragala-Pinkham, M. A. (2005) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2004.10.032 · Cited by 79
Effectiveness of Visual Feedback During Isokinetic Exercise
Kim, H. J., & Kramer, J. F. (1997) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1997.26.6.318 · Cited by 79
MRI of human entorhinal cortex: a reliable protocol for volumetric measurement
Goncharova, I. I., Dickerson, B. C., Stoub, T. R., & deToledo‐Morrell, L. (2001) · Neurobiology of Aging · doi:10.1016/s0197-4580(01)00270-6 · Cited by 80
Differences in Elbow Flexion Torque Measured Concentrically, Eccentrically, and Isometrically
Griffin, J. W. (1987) · Physical Therapy · doi:10.1093/ptj/67.8.1205 · Cited by 79
Gordis, E. B., Margolin, G., & John, R. S. (1997) · Journal of Family Psychology · doi:10.1037/0893-3200.11.1.76 · Cited by 79
Personality in the chimpanzees of Gombe National Park
Weiß, A., Wilson, M. L., Collins, D., Mjungu, D. C., Kamenya, S. M., Foerster, S., & Pusey, A. E. (2017) · Scientific Data · doi:10.1038/sdata.2017.146 · Cited by 79
Thompson-Brenner, H., Boswell, J. F., Espel-Huynh, H., Brooks, G., & Lowe, M. R. (2019) · Psychotherapy Research · doi:10.1080/10503307.2018.1446563 · Cited by 80
Assessing fidelity in individual and family therapy for adolescent substance abuse
Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C., Inclan, J., Reiner, R. H., & Liddle, H. A. (2008) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2007.09.002 · Cited by 78
Cukor, J., Wyka, K., Jayasinghe, N., Weathers, F. W., Giosan, C., Leck, P., Roberts, J., Spielman, L. A., Crane, M., & Difede, J. (2010) · Depression and Anxiety · doi:10.1002/da.20776 · Cited by 78
Peeler, J. D., & Anderson, J. E. (2008) · Journal of Orthopaedic Research® · doi:10.1002/jor.20556 · Cited by 78
Creative thinking in music: Its nature and assessment through musical exploratory behaviors.
Barbot, B., & Lubart, T. (2012) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0027307 · Cited by 78
Reliability and Validity of Finger Strength and Endurance Measurements in Rock Climbing
Michailov, M. L., Baláš, J., Tanev, S. K., Andonov, H., Kodejška, J., & Brown, L. E. (2018) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.2018.1441484 · Cited by 78
Bracke, P., Delaruelle, K., & Verhaeghe, M. (2019) · Frontiers in Sociology · doi:10.3389/fsoc.2019.00040 · Cited by 79
Dowling, C., Bavo, A. M., Faquir, N. E., Mortier, P., de Jaegere, P. P. T., Backer, O. D., Søndergaard, L., Ruile, P., Mylotte, D., McConkey, H. Z. R., et al. (2019) · Circulation Cardiovascular Imaging · doi:10.1161/circimaging.119.009178 · Cited by 77
Local Brain-Age: A U-Net Model
Popescu, S. G., Glocker, B., Sharp, D. J., & Cole, J. H. (2021) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2021.761954 · Cited by 77
Mindfulness enhances cognitive functioning: a meta-analysis of 111 randomized controlled trials
Zainal, N. H., & Newman, M. G. (2023) · Health Psychology Review · doi:10.1080/17437199.2023.2248222 · Cited by 106
Bojić, L., Zagovora, O., Zelenkauskaitė, A., Vuković, V., Čabarkapa, M., Jerković, S. V., & Jovančević, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-96508-3 · Cited by 119
Mullix, J., Warner, M. B., & Stokes, M. J. (2012) · ePrints Soton (University of Southampton) · Cited by 77
Effect of Peripheral Afferent Alteration of the Lateral Ankle Ligaments on Dynamic Stability
Myers, J. B., Riemann, B. L., Hwang, J.-H., Fu, F. H., & Lephart, S. M. (2003) · The American Journal of Sports Medicine · doi:10.1177/03635465030310040401 · Cited by 78
Smith, P. M., Saunders, R., Lifshen, M., Black, O., Lay, A. M., Breslin, F. C., LaMontagne, A. D., & Tompa, E. (2015) · Accident Analysis & Prevention · doi:10.1016/j.aap.2015.06.004 · Cited by 78
Schumacker, R. E., & Smith, E. V. (2007) · Educational and Psychological Measurement · doi:10.1177/0013164406294776 · Cited by 78
Stability and variability may respond differently to changes in walking speed
Li, L., Haddad, J. M., & Hamill, J. (2005) · Human Movement Science · doi:10.1016/j.humov.2005.03.003 · Cited by 78
Enlargement of Thalamic Nuclei in Tourette Syndrome
Miller, A. M., Bansal, R., Hao, X., Sanchez‐Peña, J., Sobel, L. J., Liu, J., Xu, D., Hongtu Zhu, B., Chakravarty, M. M., Durkin, K., et al. (2010) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2010.102 · Cited by 78
Suspicious young minds: paranoia and mistrust in 8- to 14-year-olds in the UK and Hong Kong
Wong, K. K., Freeman, D. H., & Hughes, C. H. (2014) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.113.135467 · Cited by 82
Dubi, K., Rapee, R. M., Emerton, J. L., & Schniering, C. A. (2007) · Research on Child and Adolescent Psychopathology · doi:10.1007/s10802-007-9195-3 · Cited by 91
Tensor Based Singular Spectrum Analysis for Automatic Scoring of Sleep EEG
Kouchaki, S., Sanei, S., Arbon, E. L., et al. (2014) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2014.2329557 · Cited by 78
Reliability of assessing the coronal curvature of children with scoliosis by using ultrasound images
Chen, W., Lou, E. H. M., Zhang, P. Q., et al. (2013) · Journal of Children s Orthopaedics · doi:10.1007/s11832-013-0539-y · Cited by 78
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
