The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,342
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 8,401–8,500 of 23,881
Variability of Neutral-Position Casting of the Foot
Chuter, V., Payne, C., & Miller, K. (2003) · Journal of the American Podiatric Medical Association · doi:10.7547/87507315-93-1-1 · Cited by 30
Quantifying variation in human scalp hair fiber shape and pigmentation
Lasisi, T., Ito, S., Wakamatsu, K., & Shaw, C. N. (2016) · American Journal of Physical Anthropology · doi:10.1002/ajpa.22971 · Cited by 33
Shoulder Arthroplasty for Traumatic Avascular Necrosis
Tauber, M., Karpik, S., Matis, N., et al. (2007) · Clinical Orthopaedics and Related Research · doi:10.1097/blo.0b013e318159cb7a · Cited by 30
Pain Threshold Tests in Patients With Heel Pain Syndrome
Saban, B., & Masharawi, Y. (2016) · Foot & Ankle International · doi:10.1177/1071100716642038 · Cited by 30
Schaben, J. A., Welk, G. J., Joens-Matre, R., & Hensley, L. D. (2006) · Journal of Physical Activity and Health · doi:10.1123/jpah.3.1.59 · Cited by 30
Knobloch-Fedders, L. M., Caska, C. M., Smith, T. W., & Renshaw, K. D. (2016) · Behavior Therapy · doi:10.1016/j.beth.2016.08.014 · Cited by 30
Peker, K., Köse, T. E., Güray, B., et al. (2017) · Acta Odontologica Scandinavica · doi:10.1080/00016357.2016.1278079 · Cited by 30
Kim, S., Lee, E., Nam, S. J., et al. (2017) · PLoS ONE · doi:10.1371/journal.pone.0176103 · Cited by 30
Koppelmans, V., Pasternak, O., Bloomberg, J. J., Dios, Y. E. D., Wood, S. J., Riascos, R., Reuter‐Lorenz, P. A., Kofman, I. S., Mulavara, A. P., & Seidler, R. D. (2017) · Scientific Reports · doi:10.1038/s41598-017-03311-w · Cited by 31
Wang, F., Wu, L., Xiao-lan, H., et al. (2017) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.25839 · Cited by 30
Almeida, J., Greenberg, T., Lu, H., et al. (2017) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2017.09.004 · Cited by 30
Martin, S. S., Wichmann, J. L., Pfeifer, S., et al. (2017) · European Journal of Radiology · doi:10.1016/j.ejrad.2017.10.008 · Cited by 30
Peng, K., Yücel, M. A., Aasted, C. M., Steele, S. C., Boas, D. A., Borsook, D., & Becerra, L. R. (2017) · Neurophotonics · doi:10.1117/1.nph.5.1.011018 · Cited by 31
Vreugdenhil, J., & Spek, B. (2017) · Nurse Education Today · doi:10.1016/j.nedt.2017.12.013 · Cited by 30
Expressed parental concern regarding childhood stuttering and the Test of Childhood Stuttering
Tumanova, V., Choi, D., Conture, E. G., et al. (2018) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2018.01.002 · Cited by 30
Matsuishi, Y., Hoshino, H., Shimojo, N., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0194094 · Cited by 30
Costa, Y. M., de Araújo-Júnior, E. N. S., Fiedler, L. S., de Souza, P. R. J., Silva, L. L. C. P., Ferreira, D. M. A. O., Conti, P. C. R., & Bonjardim, L. R. (2018) · European Journal of Pain · doi:10.1002/ejp.1287 · Cited by 30
Cross-cultural emotion recognition and evaluation of Radboud faces database with an Indian sample
Mishra, M., Ray, S. B., & Srinivasan, N. (2018) · PLoS ONE · doi:10.1371/journal.pone.0203959 · Cited by 30
Chen, L., Shen, F., Li, Z., et al. (2018) · Cancer Imaging · doi:10.1186/s40644-018-0177-1 · Cited by 30
Weightbearing CT International Study Group (WBCT ISG), Burssens, A., Barg, A., et al. (2019) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-019-01949-7 · Cited by 30
Bui, K.-L., Mathur, S., Dechman, G., Maltais, F., Camp, P. G., & Saey, D. (2019) · Physical Therapy · doi:10.1093/ptj/pzz059 · Cited by 30
Hagemeijer, N. C., Saengsin, J., Chang, S. H., et al. (2020) · Injury · doi:10.1016/j.injury.2020.07.060 · Cited by 30
Synonym Suggestion for Tags on Stack Overflow
Beyer, S., & Pinzger, M. (2015) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2015.18 · Cited by 30
Intra-tester and inter-tester reliability of the MicroFET 3 hand-held dynamometer
Clarke, M., Mhuircheartaigh, D. N., Gm, W., et al. (2011) · Physiotherapy Practice and Research · doi:10.3233/ppr-2011-32103 · Cited by 30
Clinical Evidence of Subcutaneous CO 2 Insufflations: A Systematic Review
Brockow, T., Hausner, T., Dillner, A., & Resch, K. (2000) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2000.6.391 · Cited by 29
GUIDELINES FOR MEASUREMENT VALIDATION IN CLINICAL TRIAL DESIGN
Sanchez, M. M., & Binkowitz, B. S. (1999) · Journal of Biopharmaceutical Statistics · doi:10.1081/bip-100101185 · Cited by 30
Chen, X., Zheng, H., Sussman, S. Y., Gong, J., Stacy, A. W., Xia, J., Gallaher, P. E., Dent, C. W., Azen, S. P., Shan, J., et al. (2002) · Psychology of Addictive Behaviors · doi:10.1037//0893-164x.16.3.260 · Cited by 31
Effect of Running on Volume of the Foot and Ankle
Cloughley, W. B., & Mawdsley, R. H. (1995) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1995.22.4.151 · Cited by 30
Schuster, C., & Smith, D. A. (2005) · Psychometrika · doi:10.1007/s11336-003-1110-4 · Cited by 30
Waldron, S., Scharf, R. D., Crouse, J. H., et al. (2004) · The Psychoanalytic Quarterly · doi:10.1002/j.2167-4086.2004.tb00193.x · Cited by 30
Team negotiation and perceptions of trustworthiness: The whole versus the sum of the parts.
Naquin, C. E., & Kurtzberg, T. R. (2009) · Group Dynamics Theory Research and Practice · doi:10.1037/a0013879 · Cited by 28
Huinck, W. J., Langevin, M., Kully, D., Graamans, K., Peters, H. F. M., & Hulstijn, W. (2006) · Journal of Fluency Disorders · doi:10.1016/j.jfludis.2005.12.001 · Cited by 30
Nieto, R. G., Blasco-Fontecilla, H., Yepes, M. P., & Baca‐García, E. (2013) · Revista de Psiquiatría y Salud Mental (English Edition) · doi:10.1016/j.rpsmen.2012.07.004 · Cited by 30
Psychometric Properties of the Chinese Version of the Psycho-educational Profile-Revised (CPEP-R)
Shek, D. T. L., Tsang, S., Lam, L. L., Tang, F. L. Y., & Cheung, P. M. P. (2005) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-004-1029-3 · Cited by 30
Lacoste, M., Therrien, M., Côté, J. N., Shrier, I., Labelle, H., & Prince, F. (2006) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2006.08.344 · Cited by 30
Dimensions of horse personality based on owner and trainer supplied personality traits
McGrogan, C., Hutchison, M. D., & King, J. E. (2007) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2007.10.006 · Cited by 32
Relations between social skills and ward behavior in chronic schizophrenia
Penn, D. L., Mueser, K. T., Doonan, R. J., & Nishith, P. (1995) · Schizophrenia Research · doi:10.1016/0920-9964(94)00084-l · Cited by 30
Variations in diagnostic criteria for carpal tunnel syndrome among Ontario specialists
Graham, B., Dvali, L. T., Regehr, G., et al. (2005) · American Journal of Industrial Medicine · doi:10.1002/ajim.20240 · Cited by 30
Interrater Reliability of the Fitness Interview Test across 4 Professional Groups
Viljoen, J. L., Roesch, R. G., & Zapf, P. A. (2002) · The Canadian Journal of Psychiatry · doi:10.1177/070674370204701006 · Cited by 30
Inter-rater Reliability and Between-days Repeatability of Eight Physical Performance Tests
Horneij, E., Holmström, E., Hemborg, B., Isberg, P., & Ekdahl, C. S. (2002) · Advances in Physiotherapy · doi:10.1080/14038190260501596 · Cited by 30
Situational Familiarity and Feature Recognition in Schizophrenia
Corrigan, P. W., Silverman, R., Stephenson, J., et al. (1996) · Schizophrenia Bulletin · doi:10.1093/schbul/22.1.153 · Cited by 30
The Influence of Tibialis Anterior Muscle Activity on Rearfoot Motion during Walking
Cornwall, M. W., & McPoil, T. G. (1994) · Foot & Ankle International · doi:10.1177/107110079401500205 · Cited by 30
The Effect of Inter-Trial Rest Interval on the Assessment of Isokinetic Thigh Muscle Torque
Stratford, P. W., Bruulsema, A., Maxwell, B. D., et al. (1990) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1990.11.8.362 · Cited by 30
Seo, B. K., Oh, Y. W., Kim, H. R., et al. (2002) · Korean Journal of Radiology · doi:10.3348/kjr.2002.3.1.38 · Cited by 30
Dilated cisternae chyli: a sign of uncompensated cirrhosis at MR imaging
Verma, S. K., Mitchell, D. G., Bergin, D., Lakhman, Y., Austin, A., Verma, M., Assis, D. N., Herrine, S. K., & Parker, L. (2008) · Abdominal Imaging · doi:10.1007/s00261-008-9369-7 · Cited by 30
Purvis, J. A., Dabbs, J. M., & Hopper, C. H. (1984) · Personality and Social Psychology Bulletin · doi:10.1177/0146167284101006 · Cited by 30
Müller, M. J., Mazánek, M., Weibrich, C., Dellani, P. R., Stoeter, P., & Fellgiebel, A. (2006) · PubMed Central · Cited by 30
Baez, K. A., Orengo, S., Gandham, S. B., et al. (1992) · Ophthalmic surgery, lasers & imaging retina · doi:10.3928/1542-8877-19920601-13 · Cited by 30
Rey, J. M., Stewart, G. W., Plapp, J. M., Bashir, M. R., & Richards, I. (1988) · American Journal of Psychiatry · doi:10.1176/ajp.145.3.286 · Cited by 29
A comparison of alternative assessments of depressive symptom severity: a pilot study
Biggs, M. M., Shores‐Wilson, K., Rush, A. J., Carmody, T. J., Trivedi, M. H., Crismon, M. L., Toprac, M. G., & Mason, M. (2000) · Psychiatry Research · doi:10.1016/s0165-1781(00)00235-3 · Cited by 30
Is the Clinical Concept of Spinal Stiffness Multidimensional?
Maher, C. G., & Adams, R. D. (1995) · Physical Therapy · doi:10.1093/ptj/75.10.854 · Cited by 30
Agoraphobia, depression, mental health locus of control, and attributional styles
Hoffart, A., & Martinsen, E. W. (1990) · Cognitive Therapy and Research · doi:10.1007/bf01184001 · Cited by 30
Walking ability, balance and accidental falls in persons with Multiple Sclerosis
Nilsagård, Y. E. (2008) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 30
Measuring Quality of Delivery in a Substance Use Prevention Program
Giles, S., Jackson-Newsom, J., Pankratz, M. M., Hansen, W. B., Ringwalt, C. L., & Dusenbury, L. (2008) · The Journal of Primary Prevention · doi:10.1007/s10935-008-0155-7 · Cited by 29
Shape and volume of craniofacial cavities in intentional skull deformations
Khonsari, R. H., Friess, M., Nysjö, J., Odri, G., Malmberg, F., Nyström, I., Messo, E., Hirsch, J. M., Cabanis, E. A. M., Kunzelmann, K. H., et al. (2013) · American Journal of Physical Anthropology · doi:10.1002/ajpa.22263 · Cited by 25
Pantalon, M. V., Martino, S., Dziura, J., Li, F.-Y., Owens, P. H., Fiellin, D. A., O'Connor, P. G., & D'Onofrio, G. (2012) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2012.08.011 · Cited by 30
“Based on previous work (Carroll et al., 2000; Martino et al., 2010; Santa Ana et al., 2009), the Shrout and Fleiss (1979) intraclass correlation coefficient (ICC) twoway mixed model, with item ratings as the fixed effect and raters as the random effect, was used to estimate inter-rater reliability…”
MethodKatz-Buonincontro, J. L., Hass, R. W., & Perignat, E. (2019) · The Journal of Creative Behavior · doi:10.1002/jocb.424 · Cited by 31
From Needs Analysis to Task Selection, Design, and Sequencing
Gilabert, R., & Malicka, A. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108868327.014 · Cited by 30
Dai, B., Lin, Y., Lai, X., He, J., Du, M., Hou, X., & Zhang, G. (2024) · BMC Psychiatry · doi:10.1186/s12888-023-05489-y · Cited by 41
Duan, K., Eyler, L. T., Pierce, K., Lombardo, M. V., Datko, M., Hagler, D. J., Taluja, V., Zahiri, J., Campbell, K. M., Barnes, C. C., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-48952-4 · Cited by 28
Peterson, K. L., Sánchez-Romero, R., Mill, R. D., & Cole, M. W. (2025) · Imaging Neuroscience · doi:10.1162/imag.a.162 · Cited by 30
Application Portfolio Management—An Integrated Framework and a Software Tool Evaluation Approach
Simon, D., Fischbach, K., & Schoder, D. (2010) · Communications of the Association for Information Systems · doi:10.17705/1cais.02603 · Cited by 29
Reproducibility of the water drinking test in treated glaucomatous patients
Babic, M., Moraes, C. G. D., Hatanaka, M., et al. (2014) · Clinical and Experimental Ophthalmology · doi:10.1111/ceo.12434 · Cited by 29
Blame it on Hollywood: The influence of films on Paris as product location
Gkritzali, A., Lampel, J., & Wiertz, C. (2015) · Journal of Business Research · doi:10.1016/j.jbusres.2015.10.005 · Cited by 33
Fehr, K. K., & Russ, S. W. (2013) · Journal of Personality Assessment · doi:10.1080/00223891.2013.838171 · Cited by 29
Reproducibility of performance and fatigue in trail running
Easthope, C. A., Nosaka, K., Caillaud, C., et al. (2013) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2013.03.009 · Cited by 29
Properties of the Patient Administered Questionnaires
Mancuso, C. A., Ranawat, A. S., Meftah, M., et al. (2011) · The Journal of Arthroplasty · doi:10.1016/j.arth.2011.07.014 · Cited by 29
Lorenzi, D. D., Masserdotti, C., Bertoncello, D., et al. (2009) · Veterinary Clinical Pathology · doi:10.1111/j.1939-165x.2009.00160.x · Cited by 29
Stasiewicz, P., Vincent, P. C., Bradizza, C., Connors, G., Maisto, S., & Mercer, N. D. (2008) · Psychology of Addictive Behaviors · doi:10.1037/0893-164x.22.1.78 · Cited by 29
Vermote, R., Lowyck, B., Luyten, P., Verhaest, Y., Vertommen, H., Vandeneede, B., Corveleyn, J., & Peuskens, J. (2011) · Clinical Psychology and Psychotherapy · doi:10.1002/cpp.713 · Cited by 29
Gomes, A. F., Nejaim, Y., Brasil, D. M., et al. (2015) · Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.joms.2015.02.020 · Cited by 29
Assessing motivational interviewing integrity for group interventions with adolescents.
D’Amico, E. J., Osilla, K. C., Miles, J. N. V., Ewing, B. A., Sullivan, K., Katz, K., & Hunter, S. B. (2012) · Psychology of Addictive Behaviors · doi:10.1037/a0027987 · Cited by 29
The moderating role of regulatory focus on the social modeling of food intake
Florack, A., Palcu, J., & Friese, M. (2013) · Appetite · doi:10.1016/j.appet.2013.05.012 · Cited by 29
Reliability and validity of the L test in participants with chronic stroke
Kim, J. S., Chu, D. Y., & Jeon, H. (2014) · Physiotherapy · doi:10.1016/j.physio.2014.09.003 · Cited by 30
Comparison of ActiGraph activity monitors in persons with multiple sclerosis and controls
Sandroff, B. M., & Motl, R. W. (2013) · Disability and Rehabilitation · doi:10.3109/09638288.2012.707745 · Cited by 29
Veltman, E. S., van den Bekerom, M. P. J., Doornberg, J. N., et al. (2014) · Injury · doi:10.1016/j.injury.2014.01.022 · Cited by 29
Jaimes, C. E., Berman, J., Delgado, J. A., Ho-Fung, V. M., & Jaramillo, D. (2014) · Radiology · doi:10.1148/radiol.14132136 · Cited by 29
Modified Test Administration Using Assistive Technology
Warschausky, S. A., Tubbergen, M. V., Asbell, S., Kaufman, J. N., Ayyangar, R. N., & Donders, J. (2011) · Assessment · doi:10.1177/1073191111402458 · Cited by 29
Dinkel, P. J., Willmes, K., Krinzinger, H., Konrad, K., & Koten, J. W., Jr. (2013) · PLoS ONE · doi:10.1371/journal.pone.0083722 · Cited by 29
Netson, K. L., Conklin, H. M., Ashford, J. M., Kahalley, L. S., Wu, S., & Xiong, X. (2010) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsq102 · Cited by 29
Honebrink, T., Dayton, V. J., Burke, M. J., et al. (2011) · Transplantation and Cellular Therapy · doi:10.1016/j.bbmt.2011.11.015 · Cited by 29
Validation of tissue microarray technology in malignant pleural mesothelioma
Kao, S., Lee, K. M. L., Armstrong, N. J., et al. (2011) · Pathology · doi:10.1097/pat.0b013e328342016c · Cited by 29
Sex-specific body configurations can be estimated from voice samples.
Hughes, S. M., Harrison, M. A., & Gallup, G. G. (2009) · Journal of Social Evolutionary and Cultural Psychology · doi:10.1037/h0099311 · Cited by 29
Bulut, H. T., Sarıca, M. A., & BAYKAN, A. H. (2014) · PubMed · Cited by 29
Serial Cognition and Personality in Macaques
Altschul, D., Terrace, H. S., & Weiß, A. (2016) · Animal Behavior and Cognition · doi:10.12966/abc.02.04.2016 · Cited by 29
Diffusion Tensor Imaging of the Normal Cervical and Thoracic Pediatric Spinal Cord
Saksena, S., Middleton, D. M., Krisa, L., et al. (2016) · American Journal of Neuroradiology · doi:10.3174/ajnr.a4883 · Cited by 29
Engagement Perception and Generation for Social Robots and Virtual Agents
Corrigan, L. J., Peters, C., Küster, D., & Castellano, G. (2016) · Intelligent systems reference library · doi:10.1007/978-3-319-31056-5_4 · Cited by 32
Creating a school context of success
Mitchell, R. M., Mendiola, B., Schumacker, R. E., & Lowery, X. (2016) · Journal of Educational Administration · doi:10.1108/jea-02-2015-0018 · Cited by 29
Pierre, T. G. S., House, M. J., Bangma, S. J., Pang, W., Bathgate, A. J., Gan, E. K., Ayonrinde, O. T., Bhathal, P. S., Clouston, A. D., Olynyk, J. K., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0160789 · Cited by 29
Inter-rater reliability of distal ureteral diameter ratio compared to grade of VUR
Swanton, A., Arlen, A. M., Alexander, S. E., et al. (2016) · Journal of Pediatric Urology · doi:10.1016/j.jpurol.2016.10.021 · Cited by 29
Standing balance and strength measurements in older adults living in residential care communities
Alqahtani, B. A., Ferchak, M. A., Huppert, T. J., Sejdić, E., Perera, S., Greenspan, S. L., & Sparto, P. J. (2016) · Aging Clinical and Experimental Research · doi:10.1007/s40520-016-0693-4 · Cited by 30
Moreno, F.-Á., Merchán-Baeza, J. A., González‐Sánchez, M., González-Jiménez, J., & Cuesta‐Vargas, A. (2017) · Sensors · doi:10.3390/s17020424 · Cited by 29
Vieira, A., Battini, M., Can, E., Mattiello, S., & Stilwell, G. (2018) · animal · doi:10.1017/s1751731117003597 · Cited by 30
Borg, D. N., Osborne, J. O., Stewart, I. B., et al. (2018) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2018.01.004 · Cited by 29
Patil, S. V. (2018) · Administrative Science Quarterly · doi:10.1177/0001839218783988 · Cited by 32
Yu, K., Wang, Z., & Huang, Y. (2018) · Frontiers of Business Research in China · doi:10.1186/s11782-018-0039-5 · Cited by 41
Lodewijks, P. C. A. M., Delawi, D., Bollen, T. L., et al. (2018) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-018-5214-x · Cited by 29
Within-Group Agreement ( r WG ): Two Theoretical Parameters and their Estimators
Newman, D. A., & Sin, H. (2018) · Organizational Research Methods · doi:10.1177/1094428118809504 · Cited by 30
Nagle, C. (2019) · Journal of Second Language Pronunciation · doi:10.1075/jslp.18016.nag · Cited by 29
Singh, N. K., Govindaswamy, R., & Jagadish, N. (2019) · Journal of the American Academy of Audiology · doi:10.3766/jaaa.17080 · Cited by 29
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
