The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,342
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 9,301–9,400 of 23,881
Santani, D., Ruíz-Correa, S., & Gática-Pérez, D. (2018) · ACM Transactions on Social Computing · doi:10.1145/3224182 · Cited by 25
Impaired Performance of the Smash Stroke in Badminton Induced by Muscle Fatigue
Mansec, Y. L., Perez, J., Rouault, Q., Doron, J., & Jubeau, M. (2019) · International Journal of Sports Physiology and Performance · doi:10.1123/ijspp.2018-0697 · Cited by 25
Boga, A., Stapleton, F. J., Briggs, N., & Golebiowski, B. (2019) · The Ocular Surface · doi:10.1016/j.jtos.2019.06.005 · Cited by 25
Quantifying human presence in a heterogeneous urban landscape
Corsini, M., Marrot, P., & Szulkin, M. (2019) · Behavioral Ecology · doi:10.1093/beheco/arz128 · Cited by 25
Rasmussen, A., Stone, D. U., Kaufman, C. E., et al. (2019) · ACR Open Rheumatology · doi:10.1002/acr2.1033 · Cited by 25
Automated Scoring of Students’ Use of Text Evidence in Writing
Correnti, R. J., Matsumura, L. C., Wang, E. L., Litman, D. J., Rahimi, Z., & Kısa, Z. (2019) · Reading Research Quarterly · doi:10.1002/rrq.281 · Cited by 28
The Relation Between Valence and Arousal in Subjective Odor Experience
Toet, A., Eijsman, S., Liu, Y., Donker, S. F., Kaneko, D., Brouwer, A.-M., & van Erp, J. B. F. (2019) · Chemosensory Perception · doi:10.1007/s12078-019-09275-7 · Cited by 30
Taba, S. T., Baran, P., Nesterets, Y. I., et al. (2020) · European Radiology · doi:10.1007/s00330-019-06567-0 · Cited by 25
Test–retest reliability of tip, key, and palmar pinch force sense in healthy adults
Li, L., Li, Y., Wu, C., & Zhang, X. (2020) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-020-3187-7 · Cited by 25
Jiang, Y., Guo, D., Li, C., Chen, T., & Li, R. (2020) · Radiology of Infectious Diseases · doi:10.1016/j.jrid.2020.05.001 · Cited by 26
Bamdad, S., Momeni‐Moghaddam, H., Abdolahian, M., & Pinero, D. P. (2020) · Journal of Optometry · doi:10.1016/j.optom.2020.08.008 · Cited by 26
Efficace, F., Iurlo, A., Patriarca, A., et al. (2020) · Leukemia & lymphoma/Leukemia and lymphoma · doi:10.1080/10428194.2020.1838509 · Cited by 25
Khac, A. D., Jourdan, C., Fazilleau, S., Palayer, C., Laffont, I., Dupeyron, A., Verdun, S., & Gélis, A. (2021) · JMIR mhealth and uhealth · doi:10.2196/26443 · Cited by 29
Korucuoglu, O., Harms, M. P., Astafiev, S. V., Golosheykin, S., Kennedy, J. T., Barch, D. M., & Anokhin, A. P. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.624911 · Cited by 25
Apps for people with autism: Assessment, classification and ranking of the best
Montes, C. D. P. G., Fuentes, A. R., & Cara, M. J. C. (2021) · Technology in Society · doi:10.1016/j.techsoc.2020.101474 · Cited by 25
Declève, P., CANT, J. V., & Cools, A. M. J. (2021) · Brazilian Journal of Physical Therapy · doi:10.1016/j.bjpt.2021.02.002 · Cited by 27
Diagnostic accuracy of gray scale muscle ultrasound screening for pediatric neuromuscular disease
Boon, A. J., Wijntjes, J., OʼBrien, T. G., et al. (2021) · Muscle & Nerve · doi:10.1002/mus.27211 · Cited by 25
Reliability of IMU-Derived Static Balance Parameters in Neurological Diseases
Hansen, C., Beckbauer, M., Romijnders, R., Warmerdam, E., Welzel, J., Geritz, J., Emmert, K., & Maetzler, W. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18073644 · Cited by 25
Janke, D., Kagelmann, N., Storm, C., Maggioni, M. A., Kienast, C., Gunga, H., & Opatz, O. S. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.666908 · Cited by 25
Kho, D.-H., Cho, B.-K., & Choi, S.-M. (2021) · Foot & Ankle International · doi:10.1177/10711007211017470 · Cited by 25
Oshri, A., Liu, S., Suveg, C. M., Caughy, M. O., & Huffman, L. G. (2021) · Development and Psychopathology · doi:10.1017/s095457942100078x · Cited by 28
Politzer, C. S., Bomar, J. D., Pehlivan, H. C., et al. (2021) · The American Journal of Sports Medicine · doi:10.1177/03635465211032986 · Cited by 25
Keiner, M., Kapsecker, A., Stefer, T., Kadlubowski, B., & WIRTH, K. J. (2021) · Sports · doi:10.3390/sports9110149 · Cited by 25
Jeon, S. K., Lee, J. M., Joo, I., et al. (2021) · ULTRASONOGRAPHY · doi:10.14366/usg.21167 · Cited by 25
Vedernikova, E., Kuppens, P., & Erbaş, Y. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.703757 · Cited by 34
Prognostic value of baseline metabolic tumour volume in advanced-stage Hodgkin’s lymphoma
Pinochet, P., Texte, E., Stamatoullas, A., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-02734-w · Cited by 25
Santero, M., Meade, A.-G., Acosta‐Dighero, R., González, L., Melendi, S., Solà, I., Urrútia, G., Quintana, M. J., & Bonfill, X. (2022) · Clinical & Translational Oncology · doi:10.1007/s12094-022-02807-w · Cited by 22
Development and Validation of a Photonumeric Scale for Evaluation of Volume Deficit of the Hand
Jones, D., DONOFRIO, L. M., Hardas, B., Murphy, D. K., CARRUTHERS, J. D. A., CARRUTHERS, A. J., Sykes, J. M., Creutz, L., Marx, A., & Dill, S. (2016) · Dermatologic Surgery · doi:10.1097/dss.0000000000000850 · Cited by 25
Robertsen, R., Lorås, H. W., Polman, R. C. J., et al. (2022) · SAGE Open · doi:10.1177/21582440221096133 · Cited by 25
Xu, C., Peng, Y., Zhu, W., et al. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.969907 · Cited by 25
Showing with whom I belong: The desire to belong publicly on social media
Büttner, C. M., Lalot, F., & Rudert, S. C. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107535 · Cited by 33
Destito, M., Marzullo, A., Leone, R., et al. (2023) · Bioengineering · doi:10.3390/bioengineering10030285 · Cited by 25
Donate, M. J., Gómez, F. G., & González‐Mohíno, M. (2023) · Journal of Knowledge Management · doi:10.1108/jkm-10-2022-0852 · Cited by 25
Is the Ventilatory Efficiency in Endurance Athletes Different?—Findings from the NOODLE Study
Kasiak, P. S., Kowalski, T., Rębiś, K., Klusiewicz, A., Ładyga, M., Sadowska, D., Wilk, A., Wiecha, S., Barylski, M., Poliwczak, A. R., et al. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13020490 · Cited by 28
Concurrent validity and reliability of the maximum step length test in older adults.
Goldberg, A., Schepens, S., & Wallace, M. (2010) · PubMed · Cited by 25
Sankar, T., Moore, N. Z., Johnson, J. B., Ashby, L. S., Scheck, A. C., Shapiro, W. R., Smith, K. A., Spetzler, R. F., & Preul, M. C. (2012) · Journal of neurosurgery · doi:10.3171/2012.2.jns102032 · Cited by 25
Development and validation of a measurement tool to assess perceptions of palliative care
Milne, D., Aranda, S. K., Jefford, M., et al. (2012) · Psycho-Oncology · doi:10.1002/pon.3071 · Cited by 25
Feder, M. M., & Diamond, G. M. (2015) · Journal of Family Therapy · doi:10.1111/1467-6427.12078 · Cited by 28
Determining knee joint alignment using digital photographs
Schmitt, H., Kappel, H., Moser, M. T., et al. (2008) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-008-0570-6 · Cited by 25
Toward the development of criteria for global flares in juvenile systemic lupus erythematosus
Brunner, H. I., Klein‐Gitelman, M. S., Higgins, G. C., Lapidus, S., Levy, D. M., Eberhard, A., Singer, N. G., Olson, J. C., Onel, K. B., Punaro, M., et al. (2010) · Arthritis Care & Research · doi:10.1002/acr.20126 · Cited by 25
Burtscher, M. J., & Meyer, B. M. (2014) · Group Processes & Intergroup Relations · doi:10.1177/1368430214522138 · Cited by 26
External Validation of Scoring Instruments for Evaluating Pediatric Resuscitation
Levy, A. R., Donoghue, A. J., Bailey, B., Thompson, N., Jamoulle, O., Gagnon, R. J., & Gravel, J. (2014) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000052 · Cited by 25
Hafkenscheid, A., & Kiesler, D. J. (2007) · Psychotherapy Research · doi:10.1080/10503300600702323 · Cited by 25
Intoxication and violent women
Weizmann‐Henelius, G., Putkonen, H., Naukkarinen, H., & Eronen, M. (2008) · Archives of Women s Mental Health · doi:10.1007/s00737-008-0038-1 · Cited by 25
Two methods for assessing oral reading prosody
Duong, M., Mostow, J., & Sitaram, S. (2011) · ACM Transactions on Speech and Language Processing · doi:10.1145/1998384.1998388 · Cited by 25
Seynnes, O., Maffiuletti, N. A., Maganaris, C. N., Boer, M. D. D., Pensini, M., di Prampero, P. E., & Narici, M. V. (2008) · Acta Physiologica · doi:10.1111/j.1748-1716.2008.01874.x · Cited by 25
Using peer assessment of fieldwork to enhance students’ practical training
El‐Mowafy, A. (2013) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2013.820823 · Cited by 27
Provider Verbal Responses to Patient Distress Cues During Ambulatory Oncology Visits
Sheldon, L. K., Hilaire, D. M., & Berry, D. L. (2011) · Oncology nursing forum · doi:10.1188/11.onf.369-375 · Cited by 25
Affectivity in the problem-solving interactions of schizophrenia patients and their family members
Blanchard, J. J. (2003) · Schizophrenia Research · doi:10.1016/j.schres.2003.07.008 · Cited by 25
Developing Expert-Derived Rating Standards for the Peer Assessment of Lectures
Newman, L. R., Brodsky, D. D., Roberts, D. H., et al. (2012) · Academic Medicine · doi:10.1097/acm.0b013e3182444fa3 · Cited by 25
Ben‐Sasson, A., Cermak, S. A., Orsmond, G. I., Carter, A. S., & Fogg, L. (2007) · Infant Mental Health Journal · doi:10.1002/imhj.20152 · Cited by 25
Scholl, I., Nicolai, J., Pahlke, S., et al. (2013) · Patient Education and Counseling · doi:10.1016/j.pec.2013.10.006 · Cited by 25
Asakura, K., Haga, M., & Sasaki, S. (2015) · Journal of Epidemiology · doi:10.2188/jea.je20140174 · Cited by 25
Atenafu, E. G., Hamid, J. S., To, T. M., Willan, A. R., Feldman, B. M., & Beyene, J. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-126 · Cited by 26
Moisture lesions: interrater agreement and reliability
Kottner, J., & Halfens, R. J. G. (2010) · Journal of Clinical Nursing · doi:10.1111/j.1365-2702.2009.03109.x · Cited by 25
Arsanjani, R., Hayes, S. W., Fish, M. B., et al. (2014) · Journal of Nuclear Cardiology · doi:10.1007/s12350-014-9895-3 · Cited by 25
Test–retest variability of [11C]raclopride‐binding potential in nontreatment‐seeking alcoholics
Yoder, K. K., Albrecht, D., Kareken, D. A., Federici, L. M., Perry, K. M., Patton, E. A., Zheng, Q., Mock, B. H., O’Connor, S., & Herring, C. M. (2010) · Synapse · doi:10.1002/syn.20874 · Cited by 25
Steinhauer, K. M., Grayhack, J. P., Smiley‐Oyen, A. L., Shaiman, S., & McNeil, M. R. (2004) · Journal of Voice · doi:10.1016/j.jvoice.2004.01.006 · Cited by 25
Concordancia intra- e interevaluadores
Campo‐Arias, A., & Herazo, E. (2010) · Revista Colombiana de Psiquiatría · doi:10.1016/s0034-7450(14)60261-4 · Cited by 26
Statistical properties of longitudinal time-activity data for use in human exposure modeling
Isaacs, K., McCurdy, T. R., Glen, G., Nysewander, M., Errickson, A., Forbes, S., Graham, S. E., McCurdy, L., Smith, L. A., Tulve, N. S., et al. (2012) · Journal of Exposure Science & Environmental Epidemiology · doi:10.1038/jes.2012.94 · Cited by 25
Wu, F. S.-Y., Hwu, Y., Lee, R. K., Li, S., Sun, F., Lin, M.-H., & Lin, S. (2011) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/j.ejogrb.2011.11.019 · Cited by 25
Schretlen, D. J. (2000) · Rehabilitation Psychology · doi:10.1037//0090-5550.45.3.260 · Cited by 25
Hage, Y. E., Politti, F., de Sousa, D., Herpich, C. M., Glória, I. P. D. S., Gomes, C. A. F. D. P., Amaral, A. P., Melo, N. C. D., da Silva, T., Arruda, É. E. C., et al. (2013) · Trials · doi:10.1186/1745-6215-14-316 · Cited by 25
Blasio, A., Narayan, A. R., Kaminski, B. J., Steardo, L., Sabino, V., & Cottone, P. (2011) · Psychopharmacology · doi:10.1007/s00213-011-2517-8 · Cited by 25
Improving the diagnosis of LV non-compaction with cardiac magnetic resonance imaging
Choudhary, P., Hsu, C., Grieve, S. M., et al. (2014) · International Journal of Cardiology · doi:10.1016/j.ijcard.2014.12.053 · Cited by 25
Using the Work Limitations Questionnaire in Patients With A Chronic Condition in the Netherlands
Verhoef, J. A. C., Miedema, H. S., Bramsen, I., & Roebroeck, M. E. (2012) · Journal of Occupational and Environmental Medicine · doi:10.1097/jom.0b013e31825cb68d · Cited by 25
Self‐Evaluation Bias of Social Comparisons in Ethical Decision Making: The Impact of Accountability
Novičević, M. M., Buckley, M. R., Harvey, M. G., & Fung, H. (2008) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2008.00339.x · Cited by 25
Fu, C.-P., Hsieh, C., Tseng, M., Chen, Y., Huang, W.-T., Wu, P.-C., & Chiang, F.-M. (2009) · Research in Autism · doi:10.1016/j.rasd.2009.09.002 · Cited by 26
Coppieters, M. W., Schmid, A. B., Kubler, P. A., et al. (2012) · Manual Therapy · doi:10.1016/j.math.2012.03.005 · Cited by 25
A multi-center study: Intra-scan and inter-scan variability of diffusion spectrum imaging
Lemkaddem, A., Daducci, A., Vulliémoz, S., O’Brien, K., Lazeyras, F., Hauf, M., Wiest, R., Meuli, R., Seeck, M., Krüger, G., et al. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.04.045 · Cited by 26
Changes in rigidity and symptoms among adolescents in psychodynamic psychotherapy
Atzil–Slonim, D., Shefler, G., Dvir-Gvirsman, S., & Tishby, O. (2011) · Psychotherapy Research · doi:10.1080/10503307.2011.602753 · Cited by 27
Shultz, S. J., Wideman, L., Montgomery, M. M., & Levine, B. J. (2009) · British Journal of Sports Medicine · doi:10.1136/bjsm.2009.064931 · Cited by 25
Gibbons, C. J., Carroll, K. M., Ball, S. A., Nich, C., Frankforter, T. L., & Martino, S. (2010) · The American Journal of Drug and Alcohol Abuse · doi:10.3109/00952990.2010.500437 · Cited by 25
O’Donnell, P. C., & Lurigio, A. J. (2008) · Criminal Justice and Behavior · doi:10.1177/0093854808324061 · Cited by 25
Cross-Sectional and Longitudinal Relations between Working Memory Performance and Worry
Bredemeier, K., & Berenbaum, H. (2013) · Journal of Experimental Psychopathology · doi:10.5127/jep.032212 · Cited by 27
Spontaneous Improvement of Radiographic Indices for Idiopathic Planovalgus with Age
Park, M. S., Kwon, S.-S., Lee, S. Y., et al. (2013) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.m.00301 · Cited by 25
Physiotherapy exercise programmes: Are instructional exercise sheets effective?
Smith, J. A., Lewis, J. S., & Prichard, D. (2005) · Physiotherapy Theory and Practice · doi:10.1080/09593980590922316 · Cited by 25
Baier, C., Maderbacher, G., Springorum, H.-R., et al. (2013) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-013-2430-2 · Cited by 25
van der Linden, F. A. H. (2006) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp.2006.090795 · Cited by 24
Visser, J. C., Smeekens, S., Rommelse, N. N. J., Verkes, R. J., van der Gaag, R. J., & Buitelaar, J. K. (2010) · Infant Mental Health Journal · doi:10.1002/imhj.20273 · Cited by 26
Ridenour, T. A., Chilenski, S. M., & Reid, E. E. (2012) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2011.12.016 · Cited by 25
Yoo, M., Kim, W.-H., Hyun, S.-J., et al. (2014) · Korean Journal of Spine · doi:10.14245/kjs.2014.11.3.127 · Cited by 25
Segura‐Jiménez, V., Álvarez‐Gallardo, I. C., Romero-Zurita, A., Camiletti‐Moirón, D., Munguía‐Izquierdo, D., Carbonell‐Baeza, A., & Ruiz, J. R. (2014) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2014.05.015 · Cited by 25
Ebling, R., & LEVENSON, R. W. (2003) · Journal of Marriage and the Family · doi:10.1111/j.1741-3737.2003.00130.x · Cited by 25
Lindner, H. Y. N., Langius‐Eklöf, A., & Hermansson, L. (2014) · The Journal of Rehabilitation Research and Development · doi:10.1682/jrrd.2013.09.0197 · Cited by 25
Quantitative assessment of the evolution of cerebellar signs in spinocerebellar ataxias
Chan, E., Charles, P., Ribaı̈, P., Goizet, C., Marelli, C., Vincitorio, C., Bayon, A. L., Guyant‐Maréchal, L., Vandenberghe, N., Anheim, M., et al. (2011) · Movement Disorders · doi:10.1002/mds.23531 · Cited by 28
An Experimental Paradigm to Assess Postural Stabilization: No More Movement and Not Yet Posture
Rabuffetti, M., Bovi, G., Quadri, P. L., et al. (2011) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2011.2159241 · Cited by 25
Kuo, C., Lu, H., Lu, T., et al. (2013) · BioMedical Engineering OnLine · doi:10.1186/1475-925x-12-131 · Cited by 25
Steketee, R. M. E., Mutsaerts, H. J. M. M., Bron, E. E., et al. (2015) · PLoS ONE · doi:10.1371/journal.pone.0132929 · Cited by 25
An investigation into resting behavior in Asian elephants in UK zoos
Williams, E., Bremner‐Harrison, S., Harvey, N. D., Evison, E., & Yon, L. (2015) · Zoo Biology · doi:10.1002/zoo.21235 · Cited by 26
Walmsley, S., Osmotherly, P. G., & Rivett, D. A. (2014) · Physical Therapy · doi:10.2522/ptj.20130398 · Cited by 25
Validation of a protocol for manual segmentation of the thalamus on magnetic resonance imaging scans
Power, B., Wilkes, F., Hunter‐Dickson, M., van Westen, D., Santillo, A. F., Walterfang, M., Nilsson, C. F., Velakoulis, D., & Looi, J. C. (2015) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2015.02.001 · Cited by 25
Diminished emotion expressivity but not experience in men and women with schizophrenia.
Mote, J., Stuart, B. K., & Kring, A. M. (2014) · Journal of Abnormal Psychology · doi:10.1037/abn0000006 · Cited by 27
Effects of Voice Therapy as Objectively Evaluated by Digitized Laryngeal Stroboscopic Imaging
Speyer, R., Kempen, P. A., Wieneke, G. H., et al. (2002) · Annals of Otology Rhinology & Laryngology · doi:10.1177/000348940211101007 · Cited by 25
Lee, Y. S., Paik, K.-H., Kim, H. W., Lee, J.-C., Kim, J., & Hwang, J. (2015) · Medicine · doi:10.1097/md.0000000000001666 · Cited by 25
Test-Retest Reliability of V-Wave Responses in the Soleus and Gastrocnemius Medialis
Solstad, G. M., Fimland, M. S., Helgerud, J., Iversen, V. M., & Hoff, J. (2011) · Journal of Clinical Neurophysiology · doi:10.1097/wnp.0b013e31821215cf · Cited by 25
Frequency of Mealtime Insulin Bolus Predicts Glycated Hemoglobin in Youths with Type 1 Diabetes
Patton, S. R., DeLurgio, S. A., Fridlington, A., Cohoon, C., Turpin, A. L., & Clements, M. A. (2014) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2013.0356 · Cited by 25
Huang, H., Zhang, D., Jiang, Y., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0155463 · Cited by 25
Sörman, K., Nilsonne, G., Howner, K., Tamm, S., Caman, S., Wang, H., Ingvar, M., Edens, J. F., Gustavsson, P. J., Lilienfeld, S. O., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0156570 · Cited by 25
Sung, J., Ousley, C., Shen, S., et al. (2016) · International Journal of Rehabilitation Research · doi:10.1097/mrr.0000000000000188 · Cited by 25
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review