The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 6,501–6,600 of 23,882
Jungo, A., Scheidegger, O., Reyes, M., & Balsiger, F. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105796 · Cited by 43
White, K. S., Payne, L. A., Gorman, J. M., Shear, M. K., Woods, S. W., Saksa, J. R., & Barlow, D. H. (2013) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0030666 · Cited by 42
Tissue Doppler imaging for detecting onset of muscle activity
Pulkovski, N., Schenk, P., Maffiuletti, N. A., & Mannion, A. F. (2008) · Muscle & Nerve · doi:10.1002/mus.20996 · Cited by 42
Randomized controlled trial of two brief alcohol interventions for OEF/OIF veterans
McDevitt-Murphy, M. E., Murphy, J. G., Williams, J. L., Monahan, C. J., Bracken-Minor, K. L., & Fields, J. A. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0036714 · Cited by 42
Imitation of facial expressions in schizophrenia
Schwartz, B. L., Mastropaolo, J., Rosse, R. B., Mathis, G., & Deutsch, S. I. (2006) · Psychiatry Research · doi:10.1016/j.psychres.2005.12.007 · Cited by 40
The "HR–line-connecting HRM system" and its effects on employee turnover
Kim, S., Su, Z., & Wright, P. M. (2018) · Human Resource Management · doi:10.1002/hrm.21905 · Cited by 47
The way films feel: Aesthetic features and mood in film.
Tarvainen, J., Westman, S., & Oittinen, P. T. (2015) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0039432 · Cited by 45
Fábrega‐Cuadros, R., Martínez‐Amat, A., Cruz‐Díaz, D., Aibar‐Almazán, A., & Hita‐Contreras, F. (2019) · Calcified Tissue International · doi:10.1007/s00223-019-00635-9 · Cited by 42
Development of Data-Driven Metrics for Balance Impairment and Fall Risk Assessment in Older Adults
McManus, K., Greene, B. R., Motti, L. G., & Caulfield, B. (2022) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2022.3142617 · Cited by 43
Warmth and competence in human-agent cooperation
McKee, K. R., Bai, X., & Fiske, S. T. (2024) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-024-09649-6 · Cited by 58
Wong, A. Y., Smith, S. L., McGrath, C. A., Flynn, L. E., & Mills, C. (2022) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2022.102098 · Cited by 48
Bernhardsson, S., Larsson, A., Bergenheim, A., Ho-Henriksson, C.-M., Ekhammar, A., Lange, E., Larsson, M. E. H., Nordeman, L., Samsson, K. S., & Bornhöft, L. (2023) · PLoS ONE · doi:10.1371/journal.pone.0283013 · Cited by 55
Redefining Trochlear Dysplasia: Normal Thresholds Vary by Measurement Technique, Landmarks, and Sex
Tanaka, M. J., Sodhi, A., Wadhavkar, I., et al. (2023) · The American Journal of Sports Medicine · doi:10.1177/03635465231158099 · Cited by 42
Savaş, S., Kilavuz, A., Koçak, F. Ö. K., & Çavdar, S. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12134260 · Cited by 50
Moghadam, A. N., Nasrin, N., Kharazmi, A., et al. (2012) · Asian Journal of Sports Medicine · doi:10.5812/asjsm.34528 · Cited by 42
Lee, M. T., & Widener, S. K. (2015) · Journal of Information Systems · doi:10.2308/isys-51298 · Cited by 42
Buysse, H. E. C., Coorevits, P. L., Maele, G. V., Hutse, A., Kaufman, J., Ruige, J. B., & Moor, G. J. E. D. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2010.05.005 · Cited by 42
Lievens, F., Schollaert, E., & Keen, G. (2014) · Journal of Applied Psychology · doi:10.1037/apl0000004 · Cited by 42
Patient affect experiencing following therapist interventions in short-term dynamic psychotherapy
Town, J. M., Hardy, G. E., McCullough, L., & Stride, C. (2011) · Psychotherapy Research · doi:10.1080/10503307.2011.637243 · Cited by 42
Test–retest reliability of the novel 5-HT1B receptor PET radioligand [11C]P943
Sarıçiçek, A., Chen, J., Planeta, B. M., Ruf, B. M., Subramanyam, K., Maloney, K. H., Matuskey, D., Labaree, D. C., Deserno, L., Neumeister, A., et al. (2014) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-014-2958-5 · Cited by 39
Gorelick, M., Bizzini, M., Maffiuletti, N. A., et al. (2009) · Clinical Physiology and Functional Imaging · doi:10.1111/j.1475-097x.2009.00864.x · Cited by 42
Guggenberger, R., Eppenberger, P., Markovic, D., et al. (2012) · European Journal of Radiology · doi:10.1016/j.ejrad.2012.03.017 · Cited by 42
Ngo, T., Stupar, M., Côté, P., et al. (2010) · European Spine Journal · doi:10.1007/s00586-010-1289-x · Cited by 42
Measuring the intensity of spontaneous facial action units with dynamic Bayesian network
Li, Y., Mavadati, S. M., Mahoor, M. H., Zhao, Y.-P., & Ji, Q. (2015) · Pattern Recognition · doi:10.1016/j.patcog.2015.04.022 · Cited by 43
Slavin-Mulford, J. M., Hilsenroth, M., Weinberger, J., & Gold, J. R. (2011) · Journal of Nervous and Mental Disease · doi:10.1097/nmd.0b013e3182125d60 · Cited by 41
Examining school culture in Flemish and Chinese primary schools
Zhu, C., Devos, G., & Tondeur, J. (2013) · Educational Management Administration & Leadership · doi:10.1177/1741143213502190 · Cited by 43
Brogårdh, C. B., Persson, A. L., & Sjölund, B. H. (2006) · Disability and Rehabilitation · doi:10.1080/09638280600747603 · Cited by 42
Higashihara, E., Nutahara, K., Okegawa, T., et al. (2015) · The Nephron journals/Nephron journals · doi:10.1159/000381476 · Cited by 42
Moore, D. R., & Florsheim, P. (2008) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2007.05.006 · Cited by 43
Jüttner, M., & Neuhaus, B. (2012) · International Journal of Science Education · doi:10.1080/09500693.2011.606511 · Cited by 49
Klep, A. H. M., Wisse, B. M., & der Flier, H. V. (2010) · European Journal of Social Psychology · doi:10.1002/ejsp.775 · Cited by 49
Shin, W. C., Lee, S. M., Lee, K. W., et al. (2015) · The Bone & Joint Journal · doi:10.1302/0301-620x.97b5.34735 · Cited by 42
Losa‐Iglesias, M. E., Vallejo, R., & Becerro-de-Bengoa-Losa, K. R. (2014) · Health Informatics Journal · doi:10.1177/1460458214540909 · Cited by 45
Development of an Assessment for Ideational Praxis
May-Benson, T. A., & Cermak, S. A. (2007) · American Journal of Occupational Therapy · doi:10.5014/ajot.61.2.148 · Cited by 42
Tang, R., Gungor, C., Sesek, R. F., et al. (2016) · European Spine Journal · doi:10.1007/s00586-016-4405-8 · Cited by 42
Iconic Gestures for Robot Avatars, Recognition and Integration with Speech
Bremner, P., & Leonards, U. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00183 · Cited by 42
Quantitative 3D magnetic resonance elastography: Comparison with dynamic mechanical analysis
Arunachalam, S. P., Rossman, P. J., Arani, A., et al. (2016) · Magnetic Resonance in Medicine · doi:10.1002/mrm.26207 · Cited by 42
Effects of age on speech and voice quality ratings
Goy, H., Pichora-Fuller, M. K., & Lieshout, P. H. H. M. V. (2016) · The Journal of the Acoustical Society of America · doi:10.1121/1.4945094 · Cited by 42
Jacxsens, M., Tongel, A. V., Henninger, H. B., Coninck, B. D., Müller, A. M., & Wilde, L. F. D. (2016) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2016.02.035 · Cited by 42
Happiness is positive welfare in brown capuchins (Sapajus apella)
Robinson, L. M., Waran, N. K., Leach, M. C., Morton, F. B., Paukner, A., Lonsdorf, E. V., Handel, I. G., Wilson, V. A. D., Brosnan, S. F., & Weiß, A. (2016) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2016.05.029 · Cited by 42
Hoyt, L. T., Ehrlich, K. B., Cham, H., & Adam, E. K. (2016) · Stress · doi:10.1080/10253890.2016.1206884 · Cited by 46
Reliability of radiographic measurements for acute distal radius fractures
Watson, N. J., Asadollahi, S., Parrish, F., Ridgway, J., Tran, P., & Keating, J. L. (2016) · BMC Medical Imaging · doi:10.1186/s12880-016-0147-7 · Cited by 42
Wakefield, C. B., O’Malley, J. M., Williams, A. J., Taylor, B. M., Nichols, R. S., Halafihi, T., Humphreys, R. L., Kaltavara, J., Nicol, S., & Newman, S. J. (2016) · ICES Journal of Marine Science · doi:10.1093/icesjms/fsw162 · Cited by 42
Guha, D., Jakubovic, R. Y., Gupta, S., Alotaibi, N. M., Cadotte, D. W., da Costa, L., George, R., Heyn, C., Howard, P., Kapadia, A. N., et al. (2016) · The Spine Journal · doi:10.1016/j.spinee.2016.10.020 · Cited by 42
Reliability of graph metrics derived from resting‐state human EEG
Kuntzelman, K. M., & Miskovic, V. (2016) · Psychophysiology · doi:10.1111/psyp.12600 · Cited by 43
Deputy, N. P., Nguyen, P., Pham, H., Nguyễn, S., Neufeld, L., Martorell, R., & Ramakrishnan, U. (2017) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-016-1192-5 · Cited by 40
Vosoughi, A. R., Roustaei, N., & Mahdaviazad, H. (2017) · Foot and Ankle Surgery · doi:10.1016/j.fas.2017.02.007 · Cited by 42
Yoon, T. (2017) · The Journal of Korean Physical Therapy · doi:10.18857/jkpt.2017.29.3.145 · Cited by 41
Thielmann, I., Zimmermann, J., Leising, D., & Hilbig, B. E. (2017) · European Journal of Personality · doi:10.1002/per.2112 · Cited by 42
Lee, Y. (2017) · Interactive Learning Environments · doi:10.1080/10494820.2017.1372483 · Cited by 48
Maeng, H., Webster, E. K., Pitchford, E. A., et al. (2017) · Adapted Physical Activity Quarterly · doi:10.1123/apaq.2016-0026 · Cited by 42
Song, J., Choe, K., Neary, M. T., Zifchock, R. A., Cameron, K. L., Trepa, M., Hannan, M. T., & Hillstrom, H. J. (2017) · Gait & Posture · doi:10.1016/j.gaitpost.2017.12.001 · Cited by 42
Harkey, M. S., Blackburn, J. T., Hackney, A. C., et al. (2017) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2017.10.009 · Cited by 42
Chimpanzees gesture to humans in mirrors: using reflection to dissociate seeing from line of gaze
Lurz, R. W., Krachun, C., Mahovetz, L. M., Wilson, M., & Hopkins, W. D. (2017) · Animal Behaviour · doi:10.1016/j.anbehav.2017.11.014 · Cited by 42
Töger, J., Arvidsson, P. M., Bock, J., Kanski, M., Pedrizzetti, G., Carlsson, M., Arheden, H., & Heiberg, E. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195597 · Cited by 42
Nishie, A., Asayama, Y., Ishigami, K., Ushijima, Y., Takayama, Y., Okamoto, D., Fujita, N., Tsurumaru, D., Togao, O., Sagiyama, K., et al. (2018) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.14315 · Cited by 43
Assessment of Subjective Visual Vertical (SVV) Using the “Bucket Test” and the Virtual SVV System
Michelson, P. L., McCaslin, D. L., Jacobson, G. P., et al. (2018) · American Journal of Audiology · doi:10.1044/2018_aja-17-0019 · Cited by 42
Bello, N. A., Schwartz, J. E., Kronish, I. M., Oparil, S., Anstey, D. E., Wei, Y., Cheung, Y. K. K., Muntner, P., & Shimbo, D. (2018) · Journal of the American Heart Association · doi:10.1161/jaha.118.008658 · Cited by 42
Andrei Irimia, A., Maher, A. S., Rostowsky, K. A., et al. (2019) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2019.00009 · Cited by 42
Weber, U., Jurik, A. G., Zejden, A., et al. (2019) · Lara D. Veeken · doi:10.1093/rheumatology/kez458 · Cited by 42
Prabhu, R., Miller, S. R., Simpson, T. W., & Meisel, N. A. (2019) · Journal of Mechanical Design · doi:10.1115/1.4045127 · Cited by 43
Mandigout, S., Lacroix, J., Perrochon, A., Svoboda, Z., Aubourg, T., & Vuillerme, N. (2019) · Frontiers in Medicine · doi:10.3389/fmed.2019.00252 · Cited by 43
Romero, R. L., Kates, F., Hart, M. W., Ojeda, A. E., Meirom, I., & Hardy, S. J. (2019) · JMIR mhealth and uhealth · doi:10.2196/14198 · Cited by 44
Lower digit ratio and higher endogenous testosterone are associated with lower empathic accuracy
Nitschke, J. P., & Bartz, J. A. (2019) · Hormones and Behavior · doi:10.1016/j.yhbeh.2019.104648 · Cited by 41
Kim, J.-H., Kim, Y., Park, I., et al. (2019) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2019.08.049 · Cited by 42
Hermassi, S., Laudner, K. G., & Schwesig, R. (2020) · Journal of Human Kinetics · doi:10.2478/hukin-2019-0083 · Cited by 42
FDG PET versus CT radiomics to predict outcome in malignant pleural mesothelioma patients
Pavic, M., Bogowicz, M., Kraft, J., et al. (2020) · EJNMMI Research · doi:10.1186/s13550-020-00669-3 · Cited by 42
Morris, W. Z., Hinds, S., Worrall, H. M., et al. (2020) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.20.00562 · Cited by 42
Wang, J., Zheng, X., Zhang, J., Xue, H., Wang, L., Jing, R., Chen, S., Che, F., Heng, X., Li, G., et al. (2021) · European Radiology · doi:10.1007/s00330-020-07581-3 · Cited by 43
Chu, C.-A., Chen, Y., Chang, K. C., Wu, W., & Özçakar, L. (2021) · Frontiers in Physiology · doi:10.3389/fphys.2021.654667 · Cited by 42
Monitoring of Gait Parameters in Post-Stroke Individuals: A Feasibility Study Using RGB-D Sensors
Ferraris, C., Cimolin, V., Vismara, L., Votta, V., Amprimo, G., Cremascoli, R., Galli, M., Nerino, R., Mauro, A., & Priano, L. (2021) · Sensors · doi:10.3390/s21175945 · Cited by 41
Schless, S., Desloovere, K., Aertbeliën, E., Molenaers, G., Huenaerts, C., & Bar‐On, L. (2015) · PLoS ONE · doi:10.1371/journal.pone.0131011 · Cited by 41
Lutz, C. J., & Lakey, B. (2001) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.81.6.1070 · Cited by 42
Reliability of a novel serious game using dual-task gait profiles to early characterize aMCI
Tarnanas, I., Papagiannopoulos, S., Kazis, D., Wiederhold, M., Widerhold, B., & Tsolaki, M. (2015) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2015.00050 · Cited by 47
Kim, C.-Y., Choi, J.-D., Kim, S.-Y., et al. (2014) · Journal of Electromyography and Kinesiology · doi:10.1016/j.jelekin.2014.07.002 · Cited by 42
Effect of Head Position on Vertical Mandibular Opening
Higbie, E. J., Seidel-Cobb, D., Taylor, L. F., et al. (1999) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1999.29.2.127 · Cited by 42
Xue, D., Bernard De Baets, C., Botte, J., Vermeulen, J., Cleemput, O. V., & Boeckx, P. (2010) · Rapid Communications in Mass Spectrometry · doi:10.1002/rcm.4445 · Cited by 41
Reliability of the Passive Knee Flexion and Extension Tests in Healthy Subjects
Gnat, R., Kuszewski, M., Koczar, R., & Dziewońska, A. (2010) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2010.09.001 · Cited by 42
Klass, O., Walker, M. J. P., Siebach, A., et al. (2009) · European Radiology · doi:10.1007/s00330-009-1652-7 · Cited by 42
Anderson, A. N., Pavese, N., Edison, P., Tai, Y., Hammers, A., Gerhard, A., Brooks, D. J., & Turkheimer, F. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2007.02.017 · Cited by 42
Davis, O. S. P., Kovas, Y. V., Harlaar, N., Busfield, P., McMillan, A. J., Frances, J., Petrill, S. A., Dale, P. S., & Plomin, R. J. (2007) · Genes Brain & Behavior · doi:10.1111/j.1601-183x.2007.00370.x · Cited by 42
Maximum Voluntary Isometric Contraction: Investigation of Reliability and Learning Effect
Meldrum, D., Cahalane, E., Keogan, F., & Hardiman, O. M. (2003) · Amyotrophic Lateral Sclerosis and Other Motor Neuron Disorders · doi:10.1080/14660820310006715 · Cited by 41
Validation of a Photonumeric Wrinkle Assessment Scale for Assessing Nasolabial Fold Wrinkles
Buchner, L., Vamvakias, G., & ROM, D. M. (2010) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0b013e3181de243b · Cited by 42
Unfairness begets unfairness: Victim derogation bias in employee ratings
Skarlicki, D. P., & Turner, R. A. (2014) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2013.11.004 · Cited by 41
Validity and reliability of a new in vivo ankle stiffness measurement device
Zinder, S. M., Granata, K. P., Padua, D. A., et al. (2006) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2005.12.007 · Cited by 42
Utility of the Rosenberg Self-Esteem Scale
Davis, C., Kellett, S., & Beail, N. (2009) · American Journal on Intellectual and Developmental Disabilities · doi:10.1352/1944-7558-114.3.172 · Cited by 44
Leppänen, M. H., Ekholm, E. M. K., Palo, P., Maunu, J., Munck, P., Parkkola, R. K., Matomäki, J., Lapinleimu, H., Haataja, L. M., Lehtonen, L., et al. (2010) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.7694 · Cited by 42
Test–retest reliability of event-related functional MRI in a probabilistic reversal learning task
Freyer, T., Valerius, G., Kuelz, A.-K., Speck, O., Glauche, V., Hull, M., & Voderholzer, U. (2009) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2009.03.003 · Cited by 43
When goals constrain: Eye movements and memory for goal‐Oriented map study
Brunyé, T. T., & Taylor, H. A. (2008) · Applied Cognitive Psychology · doi:10.1002/acp.1508 · Cited by 44
Differential muscle strength decline in osteoarthritis of the knee. A developing hypothesis
Hayes, K. W., & Falconer, J. (1992) · Arthritis & Rheumatism · doi:10.1002/art.1790050107 · Cited by 42
Inter- and intra-rater agreement of the rehabilitation activities profile
Jelles, F. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00152-g · Cited by 42
Ruberg, F. L., Chen, Z., Hua, N., Bigornia, S., Guo, Z., Hallock, K., Jara, H. J., LaValley, M. P., Phinikaridou, A., Qiao, Y., et al. (2009) · Obesity · doi:10.1038/oby.2009.363 · Cited by 43
Reliability and Validity of the Standing Heel-Rise Test
Yocum, A., McCoy, S. W., Bjornson, K. F., et al. (2010) · Physical & Occupational Therapy In Pediatrics · doi:10.3109/01942631003761380 · Cited by 42
Interrater Reliability among Epilepsy Centers: Multicenter Study of Epilepsy Surgery
Haut, S. R., Berg, A. T., Shinnar, S., et al. (2002) · Epilepsia · doi:10.1046/j.1528-1157.2002.20902.x · Cited by 42
The effect of illness experience and demographic factors on children's illness representations
Paterson, J. E., Moss-Morris, R. E., & Butler, S. J. (1999) · Psychology and Health · doi:10.1080/08870449908407318 · Cited by 42
Ethnic differences in ventricular hypertrabeculation on cardiac MRI in elite football players
Luijkx, T., Cramer, M. J., Zaidi, A., Rienks, R., Senden, P. J., Sharma, S., van Hellemondt, F. J., Buckens, C. F. M., Mali, W. P. T. M., & Velthuis, B. K. (2012) · Netherlands Heart Journal · doi:10.1007/s12471-012-0305-7 · Cited by 42
Reliability of Radiographic Measures for Infantile Idiopathic Scoliosis
Corona, J., Sanders, J. O., Luhmann, S. J., et al. (2012) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.k.00311 · Cited by 42
Steenhuis, M., Serra, M., Minderaa, R. B., & Hartman, C. A. (2009) · Psychological Assessment · doi:10.1037/a0015925 · Cited by 42
Xiong, S., Goonetilleke, R. S., Witana, C. P., et al. (2010) · The Journal of Rehabilitation Research and Development · doi:10.1682/jrrd.2009.09.0152 · Cited by 42
Confidant and breast cancer patient reports of quality of life
Sandgren, A. K., Mullens, A. B., Erickson, S. C., Romanek, K. M., & McCaul, K. D. (2004) · Quality of Life Research · doi:10.1023/b:qure.0000015287.90952.95 · Cited by 42
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
