The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 6,001–6,100 of 23,883
Clark, D. J., Condliffe, E. G., & Patten, C. (2006) · Clinical Biomechanics · doi:10.1016/j.clinbiomech.2005.11.004 · Cited by 47
Change During and After Long-term Analytic Group Psychotherapy
Lorentzen, S., Bøgwald, K. P., & Høglend, P. A. (2002) · International Journal of Group Psychotherapy · doi:10.1521/ijgp.52.3.419.45511 · Cited by 47
Utility of Flexion-Extension Radiographs in Evaluating the Degenerative Cervical Spine
White, A. P., Biswas, D., Smart, L. R., et al. (2007) · Spine · doi:10.1097/01.brs.0000261409.45251.a2 · Cited by 47
Interobserver and Intraobserver Reliability in the Evaluation of Mechanical Axis Deviation
Gordon, J. E., Chen, R. C., Dobbs, M. B., et al. (2009) · Journal of Pediatric Orthopaedics · doi:10.1097/bpo.0b013e31819b9188 · Cited by 47
Residents' and burglars' assessment of burglary risk from defensible space cues
Shaw, K. T., & Gifford, R. (1994) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(94)80055-3 · Cited by 47
THE EFFECTS OF RESIDENTIAL TURNOVER ON HOUSEHOLD VICTIMIZATION*
Xie, M., & McDowall, D. (2008) · Criminology · doi:10.1111/j.1745-9125.2008.00123.x · Cited by 47
The effect of arm sling on balance in patients with hemiplegia
Acar, M., & Karataş, G. K. (2010) · Gait & Posture · doi:10.1016/j.gaitpost.2010.09.008 · Cited by 47
Ridenour, T. A., Clark, D. B., & Cottler, L. B. (2009) · The American Journal of Drug and Alcohol Abuse · doi:10.1080/00952990902998715 · Cited by 47
A New Method to Quantify Male Pelvic Floor Displacement From 2D Transperineal Ultrasound Images
Stafford, R. E., Ashton‐Miller, J. A., Constantinou, C. E., & Hodges, P. W. (2013) · Urology · doi:10.1016/j.urology.2012.11.034 · Cited by 47
Indirect Soft-Tissue and Osseous Signs on Knee MRI of Surgically Proven Meniscal Tears
Bergin, D., Hochberg, H., Zoga, A. C., et al. (2008) · American Journal of Roentgenology · doi:10.2214/ajr.07.3313 · Cited by 47
Wondra, V. C., Pitetti, K. H., & Beets, M. W. (2007) · Pediatric Physical Therapy · doi:10.1097/pep.0b013e3181577d6d · Cited by 47
Lucas, C., Classen, J., Harrison, D., & de Haan, R. J. (2002) · Advances in Skin & Wound Care · doi:10.1097/00129334-200201000-00009 · Cited by 47
Comparison of Feminist Versus Psychoanalytic/Dynamic and Other Therapists on Self-Disclosure
Simi, N. L., & Mahalik, J. R. (1997) · Psychology of Women Quarterly · doi:10.1111/j.1471-6402.1997.tb00125.x · Cited by 47
Chesworth, B. M., & Vandervoort, A. A. (1995) · Physical Therapy · doi:10.1093/ptj/75.4.253 · Cited by 47
Neumann, D. A., Cook, T. M., Sholty, R. L., et al. (1992) · Physical Therapy · doi:10.1093/ptj/72.3.207 · Cited by 47
Significance of neuron-specific enolase levels before and during therapy for small cell lung cancer.
Bonner, J. A., Sloan, J. A., Rowland, K. M., et al. (2000) · PubMed · Cited by 47
A Standardized Technique for Establishing Onset and Duration of Symptoms of Alzheimer's Disease
Sano, M. C., Devanand, D. P., Richards, M., Miller, L. W., Marder, K. S., Bell, K. L., Dooneief, G., Bylsma, F., Lafleche, G., Albert, M., et al. (1995) · Archives of Neurology · doi:10.1001/archneur.1995.00540340045012 · Cited by 47
Défossez, G., Mathoulin‐Pélissier, S., Ingrand, I., Gasquet, I., Sifer-Rivière, L., Ingrand, P., Salamon, R., & Migeot, V. (2007) · BMC Cancer · doi:10.1186/1471-2407-7-129 · Cited by 47
Effects of spatial transformation on regional brain volume estimates
Allen, J. S., Bruss, J., Mehta, S. H., Grabowski, T. J., Brown, C. K., & Damásio, H. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.05.047 · Cited by 47
Guigui, P., Barre, E., Benoist, M., et al. (1999) · Spine · doi:10.1097/00007632-199902010-00018 · Cited by 47
Tonic Stretch Reflex Threshold as a Measure of Ankle Plantar-Flexor Spasticity After Stroke
Blanchette, A. K., Mullick, A. A., Moïn-Darbari, K., et al. (2015) · Physical Therapy · doi:10.2522/ptj.20140243 · Cited by 47
A new assessment for elders admitted to acute care: Reliabilityofthe MDS-AC
Carpenter, G., Teare, G. F., Steel, K., Berg, K., Murphy, K., Bjornson, J., Jónsson, P. V., & Hirdes, J. P. (2001) · Aging Clinical and Experimental Research · doi:10.1007/bf03353428 · Cited by 47
Defensive Verbal Behavior Assessment
Barrett, L. F., Williams, N. L., & Fong, G. T. (2002) · Personality and Social Psychology Bulletin · doi:10.1177/0146167202289007 · Cited by 47
Field, S. D., Barkham, M., Shapiro, D. A., & Stiles, W. B. (1994) · Journal of Counseling Psychology · doi:10.1037/0022-0167.41.3.397 · Cited by 47
Nonverbal behaviors and social evaluation1
Shrout, P. E., & Fiske, D. W. (1981) · Journal of Personality · doi:10.1111/j.1467-6494.1981.tb00732.x · Cited by 47
Self-citationToddler language and play in the second year: stability, covariation and influences of parenting
Bornstein, M. H., Vibbert, M., Tal, J., & O'Donnell, K. (1992) · First Language · doi:10.1177/014272379201203607 · Cited by 47
Accuracy, reproducibility, and variability of hand-held dynamometry in motor neuron disease.
Goonetilleke, A., Modarres-Sadeghi, H., & Guiloff, R. J. (1994) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp.57.3.326 · Cited by 47
Singleton, E. G., Anderson, L. M., & Heishman, S. J. (2003) · Addiction · doi:10.1046/j.1360-0443.2003.00449.x · Cited by 47
Cognitive processes involved in similarity judgments of emotions.
Schimmack, U., & Reisenzein, R. (1997) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.73.4.645 · Cited by 47
Statistical Models for Behavioral Observations
Rogosa, D., & Ghandour, G. (1991) · Journal of Educational Statistics · doi:10.2307/1165191 · Cited by 47
Llop, C., Pujol, I., Aguilar, C., Sala, J., Riba, D., & Guarro, J. (2000) · Antimicrobial Agents and Chemotherapy · doi:10.1128/aac.44.2.239-242.2000 · Cited by 47
Reliability of tissue volumes and their spatial distribution for segmented magnetic resonance images
Cardenas, V. A., Ezekiel, F., Sclafani, V. D., et al. (2001) · Psychiatry Research Neuroimaging · doi:10.1016/s0925-4927(01)00075-0 · Cited by 47
Innovation Measurement: A Decision Framework to Determine Innovativeness of a Company
Phan, K. (2000) · doi:10.15760/etd.1017 · Cited by 47
Peñarroja, V., Orengo, V., & Zornoza, A. (2017) · Journal of Applied Social Psychology · doi:10.1111/jasp.12449 · Cited by 47
Kramer, U., Pascual‐Leone, A., Rohde, K. B., & Sachse, R. (2018) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2160 · Cited by 45
Internal validity of project MATCH treatments: Discriminability and integrity.
Carroll, K. M., Connors, G. J., Cooney, N. L., DiClemente, C. C., Donovan, D. M., Kadden, R. R., Longabaugh, R. L., Rounsaville, B. J., Wirtz, P. W., & Zweben, A. (1998) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.66.2.290 · Cited by 208
Chan, R., Steel, Z., Brooks, R., Heung, T., Erlich, J., Chow, J., & Suranyi, M. (2011) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2011.05.002 · Cited by 54
Age-related differences when walking downhill on different sloped terrains
Scaglioni-Solano, P., & Vargas, A. (2014) · Gait & Posture · doi:10.1016/j.gaitpost.2014.09.022 · Cited by 46
The Relationship Between Body Size and Depressed Mood
Granberg, E. M., Simons, R. L., Gibbons, F. X., & Melby, J. N. (2007) · Youth & Society · doi:10.1177/0044118x07301952 · Cited by 47
Reliability of Judging in DanceSport
Premelč, J., Vučković, G., James, N., & Leskošek, B. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01001 · Cited by 47
Bammann, K., Thomson, N. K., Albrecht, B. M., Buchan, D. S., & Easton, C. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252615 · Cited by 46
Evaluating the layout quality of UML class diagrams using machine learning
Bergström, G., Hujainah, F., Ho-Quang, T., Jolak, R., Rukmono, S. A., Nurwidyantoro, A., & Chaudron, M. R. V. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111413 · Cited by 46
Saxer, K., Schnell, J., Morinaj, J., & Hascher, T. (2023) · International Journal of Educational Research Open · doi:10.1016/j.ijedro.2023.100318 · Cited by 76
Analyzing Dataset Annotation Quality Management in the Wild
Klie, J.-C., de Castilho, R. E., & Gurevych, I. (2024) · Computational Linguistics · doi:10.1162/coli_a_00516 · Cited by 66
Objective Behavioral Tests of Temperament in Nonhuman Primates
Fairbanks, L. A., & Jorgensen, M. J. (2011) · Springer eBooks · doi:10.1007/978-1-4614-0176-6_5 · Cited by 46
Managing the Front End of Innovation—Part I: Results From a Three-Year Study
Koen, P. A., Bertels, H. M. J., & Kleinschmidt, E. J. (2015) · Research-Technology Management · doi:10.5437/08956308x5702145 · Cited by 46
Self-efficacy, perceptions of context, and burnout: a multilevel study on nurses.
Consiglio, C., Боргогни, Л., Vecchione, M., & Maslach, C. (2014) · PubMed · Cited by 46
Are there subtypes of panic disorder? An interpersonal perspective.
Zilcha‐Mano, S., McCarthy, K. S., Dinger, U., Chambless, D. L., Milrod, B. L., Kunik, L., & Barber, J. P. (2015) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0039373 · Cited by 45
Using Mathematical Competencies to Predict Item Difficulty in PISA: A MEG Study
Turner, R., Dossey, J. A., Blum, W. F., et al. (2012) · Springer eBooks · doi:10.1007/978-94-007-4458-5_2 · Cited by 46
Bi, J., & Kuesten, C. L. (2012) · Journal of Sensory Studies · doi:10.1111/j.1745-459x.2012.00399.x · Cited by 46
Lilliengren, P., Johansson, R., Lindqvist, K., Mechler, J., & Andersson, G. (2015) · Psychotherapy · doi:10.1037/pst0000024 · Cited by 54
Vaskinn, A., Ventura, J., Andreassen, O. A., Melle, I. S., & Sundet, K. S. (2015) · Psychiatry Research · doi:10.1016/j.psychres.2015.05.019 · Cited by 47
An open trial of motivational interviewing to address suicidal ideation with hospitalized veterans.
Britton, P. C., Conner, K., & Maisto, S. (2012) · Journal of Clinical Psychology · doi:10.1002/jclp.21885 · Cited by 44
Nieto, R. G., Blasco-Fontecilla, H., Yepes, M. P., & Baca‐García, E. (2012) · Revista de Psiquiatría y Salud Mental · doi:10.1016/j.rpsm.2012.07.001 · Cited by 46
Crowley, A. R., Dong, J., McHaffie, A., et al. (2011) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.22425 · Cited by 46
A smart device inertial-sensing method for gait analysis
Steins, D., Sheret, I., Dawes, H., Esser, P., & Collett, J. (2014) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2014.06.014 · Cited by 46
Örtqvist, M., Moström, E. B., Roos, E. M., Lundell, P., Janarv, P., Werner, S., & Broström, E. W. (2011) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-011-1542-9 · Cited by 46
Fluoroscopic Diagnosis of Scapholunate Interosseous Ligament Injuries in Distal Radius Fractures
Kwon, B. C., & Baek, G. H. (2008) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-008-0126-6 · Cited by 46
Ginn, K. A., & Cohen, M. L. (2004) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2003.09.013 · Cited by 46
Reliability of scapular positioning measurement procedure using the Palpation Meter (PALM)
da Costa, B. R., Armijo-Olivo, S. L., Gadotti, I. C., Warren, S., Reid, D. C., & Magee, D. J. (2009) · Physiotherapy · doi:10.1016/j.physio.2009.06.007 · Cited by 45
Baad‐Hansen, L., Pigg, M., Yang, G., et al. (2014) · Journal of Oral Rehabilitation · doi:10.1111/joor.12245 · Cited by 46
Rodrigues, E. K. D. S., Fonseca, M. D. C. R., & MacDermid, J. C. (2014) · Journal of Hand Therapy · doi:10.1016/j.jht.2014.09.008 · Cited by 48
González, P., Bernal, V., Pérez, S. I., & Barrientos, G. (2007) · Journal of Archaeological Science · doi:10.1016/j.jas.2006.12.013 · Cited by 50
Westerberg, C. E., Mayes, A. R., Florczak, S. M., Chen, Y. J., Creery, J. D., Parrish, T. B., Weıntraub, S., Mesulam, M. M., Reber, P. J., & Paller, K. A. (2013) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2013.06.025 · Cited by 49
Persson, N., Ghisletta, P., Dahle, C. L., Bender, A. R., Yang, Y., Yuan, P., Daugherty, A. M., & Raz, N. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.09.042 · Cited by 45
The Test–Retest Reliability of the Photopic Negative Response (PhNR)
Tang, J. J., Edwards, T. L., Crowston, J. G., & Sarossy, M. (2014) · Translational Vision Science & Technology · doi:10.1167/tvst.3.6.1 · Cited by 46
Broekhuis, S. R., Kluivers, K. B., Hendriks, J. C. M., Vierhout, M. E., Barentsz, J. O., & Fütterer, J. J. (2008) · International Urogynecology Journal · doi:10.1007/s00192-008-0760-2 · Cited by 46
Some Relationships Between Cronbach’s Alpha and the Spearman-Brown Formula
Warrens, M. J. (2015) · Journal of Classification · doi:10.1007/s00357-015-9168-0 · Cited by 46
Maller, J. J., Anstey, K. J., Réglade‐Meslin, C., Christensen, H. M., Wen, W. X., & Sachdev, P. S. (2007) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2007.06.005 · Cited by 46
Linek, P., Saulicz, E., Wolny, T., & Myśliwiec, A. P. (2014) · PM&R · doi:10.1016/j.pmrj.2014.07.007 · Cited by 46
Devine, A., Taylor, S. J., Spencer, A. E., Diaz‐Ordaz, K., Eldridge, S. M., & Underwood, M. R. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.04.005 · Cited by 46
Temperament as a moderator of the relation between neighborhood and children's adjustment
Bush, N. R., Lengua, L. J., & Colder, C. R. (2010) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2010.06.004 · Cited by 48
Bulloch, B., Garcia‐Filion, P., Notrica, D. M., et al. (2009) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2009.00404.x · Cited by 46
Richter, M. M. (2010) · Biological Psychology · doi:10.1016/j.biopsycho.2010.02.014 · Cited by 46
Sinani, C., Sugden, D., & Hill, E. L. (2011) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2011.01.030 · Cited by 47
Maass, S. W. M. C., Bagher, S., Hofer, S. O. P., Baxter, N. N., & Zhong, T. (2015) · Annals of Surgical Oncology · doi:10.1245/s10434-015-4434-2 · Cited by 46
Gasq, D., Labrunée, M., Amarantini, D., Dupui, P., Montoya, R., & Marqué, P. (2014) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-11-39 · Cited by 46
Test–Retest Reliability of the Standing Heel-Rise Test
Ross, M. D., & Fontenot, E. G. (2000) · Journal of Sport Rehabilitation · doi:10.1123/jsr.9.2.117 · Cited by 46
Reproducibility of heart rate variability and baroreflex sensitivity measurements in children
Dietrich, A., Rosmalen, J. G. M., Althaus, M., van Roon, A. M., Mulder, L. J. M., Minderaa, R. B., Oldehinkel, A. J., & Riese, H. (2010) · Biological Psychology · doi:10.1016/j.biopsycho.2010.05.005 · Cited by 45
The Effect of Feeling Stereotyped on Social Power and Inhibition
Cook, J., Arrow, H., & Malle, B. F. (2011) · Personality and Social Psychology Bulletin · doi:10.1177/0146167210390389 · Cited by 46
Does Movement Recognition Precision Affect the Player Experience in Exertion Games?
Nijhar, J., Bianchi‐Berthouze, N., & Boguslawski, G. (2012) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-642-30214-5_9 · Cited by 46
Reiter‐Palmon, R., & Arreola, N. J. (2015) · Creativity Research Journal · doi:10.1080/10400419.2015.1087274 · Cited by 46
Baylis, A. L., Munson, B., & Moller, K. T. (2010) · The Cleft Palate-Craniofacial Journal · doi:10.1597/09-201 · Cited by 46
Relationships between Contextual and Task Performance and Interrater Agreement: Are There Any?
Vilela, L. F. D., Delgado, N., Isla-Díaz, R., Díaz‐Cabrera, D., Hernández‐Fernaud, E., & Rosales, C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0139898 · Cited by 60
Improving problem solving with subgoal labels in expository text and worked examples
Margulieux, L. E., & Catrambone, R. (2015) · Learning and Instruction · doi:10.1016/j.learninstruc.2015.12.002 · Cited by 46
Sometimes a celebrity holding a negative public image is the best product endorser
Sääksjärvi, M., Hellén, K., & Balabanis, G. I. (2016) · European Journal of Marketing · doi:10.1108/ejm-06-2014-0346 · Cited by 59
Relatives’ emotional involvement moderates the effects of family therapy for bipolar disorder.
Fredman, S. J., Baucom, D. H., Boeding, S. E., & Miklowitz, D. J. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0037713 · Cited by 46
Christensen, M. S., Kronborg, C. J. S., Eldrup, N., Rossen, N. B., & Knudsen, U. B. (2016) · Pregnancy Hypertension · doi:10.1016/j.preghy.2016.04.001 · Cited by 46
Novel 3-Dimensional Motion Analysis Method for Measuring the Lumbar Spine Range of Motion
Tojima, M., Ogata, N., Yozu, A., Sumitani, M., & Haga, N. (2013) · Spine · doi:10.1097/brs.0b013e3182a0dbc5 · Cited by 45
MEG sensor and source measures of visually induced gamma-band oscillations are highly reliable
Tan, H. M., Groß, J., & Uhlhaas, P. J. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.05.006 · Cited by 47
The Reliability and Predictive Validity of the Stalking Risk Profile
McEwan, T. E., Shea, D. E., Daffern, M., MacKenzie, R. D., Ogloff, J. R. P., & Mullen, P. E. (2016) · Assessment · doi:10.1177/1073191116653470 · Cited by 46
Fraquelli, M., Baccarin, A., Casazza, G., Conti, C. B., Giunta, M., Massironi, S., Invernizzi, F., Donato, M. F., Maggioni, M., Aghemo, A., et al. (2016) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.13711 · Cited by 47
An overview of Rorschach psychometrics for forensic practice.
Viglione, D. J., & Meyer, G. J. (2016) · doi:10.4324/9780203810071.ch2 · Cited by 46
Kasturi, S., Szymonifka, J. D., Koltsov, J. C. B., et al. (2017) · The Journal of Rheumatology · doi:10.3899/jrheum.161202 · Cited by 46
Malcolm, B. R., Foxe, J. J., Butler, J. S., Mowrey, W. B., Molholm, S., & Sanctis, P. D. (2017) · Brain Research · doi:10.1016/j.brainres.2017.05.021 · Cited by 48
Kimball, A. B., Sundaram, M., Banderas, B. F., Foley, C., & Shields, A. L. (2017) · Journal of Dermatological Treatment · doi:10.1080/09546634.2017.1341614 · Cited by 46
Validation of the HUMAC Balance System in Comparison with Conventional Force Plates
Koltermann, J. J., Gerber, M., Beck, H., & Beck, M. (2017) · Technologies · doi:10.3390/technologies5030044 · Cited by 46
Determining the sample size in agreement studies
Temel, G. Ö., & Erdoğan, S. (2017) · Marmara medical journal · doi:10.5472/marumj.344822 · Cited by 50
Developmental variation in regional brain iron and its relation to cognitive functions in childhood
Hect, J. L., Daugherty, A. M., Hermez, K., & Thomason, M. E. (2018) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2018.05.004 · Cited by 46
Macdonald, E. A., Cushman, S. A., Landguth, E. L., Hearn, A. J., Malhi, Y. S., & Macdonald, D. W. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196974 · Cited by 47
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
