The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 4,301–4,400 of 23,883
Yu, X., & Xiong, S. (2019) · Sensors · doi:10.3390/s19132882 · Cited by 70
Higher body mass index is linked to altered hypothalamic microstructure
Thomas, K., Beyer, F., Lewe, G., Zhang, R., Schindler, S., Schönknecht, P., Stümvoll, M., Villringer, A., & Witte, A. V. (2019) · Scientific Reports · doi:10.1038/s41598-019-53578-4 · Cited by 71
Examination of Reaction Time Deficits Following Concussion: A Systematic Review and Meta-analysis
Lempke, L. B., Howell, D. R., Eckner, J. T., & Lynall, R. C. (2020) · Sports Medicine · doi:10.1007/s40279-020-01281-0 · Cited by 68
Mahadevan, A. S., Tooley, U. A., Bertolero, M. A., Mackey, A. P., & Bassett, D. S. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118408 · Cited by 67
Ran, L., Xie, T., Zhao, L., et al. (2022) · The Spine Journal · doi:10.1016/j.spinee.2022.01.018 · Cited by 67
Reliability of an fMRI paradigm for emotional processing in a multisite longitudinal study
Gee, D. G., McEwen, S. C., Forsyth, J. K., Haut, K. M., Bearden, C. E., Addington, J., Goodyear, B. G., Cadenhead, K. S., Mirzakhanian, H., Cornblatt, B. A., et al. (2015) · Human Brain Mapping · doi:10.1002/hbm.22791 · Cited by 67
McCrimmon, C. M., King, C., Wang, P. T., Cramer, S. C., Nenadić, Z., & Do, A. H. (2015) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-015-0050-4 · Cited by 67
Accurate Judgments of Neuroticism at Zero Acquaintance: A Question of Relevance
Hirschmüller, S., Egloff, B., Schmukle, S. C., Nestler, S., & Back, M. D. (2014) · Journal of Personality · doi:10.1111/jopy.12097 · Cited by 74
Rosen, V. M., Sunderland, T., Levy, J., Harwell, A., McGee, L., Hammond, C., Bhupali, D., Putnam, K. T., Bergeson, J., & Lefkowitz, C. (2004) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2004.06.022 · Cited by 70
Examining the alliance using the Psychotherapy Process Q-Set.
Price, P. B., & Jones, E. E. (1998) · Psychotherapy · doi:10.1037/h0087654 · Cited by 67
Bouwens, S., Heugten, C. M. V., & Verhey, F. R. J. (2008) · Dementia and Geriatric Cognitive Disorders · doi:10.1159/000178757 · Cited by 68
Coaches and Clients in Action: A Sequential Analysis of Interpersonal Coach and Client Behavior
Ianiro-Dahm, P. M., Lehmann‐Willenbrock, N., & Kauffeld, S. (2014) · Journal of Business and Psychology · doi:10.1007/s10869-014-9374-5 · Cited by 78
A cross-cultural study of preschool quality in South Korea and Sweden: ECERS evaluations
Sheridan, S., Giota, J., Han, Y.-M., et al. (2009) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2009.03.004 · Cited by 67
Peltier, A., Smith, A. G., Russell, J. W., Sheikh, K., Bixby, B. A., Howard, J., Goldstein, J. M., Song, Y., Wang, L., Feldman, E. L., et al. (2009) · Muscle & Nerve · doi:10.1002/mus.21210 · Cited by 70
Temporal Reliability and Lateralization of the Resting-State Language Network
Zhu, L., Yang, F., Zou, Q., Wang, J., Gao, J., & Niu, Z. (2014) · PLoS ONE · doi:10.1371/journal.pone.0085880 · Cited by 72
Núñez-Nateras, R., Castle, E. P., Protheroe, C. A., et al. (2013) · Urologic Oncology Seminars and Original Investigations · doi:10.1016/j.urolonc.2013.06.008 · Cited by 67
Malin, A. J., & Pos, A. E. (2014) · Psychotherapy Research · doi:10.1080/10503307.2014.901572 · Cited by 84
Negahban, H., Ahmadi, P., Salehi, R., et al. (2013) · Neuroscience Letters · doi:10.1016/j.neulet.2013.10.022 · Cited by 67
The unique relations between emotional awareness and facets of affective instability
Thompson, R. J., Dizén, M., & Berenbaum, H. (2009) · Journal of Research in Personality · doi:10.1016/j.jrp.2009.07.006 · Cited by 76
A real-time assessment of interpersonal complementarity
Markey, P. M., Lowmaster, S. E., & Eichler, W. C. (2010) · Personal Relationships · doi:10.1111/j.1475-6811.2010.01249.x · Cited by 68
Pivotal Response Treatment Parent Training for Autism: Findings from a 3-Month Follow-Up Evaluation
Gengoux, G. W., Berquist, K. L., Salzman, E., Schapp, S., Phillips, J. M., Frazier, T. W., Minjarez, M., & Hardan, A. Y. (2015) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-015-2452-3 · Cited by 68
Fiore, J., Faragher, I., Bialocerkowski, A., Browning, L., & Denehy, L. (2013) · World Journal of Surgery · doi:10.1007/s00268-013-2208-1 · Cited by 69
Stacey, D., Taljaard, M., Drake, E. R., & O'Connor, A. M. C. (2008) · Patient Education and Counseling · doi:10.1016/j.pec.2008.07.016 · Cited by 66
Eckardt, W., Steklis, H. D., Steklis, N. G., Fletcher, A. W., Stoinski, T. S., & Weiß, A. (2014) · Journal of comparative psychology · doi:10.1037/a0038370 · Cited by 67
Newton, J. M., Dong, Y., Hidler, J. M., Plummer, P., Marehbian, J., Albistegui-Dubois, R., Woods, R. P., & Dobkin, B. H. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.07.001 · Cited by 68
Ettinger, U., Williams, S., Patel, D., Michel, T. M., Nwaigwe, A., Cáceres, A., Mehta, M. A., Anilkumar, A. P., & Kumari, V. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.12.029 · Cited by 65
Hudswell, S., von Mengersen, M., & Lucas, N. P. (2005) · International journal of osteopathic medicine · doi:10.1016/j.ijosm.2005.07.003 · Cited by 67
Modelling Knee Range of Motion Post Arthroplasty: Clinical Applications
Stratford, P. W., Kennedy, D. M., & Robarts, S. F. (2010) · Physiotherapy Canada · doi:10.3138/physio.62.4.378 · Cited by 67
A comparison of fitness and skill among playing positions in sub-elite rugby league players
Gabbett, T. J., Kelly, J., & Pezet, T. (2007) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2007.07.004 · Cited by 67
Child and informant influences on behavioral ratings of preschool children
Phillips, B. M., & Lonigan, C. J. (2010) · Psychology in the Schools · doi:10.1002/pits.20476 · Cited by 67
Early treatment for Class II malocclusion and perceived improvements in facial profile
O'Brien, K. D., Macfarlane, T. V., Wright, J., et al. (2009) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2008.02.020 · Cited by 67
Radiographic assessment of cervical subaxial fusion
Kaiser, M. G., Mummaneni, P. V., Matz, P. G., et al. (2009) · Journal of Neurosurgery Spine · doi:10.3171/2009.3.spine08719 · Cited by 67
Validation of a Parent Report Questionnaire
Riley, A. W., Trabulsi, J., Yao, M., et al. (2015) · Clinical Pediatrics · doi:10.1177/0009922815574075 · Cited by 67
Baer, M., Vadera, A. K., Leenders, R., & Oldham, G. R. (2013) · Organization Science · doi:10.1287/orsc.2013.0878 · Cited by 67
A Contextual Assessment Inventory for Problem Behavior
McAtee, M. L., Carr, E. G., Schulte, C., & Dunlap, G. (2004) · Journal of Positive Behavior Interventions · doi:10.1177/10983007040060030301 · Cited by 67
Reliability of Clinical Measurements of Lumbar Lordosis Taken with a Flexible Rule
Lovell, F. W., Rothstein, J. M., & Personius, W. J. (1989) · Physical Therapy · doi:10.1093/ptj/69.2.96 · Cited by 67
Madrigano, J., Baccarelli, A., Mittleman, M. A., Sparrow, D., Spiro, A., Vokonas, P., Cantone, L., Kubzansky, L. D., & Schwartz, J. D. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kwr523 · Cited by 69
Test–retest reliability of functional near infrared spectroscopy in infants
Blasi, A., Lloyd‐Fox, S., Johnson, M. H., & Elwell, C. E. (2014) · Neurophotonics · doi:10.1117/1.nph.1.2.025005 · Cited by 68
Interrater Reliability of Students Using Hand and Pinch Dynamometers
Lindstrom-Hazel, D., Kratt, A., & Bix, L. (2009) · American Journal of Occupational Therapy · doi:10.5014/ajot.63.2.193 · Cited by 67
Merino-Marbán, R., Mayorga-Vega, D., Rodríguez, E. F., Estrada, F. V., & Viciana, J. (2014) · European Physical Education Review · doi:10.1177/1356336x14550942 · Cited by 66
Smith, J. L., Carpenter, K. M., Amrhein, P. C., Brooks, A. C., Levin, D., Schreiber, E. A., Travaglini, L. A., Hu, M., & Nunes, E. V. (2012) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0028176 · Cited by 68
Persson, N., Ghisletta, P., Dahle, C. L., Bender, A. R., Yang, Y., Yuan, P., Daugherty, A. M., & Raz, N. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.11.028 · Cited by 63
Schulte, F., Wurz, A. J., Reynolds, K. A., Strother, D. R., & Dewey, D. (2016) · Pediatric Blood & Cancer · doi:10.1002/pbc.25868 · Cited by 67
Grimm, M. R., Lazariotou, M., Kircher, S., Höfelmayr, A., Germer, C.-T., von Rahden, B. H. A., Waaga-Gasser, A. M., & Gasser, M. (2011) · Cellular Oncology · doi:10.1007/s13402-011-0027-7 · Cited by 67
Yue, R., Arendt, E. A., & Tompkins, M. A. (2017) · The Journal of Knee Surgery · doi:10.1055/s-0037-1599249 · Cited by 67
Varieties of Confidence Intervals
Cousineau, D. (2017) · Advances in Cognitive Psychology · doi:10.5709/acp-0214-z · Cited by 67
Test-Retest Reliability of “High-Order” Functional Connectivity in Young Healthy Adults
Zhang, H., Chen, X., Zhang, Y., & Shen, D. G. (2017) · Frontiers in Neuroscience · doi:10.3389/fnins.2017.00439 · Cited by 67
Laurin, J. C., & Joussemet, M. (2017) · Motivation and Emotion · doi:10.1007/s11031-017-9627-5 · Cited by 81
Exercise-Induced Hypoalgesia After Isometric Wall Squat Exercise: A Test-Retest Reliabilty Study
Vægter, H. B., Lyng, K. D., Yttereng, F. W., Christensen, M. H., Sørensen, M. B., & Graven‐Nielsen, T. (2018) · Pain Medicine · doi:10.1093/pm/pny087 · Cited by 72
Dynamic capabilities and employee participation: The role of trust and informal control
Wohlgemuth, V., Wenzel, M., Berger, E. S. C., & Eisend, M. (2019) · European Management Journal · doi:10.1016/j.emj.2019.02.005 · Cited by 72
Arain, G. A., Hameed, I., & Crawshaw, J. R. (2019) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2019.1609946 · Cited by 74
So, S.-Y., Lee, S., Jung, E. Y., et al. (2019) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-019-05555-7 · Cited by 67
Shogren, K. A., & Rojahn, J. (2003) · Journal of Developmental and Physical Disabilities · doi:10.1023/a:1026314316977 · Cited by 67
Content Analysis of Worrisome Streams of Consciousness in Anxious and Dysphoric Participants
Molina, S., Borkovec, T. D., Peasley, C., & Person, D. C. (1998) · Cognitive Therapy and Research · doi:10.1023/a:1018772104808 · Cited by 68
Paulsen, J. S., & Altmaier, E. M. (1995) · Pain · doi:10.1016/0304-3959(94)00096-w · Cited by 67
Effect of Temperature on Muscle Force and Rate of Muscle Force Production in Men and Women
Cornwall, M. W. (1994) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1994.20.2.74 · Cited by 67
Guides for Reading and Interpreting Systematic Reviews
Moher, D., Jadad, A. R., & Klassen, T. P. (1998) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/archpedi.152.9.915 · Cited by 66
Peer Assessment of Personality Traits and Pathology in Female College Students
Oltmanns, T. F., & Strauss, M. E. (1998) · Assessment · doi:10.1177/107319119800500108 · Cited by 67
Wolf, O. T., Dyakin, V. V., Patel, A., Vadász, C., de Leon, M. J., McEwen, B. S., & Bulloch, K. (2002) · Brain Research · doi:10.1016/s0006-8993(02)02363-6 · Cited by 67
Hand-held dynamometry reliability in persons with neuropathic weakness
Kilmer, D. D., McCrory, M. A., Wright, N. C., et al. (1997) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(97)90311-7 · Cited by 67
Three-Dimensional Flexibility Characteristics of the Human Cervical Spine In Vivo
McClure, P. W., Siegler, S., & Nobilini, R. (1998) · Spine · doi:10.1097/00007632-199801150-00013 · Cited by 67
An Assessment of Methods to Combine Published Survival Curves
Earle, C. C., & Wells, G. A. (2000) · Medical Decision Making · doi:10.1177/0272989x0002000113 · Cited by 67
Reliability of a Clinical Kinematic Assessment of the Sit-to-Stand Movement
Jeng, S., Schenkman, M. L., Riley, P. O., et al. (1990) · Physical Therapy · doi:10.1093/ptj/70.8.511 · Cited by 67
Reliability and Running Speed Effects of In-shoe Loading Measurements During Slow Treadmill Running
Kemozek, T. W., & Zimmer, K. A. (2000) · Foot & Ankle International · doi:10.1177/107110070002100906 · Cited by 67
Elliott, M. L., Knodt, A. R., Ireland, D. R., Morris, M. L., Poulton, R., Ramrakha, S., Sison, M. L., Moffitt, T. E., Caspi, A., & Hariri, A. R. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/681700 · Cited by 71
Occupational characteristics and cognitive performance among elderly male twins
Potter, G. G., Plassman, B. L., Helms, M. J., Foster, S. M., & Edwards, N. W. (2006) · Neurology · doi:10.1212/01.wnl.0000240061.51215.ed · Cited by 65
Social and physical anhedonia and valence and arousal aspects of emotional experience
Kerns, J. G., Docherty, A. R., & Martin, E. A. (2008) · Journal of Abnormal Psychology · doi:10.1037/a0013601 · Cited by 66
Melwani, S., & Barsade, S. G. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0023492 · Cited by 81
Good keeper-elephant relationships in North American zoos are mutually beneficial to welfare
Carlstead, K., Paris, S., & Brown, J. L. (2019) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2018.11.003 · Cited by 75
Loison, B., Moussaddaq, A.-S., Cormier, J.-P., Richard, I., Ferrapie, A.-L., Ramond‐Roquin, A., & Dinomais, M. (2013) · Annals of Physical and Rehabilitation Medicine · doi:10.1016/j.rehab.2013.01.001 · Cited by 66
Comparison of Rater’s Reliability on Perceptual Evaluation of Different Types of Voice Sample
Law, T. K. T., Kim, J. H., Lee, K., et al. (2012) · Journal of Voice · doi:10.1016/j.jvoice.2011.08.003 · Cited by 66
Zaben, F. A., Sehlo, M. G., Khalifa, D. A., et al. (2015) · Journal of Religion and Health · doi:10.1007/s10943-015-0025-6 · Cited by 66
Finding a Life Worth Living: Meaning in Life and Graduation from College
Wilt, J. A., Bleidorn, W., & Revelle, W. (2016) · European Journal of Personality · doi:10.1002/per.2046 · Cited by 71
Zajic, M. C., McIntyre, N. S., Lerro, L., Novotny, S. E., Oswald, T. M., & Mundy, P. C. (2016) · Autism · doi:10.1177/1362361316675121 · Cited by 66
Markiewicz, Ł., Witkowski, M., Sitnik, R., & Mielicka, E. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.04.052 · Cited by 66
Rand, S. E., Malley, J., Towers, A., Netten, A., & Forder, J. (2017) · Health and Quality of Life Outcomes · doi:10.1186/s12955-017-0739-0 · Cited by 66
Juvenile Idiopathic Inflammatory Myopathies
Huber, A. M. (2018) · Pediatric Clinics of North America · doi:10.1016/j.pcl.2018.04.006 · Cited by 66
Yu, J. J. (2018) · Journal of Marketing Communications · doi:10.1080/13527266.2018.1514317 · Cited by 71
Boost in Test–Retest Reliability in Resting State fMRI with Predictive Modeling
Taxali, A., Angstadt, M., Rutherford, S. E., & Sripada, C. (2020) · Cerebral Cortex · doi:10.1093/cercor/bhaa390 · Cited by 76
The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification
Alva-Manchego, F. E., Scarton, C., & Specia, L. (2021) · Computational Linguistics · doi:10.1162/coli_a_00418 · Cited by 68
de Farias, E. C., di Noia, C., Han, C., Sala, E., Castelli, M., & Rundo, L. (2021) · Scientific Reports · doi:10.1038/s41598-021-00898-z · Cited by 66
Nuwailati, R., Bobos, P., Drangsholt, M. T., & Curatolo, M. (2022) · Scandinavian Journal of Pain · doi:10.1515/sjpain-2021-0149 · Cited by 66
Genetic architecture of the white matter connectome of the human brain
Sha, Z., Schijven, D., Fisher, S. E., & Francks, C. (2023) · Science Advances · doi:10.1126/sciadv.add2870 · Cited by 68
Evaluating Performance and Agreement of Coronary Heart Disease Polygenic Risk Scores
Abramowitz, S. A., Boulier, K. M., Keat, K., Cardone, K. M., Shivakumar, M., DePaolo, J. S., Judy, R., Bermudez, F., Mimouni, N., Neylan, C. J., et al. (2024) · JAMA · doi:10.1001/jama.2024.23784 · Cited by 69
Gill, J. K., Maskarinec, G., Pagano, I., et al. (2006) · Breast Cancer Research · doi:10.1186/bcr1507 · Cited by 66
Peng, D. X., Verghese, A. J., Shah, R., & Schroeder, R. G. (2013) · Journal of Supply Chain Management · doi:10.1111/j.1745-493x.2012.03286.x · Cited by 66
Lindquist, N. J., Loudon, P. E., Magis, T. F., Rispin, J. E., Kirby, R. L., & Manns, P. J. (2010) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2010.07.226 · Cited by 66
Auger, C., Demers, L., Gélinas, I., Routhier, F., Jutai, J. W., Guérette, C., & DeRuyter, F. (2009) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483100802543064 · Cited by 66
A Historiometric Examination of Machiavellianism and a New Taxonomy of Leadership
Bedell, K. E., Hunter, S. T., Angie, A. D., et al. (2006) · Journal of Leadership & Organizational Studies · doi:10.1177/107179190601200404 · Cited by 66
Shelef, K., & Diamond, G. M. (2008) · Psychotherapy Research · doi:10.1080/10503300701810801 · Cited by 66
Focus on therapeutic alliance: The psychometric properties of six measures across three treatments.
Cecero, J. J., Fenton, L. R., Frankforter, T. L., Nich, C., & Carroll, K. M. (2001) · Psychotherapy · doi:10.1037/0033-3204.38.1.1 · Cited by 67
Selective reduction of the posterior superior vermis in men with chronic schizophrenia
Okugawa, G., Sedvall, G., Nordström, M., Andreasen, N. C., Pierson, R. K., Magnotta, V. A., & Agartz, I. (2002) · Schizophrenia Research · doi:10.1016/s0920-9964(01)00248-1 · Cited by 66
Development and Evaluation of a Simulation-Based Pediatric Emergency Medicine Curriculum
Adler, M., Vozenilek, J. A., Trainor, J. L., Eppich, W., Wang, E. E., Beaumont, J. L., Aitchison, P. R., Erickson, T. B., Edison, M. I., & McGaghie, W. C. (2009) · Academic Medicine · doi:10.1097/acm.0b013e3181a813ca · Cited by 66
The minimal detectable change of the simplified stroke rehabilitation assessment of movement measure
Lu, W., Wang, C., Lin, J., Sheu, C., & Hsieh, C. (2008) · Journal of Rehabilitation Medicine · doi:10.2340/16501977-0230 · Cited by 67
Psychometric evaluation of the Fatigue Severity Scale in patients with major depression
Ferentinos, P. P., Kontaxakis, V. P., Havaki-Kontaxaki, B. J., Dikeos, D. G., & Lykouras, L. (2010) · Quality of Life Research · doi:10.1007/s11136-010-9769-3 · Cited by 66
Ana, E. J. S., Wulfert, E., & Nietert, P. J. (2007) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.75.5.816 · Cited by 66
Prefrontal cortex atrophy predicts dementia over a six-year period
Burgmans, S., van Boxtel, M. P. J., Smeets, F., Vuurman, E. F. P. M., Gronenschild, E. H. B. M., Verhey, F. R. J., Uylings, H. B. M., & Jolles, J. (2008) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2007.11.028 · Cited by 66
Group versus individual exercise interventions for women with breast cancer: a meta-analysis
Floyd, A. H. L., & Moyer, A. E. (2009) · Health Psychology Review · doi:10.1080/17437190903384291 · Cited by 66
A comparison of four disability scales for Turkish patients with neck pain
Kose, G., Hepgüler, S., Atamaz, F. C., et al. (2007) · Acta Dermato Venereologica · doi:10.2340/16501977-0060 · Cited by 66
Translation and validation of the German version of the foot and ankle outcome score
van Bergen, C. J. A., Sierevelt, I. N., Hoogervorst, P., Waizy, H., van Dijk, C. N., & Becher, C. (2014) · Archives of Orthopaedic and Trauma Surgery · doi:10.1007/s00402-014-1994-8 · Cited by 66
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
