The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 201–300 of 23,883
JUSTICE IN TEAMS: ANTECEDENTS AND CONSEQUENCES OF PROCEDURAL JUSTICE CLIMATE
Colquitt, J. A., Noe, R. A., & JACKSON, C. L. (2002) · Personnel Psychology · doi:10.1111/j.1744-6570.2002.tb00104.x · Cited by 734
Crossley, K. M., Bennell, K. L., Cowan, S., & Green, S. (2004) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(03)00613-0 · Cited by 711
Psychopathy in instrumental and reactive violent offenders.
Cornell, D. G., Warren, J. I., Hawk, G. L., Stafford, E., Oram, G., & Pine, D. (1996) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.64.4.783 · Cited by 703
Sample size requirements for reliability studies
Donner, A. P., & Eliasziw, M. (1987) · Statistics in Medicine · doi:10.1002/sim.4780060404 · Cited by 693
DISFA: A Spontaneous Facial Action Intensity Database
Mavadati, S. M., Mahoor, M. H., Bartlett, K., Trinh, P., & Cohn, J. F. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2013.4 · Cited by 793
Homan, A. C., van Knippenberg, D., Van Kleef, G. A., & De Dreu, C. K. W. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.5.1189 · Cited by 685
Linde, K., Clausius, N., Ramirez, G., Melchart, D., Eitel, F., Hedges, L. V., & Jonas, W. B. (1997) · The Lancet · doi:10.1016/s0140-6736(97)02293-9 · Cited by 979
Chesterton, L. S., Sim, J., Wright, C. C., & Foster, N. E. (2007) · Clinical Journal of Pain · doi:10.1097/ajp.0b013e318154b6ae · Cited by 690
Grant, B., Harford, T., Dawson, D., Chou, P., & Pickering, R. (1995) · Drug and Alcohol Dependence · doi:10.1016/0376-8716(95)01134-k · Cited by 689
Has anything changed? Current characterizations of men, women, and managers.
Heilman, M. E., Block, C. J., Martell, R. F., & Simon, M. C. (1989) · Journal of Applied Psychology · doi:10.1037/0021-9010.74.6.935 · Cited by 689
Menz, H. B., Latt, M. D., Tiedemann, A., Kwan, M. M. S., & Lord, S. R. (2003) · Gait & Posture · doi:10.1016/s0966-6362(03)00068-7 · Cited by 682
Grant, P., Ryan, C. G., Tigbe, W. W., & Granat, M. H. (2006) · British Journal of Sports Medicine · doi:10.1136/bjsm.2006.030262 · Cited by 711
Gerhart, B. A., Wright, P. M., MAHAN, G. C. M., & Snell, S. A. (2000) · Personnel Psychology · doi:10.1111/j.1744-6570.2000.tb02418.x · Cited by 679
Choi, S. Y., Lee, H., & Yoo, Y. (2010) · MIS Quarterly · doi:10.2307/25750708 · Cited by 697
Magnetic resonance for assessment of disease activity and severity in ileocolonic Crohn’s disease
Rimola, J., Rodríguez, S., García‐Bosch, O., Ordás, Í., Ayala, E., Aceituno, M., Pellisé, M., Ayuso, C., Ricart, E., Donoso, L., et al. (2009) · Gut · doi:10.1136/gut.2008.167957 · Cited by 673
The development of an instrument to measure neighborhood cohesion
Buckner, J. C. (1988) · American Journal of Community Psychology · doi:10.1007/bf00930892 · Cited by 674
Younossi, Z. M., Guyatt, G. H., Kiwi, M. L., Boparai, N., & King, D. (1999) · Gut · doi:10.1136/gut.45.2.295 · Cited by 678
Keller, R. T. (2001) · Academy of Management Journal · doi:10.2307/3069369 · Cited by 658
Social Identity and the Service-Profit Chain
Homburg, C., Wieseke, J., & Hoyer, W. D. (2009) · Journal of Marketing · doi:10.1509/jmkg.73.2.38 · Cited by 699
Intra-rater and inter-rater reliability of a weight-bearing lunge measure of ankle dorsiflexion
Bennell, K. L., Talbot, R., Wajswelner, H., et al. (1998) · Australian Journal of Physiotherapy · doi:10.1016/s0004-9514(14)60377-9 · Cited by 664
Reliability of Measurements Obtained With the Timed “Up & Go” Test in People With Parkinson Disease
Morris, S., Morris, M. E., & Iansek, R. (2001) · Physical Therapy · doi:10.1093/ptj/81.2.810 · Cited by 664
Zuo, X., & Xing, X.-X. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.05.009 · Cited by 678
Peterson, R. S., Smith, D. B., Martorana, P. V., & Owens, P. D. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.5.795 · Cited by 654
Reliability, Internal Consistency, and Validity of Data Obtained With the Functional Gait Assessment
Wrisley, D. M., Marchetti, G. F., Kuharsky, D. K., & Whitney, S. L. (2004) · Physical Therapy · doi:10.1093/ptj/84.10.906 · Cited by 660
Seibert, S. E., Silver, S. R., & Randolph, W. A. (2004) · Academy of Management Journal · doi:10.5465/20159585 · Cited by 652
Russell, D. J., Avery, L. M., Rosenbaum, P., Raina, P. S., Walter, S. D., & Palisano, R. J. (2000) · Physical Therapy · doi:10.1093/ptj/80.9.873 · Cited by 649
Zhou, J. (1998) · Journal of Applied Psychology · doi:10.1037/0021-9010.83.2.261 · Cited by 676
Ruhe, A., Fejer, R., & Walker, B. F. (2010) · Gait & Posture · doi:10.1016/j.gaitpost.2010.09.012 · Cited by 645
Relation of Prefrontal Cortex Dysfunction to Working Memory and Symptoms in Schizophrenia
Perlstein, W. M., Carter, C. S., Noll, D. C., & Cohen, J. D. (2001) · American Journal of Psychiatry · doi:10.1176/appi.ajp.158.7.1105 · Cited by 652
Sholomskas, D. E., Syracuse-Siewert, G., Rounsaville, B. J., Ball, S. A., Nuro, K. F., & Carroll, K. M. (2005) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.73.1.106 · Cited by 628
Nieuwboer, A., Rochester, L., Herman, T., Vandenberghe, W., Emil, G. E., Thomaes, T., & Giladi, N. (2009) · Gait & Posture · doi:10.1016/j.gaitpost.2009.07.108 · Cited by 637
Coparenting: A link between marital conflict and parenting in two-parent families.
Margolin, G., Gordis, E. B., & John, R. S. (2001) · Journal of Family Psychology · doi:10.1037/0893-3200.15.1.3 · Cited by 640
Crossland, C., & Hambrick, D. C. (2010) · Strategic Management Journal · doi:10.1002/smj.913 · Cited by 655
Isenberg, D. A., Rahman, A., Allen, E., Farewell, V. T., Akil, M., Bruce, I. N., D’Cruz, D. P., Griffiths, B., Khamashta, M. A., Maddison, P. J., et al. (2005) · Lara D. Veeken · doi:10.1093/rheumatology/keh624 · Cited by 638
Group Size, ICC Values, and Group-Level Correlations: A Simulation
Bliese, P. D. (1998) · Organizational Research Methods · doi:10.1177/109442819814001 · Cited by 636
Development of a Patient-Reported Measure of Function of the Knee*
Irrgang, J. J., Snyder‐Mackler, L., Wainner, R. S., Fu, F. H., & Harner, C. D. (1998) · Journal of Bone and Joint Surgery · doi:10.2106/00004623-199808000-00006 · Cited by 636
Eliasziw, M., Young, S. L., Woodbury, M. G., & Fryday-Field, K. (1994) · Physical Therapy · doi:10.1093/ptj/74.8.777 · Cited by 632
Hobart, J., & Cano, S. (2009) · Health Technology Assessment · doi:10.3310/hta13120 · Cited by 621
Two replicable suppressor situations in personality research
Paulhus, D. L., Robins, R. W., Trzesniewski, K. H., & Tracy, J. L. (2004) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3902_7 · Cited by 631
Ng, S. S. M., & Hui-Chan, C. W. Y. (2005) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2005.01.011 · Cited by 635
Trajectories of brain aging in middle-aged and older adults: Regional and individual differences
Raz, N., Ghisletta, P., Rodrigue, K. M., Kennedy, K. M., & Lindenberger, U. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.03.020 · Cited by 626
Handbook of Industrial, Work & Organizational Psychology
Anderson, N. R., Öneş, D. S., Sinangil, H. K., & Viswesvaran, C. (2001) · Sage eBooks · Cited by 632
A simple method for noninvasive estimation of pulmonary vascular resistance
Abbas, A. E., Fortuin, F. D., Schiller, N. B., et al. (2003) · Journal of the American College of Cardiology · doi:10.1016/s0735-1097(02)02973-x · Cited by 629
How to Assess the Reliability of Measurements in Rehabilitation
Lexell, J. E., & Downham, D. Y. (2005) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/01.phm.0000176452.17771.20 · Cited by 629
Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008) · Value in Health · doi:10.1111/j.1524-4733.2007.00231.x · Cited by 623
Composition, process, and performance in self-managed groups: The role of personality.
Barry, B., & Stewart, G. L. (1997) · Journal of Applied Psychology · doi:10.1037/0021-9010.82.1.62 · Cited by 619
The heritability of general cognitive ability increases linearly from childhood to young adulthood
Haworth, C. M. A., Wright, M. J., Luciano, M., Martin, N. G., de Geus, E. J. C., van Beijsterveldt, C. E. M., Bartels, M., Posthuma, D., Boomsma, D. I., Davis, O. S. P., et al. (2009) · Molecular Psychiatry · doi:10.1038/mp.2009.55 · Cited by 618
Consequences of violating the independence assumption in analysis of variance.
Kenny, D. A., & Judd, C. M. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.99.3.422 · Cited by 616
Violent Video Games and Hostile Expectations: A Test of the General Aggression Model
Bushman, B. J., & Anderson, C. A. (2002) · Personality and Social Psychology Bulletin · doi:10.1177/014616702237649 · Cited by 670
A clinical rating scale for progressive supranuclear palsy
Golbe, L. I., & Ohman‐Strickland, P. (2007) · Brain · doi:10.1093/brain/awm032 · Cited by 628
Project AIM: Autism intervention meta-analysis for studies of young children.
Sandbank, M., Bottema-Beutel, K., Crowley, S., Cassidy, M., Dunham, K., Feldman, J. I., Crank, J., Albarran, S. A., Raj, S., Mahbub, P., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000215 · Cited by 606
Trzeciak, S. W., Dellinger, R. P., Parrillo, J. E., Guglielmi, M., Bajaj, J., Abate, N. L., Arnold, R. C., Colilla, S., Zanotti, S. L., & Hollenberg, S. M. (2006) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2006.08.021 · Cited by 615
The role of working memory capacity in retrieval.
Rosen, V. M., & Engle, R. W. (1997) · Journal of Experimental Psychology General · doi:10.1037//0096-3445.126.3.211 · Cited by 627
Myers, J. B., Laudner, K. G., Pasquale, M. R., et al. (2005) · The American Journal of Sports Medicine · doi:10.1177/0363546505281804 · Cited by 609
Development and validation of the Unified Multiple System Atrophy Rating Scale (UMSARS)
Wenning, G. K., Tison, F., Seppi, K., Sampaio, C., Diem, A., Yekhlef, F., Ghorayeb, I., Ory‐Magne, F., Galitzky, M., Scaravilli, T., et al. (2004) · Movement Disorders · doi:10.1002/mds.20255 · Cited by 606
Perceptual measures of performance: fact or fiction?
Ketokivi, M. A., & Schroeder, R. G. (2004) · Journal of Operations Management · doi:10.1016/j.jom.2002.07.001 · Cited by 607
Wainner, R. S., Fritz, J. M., Irrgang, J. J., Boninger, M. L., Delitto, A., & Allison, S. C. (2003) · Spine · doi:10.1097/00007632-200301010-00014 · Cited by 604
Mokkink, L. B., Boers, M., van der Vleuten, C. P. M., Bouter, L. M., Alonso, J., Patrick, D. L., de Vet, H. C. W., & Terwee, C. B. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01179-5 · Cited by 588
Yushkevich, P. A., Pluta, J. B., Wang, H., Xie, L., Ding, S., Gertje, E. C., Mancuso, L., Kliot, D., Das, S. R., & Wolk, D. A. (2014) · Human Brain Mapping · doi:10.1002/hbm.22627 · Cited by 587
Knee injury and Osteoarthritis Outcome Score (KOOS) ‐ validation of a Swedish version
Roos, E. M., Roos, H. P., Ekdahl, C. S., & Lohmander, S. L. (1998) · Scandinavian Journal of Medicine and Science in Sports · doi:10.1111/j.1600-0838.1998.tb00465.x · Cited by 596
Macrophage Polarisation: an Immunohistochemical Approach for Identifying M1 and M2 Macrophages
Barros, M. H. M., Hauck, F., Dreyer, J. H., Kempkes, B., & Niedobitek, G. (2013) · PLoS ONE · doi:10.1371/journal.pone.0080908 · Cited by 616
Adler, J. M. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0025289 · Cited by 588
Social Relationships and Mortality Risk: A Meta-analytic Review
Holt-Lunstad, J., Smith, T., & Layton, J. (2010) · SciVee · doi:10.4016/19865.01 · Cited by 613
Lin, M., Hwang, H., Hu, M., Wu, H. I., Wang, Y., & Huang, F. (2004) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2004.52366.x · Cited by 585
Parent–child interactions and anxiety disorders: an observational study
Hudson, J. L., & Rapee, R. M. (2001) · Behaviour Research and Therapy · doi:10.1016/s0005-7967(00)00107-8 · Cited by 636
Functional Assessment Staging (FAST) in Alzheimer’s Disease: Reliability, Validity, and Ordinality
Sclan, S. G., & Reisberg, . (1992) · International Psychogeriatrics · doi:10.1017/s1041610292001157 · Cited by 596
Gender Dimensions of Brand Personality
Grohmann, B. (2009) · Journal of Marketing Research · doi:10.1509/jmkr.46.1.105 · Cited by 593
Competitive Tension: The Awareness-Motivation-Capability Perspective
Chen, M.-J., Su, K., & Tsai, W. (2007) · Academy of Management Journal · doi:10.5465/amj.2007.24162081 · Cited by 583
Noble, S., Scheinost, D., & Constable, R. T. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116157 · Cited by 601
Management team learning orientation and business unit performance.
Bunderson, J. S., & Sutcliffe, K. M. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.3.552 · Cited by 591
Sample Size Requirements for Testing and Estimating Coefficient Alpha
Bonett, D. G. (2002) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986027004335 · Cited by 593
How reliable are the results from functional magnetic resonance imaging?
Bennett, C. M., & Miller, M. B. (2010) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2010.05446.x · Cited by 579
Taggar, S. (2002) · Academy of Management Journal · doi:10.2307/3069349 · Cited by 834
The Psychology of Rivalry: A Relationally Dependent Analysis of Competition
Kilduff, G. J., Elfenbein, H. A., & Staw, B. M. (2010) · Academy of Management Journal · doi:10.5465/amj.2010.54533171 · Cited by 602
Computerized Virtual Patients in Health Professions Education: A Systematic Review and Meta-Analysis
Cook, D. A., Erwin, P. J., & Triola, M. M. (2010) · Academic Medicine · doi:10.1097/acm.0b013e3181edfe13 · Cited by 570
Social responsibility in new ventures: profiting from a long‐term orientation
Wang, T. T., & Bansal, P. (2012) · Strategic Management Journal · doi:10.1002/smj.1962 · Cited by 596
Indoor time–microenvironment–activity patterns in seven regions of Europe
Schweizer, C., Edwards, R. D., Bayer-Oglesby, L., Gauderman, W. J., Ilacqua, V. A., Jantunen, M. J., Lai, H. K., Nieuwenhuijsen, M. J., & Künzli, N. (2006) · Journal of Exposure Science & Environmental Epidemiology · doi:10.1038/sj.jes.7500490 · Cited by 626
Construct validity of an objective (entropy) categorical measure of diversification strategy
Hoskisson, R. E., Hitt, M. A., Johnson, R. A., & Moesel, D. D. (1993) · Strategic Management Journal · doi:10.1002/smj.4250140305 · Cited by 569
The revised cognitive therapy scale (CTS-R): psychometric properties.
Blackburn, I.-M., James, I. A., Milne, D. L., Baker, C., Standart, S., Garland, A., & Reichelt, F. K. (2001) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465801004040 · Cited by 568
An empirical investigation of KM styles and their effect on corporate performance
Choi, B., & Lee, H. (2003) · Information & Management · doi:10.1016/s0378-7206(02)00060-5 · Cited by 642
Binkley, J. M., Stratford, P. W., Lott, S. A., & Riddle, D. L. (1999) · Physical Therapy · doi:10.1093/ptj/79.4.371 · Cited by 573
Dobson, K. S., Hollon, S. D., Dimidjian, S., Schmaling, K. B., Kohlenberg, R. J., Gallop, R. J., Rizvi, S. L., Gollan, J. K., Dunner, D. L., & Jacobson, N. S. (2008) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.76.3.468 · Cited by 560
Aggregate Confusion: The Divergence of ESG RatingsPreprint
Berg, F., Koelbel, J., & C, R. R. B. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3438533 · Cited by 566
Moyers, T. B., Rowell, L. N., Manuel, J. K., Ernst, D., & Houck, J. M. (2016) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2016.01.001 · Cited by 567
Ruan, W. J., Goldstein, R. B., Chou, S. P., Smith, S. M., Saha, T. D., Pickering, R. P., Dawson, D. A., Huang, B., Stinson, F. S., & Grant, B. F. (2008) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2007.06.001 · Cited by 557
Aarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014) · Implementation Science · doi:10.1186/1748-5908-9-45 · Cited by 559
Janz, B. D., Colquitt, J. A., & Noe, R. A. (1997) · Personnel Psychology · doi:10.1111/j.1744-6570.1997.tb01486.x · Cited by 570
Fourney, D. R., V, Frangou, E., Ryken, T. C., DiPaola, C. P., Shaffrey, C. I., Berven, S., Bilsky, M. H., Harrop, J. S., Fehlings, M. G., Boriani, S., et al. (2011) · Journal of Clinical Oncology · doi:10.1200/jco.2010.34.3897 · Cited by 559
Stice, E., Shaw, H., Bohon, C., Marti, C. N., & Rohde, P. (2009) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0015168 · Cited by 553
Hekkert, P., Snelders, D., & van Wieringen, P. C. W. (2003) · British Journal of Psychology · doi:10.1348/000712603762842147 · Cited by 588
Tiedemann, A., Shimada, H., Sherrington, C., Murray, S. M., & Lord, S. R. (2008) · Age and Ageing · doi:10.1093/ageing/afn100 · Cited by 550
Goniometric Reliability in a Clinical Setting
Rothstein, J. M., Miller, P. J., & Roettger, R. F. (1983) · Physical Therapy · doi:10.1093/ptj/63.10.1611 · Cited by 549
Gustavsson, A., Neeter, C., Thomeé, P., Silbernagel, K. G., Augustsson, J., Thomeé, R., & Karlsson, J. (2006) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-006-0045-6 · Cited by 549
Keller, R. T. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.1.202 · Cited by 547
Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2010) · Academic Medicine · doi:10.1097/acm.0b013e3181d6c319 · Cited by 550
Validation of the Headache Impact Test (HIT-6™) across episodic and chronic migraine
Yang, M. S., Rendas‐Baum, R., Varon, S. F., & Kosinski, M. R. (2010) · Cephalalgia · doi:10.1177/0333102410379890 · Cited by 579
The reliability of the Wolf Motor Function Test for assessing upper extremity function after stroke
Morris, D. M., Uswatte, G., Crago, J. E., Cook, E. W., & Taub, E. (2001) · Archives of Physical Medicine and Rehabilitation · doi:10.1053/apmr.2001.23183 · Cited by 543
Rating the ratings: Assessing the psychometric quality of rating data.
Saal, F. E., Downey, R. G., & Lahey, M. A. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.2.413 · Cited by 545
Operacionalização de adaptação transcultural de instrumentos de aferição usados em epidemiologia
Reichenheim, M. E., & de Moraes, C. L. (2007) · Revista de Saúde Pública · doi:10.1590/s0034-89102006005000035 · Cited by 521
Development and verification of the PAM50-based Prosigna breast cancer gene signature assay
Wallden, B., Storhoff, J. J., Nielsen, T. O., Dowidar, N., Schaper, C., Ferree, S., Liu, S., Leung, S. C. Y., Geiss, G. K., Snider, J., et al. (2015) · BMC Medical Genomics · doi:10.1186/s12920-015-0129-6 · Cited by 563
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
