The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 2,701–2,800 of 23,883
Weaver, R. R., Lemonde, M., Payman, N., & Goodman, W. M. (2013) · Social Science & Medicine · doi:10.1016/j.socscimed.2013.11.033 · Cited by 107
Validation of the simplified therapeutic intervention scoring system on an independent database
Moreno, R. P., & Morais, P. (1997) · Intensive Care Medicine · doi:10.1007/s001340050387 · Cited by 105
Daugherty, A. M., Haacke, E. M., & Raz, N. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.4717-14.2015 · Cited by 105
Minimally Invasive Total Knee Arthroplasty: A Systematic Review
Khanna, A., Gougoulias, N. E., Longo, U. G., & Maffulli, N. (2009) · Orthopedic Clinics of North America · doi:10.1016/j.ocl.2009.05.003 · Cited by 104
Hodge, D. R., & Lacasse, J. R. (2011) · Journal of Social Work Education · doi:10.5175/jswe.2011.201000024 · Cited by 105
Thoma, N. C., McKay, D., Gerber, A. J., Milrod, B. L., Edwards, A. R., & Kocsis, J. H. (2011) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2011.11030433 · Cited by 108
Why do spatial abilities predict mathematical performance?
Tosto, M. G., Hanscombe, K. B., Haworth, C., Davis, O., Petrill, S., Dale, P. S., Malykh, S., Plomin, R., & Kovas, Y. (2014) · Developmental Science · doi:10.1111/desc.12138 · Cited by 118
Concordance of DSM-IV Axis I and II diagnoses by personal and informant's interview
Schneider, B., Maurer, K., Sargk, D., Heiskel, H., Weber, B., Frölich, L., Georgi, K., Fritze, J., & Seidler, A. (2004) · Psychiatry Research · doi:10.1016/j.psychres.2004.02.015 · Cited by 105
Mandibular sexual dimorphism analysis in CBCT scans
Gamba, T. D. O., Alves, M. C., & Neto, F. H. (2015) · Journal of Forensic and Legal Medicine · doi:10.1016/j.jflm.2015.11.024 · Cited by 105
Smith‐Ryan, A. E., Mock, M. G., Ryan, E. D., Gerstner, G. R., Trexler, E. T., & Hirsch, K. R. (2016) · Clinical Nutrition · doi:10.1016/j.clnu.2016.05.006 · Cited by 104
Croall, I. D., Lohner, V., Moynihan, B., Khan, U., Hassan, A., O'Brien, J. T., Morris, R. G., Tozer, D. J., Cambridge, V. C., Harkness, K., et al. (2017) · Clinical Science · doi:10.1042/cs20170146 · Cited by 105
Zhou, N., & Yadav, A. K. (2017) · Educational Technology Research and Development · doi:10.1007/s11423-017-9533-2 · Cited by 122
Lohr, C., Braumann, K.-M., Reer, R., Schroeder, J., & Schmidt, T. (2018) · European Journal of Applied Physiology · doi:10.1007/s00421-018-3867-2 · Cited by 105
Kompetenzmessung in der beruflichen Bildung
Winther, E. (2010) · doi:10.3278/6004148w · Cited by 107
Williams, Z. J., Suzman, E., & Woynaroski, T. G. (2021) · Ear and Hearing · doi:10.1097/aud.0000000000001005 · Cited by 120
Palpation identification of spinous processes in the lumbar spine
Harlick, J. C., Milosavljevic, S., & Milburn, P. D. (2006) · Manual Therapy · doi:10.1016/j.math.2006.02.008 · Cited by 105
Keller, K. L., Pietrobelli, A., Johnson, S. L., & Faith, M. S. (2006) · International Journal of Obesity · doi:10.1038/sj.ijo.0803318 · Cited by 110
A general procedure for the estimation of interdependence.
Kenny, D. A., & Judd, C. M. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.1.138 · Cited by 105
Performance and Repeatability of the NEI-VFQ-25 in Patients With Dry Eye
Nichols, K. K., Mitchell, G. L., & Zadnik, K. (2002) · Cornea · doi:10.1097/00003226-200208000-00009 · Cited by 104
Enhancing the reading of dyslexic children by reading acceleration and auditory masking.
Breznitz, Z. (1997) · Journal of Educational Psychology · doi:10.1037/0022-0663.89.1.103 · Cited by 107
Kottner, J., & Dassen, T. W. N. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.11.005 · Cited by 107
Lundon, K., Li, A. M. W. Y., & Bibershtein, S. (1998) · Spine · doi:10.1097/00007632-199809150-00013 · Cited by 105
Regional White Matter Pathology in Mild Cognitive Impairment
Delano‐Wood, L., Abeles, N., Sacco, J. M., Wierenga, C. E., Horne, N. R., & Bozoki, A. (2008) · Stroke · doi:10.1161/strokeaha.107.502534 · Cited by 109
Female physical attractiveness in Britain and Japan: a cross‐cultural study
Swami, V., Caprario, C., Tovée, M. J., & Furnham, A. F. (2006) · European Journal of Personality · doi:10.1002/per.568 · Cited by 102
A new method for measuring passive length–tension properties of human gastrocnemius muscle in vivo
Hoang, P. D., Gorman, R. B., Todd, G., Gandevia, S. C., & Herbert, R. (2004) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2004.05.046 · Cited by 105
In-chair movement: validity, reliability and implications for measuring sitting discomfort
Fenety, P. A., Putnam, C. A., & Walker, J. M. T. (2000) · Applied Ergonomics · doi:10.1016/s0003-6870(00)00003-x · Cited by 105
Stöber, J., Tepperwien, S., & Staak, M. (2000) · Anxiety Stress & Coping · doi:10.1080/10615800008549263 · Cited by 114
Human fetal and neonatal movement patterns: Gender differences and fetal‐to‐neonatal continuity
Almli, C. R., Ball, R. H., & Wheeler, M. E. (2001) · Developmental Psychobiology · doi:10.1002/dev.1019 · Cited by 106
Hand, B. M., & Choi, A. (2010) · Research in Science Education · doi:10.1007/s11165-009-9155-8 · Cited by 105
The presence of friends increases food intake in youth
Salvy, S., Howard, M., Read, M., & Mele, E. (2009) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.2009.27658 · Cited by 107
Goetghebeur, M. M., Wagner, M., Khoury, H., Rindress, D., Grégoire, J., & Deal, C. (2010) · Cost Effectiveness and Resource Allocation · doi:10.1186/1478-7547-8-4 · Cited by 105
Energy Cost of Propulsion in Standard and Ultralight Wheelchairs in People With Spinal Cord Injuries
Beekman, C. E., Miller-Porter, L., & Schoneberger, M. (1999) · Physical Therapy · doi:10.1093/ptj/79.2.146 · Cited by 105
Barch, D. M. (1999) · Schizophrenia Research · doi:10.1016/s0920-9964(99)00025-0 · Cited by 105
Trust dynamics in acquisitions: A case survey
Stahl, G. K., Larsson, R., Kremershof, I., & Sitkin, S. B. (2011) · Human Resource Management · doi:10.1002/hrm.20448 · Cited by 104
Comparison of therapeutic factors in group and individual treatment processes.
Holmes, S. E., & Jr. Kivlighan Dennis, M. (2000) · Journal of Counseling Psychology · doi:10.1037/0022-0167.47.4.478 · Cited by 105
van Riet, R. P., Glabbeek, F. V., Weerdt, W. D., et al. (2007) · Journal of Bone and Joint Surgery - British Volume · doi:10.1302/0301-620x.89b3.18099 · Cited by 105
Head, D. M., Raz, N., Gunning‐Dixon, F. M., Williamson, A. L., & Acker, J. D. (2002) · Psychology and Aging · doi:10.1037//0882-7974.17.1.72 · Cited by 105
Devanand, D. P., Sackeïm, H. A., Brown, R. P., & Mayeux, R. P. (1989) · Archives of Neurology · doi:10.1001/archneur.1989.00520440036018 · Cited by 105
Análise de concordância em estudos clínicos e experimentais
Miot, H. A. (2016) · Jornal Vascular Brasileiro · doi:10.1590/1677-5449.004216 · Cited by 103
Aas, M., Steen, N. E., Agartz, I., Aminoff, S. R., Lorentzen, S., Sundet, K., Andreassen, O. A., & Melle, I. (2012) · Psychiatry Research · doi:10.1016/j.psychres.2011.12.045 · Cited by 103
Wu, J.-Y. (2017) · Computers & Education · doi:10.1016/j.compedu.2016.10.010 · Cited by 104
Lanza, I. R., Bhagra, S., Nair, K. S., & Port, J. D. (2011) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.22733 · Cited by 103
The lay assessment of subclinical depression in daily life.
Mehl, M. R. (2006) · Psychological Assessment · doi:10.1037/1040-3590.18.3.340 · Cited by 105
Lenhard, F., Vigerland, S., Andersson, E., Rück, C., Mataix-Cols, D., Thulin, U., Ljótsson, B., & Serlachius, E. (2014) · PLoS ONE · doi:10.1371/journal.pone.0100773 · Cited by 104
Trusting problem gamblers: Reliability and validity of self-reported gambling behavior.
Hodgins, D. C., & Makarchuk, K. (2003) · Psychology of Addictive Behaviors · doi:10.1037/0893-164x.17.3.244 · Cited by 131
Time-based indicators of emotional complexity: Interrelations and correlates.
Grühn, D., Lumley, M. A., Diehl, M. K., & Labouvie‐Vief, G. (2012) · Emotion · doi:10.1037/a0030363 · Cited by 123
Marras, S., Killen, S. S., Claireaux, G., et al. (2011) · Journal of Experimental Biology · doi:10.1242/jeb.056648 · Cited by 104
Fujino, Y., Uehara, M., Izumi, H., Nagata, T., Muramatsu, K., Kubo, T., Oyama, I., & Matsuda, S. (2015) · Journal of Occupational Health · doi:10.1539/joh.15-0135-oa · Cited by 109
Governance-Knowledge Fit in Systems Development Projects
Tiwana, A. (2008) · Information Systems Research · doi:10.1287/isre.1070.0164 · Cited by 104
Technical Adequacy of the SWPBIS Tiered Fidelity Inventory
McIntosh, K., Massar, M. M., Algozzine, R. F., George, H. P., Horner, R. H., Lewis, T. J., & Swain-Bradway, J. (2016) · Journal of Positive Behavior Interventions · doi:10.1177/1098300716637193 · Cited by 104
Ammann, C., Lindquist, M. A., & Celnik, P. A. (2017) · Brain stimulation · doi:10.1016/j.brs.2017.04.003 · Cited by 104
Analysis of sexual dimorphism of craniofacial traits using geometric morphometric techniques
González, P., Bernal, V., & Pérez, S. I. (2009) · International Journal of Osteoarchaeology · doi:10.1002/oa.1109 · Cited by 108
Boer, K. V. C. W.-D., Visser, K., Heimans, L., et al. (2012) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2011-200736 · Cited by 104
School type and academic culture: evidence for the differentiation–polarization theory
Houtte, M. V. (2006) · Journal of Curriculum Studies · doi:10.1080/00220270500363661 · Cited by 104
Reflective Functioning in Panic Disorder Patients: a Pilot Study
Rudden, M. G., Milrod, B. L., Target, M., Ackerman, S., & Graf, E. (2006) · Journal of the American Psychoanalytic Association · doi:10.1177/00030651060540040109 · Cited by 104
Rasch analysis provides new insights into the measurement properties of the neck disability index
van der Velde, G., Beaton, D., Hogg‐Johnson, S., Hurwitz, E. L., & Tennant, A. (2009) · Arthritis Care & Research · doi:10.1002/art.24399 · Cited by 104
Dade, L., Gao, F.-Q., Kovacevic, N., Roy, P., Rockel, C., O'Toole, C., Lobaugh, N., Feinstein, A., Levine, B., & Black, S. E. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.03.023 · Cited by 108
Group Composition, Creative Synergy, and Group Performance
Taggar, S. (2001) · The Journal of Creative Behavior · doi:10.1002/j.2162-6057.2001.tb01050.x · Cited by 104
Relative and Absolute Reliability of the KT-2000 Arthrometer for Uninjured Knees
Myrer, J. W., Schulthies, S. S., & Fellingham, G. W. (1996) · The American Journal of Sports Medicine · doi:10.1177/036354659602400119 · Cited by 104
Effectiveness of psychotherapy for severe somatoform disorder: meta-analysis
Koelen, J. A., Houtveen, J. H., Abbass, A. A., Luyten, P., Eurelings-Bontekoe, E. H. M., Kloth, S. A. M., Bühring, M. E. F., & Geenen, R. (2014) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.112.121830 · Cited by 104
Taniguchi, T., Sakata, Y., Ohtani, T., Mizote, I., Takeda, Y., Asano, Y., Masuda, M., Minamiguchi, H., Kanzaki, M. M., Ichibori, Y., et al. (2013) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2013.10.018 · Cited by 104
Vangeneugden, T., Laenen, A., Geys, H., Renard, D., & Molenberghs, G. (2005) · Biometrics · doi:10.1111/j.0006-341x.2005.031040.x · Cited by 104
Qualitative assessment of brain morphology in acute and chronic schizophrenia
Lieberman, J. A., Bogerts, B., Degreef, G., Ashtari, M., Lantos, G., & Alvir, J. M. (1992) · American Journal of Psychiatry · doi:10.1176/ajp.149.6.784 · Cited by 104
AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and ChallengePreprint
Valstar, M., Gratch, J. S., Schuller, B. W., Ringeval, F., Lalanne, D., Torres, M. T., Scherer, S. J., Stratou, G., Cowie, R., & Pantić, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01600 · Cited by 129
Shakeshaft, N. G., Trzaskowski, M., McMillan, A., Rimfeld, K., Krapohl, E., Haworth, C., Dale, P. S., & Plomin, R. (2013) · PLoS ONE · doi:10.1371/journal.pone.0080341 · Cited by 112
Single subject fMRI test–retest reliability metrics and confounding factors
Gorgolewski, K. J., Storkey, A., Bastin, M. E., Whittle, I. R., & Pernet, C. R. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.10.085 · Cited by 103
The concentration of glutathione in human erythrocytes is a heritable trait
Erve, T. J. V. ‘., Wagner, B. A., Ryckman, K. K., Raife, T. J., & Buettner, G. R. (2013) · Free Radical Biology and Medicine · doi:10.1016/j.freeradbiomed.2013.08.002 · Cited by 108
Ianiro-Dahm, P. M., Schermuly, C. C., & Kauffeld, S. (2012) · Coaching An International Journal of Theory Research and Practice · doi:10.1080/17521882.2012.740489 · Cited by 116
Bertels, H. M. J., Kleinschmidt, E. J., & Koen, P. A. (2011) · Journal of Product Innovation Management · doi:10.1111/j.1540-5885.2011.00836.x · Cited by 103
Pound, N., Lawson, D. W., Toma, A. M., Richmond, S., Zhurov, A. I., & Penton‐Voak, I. S. (2014) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2014.1639 · Cited by 103
Tseng, C. W., Kyme, P. A., Low, J., Rocha, M. A., Alsabeh, R., Miller, L. G., Otto, M. J., Arditi, M., Diep, B. A., Nizet, V., et al. (2009) · PLoS ONE · doi:10.1371/journal.pone.0006387 · Cited by 103
Continuous Passive Motion after Repair of the Rotator Cuff. A Prospective Outcome Study*
LaStayo, P. C., Wright, T. W., Jaffe, R., et al. (1998) · Journal of Bone and Joint Surgery · doi:10.2106/00004623-199807000-00009 · Cited by 103
Jiang, L., & Probst, T. M. (2016) · Journal of Safety Research · doi:10.1016/j.jsr.2016.03.002 · Cited by 103
Point-of-care wound visioning technology: Reproducibility and accuracy of a wound measurement app
Wang, S. C., Anderson, J. A. E., Evans, R., et al. (2017) · PLoS ONE · doi:10.1371/journal.pone.0183139 · Cited by 103
Marchetti, A., Barberis, M. C. P., Franco, R., Luca, G. D., Pace, M. V., Staibano, S., Volante, M., Buttitta, F., Guerini‐Rocco, E., Righi, L., et al. (2017) · Journal of Thoracic Oncology · doi:10.1016/j.jtho.2017.07.031 · Cited by 103
Young, I. A., Dunning, J. R., Butts, R., Cleland, J. A., & Fernández‐de‐las‐Peñas, C. (2018) · Cephalalgia · doi:10.1177/0333102418772584 · Cited by 103
Mojtahed, A., Kelly, M., Herlihy, A. H., Kin, S., Wilman, H., McKay, A., Kelly, M., Milanesi, M., Neubauer, S., Thomas, E. L., et al. (2018) · Abdominal Radiology · doi:10.1007/s00261-018-1701-2 · Cited by 103
EmojiGrid: A 2D Pictorial Scale for the Assessment of Food Elicited Emotions
Toet, A., Kaneko, D., Ushiama, S., Hoving, S., de Kruijf, I., Brouwer, A.-M., Kallen, V., & van Erp, J. B. F. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.02396 · Cited by 105
Did Ezekiel Have Temporal Lobe Epilepsy?
Altschuler, E. L. (2002) · Archives of General Psychiatry · doi:10.1001/archpsyc.59.6.560 · Cited by 103
Center of pressure progression characteristics under the plantar region for elderly adults
Chiu, M.-C., Wu, H., Chang, L.-Y., et al. (2012) · Gait & Posture · doi:10.1016/j.gaitpost.2012.08.010 · Cited by 103
An examination of the interactions between mothers and children with anxiety disorders
Gar, N. S., & Hudson, J. L. (2008) · Behaviour Research and Therapy · doi:10.1016/j.brat.2008.08.006 · Cited by 120
The acoustic startle reflex and its modulation: effects of age and gender in humans
Ludewig, K., Ludewig, S., Seitz, A., Obrist, M., Geyer, M. A., & Vollenweider, F. X. (2003) · Biological Psychology · doi:10.1016/s0301-0511(03)00074-7 · Cited by 103
Griffin, J. M., Greiner, B. A., Stansfeld, S. A., & Marmot, M. G. (2007) · Journal of Occupational Health Psychology · doi:10.1037/1076-8998.12.4.334 · Cited by 112
Hu, S., Pruessner, J. C., Coupé, P., & Collins, D. L. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.02.032 · Cited by 108
Story, T. N., Berg, C. A., Smith, T. W., Beveridge, R. M., Henry, N. J. M., & Pearce, G. E. (2007) · Psychology and Aging · doi:10.1037/0882-7974.22.4.719 · Cited by 103
Weber, U., Østergaard, M., Lambert, R. G. W., et al. (2014) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2014-205408 · Cited by 103
Reliability of simple portable tests of physical performance in older people after hip fracture
Sherrington, C., & Lord, S. R. (2005) · Clinical Rehabilitation · doi:10.1191/0269215505cr833oa · Cited by 103
A Validated Grading Scale for Crow's Feet
CARRUTHERS, A. J., CARRUTHERS, J. D. A., Hardas, B., et al. (2008) · Dermatologic Surgery · doi:10.1111/j.1524-4725.2008.34367.x · Cited by 103
Richardson, J. T. E. (1996) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03203631 · Cited by 103
Unraveling scientific impact: Citation types in marketing journals
Stremersch, S., Camacho, N., Vanneste, S., & Verniers, I. (2015) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2014.09.004 · Cited by 102
Harden, R. N., Maihofner, C., Abousaad, E., Vatine, J.-J., Kirsling, A., Perez, R. S. G. M., Kuroda, M., Brunner, F., Stanton-Hicks, M., Marinus, J., et al. (2017) · Pain · doi:10.1097/j.pain.0000000000000927 · Cited by 102
The Relationship between Symbolic Play and Executive Function in Young Children
Kelly, R., Dissanayake, C., Ihsen, E., & Hammond, S. (2011) · Australasian Journal of Early Childhood · doi:10.1177/183693911103600204 · Cited by 112
Zekveld, A., George, E., Kramer, S., Goverts, S., & Houtgast, T. (2007) · Journal of Speech, Language and Hearing Research · doi:10.1044/1092-4388(2007/040 · Cited by 105
Yang, Z., Craddock, R. C., Margulies, D. S., Yan, C., & Milham, M. P. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.02.014 · Cited by 102
Zevin, B., Bonrath, E. M., Aggarwal, R. K., Dedy, N. J., Ahmed, N. A., & Grantcharov, T. (2013) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2013.01.003 · Cited by 102
Wolfe, E. W. (1997) · Assessing Writing · doi:10.1016/s1075-2935(97)80006-2 · Cited by 102
Smith, J. D., Dishion, T. J., Shaw, D. S., & Wilson, M. N. (2013) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0033950 · Cited by 106
Assessing How Teachers Enhance Self-Regulated Learning: A Multiperspective Approach
Dignath, C., Dickhäuser, O., & Büttner, G. (2013) · Journal of Cognitive Education and Psychology · doi:10.1891/1945-8959.12.3.338 · Cited by 116
Smith-Bynum, M. A., Anderson, R. E., Davis, B. L., Franco, M. G., & English, D. (2016) · Child Development · doi:10.1111/cdev.12562 · Cited by 80
Kainz, H., Hoang, H., Stockton, C., et al. (2017) · Journal of Applied Biomechanics · doi:10.1123/jab.2016-0282 · Cited by 102
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
