The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 4,501–4,600 of 23,883
Hoffart, A., Borge, F.-M., Sexton, H., & Clark, D. M. (2009) · Behavior Therapy · doi:10.1016/j.beth.2007.12.003 · Cited by 64
Leopold, R., & Backenstrass, M. (2015) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2014.12.016 · Cited by 63
Aas, M., Aminoff, S. R., Vik Lagerberg, T., Etain, B., Agartz, I., Andreassen, O. A., & Melle, I. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.03.046 · Cited by 64
Interrater relability of cognitive-behavioral case formulations
Persons, J. B., Mooney, K. A., & Padesky, C. A. (1995) · Cognitive Therapy and Research · doi:10.1007/bf02229674 · Cited by 64
Influence of Music Therapy on Coping Skills and Anger Management in Forensic Psychiatric Patients
Hakvoort, L., Bogaerts, S., Thaut, M. H., & Spreen, M. (2013) · International Journal of Offender Therapy and Comparative Criminology · doi:10.1177/0306624x13516787 · Cited by 64
Cha, Y., Baek, S., Ahn, G., Hyoungsuk, L., Lee, B., Shin, J.-E., & Jang, D. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.08.027 · Cited by 68
Laguna, P. A. L., Combes, A. J. E., Streffer, J. R., Einstein, S. G., Timmers, M., Williams, S., & Dell’Acqua, F. (2020) · NeuroImage Clinical · doi:10.1016/j.nicl.2020.102168 · Cited by 64
Moreta, M. C., Fleet, A., Reebye, R. N., et al. (2020) · Archives of Rehabilitation Research and Clinical Translation · doi:10.1016/j.arrct.2020.100071 · Cited by 64
Fakih‐Gomez, N., & Kadouch, J. A. (2021) · Aesthetic Plastic Surgery · doi:10.1007/s00266-021-02479-x · Cited by 64
Zhu, H., Fang, X., Zhao, F., et al. (2021) · Spine · doi:10.1097/brs.0000000000004303 · Cited by 64
Accelerometers in Our Pocket: Does Smartphone Accelerometer Technology Provide Accurate Data?
Grouios, G., Ziagkas, E., Loukovitis, A., et al. (2022) · Sensors · doi:10.3390/s23010192 · Cited by 64
Replicable brain–phenotype associations require large-scale neuroimaging data
Liu, S., Abdellaoui, A., Verweij, K. J. H., & van Wingen, G. A. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01642-5 · Cited by 65
Olesen, S. S., van Goor, H., Bouwense, S. A. W., et al. (2012) · Regional Anesthesia & Pain Medicine · doi:10.1097/aap.0b013e3182632c40 · Cited by 64
Brady, N. C., Warren, S. F., Fleming, K., Keller, J. P., & Sterling, A. M. (2013) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2013/12-0341 · Cited by 64
Chou, S., & Min, H. (2008) · Computers & Education · doi:10.1016/j.compedu.2008.09.006 · Cited by 64
Normal kinematics of the neck: The interplay between the cervical and thoracic spines
Tsang, S. M. H., Szeto, G. P. Y., & Lee, R. (2013) · Manual Therapy · doi:10.1016/j.math.2013.03.002 · Cited by 64
Cameron, D. M., Bohannon, R. W., Garrett, G. E., Owen, S. V., & Cameron, D. A. (2003) · Clinical Biomechanics · doi:10.1016/s0268-0033(03)00023-8 · Cited by 64
Observational assessment of situation awareness, team differences and training implications
Patrick, J., James, N., Ahmed, A., & Halliday, P. (2006) · Ergonomics · doi:10.1080/00140130600576272 · Cited by 64
A COMPARISON OF VOLUNTARY AND INVOLUNTARY MEASURES OF ELECTROMECHANICAL DELAY
Hopkins, J. T., Feland, J. B., & Hunter, I. (2007) · International Journal of Neuroscience · doi:10.1080/00207450600773764 · Cited by 65
The interrater reliability of the Structured Interview for DSM-IV Personality
Jane, J. S., Pagan, J. L., Turkheimer, E. N., et al. (2006) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2006.01.009 · Cited by 64
Psychometric Characteristics of Role-Play Assessments of Social Skill in Schizophrenia
Bellack, A. S., Brown, C. H., & Thomas-Lohrman, S. (2006) · Behavior Therapy · doi:10.1016/j.beth.2006.01.005 · Cited by 64
Establishing the Criterion Validity and Reliability of Common Methods for Quantifying Training Load
Wallace, L., Slattery, K., Impellizzeri, F. M., & Coutts, A. J. (2014) · The Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000000416 · Cited by 64
Lindqvist, S. M., Duncan, A., Shepstone, L., Watts, F., & Pearce, S. A. (2005) · Journal of Interprofessional Care · doi:10.1080/13561820400026071 · Cited by 64
Dawson, A. P., Schlüter, P. J., Hodges, P. W., Stewart, S., & Turner, C. T. (2011) · Pain · doi:10.1016/j.pain.2011.02.041 · Cited by 69
Interpretation Bias and Anxiety in Childhood: Stability, Specificity and Longitudinal Associations
Creswell, C., & O’Connor, T. G. (2010) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465810000494 · Cited by 64
Comparison of mammographic densities and their determinants in women from Japan and Hawaii
Maskarinec, G., Nagata, C., Shimizu, H., et al. (2002) · International Journal of Cancer · doi:10.1002/ijc.10673 · Cited by 64
Schreuders, T. A. R., Roebroeck, M. E., Jaquet, J.-B., Hovius, S. E. R., & Stam, H. J. (2004) · The Journal Of Hand Surgery · doi:10.1016/j.jhsa.2003.10.024 · Cited by 64
Optimization of metal artefact reduction (MAR) sequences for MRI of total hip prostheses
Toms, A. P., Smith-Bateman, C., Malcolm, P. N., et al. (2010) · Clinical Radiology · doi:10.1016/j.crad.2009.12.014 · Cited by 64
Heterosexual Men’s Confrontation of Sexual Prejudice: The Role of Precarious Manhood
Kroeper, K. M., Sanchez, D. T., & Himmelstein, M. S. (2013) · Sex Roles · doi:10.1007/s11199-013-0306-z · Cited by 64
Zhang, Y., Schuff, N. W., Woolley, S. C., Chiang, G. C.-Y., Boreta, L. C., Laxamana, J., Katz, J., & Weiner, M. W. (2011) · Amyotrophic Lateral Sclerosis · doi:10.3109/17482968.2011.593036 · Cited by 64
Newton, M., & Waddell, G. (1991) · Physiotherapy · doi:10.1016/s0031-9406(10)61906-2 · Cited by 64
Blondheim, D. S., Beeri, R., Feinberg, M. S., et al. (2010) · Journal of the American Society of Echocardiography · doi:10.1016/j.echo.2009.12.020 · Cited by 64
Husebo, B. S., Strand, L. I., Moe‐Nilssen, R., Husebo, S. B., & Ljunggren, A. E. (2009) · Scandinavian Journal of Caring Sciences · doi:10.1111/j.1471-6712.2008.00606.x · Cited by 64
Characterization of Adrenal Masses With Diffusion-Weighted Imaging
Sandrasegaran, K., Patel, A. A., Ramaswamy, R. S., Samuel, V. P., Northcutt, B. G., Frank, M. S., & Francis, I. R. (2011) · American Journal of Roentgenology · doi:10.2214/ajr.10.4583 · Cited by 64
Crespi, C. M., Alfonso, V. H., Whaley, S. E., & Wang, M. C. (2012) · Pediatric Research · doi:10.1038/pr.2011.37 · Cited by 64
Pilot study of the accuracy of bedside glucometry in the intensive care unit
Ray, J. G., Hamielec, C. M., & Mastracci, T. M. (2001) · Critical Care Medicine · doi:10.1097/00003246-200111000-00025 · Cited by 64
Development of a School Nutrition–Environment State Policy Classification System (SNESPCS)
Mâsse, L. C., Frosh, M. M., Chriqui, J. F., Yaroch, A. L., Agurs‐Collins, T., Blanck, H. M., Atienza, A. A., McKenna, M. L., & Igoe, J. F. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.07.017 · Cited by 64
Vollenweider, F. X., Barro, M., Csomor, P., & Feldon, J. (2006) · Biological Psychiatry · doi:10.1016/j.biopsych.2006.03.058 · Cited by 65
The Role of Students’ Home Language in Science Achievement: A multilevel approach
Laere, E. V., Aesaert, K., & van Braak, J. (2014) · International Journal of Science Education · doi:10.1080/09500693.2014.936327 · Cited by 76
An experimental test of parenting practices as a mediator of early childhood physical aggression
Brotman, L. M., O’Neal, C. R., Huang, K., Gouley, K. K., Rosenfelt, A., & Shrout, P. E. (2009) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2008.01929.x · Cited by 65
Self-citationKühl, M., Wagner, R. J., Bauder, M., et al. (2012) · BMC Medical Education · doi:10.1186/1472-6920-12-101 · Cited by 64
Detailed semiautomated MRI based morphometry of the neonatal brain: Preliminary results
Nishida, M., Makris, N., Kennedy, D. N., Vangel, M. G., Fischl, B. R., Krishnamoorthy, K. S., Caviness, V. S., & Grant, P. E. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.05.020 · Cited by 63
Entwicklung und Validierung eines Fragebogens zu räumlichen Strategien
Münzer, S., & Hölscher, C. (2011) · Diagnostica · doi:10.1026/0012-1924/a000040 · Cited by 64
Hama, Y., Kanazawa, M., Minakuchi, S., et al. (2014) · PubMed · Cited by 64
Automatic Quantification of Subarachnoid Hemorrhage on Noncontrast CT
Boers, A., Zijlstra, I. J., Gathier, C., van den Berg, R., Slump, C., Marquering, H., & Majoie, C. (2014) · American Journal of Neuroradiology · doi:10.3174/ajnr.a4042 · Cited by 62
Carter, A. J., Marshall, H. H., Heinsohn, R., et al. (2011) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-011-1263-6 · Cited by 64
Gådin, K. G., & Hammarström, A. (2005) · European Journal of Public Health · doi:10.1093/eurpub/cki097 · Cited by 67
Smith, E., Walsh, L. C., Doyle, J., Greene, B. R., & Blake, C. M. (2015) · Gait & Posture · doi:10.1016/j.gaitpost.2015.10.004 · Cited by 64
Measuring the face-sensitive N170 with a gaming EEG system: A validation study
de Lissa, P., Sörensen, S., Badcock, N. A., Thie, J., & McArthur, G. M. (2015) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2015.05.025 · Cited by 62
Echegaray, S., Gevaert, O., Shah, R. P., Kamaya, A., Louie, J. D., Kothary, N., & Napel, S. A. (2015) · Journal of medical imaging · doi:10.1117/1.jmi.2.4.041011 · Cited by 63
Travers, G. J. S., Nichols, D. S., Farooq, A., Racinais, S., & Périard, J. D. (2016) · Temperature · doi:10.1080/23328940.2016.1171281 · Cited by 66
Althuizen, N., & Reichel, A. (2016) · Journal of Management Information Systems · doi:10.1080/07421222.2016.1172439 · Cited by 76
Cridge, H., MacLeod, A. G., Pachtinger, G. E., et al. (2018) · Journal of Veterinary Internal Medicine · doi:10.1111/jvim.15039 · Cited by 64
Bril, S. I., Wendrich, A. W., Swartz, J. E., Wegner, I., Pameijer, F. A., Smid, E. J., Bol, G. H., Pothen, A. J., & de Bree, R. (2019) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-019-05307-w · Cited by 67
Lee, K. C., Bamford, A. L., Gardiner, F., Agovino, A., ter Horst, B., Bishop, J. R. B., Sitch, A. J., Grover, L. M., Logan, A., & Moiemen, N. S. (2019) · Burns · doi:10.1016/j.burns.2019.02.002 · Cited by 67
Bischoff, K. M., Gielnik, M. M., & Frese, M. (2020) · Strategic Entrepreneurship Journal · doi:10.1002/sej.1364 · Cited by 64
The Handbook of Research on Top Management Teams
(2011) · Edward Elgar Publishing eBooks · doi:10.4337/9780857933201 · Cited by 63
Effect of a neoprene sleeve on knee joint kinesthesis: influence of different testing procedures
Birmingham, T. B., Inglis, J. T., KRAMER, J. F., et al. (2000) · Medicine & Science in Sports & Exercise · doi:10.1097/00005768-200002000-00009 · Cited by 64
A twin study of weight loss and metabolic efficiency
Hainer, V., Stunkard, A. J., Kunešová, M., Pařízková, J., Štich, V., & Allison, D. B. (2001) · International Journal of Obesity · doi:10.1038/sj.ijo.0801559 · Cited by 67
Use of the Longitudinal Arch Angle to Predict Dynamic Foot Posture in Walking
McPoil, T. G., & Cornwall, M. W. (2005) · Journal of the American Podiatric Medical Association · doi:10.7547/0950114 · Cited by 64
Retrospective Self‐Report of Alcohol Consumption: Test‐Retest Reliability by Telephone
Cohen, B. B., & Vinson, D. C. (1995) · Alcoholism Clinical and Experimental Research · doi:10.1111/j.1530-0277.1995.tb01595.x · Cited by 64
Reliability of the time trade-off technique of utility assessment in patients with retinal disease
Hollands, H., Lam, M., Pater, J., et al. (2001) · Canadian Journal of Ophthalmology · doi:10.1016/s0008-4182(01)80041-7 · Cited by 64
Auleley, G.-R., Benbouazza, K., Spoorenberg, A. J. P. L., Collantes‐Estévez, E., Hajjaj-Hassouni, N., van der Heijde, D. M. F. M., & Dougados, M. (2002) · Arthritis Care & Research · doi:10.1002/art.10798 · Cited by 63
Rossini, E. D., & O'Connor, M. A. (1995) · Psychological Reports · doi:10.2466/pr0.1995.77.3.751 · Cited by 64
Somers, D. L., Hanson, J. A., Kedzierski, C. M., et al. (1997) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1997.25.3.192 · Cited by 64
Bandy, W. D., & Hanten, W. P. (1993) · Physical Therapy · doi:10.1093/ptj/73.7.455 · Cited by 64
Carpenter, K. M., Brooks, A. C., Vosburg, S. K., & Nunes, E. V. (2004) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2003.11.015 · Cited by 64
The influence of gatekeeping and utilization review on patient satisfaction
Kerr, E. A., Hays, R. D., Mitchinson, A., Lee, M. L., & Siu, A. L. (1999) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.1999.00336.x · Cited by 64
Measurement of stiffness during simulated spinal physiotherapy
Lee, M., & Svensson, N. L. (1990) · Clinical Physics and Physiological Measurement · doi:10.1088/0143-0815/11/3/002 · Cited by 64
Test-taking engagement in PIAAC
Goldhammer, F., Martens, T., Christoph, G., & Lüdtke, O. (2016) · OECD education working papers · doi:10.1787/5jlzfl6fhxs2-en · Cited by 82
Fox, A. S., Oler, J. A., Shelton, S. E., Nanda, S. A., Davidson, R. J., Roseboom, P. H., & Kalin, N. H. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1206723109 · Cited by 57
Toivonen, J., Merisaari, H., Pesola, M., et al. (2014) · Magnetic Resonance in Medicine · doi:10.1002/mrm.25482 · Cited by 63
Identity consolidation and internalizing and externalizing problem behaviors in early adolescence.
Avi Besser, A., & Blatt, S. J. (2007) · Psychoanalytic Psychology · doi:10.1037/0736-9735.24.1.126 · Cited by 63
Schudlich, T. D. R., Shamir, H., & Cummings, E. M. (2004) · Social Development · doi:10.1111/j.1467-9507.2004.000262.x · Cited by 72
Biais, M., Larghi, M., Henriot, J., de Courson, H., Sesay, M., & Nouette‐Gaulain, K. (2017) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000002322 · Cited by 65
Language Experience during Infancy Predicts White Matter Myelination at Age 2 Years
Huber, E., Corrigan, N. M., Yarnykh, V. L., Ramírez, N. F., & Kuhl, P. K. (2023) · Journal of Neuroscience · doi:10.1523/jneurosci.1043-22.2023 · Cited by 65
MOLLINGER, L. A., & Steffen, T. M. (1993) · Physical Therapy · doi:10.1093/ptj/73.7.437 · Cited by 63
Acute Effects of a Vibration-like Stimulus during Knee Extension Exercise
Mileva, K. N., NALEEM, A. A., BISWAS, S. K., Marwood, S., & Bowtell, J. L. (2006) · Medicine & Science in Sports & Exercise · doi:10.1249/01.mss.0000227318.39094.b6 · Cited by 61
Internet- vs. telephone-administered questionnaires in a randomized trial of smoking cessation
Graham, A. L., Papandonatos, G. D., Bock, B. C., Cobb, N. K., Baskin-sommers, A. R., NIAURA, R. S., & Abrams, D. B. (2006) · Nicotine & Tobacco Research · doi:10.1080/14622200601045367 · Cited by 63
Test-Retest Reliability of the 6-Minute Walk Test in Individuals With Down Syndrome
Casey, A., Wang, X., & Osterling, K. (2012) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2012.04.022 · Cited by 63
Improving the Reliability of Physician Performance Assessment
Kaplan, S. H., Griffith, J. L., Price, L. L., Pawlson, L. G., & Greenfield, S. (2009) · Medical Care · doi:10.1097/mlr.0b013e31818dce07 · Cited by 63
Ventilation‐based segmentation of the lungs using hyperpolarized3He MRI
Tustison, N. J., Avants, B., Flors, L., Altes, T. A., de Lange, E. E., Mugler, J. P., & Gee, J. C. (2011) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.22738 · Cited by 63
Symmetrical and asymmetrical hip rotation and its relationship to hip rotator muscle strength
Cibulka, M. T., Strube, M., Meier, D., et al. (2009) · Clinical Biomechanics · doi:10.1016/j.clinbiomech.2009.09.006 · Cited by 63
Shame aversion and shame-proneness in Cluster C personality disorders.
Schoenleber, M. L., & Berenbaum, H. (2010) · Journal of Abnormal Psychology · doi:10.1037/a0017982 · Cited by 63
Montoya, L., Junger, M., & Ongena, Y. (2014) · Environment and Behavior · doi:10.1177/0013916514551047 · Cited by 63
Bodie, G. D., Burleson, B. R., Holmstrom, A. J., et al. (2011) · Human Communication Research · doi:10.1111/j.1468-2958.2011.01405.x · Cited by 63
Pohl, M. B., & Farr, L. (2010) · Journal of Foot and Ankle Research · doi:10.1186/1757-1146-3-14 · Cited by 63
Reisberg, ., Monteiro, I. M., Torossian, C. L., Auer, S., Shulman, M. B., Ghimire, S., Boksay, I., Guillo‐Benarous, F., Osorio, R. S., Vengassery, A., et al. (2014) · Dementia and Geriatric Cognitive Disorders · doi:10.1159/000357839 · Cited by 67
Process stationarity and reliability of trunk postural stability
Lee, H., & Granata, K. P. (2008) · Clinical Biomechanics · doi:10.1016/j.clinbiomech.2008.01.008 · Cited by 63
Are cervical physical outcome measures influenced by the presence of symptomatology?
Sterling, M. M., Jull, G. A., Carlsson, Y., & Crommert, L. (2002) · Physiotherapy Research International · doi:10.1002/pri.248 · Cited by 63
Development and validation of the patient-rated ulnar nerve evaluation
MacDermid, J. C., & Grewal, R. S. (2013) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-14-146 · Cited by 63
Parental Validating and Invalidating Responses and Adolescent Psychological Functioning
Shenk, C. E., & Fruzzetti, A. E. (2013) · The Family Journal · doi:10.1177/1066480713490900 · Cited by 69
Johnson, S. R., Naden, R. P., Fransen, J., van den Hoogen, F. H. J., Pope, J. E., Baron, M. G., Tyndall, A., Matucci‐Cerinic, M., Denton, C. P., Distler, O., et al. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.12.009 · Cited by 63
A Multilevel Investigation of the Influences of Employees' Resistance to Empowerment
Maynard, M. T., Mathieu, J. E., Marsh, W. M., & Ruddy, T. M. (2007) · Human Performance · doi:10.1080/08959280701332885 · Cited by 65
Hill, C., Maskowitz, K., Danis, B., & Wakschlag, L. S. (2008) · Parenting · doi:10.1080/15295190802045469 · Cited by 67
Development of a scoring system to assess mind maps
Evrekli, E., İnel, D., & Balım, A. G. (2010) · Procedia - Social and Behavioral Sciences · doi:10.1016/j.sbspro.2010.03.331 · Cited by 65
A criterion-based stepwise approach for training counselors in motivational interviewing
Martino, S., Canning‐Ball, M., Carroll, K. M., & Rounsaville, B. J. (2011) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2010.12.004 · Cited by 63
The Influence of Picture Size on Recognition and Exploratory Behaviour in Raised-Line Drawings
Wijntjes, M., van Lienen, T., Verstijnen, I. M., & Kappers, A. M. L. (2008) · Perception · doi:10.1068/p5714 · Cited by 63
Reliability Analysis in Therapeutic Research: Practice and Procedures
Ottenbacher, K. J., & Tomchek, S. D. (1993) · American Journal of Occupational Therapy · doi:10.5014/ajot.47.1.10 · Cited by 63
How to perform multiblock component analysis in practice
Roover, K. D., Ceulemans, E., & Timmerman, M. E. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0129-1 · Cited by 69
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
