The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 4,801–4,900 of 23,883
Reproducibility of ambulatory blood pressure in treated and untreated hypertensive patients
Eguchi, K., Hoshide, S., Hoshide, Y., Ishikawa, S., Shimada, K., & Kario, K. (2010) · Journal of Hypertension · doi:10.1097/hjh.0b013e3283378477 · Cited by 60
Left ventricular remodeling and torsion dynamics in hypertensive patients
Cameli, M., Lisi, M., Righini, F. M., Massoni, A., & Mondillo, S. (2012) · International journal of cardiac imaging · doi:10.1007/s10554-012-0054-0 · Cited by 60
Chon, S.-C., Chang, K.-Y., & You, J. H. (2009) · Physiotherapy · doi:10.1016/j.physio.2009.09.007 · Cited by 61
The utility and accuracy of computed tomography in the diagnosis of occipitocervical dissociation
Gire, J. D., Roberto, R. F., Bobinski, M., et al. (2013) · The Spine Journal · doi:10.1016/j.spinee.2013.01.023 · Cited by 60
Willcox, E. L., & Burden, A. (2013) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2013.4004 · Cited by 60
Karnik, M. S., Wang, L., Barch, D. M., Harms, M. P., Campanella, C., & Csernansky, J. G. (2012) · Schizophrenia Research · doi:10.1016/j.schres.2012.03.015 · Cited by 60
Heritability of glutathione and related metabolites in stored red blood cells
Erve, T. J. V. ‘., Doskey, C. M., Wagner, B. A., Hess, J. R., Darbro, B. W., Ryckman, K. K., Murray, J. C., Raife, T. J., & Buettner, G. R. (2014) · Free Radical Biology and Medicine · doi:10.1016/j.freeradbiomed.2014.07.040 · Cited by 61
Fernández, Á. L. R., Garrido-Santofimia, V., Güeita‐Rodríguez, J., et al. (2011) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2011.04.010 · Cited by 60
Hirai, T., Kitajima, M., Nakamura, H., Okuda, T., Sasao, A., Shigematsu, Y., Utsunomiya, D., Oda, S., Uetani, H., Morioka, M., et al. (2011) · American Journal of Neuroradiology · doi:10.3174/ajnr.a2725 · Cited by 59
Inter-examiner reproducibility of tests for lumbar motor control
Enoch, F., Kjær, P., Elkjær, A., Remvig, L., & Juul‐Kristensen, B. (2011) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-12-114 · Cited by 61
Validation of a clinical assessment score for pediatric sleep‐disordered breathing
Goldstein, N. A., Stefanov, D. G., Graw-Panzer, K. D., Fahmy, S. A., Fishkin, S., Jackson, A., Sarhis, J. S., & Weedon, J. (2012) · The Laryngoscope · doi:10.1002/lary.23455 · Cited by 59
Valencia, C., Kindler, L. L., Fillingim, R. B., & George, S. Z. (2013) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-14-182 · Cited by 60
MacDermid, J. C., Sinden, K. E., Jenkyn, T., et al. (2014) · International Journal of Shoulder Surgery · doi:10.4103/0973-6042.131847 · Cited by 60
Brenton‐Rule, A., Mattock, J. P. M., Carroll, M., et al. (2012) · Journal of Foot and Ankle Research · doi:10.1186/1757-1146-5-21 · Cited by 60
Psychometric Properties of the French Version of the Adult Dispositional Hope Scale
Gana, K., Daigre, S., & Ledrich, J. (2012) · Assessment · doi:10.1177/1073191112468315 · Cited by 60
Rathschlag, M., & Memmert, D. (2013) · Journal of Sport and Exercise Psychology · doi:10.1123/jsep.35.2.197 · Cited by 67
The Test–Retest Reliability of the Child-Initiated Pretend Play Assessment
Stagnitti, K. E., & Unsworth, C. A. (2004) · American Journal of Occupational Therapy · doi:10.5014/ajot.58.1.93 · Cited by 60
Observations of the middle school environment: The context for student behavior beyond the classroom
Rusby, J. C., Crowley, R. N., Sprague, J. R., & Biglan, A. (2011) · Psychology in the Schools · doi:10.1002/pits.20562 · Cited by 60
Community implementation outcomes of Attachment and Biobehavioral Catch-up
Caron, E., Weston-Lee, P., Haggerty, D., & Dozier, M. (2015) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2015.11.010 · Cited by 70
Uher, J., & Visalberghi, E. M. (2016) · Journal of Research in Personality · doi:10.1016/j.jrp.2016.02.003 · Cited by 57
New Venture Performance in Transition Economies from Different Institutional Perspectives
Bruton, G. D., Su, Z., & Filatotchev, I. V. (2016) · Journal of Small Business Management · doi:10.1111/jsbm.12266 · Cited by 65
Are Eyes a Mirror of the Soul? What Eye Wrinkles Reveal about a Horse’s Emotional State
Hintze, S., Smith, S. E., Patt, A., Bachmann, I., & Würbel, H. (2016) · PLoS ONE · doi:10.1371/journal.pone.0164017 · Cited by 60
Reliability of physical functioning tests in patients with low back pain: a systematic review
Denteneer, L., Daele, U. V., Truijen, S., Hertogh, W. D., Meirte, J., & Stassijns, G. (2017) · The Spine Journal · doi:10.1016/j.spinee.2017.08.257 · Cited by 60
Vereijken, M. W. C., van der Rijst, R. M., DRIEL, J. H. V., & Dekker, F. W. (2017) · Advances in Health Sciences Education · doi:10.1007/s10459-017-9803-0 · Cited by 60
Schmitz, E. A., Baum, M., Huett, P., & Kabst, R. (2017) · Organization & Environment · doi:10.1177/1086026617745992 · Cited by 63
Shatu, F. M., & Yiğitcanlar, T. (2018) · Journal of Transport Geography · doi:10.1016/j.jtrangeo.2018.06.004 · Cited by 60
Weber, M., Ancum, J. M. V., Bergquist, R., Taraldsen, K., Gordt, K., Mikolaizak, A. S., Nerz, C., Pijnappels, M. A. G. M., Jonkman, N. H., Maier, A. B., et al. (2018) · BMC Geriatrics · doi:10.1186/s12877-018-0845-9 · Cited by 60
Beamish, N., Green, N., Nieuwold, E., et al. (2019) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2019.8543 · Cited by 60
Age and gender influence on foveal avascular zone in healthy eyes
Gómez‐Ulla, F., Cutrín, P., Santos-Ramos, P., et al. (2019) · Experimental Eye Research · doi:10.1016/j.exer.2019.107856 · Cited by 60
Arslan, R. C., Reitz, A. K., Driebe, J. C., Gerlach, T. M., & Penke, L. (2020) · Psychological Methods · doi:10.1037/met0000294 · Cited by 60
Veronese, M., Santangelo, B., Jauhar, S., D’Ambrosio, E., Demjaha, A., Salimbeni, H., Jin, H., McCrone, P., Turkheimer, F., & Howes, O. D. (2020) · Neuropsychopharmacology · doi:10.1038/s41386-020-00866-7 · Cited by 58
User-Guided Level Set Segmentation of Anatomical Structures with ITK-SNAP
Yushkevich, P. A., Piven, J., Cody, H., Ho, S., Gee, J. C., & Gerig, G. (2005) · The Insight Journal · doi:10.54294/13sgje · Cited by 60
Lima, J. C., Caughy, M. O., Nettles, S. M., & O'Campo, P. J. (2010) · Social Science & Medicine · doi:10.1016/j.socscimed.2010.06.022 · Cited by 60
One-Arm Hop Test: Reliability and Effects of Arm Dominance
Falsone, S., Gross, M. T., Guskiewicz, K. M., et al. (2002) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2002.32.3.98 · Cited by 60
Correlates of functional status in older patients with schizophrenia
Patterson, T. L., Klapow, J. C., Eastham, J. H., et al. (1998) · Psychiatry Research · doi:10.1016/s0165-1781(98)00060-2 · Cited by 60
Expectancy, valence, and motivational force functions in goal-setting research: An empirical test.
Tubbs, M. E., Boehne, D. M., & Dahl, J. G. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.3.361 · Cited by 69
Psychometric properties of the functional rating index in patients with low back pain
Childs, M. J. D., & Piva, S. R. (2005) · European Spine Journal · doi:10.1007/s00586-005-0900-z · Cited by 60
Ghyselinck, M., Custers, R., & Brysbaert, M. (2004) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.30.2.550 · Cited by 60
Mikkelsen, M. E., Dedhiya, P., Kalhan, R., et al. (2008) · Respiratory Care · doi:10.4187/respcare.08530455 · Cited by 60
Applying Impairment Criteria to Children's Psychiatric Diagnosis
Weissman, M. M., Warner, V., & Fendrich, M. (1990) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199009000-00019 · Cited by 60
The Blank Page: Effects of Constraint on CreativityPreprint
Joyce, C. K. (2010) · SSRN Electronic Journal · doi:10.2139/ssrn.1552835 · Cited by 60
Gorgolewski, K. J., Mendes, N., Wilfling, D., Wladimirow, E., Gauthier, C. J., Bonnen, T., Ruby, F. J. M., Trampel, R., Bazin, P., Cozatl, R., et al. (2015) · Scientific Data · doi:10.1038/sdata.2014.54 · Cited by 60
A randomized controlled trial of preoperative psychological preparation for mastectomy
Burton, M. V., Parker, R. W., Farrell, A., Bailey, D., Conneely, J., Booth, S., & Elcombe, S. (1995) · Psycho-Oncology · doi:10.1002/pon.2960040102 · Cited by 59
Self-regulation and control in personality functioning
Carver, C. S., & Scheier, M. F. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9780511596544.027 · Cited by 59
Posttraumatic stress in deployed Marines: Prospective trajectories of early adaptation.
Nash, W. P., Boasso, A. M., Steenkamp, M. M., Larson, J. L., Lubin, R. E., & Litz, B. T. (2014) · Journal of Abnormal Psychology · doi:10.1037/abn0000020 · Cited by 59
Barker, K., Bowman, B., Galloway, H., Oliashirazi, N., Oliashirazi, A., & Mehta, S. P. (2016) · The Journal of Knee Surgery · doi:10.1055/s-0036-1593877 · Cited by 59
Wieland, L. S., Berman, B., Altman, D. G., Barth, J., Bouter, L. M., D’Adamo, C. R., Linde, K., Moher, D., Mullins, C. D., Treweek, S. P., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.01.010 · Cited by 59
Exploring the reliability and sensitivity of the EEG power spectrum as a biomarker
Pathania, A., Schreiber, M. A., Miller, M. W., Euler, M. J., & Lohse, K. R. (2020) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2020.12.002 · Cited by 59
McDougall, F., Edgar, C. J., Mertes, M., Delmar, P., da Fontoura, P. P., Abi‐Saab, D., Lansdall, C. J., Boada, M., & Doody, R. S. (2021) · The Journal of Prevention of Alzheimer s Disease · doi:10.14283/jpad.2020.73 · Cited by 60
Temporal complexity of fMRI is reproducible and correlates with higher order cognition
Omidvarnia, A., Zalesky, A., Sina Mansour, L., Ville, D. V. D., Jackson, G. D., & Pedersen, M. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.117760 · Cited by 60
Rudisch, J., Jöllenbeck, T., Vogt, L., et al. (2021) · Gait & Posture · doi:10.1016/j.gaitpost.2021.01.013 · Cited by 59
Zhao, K., Zheng, Q., Che, T., Dyrba, M., Li, Q., Ding, Y., Zheng, Y., Liu, Y., & Li, S. (2021) · Network Neuroscience · doi:10.1162/netn_a_00200 · Cited by 58
A Novel Tool for Gait Analysis: Validation Study of the Smart Insole PODOSmart®
Ziagkas, E., Loukovitis, A., Zekakos, D. X., Chau, T. D.-P., Petrelis, A., & Grouios, G. (2021) · Sensors · doi:10.3390/s21175972 · Cited by 59
Proxy‐report in individuals with intellectual disability: A scoping review
Santoro, S. L., Donelan, K., & Constantine, M. L. (2022) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.13013 · Cited by 59
Huttunen, R., Leppänen, T., Duce, B., Arnardóttir, E. S., Nikkonen, S., Myllymaa, S., Töyräs, J., & Korkalainen, H. (2022) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2022.3225268 · Cited by 59
Improving the reliability of measurements in orthopaedics and sports medicine
Królikowska, A., Reichert, P., Karlsson, J., Mouton, C., Becker, R., & Prill, R. (2023) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-023-07635-1 · Cited by 65
Fatigue in adolescents with cancer compared to healthy adolescents
Daniel, L. C., Brumley, L. D., & Schwartz, L. A. (2013) · Pediatric Blood & Cancer · doi:10.1002/pbc.24706 · Cited by 61
Reliability of electromyographic activity vs. bite-force from human masticatory muscles
Gonzalez, Y. M., Iwasaki, L. R., McCall, W. D., et al. (2011) · European Journal Of Oral Sciences · doi:10.1111/j.1600-0722.2011.00823.x · Cited by 59
Reliability of kinematic measures of functional reaching in children with cerebral palsy
Schneiberg, S., McKinley, P. A., Gisel, E. G., et al. (2010) · Developmental Medicine & Child Neurology · doi:10.1111/j.1469-8749.2010.03635.x · Cited by 59
A Method to Optimize the Response Range While Maintaining Rorschach Comprehensive System Validity
Dean, K. L., Viglione, D. J., Perry, W., & Meyer, G. J. (2007) · Journal of Personality Assessment · doi:10.1080/00223890701468543 · Cited by 59
Melchers, K. G., Klehe, U., Richter, G. M., Kleinmann, M., König, C. J., & Lievens, F. (2009) · Human Performance · doi:10.1080/08959280903120295 · Cited by 59
Salavati, M., Negahban, H., Mazaheri, M., Soleimanifar, M., Hadadi, M., Sefiddashti, L., Zahraee, M. H., Davatgaran, K., & Feizi, A. (2012) · Disability and Rehabilitation · doi:10.3109/09638288.2012.660604 · Cited by 59
Taylor, O., Armfield, N. R., Dodrill, P., & Smith, A. C. (2014) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x14552388 · Cited by 60
Sebbag, L., Kass, P. H., & Maggs, D. J. (2015) · Journal of the American Veterinary Medical Association · doi:10.2460/javma.246.4.426 · Cited by 60
Intra- and inter-observer reliability of the Trunk Impairment Scale for children with cerebral palsy
Sæther, R., & Jørgensen, L. (2010) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2010.11.007 · Cited by 58
A critical review of the effects of entrepreneurship training in developing countries
Matthias Glaub, M. G., & Michael Frese, M. F. (2011) · Enterprise Development & Microfinance · doi:10.3362/1755-1986.2011.035 · Cited by 59
A Comparison of Spelling Performance Across Young Adults With and Without Dyslexia
Coleman, C., Gregg, N., McLain, L., et al. (2008) · Assessment for Effective Intervention · doi:10.1177/1534508408318808 · Cited by 59
Nemanic, S., Alvarado, M. C., Price, R. E., Jackson, E. F., & Bachevalier, J. (2002) · Journal of Neuroscience Methods · doi:10.1016/s0165-0270(02)00264-9 · Cited by 59
Brunner, J. F., Hansen, T. I., Olsen, A., et al. (2013) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2013.06.005 · Cited by 59
Attachment‐Based Family Therapy: Adherence and Differentiation
Diamond, G. M., Diamond, G. S., & Hogue, A. (2007) · Journal of Marital and Family Therapy · doi:10.1111/j.1752-0606.2007.00015.x · Cited by 59
Reliability of position sense testing assessed with a fully automated system
Lönn, Crenshaw, Djupsjöbacka, et al. (2000) · Clinical Physiology · doi:10.1046/j.1365-2281.2000.00218.x · Cited by 59
An Inter-Rater Reliability Study for the Rorschach Performance Assessment System
Viglione, D. J., Blume-Marcovici, A. C., Miller, H. L., Giromini, L., & Meyer, G. J. (2012) · Journal of Personality Assessment · doi:10.1080/00223891.2012.684118 · Cited by 59
Storto, G., Cirillo, P. L., Vicario, M. L. E., Pellegrino, T., Sorrentino, A. R., Petretta, M., Galasso, G., Sanctis, V. D., Piscione, F., & Cuocolo, A. (2004) · Journal of Nuclear Cardiology · doi:10.1016/j.nuclcard.2004.08.007 · Cited by 59
Two-Week Test-Retest Reliability of the Duke Religion Index
Storch, E. A., Strawser, M. S., & Storch, J. B. (2004) · Psychological Reports · doi:10.2466/pr0.94.3.993-994 · Cited by 59
In the Eye of the Beholder: Differences Across Ethnicity and Gender in Evaluating Creative Work
Kaufman, J. C., Niu, W., Sexton, J. D., & Cole, J. C. (2010) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2009.00584.x · Cited by 59
Retest reliability of reward-related BOLD signals
Fließbach, K., Rohe, T., Linder, N. S., Trautner, P., Elger, C. E., & Weber, B. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.01.036 · Cited by 60
Contralateral Elbow Radiographs Can Reliably Diagnose Radial Head Implant Overlengthening
Athwal, G. S., Rouleau, D. M., MacDermid, J. C., & King, G. J. W. (2011) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.j.01244 · Cited by 59
Reliability of clinician judgements of bruxism
Marbach, J. J., Raphael, K. G., Janal, M. N., et al. (2003) · Journal of Oral Rehabilitation · doi:10.1046/j.1365-2842.2003.01146.x · Cited by 59
Test–retest reliability of cardinal plane isokinetic hip torque and EMG
Claiborne, T. L., Timmons, M. K., & Pincivero, D. M. (2008) · Journal of Electromyography and Kinesiology · doi:10.1016/j.jelekin.2008.07.005 · Cited by 59
Paraïso, M. N., Houinato, D. S., Guerchet, M., Aguèh, V., Nubukpo, P., Preux, P., & Marin, B. (2010) · Neuroepidemiology · doi:10.1159/000301715 · Cited by 59
The Impact of Item Readability on the Endorsement of the Midpoint Response in Surveys
Velez, P. M., & Ashworth, S. D. (2007) · Survey research methods · doi:10.18148/srm/2007.v1i2.76 · Cited by 60
van Aken, J., Heimans, L., Dongen, H. G.-V., et al. (2013) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2012-202967 · Cited by 59
Prevalence and determinants of QT interval prolongation in medical inpatients
Pasquier, M., Pantet, O., Hügli, O., Pruvot, E. J., Buclin, T., Waeber, G., & Aujesky, D. (2011) · Internal Medicine Journal · doi:10.1111/j.1445-5994.2011.02447.x · Cited by 59
Reliability and Validity of Nonverbal Thin Slices in Social Interactions
Murphy, N. A., Hall, J. A., Mast, M. S., et al. (2014) · Personality and Social Psychology Bulletin · doi:10.1177/0146167214559902 · Cited by 59
Kim, Y. B., Kim, Y. J., Kim, Y. J., et al. (2014) · European Spine Journal · doi:10.1007/s00586-014-3236-8 · Cited by 59
Choice of Words in Doctor–Patient Communication: An Analysis of Health-Related Internet Sites
Jucks, R., & Bromme, R. (2007) · Health Communication · doi:10.1080/10410230701307865 · Cited by 59
Rener-Sitar, K., John, M. T., Bandyopadhyay, D., Howell, M. J., & Schiffman, E. L. (2014) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-12-10 · Cited by 59
Assessment of cartilage loss at the wrist in rheumatoid arthritis using a new MRI scoring system
McQueen, F. M., Clarke, A. W., McHaffie, A., et al. (2010) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2009.127324 · Cited by 59
Baird, D. D., Saldana, T. M., Nepomnaschy, P. A., Hoppin, J. A., Longnecker, M. P., Weinberg, C. R., & Wilcox, A. J. (2009) · Journal of Exposure Science & Environmental Epidemiology · doi:10.1038/jes.2009.17 · Cited by 59
Houtte, M. V., & Demanet, J. (2015) · Teaching and Teacher Education · doi:10.1016/j.tate.2015.12.003 · Cited by 72
Wilson, A., Hides, J. A., Blizzard, L., Callisaya, M. L., Cooper, A. J., Srikanth, V. K., & Winzenberg, T. M. (2016) · Manual Therapy · doi:10.1016/j.math.2016.01.004 · Cited by 59
Erbaş, Y., Sels, L., Ceulemans, E., & Kuppens, P. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616633504 · Cited by 72
Merisaari, H., Movahedi, P., Perez, I. M., et al. (2016) · Magnetic Resonance in Medicine · doi:10.1002/mrm.26169 · Cited by 59
Anacta, V. J. A., Schwering, A., Li, R., & Münzer, S. (2016) · GeoJournal · doi:10.1007/s10708-016-9703-5 · Cited by 59
Benjamin‐Thomas, T. E., Lucas‐Thompson, R. G., Little, L. M., Davies, P. L., & Khetani, M. A. (2016) · Physical & Occupational Therapy In Pediatrics · doi:10.3109/01942638.2015.1130007 · Cited by 61
Chen, C., Bojikian, K. D., Chen, X., et al. (2016) · Journal of Biomedical Optics · doi:10.1117/1.jbo.21.6.065002 · Cited by 59
Vaughn, A., Dearth‐Wesley, T., Tabak, R. G., et al. (2016) · Journal of the Academy of Nutrition and Dietetics · doi:10.1016/j.jand.2016.07.021 · Cited by 59
Alter, A. L., Stern, C., Granot, Y., & Balcetis, E. (2016) · Personality and Social Psychology Bulletin · doi:10.1177/0146167216669123 · Cited by 60
Snellgrove, C. A. (2005) · Cited by 59
Yao, L., Chen, Y., Wang, X., Shi, X., Wang, Y., Guo, T., & Yang, K. (2017) · International Journal of Clinical Practice · doi:10.1111/ijcp.12931 · Cited by 58
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
