The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 7,101–7,200 of 23,882
Vermeulen, H. M., Boonman, D. C. G., Schüller, H. M., Obermann, W. R., van Houwelingen, H. C., Rozing, P. M., & Vlieland, T. P. M. V. (2005) · Clinical Rehabilitation · doi:10.1191/0269215505cr811oa · Cited by 38
Correnti, A. J., Wollstein, G., Price, L. L., & Schuman, J. S. (2003) · Ophthalmology · doi:10.1016/s0161-6420(03)00496-2 · Cited by 38
PR interval: A comparison of electrical and mechanical methods in the fetus
Pasquini, L., Seale, A. N., Belmar, C., Oseku‐Afful, S., Thomas, M. J., Taylor, M. J. O., Roughton, M., & Gardiner, H. M. (2006) · Early Human Development · doi:10.1016/j.earlhumdev.2006.05.020 · Cited by 38
Standardization and Validation of the Visual Evaluation of Biocrystallizations
Huber, M., Andersen, J., Kahl, J., et al. (2010) · Biological Agriculture & Horticulture · doi:10.1080/01448765.2010.10510428 · Cited by 38
Reliability of a method to conduct upper airway analysis in cone-beam computed tomography
de Souza, K. R. S., Oltramari-Navarro, P. V. P., Navarro, R. D. L., Conti, A. C. D. C. F., & de Almeida, M. R. (2013) · Brazilian Oral Research · doi:10.1590/s1806-83242013000100009 · Cited by 38
A Model of Instrumental Networks: The Roles of Socialized Charismatic Leadership and Group Behavior
Varella, P., Javidan, M., & Waldman, D. A. (2011) · Organization Science · doi:10.1287/orsc.1100.0604 · Cited by 39
Reliability and validity of a qualitative and quantitative motor test for 5- to 6-year-old children
Kroes, M., Vissers, Y. L. J., Sleijpen, F. A. M., et al. (2004) · European Journal of Paediatric Neurology · doi:10.1016/j.ejpn.2004.01.007 · Cited by 38
Reliability of Dermatologists in Acne Lesion Counts and Global Assessments
Tan, J. K. L., Fung, K., & Bulger, L. (2006) · Journal of Cutaneous Medicine and Surgery · doi:10.2310/7750.2006.00044 · Cited by 38
Koenig, L. J., Parks, E. T., Analoui, M., et al. (2004) · Dentomaxillofacial Radiology · doi:10.1259/dmfr/30099843 · Cited by 38
Bogovic, J., Jedynak, B., Rigg, R. A., Du, A. X., Landman, B. A., Prince, J. L., & Ying, S. H. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.08.075 · Cited by 38
Hanke, T. A., & Rogers, M. W. (1992) · Physical Therapy · doi:10.1093/ptj/72.11.810 · Cited by 38
Martino, S. C., Elliott, M. N., Cleary, P. D., Kanouse, D. E., Brown, J. A., Spritzer, K. L., Heller, A., & Hays, R. D. (2009) · PubMed Central · Cited by 38
Test-Retest Reliability of Dual-Task Outcome Measures in People With Parkinson Disease
Strouwen, C., Molenaar, E., Keus, S. H. J., Münks, L., Bloem, B. R., & Nieuwboer, A. (2016) · Physical Therapy · doi:10.2522/ptj.20150244 · Cited by 39
Vira, S. N., Husain, Q., Jalai, C. M., et al. (2016) · Journal of Pediatric Orthopaedics · doi:10.1097/bpo.0000000000000891 · Cited by 38
Does Reassessment of Risk Improve Predictions? A Framework and Examination of the SAVRY and YLS/CMI
Viljoen, J. L., Gray, A. L., Shaffer, C. S., Bhanwer, A. K., Tafreshi, D., & Douglas, K. (2017) · Psychological Assessment · doi:10.1037/pas0000402 · Cited by 34
Validation of an ambient system for the measurement of gait parameters
Dubois, A., & Bresciani, J. (2018) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2018.01.024 · Cited by 38
Chargi, N., Ansari, E., Huiskamp, L. F. J., et al. (2019) · Oral Oncology · doi:10.1016/j.oraloncology.2019.06.022 · Cited by 38
Holm, I., Friis, A., Storheim, K., & Brox, J. I. (2003) · Spine · doi:10.1097/01.brs.0000058931.31599.e2 · Cited by 85
A study of the reliability of the Canada Fitness Survey questionnaire
Weller, I. M. R., & Corey, P. (1998) · Medicine & Science in Sports & Exercise · doi:10.1097/00005768-199810000-00010 · Cited by 38
Shields, R. K., Ruhland, J. L., Ross, M. A., Saehler, M. M., Smith, K. B., & Heffner, M. L. (1998) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(98)90370-7 · Cited by 38
A Comparison of the Dynamic Pedobarograph and EMED Systems for Measuring Dynamic Foot Pressures
Quaney, B. M., Meyer, K., Cornwall, M. W., & McPoil, T. G. (1995) · Foot & Ankle International · doi:10.1177/107110079501600909 · Cited by 38
Computerised estimation of the baseline fetal heart rate in labour: the low frequency line
Mongelli, M., Dawkins, R. R., Chung, T. K. H., Sahota, D. S., Spencer, J. A. D., & Chang, A. M. Z. (1997) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.1997.tb10935.x · Cited by 37
Instrumented measurement of glenohumeral joint laxity: reliability and normative data
Sauers, E. L., Borsa, P. A., Herling, D. E., & Stanley, R. D. (2001) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s001670000174 · Cited by 38
A comparison of alternative assessments of depressive symptom severity: a pilot study
Biggs, M. M., Shores‐Wilson, K., Rush, A. J., Carmody, T. J., Trivedi, M. H., Crismon, M. L., Toprac, M. G., & Mason, M. (2000) · Psychiatry Research · doi:10.1016/s0165-1781(00)00159-1 · Cited by 38
Celano, M., Bakeman, R. A., Gaytan, O., Smith, C. O., Koci, A., & HENDERSON, S. (2008) · Family Process · doi:10.1111/j.1545-5300.2008.00236.x · Cited by 42
Holmbäck, A. M., Askaner, K., Holtås, S., Downham, D. Y., & Lexell, J. E. (2002) · Muscle & Nerve · doi:10.1002/mus.10031 · Cited by 37
Predictors and Correlates of Psychopathic Characteristics in Substance Abusing Adolescents
O’Neill, M. L., Lidz, V., & Heilbrun, K. (2003) · International Journal of Forensic Mental Health · doi:10.1080/14999013.2003.10471177 · Cited by 38
IDENTIFICATION OF A CORE OF IMPORTANT ENABLING SKILLS FOR THE NTE SUCCESSOR STAGE I EXAMINATION*
Rosenfeld, M., & Tannenbaum, R. J. (1991) · ETS Research Report Series · doi:10.1002/j.2333-8504.1991.tb01404.x · Cited by 38
Reliability of Speech Naturalness Ratings of Stuttered Speech During Treatment
Onslow, M., Adams, R. D., & Ingham, R. J. (1992) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3505.994 · Cited by 38
Embrey, D. G., Yates, L., & Mott, D. H. (1990) · Physical Therapy · doi:10.1093/ptj/70.10.626 · Cited by 38
Rodriquez, A. A., & Agre, J. C. (1991) · PubMed · Cited by 38
Self-conscious emotions in worry and generalized anxiety disorder
Schoenleber, M., Chow, P. I., & Berenbaum, H. (2014) · British Journal of Clinical Psychology · doi:10.1111/bjc.12047 · Cited by 36
Rogers, E. M., Banks, N. F., & Jenkins, N. D. M. (2022) · Experimental Physiology · doi:10.1113/ep090540 · Cited by 37
Cortical gradients during naturalistic processing are hierarchical and modality-specific
Samara, A., Eilbott, J. A., Margulies, D. S., Xu, T., & Vanderwal, T. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120023 · Cited by 45
Venturin, D., Giannotta, G., Pellicciari, L., Rossi, A., Pennella, D., Goffredo, M., & Poser, A. (2023) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-023-06268-2 · Cited by 49
Mai, J., Lachinov, D., Riedl, S., Reiter, G. S., Vogl, W., Bogunović, H., & Schmidt-Erfurth, U. M. (2023) · Scientific Reports · doi:10.1038/s41598-023-34139-2 · Cited by 37
Motahari, A., Barr, R. G., Han, M. K., Anderson, W. H., Barjaktarević, I., Bleecker, E. R., Comellas, A. P., Cooper, C. B., Couper, D. J., Hansel, N. N., et al. (2023) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.202209-1698pp · Cited by 39
Prinzi, F., Militello, C., Scichilone, N., Gaglio, S., & Vitabile, S. (2023) · IEEE Access · doi:10.1109/access.2023.3327808 · Cited by 38
How measurement noise limits the accuracy of brain-behaviour predictions
Gell, M., Eickhoff, S. B., Omidvarnia, A., Küppers, V., Patil, K. R., Satterthwaite, T. D., Müller, V. I., & Langner, R. (2024) · Nature Communications · doi:10.1038/s41467-024-54022-6 · Cited by 89
Using Large Language Models for Automated Grading of Student Writing about Science
Impey, C. D., Wenger, M. C., Garuda, N., Golchin, S., & Stamer, S. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00453-7 · Cited by 57
Timmins, R. G., SHIELD, A. J., Williams, M. D., & Opar, D. A. (2015) · Sports Medicine · doi:10.1007/s40279-015-0378-8 · Cited by 37
Sung, P. S., & Leininger, P. M. (2015) · Clinical Biomechanics · doi:10.1016/j.clinbiomech.2015.05.003 · Cited by 37
Visuospatial Ability as a Predictor of Novice Performance in Ultrasound-guided Regional Anesthesia
Shafqat, A., Ferguson, E., Thanawala, V., et al. (2015) · Anesthesiology · doi:10.1097/aln.0000000000000870 · Cited by 37
Wellmon, R., Degano, A., Rubertone, J. A., et al. (2015) · Archives of Physiotherapy · doi:10.1186/s40945-015-0011-z · Cited by 37
Adams, M. J., Robinson, M. R., Mannarelli, M., et al. (2015) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2015.0689 · Cited by 37
Andreychik, M., & Gill, M. (2012) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2012.05.006 · Cited by 37
The risk factors associated with subluxation of the distal interphalangeal joint in mallet fracture
Kim, J. K., & Kim, D. J. (2014) · Journal of Hand Surgery (European Volume) · doi:10.1177/1753193414554556 · Cited by 37
Hu, S., Pierrick Coupé, A., Pruessner, J. C., & Collins, D. L. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.06.054 · Cited by 37
Daugherty, A. M., Bender, A. R., Yuan, P., & Raz, N. (2015) · Cerebral Cortex · doi:10.1093/cercor/bhv061 · Cited by 37
Accuracy and Reliability of Three Methods of Recording Scapular Motion Using Reflective Skin Markers
Brochard, S., Lempereur, M., & Rémy‐Néris, O. (2010) · Proceedings of the Institution of Mechanical Engineers Part H Journal of Engineering in Medicine · doi:10.1243/09544119jeim830 · Cited by 37
Sohn, S., Chung, C. K., Park, S., et al. (2013) · Journal of Neuro-Oncology · doi:10.1007/s11060-013-1190-7 · Cited by 37
Hollen, P. J., Gralla, R. J., Stewart, J. A., Meharchand, J. M., Wierzbicki, R. F., & Leighl, N. B. (2012) · Supportive Care in Cancer · doi:10.1007/s00520-012-1507-7 · Cited by 36
A comprehensive Fabry-related pain questionnaire for adult patients
Üçeyler, N., Magg, B., Thomas, P., et al. (2014) · Pain · doi:10.1016/j.pain.2014.08.024 · Cited by 37
Luan, S., Min, Y., Li, G., Lin, C., Li, X., Wu, S., Ma, C., & Hill, J. C. (2014) · Spine · doi:10.1097/brs.0000000000000413 · Cited by 37
Proxy respondents reliably assessed the quality of life of elective cardiac surgery patients
Elliott, D., Lazarus, R. M., & Leeder, S. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.06.010 · Cited by 37
Goñi, J., Sporns, O., Cheng, H., Aznárez‐Sanado, M., Wang, Y., Josa, S., Arrondo, G., Mathews, V. P., Hummer, T. A., Kronenberger, W. G., et al. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.06.072 · Cited by 37
Healthy Volunteers Can Be Phenotyped Using Cutaneous Sensitization Pain Models
Werner, M. U., Petersen, K. L., Rowbotham, M. C., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0062733 · Cited by 37
Using multimedia to teach inservice teachers: Impacts on learning, application, and retention
Pryor, C. R., & Bitter, G. G. (2008) · Computers in Human Behavior · doi:10.1016/j.chb.2008.03.007 · Cited by 37
Guidelines for infection control in nursing homes: a Delphi consensus web-based survey
Chami, K., Gavazzi, G., de Wazières, B., et al. (2011) · Journal of Hospital Infection · doi:10.1016/j.jhin.2011.04.014 · Cited by 37
Bartelink, C., van Yperen, T. A., ten Berge, I. J., de Kwaadsteniet, L., & Witteman, C. L. M. (2014) · Child & Youth Care Forum · doi:10.1007/s10566-014-9259-9 · Cited by 37
The course of prosthodontic patients’ oral health-related quality of life over a period of 2 years
Aarabi, G., John, M. T., Schierz, O., Heydecke, G., & Reissmann, D. R. (2014) · Journal of Dentistry · doi:10.1016/j.jdent.2014.09.006 · Cited by 39
Hume, K., Boyd, B. A., McBee, M., Coman, D. C., Gutierrez, A., Shaw, E., Sperry, L. A., Alessandri, M., & Odom, S. L. (2011) · Research in Autism · doi:10.1016/j.rasd.2011.02.002 · Cited by 38
Effect of fatigue on force fluctuations in knee extensors in young adults
Singh, N. B., Arampatzis, A., Duda, G. N., Heller, M. O. W., & Taylor, W. R. (2010) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2010.0091 · Cited by 40
Examining retaliatory responses to justice violations and recovery attempts in teams.
Christian, J. S., Christian, M. S., Garza, A. S., & Ellis, A. P. J. (2012) · Journal of Applied Psychology · doi:10.1037/a0029450 · Cited by 40
Chaudoir, S. R., & Quinn, D. M. (2015) · Self and Identity · doi:10.1080/15298868.2015.1091378 · Cited by 43
Towards the assessment of local dynamic stability of level-grounded walking in an older population
Hamacher, D., Hamacher, D., Singh, N. B., Taylor, W. R., & Schega, L. (2015) · Medical Engineering & Physics · doi:10.1016/j.medengphy.2015.09.007 · Cited by 37
Seelandt, J., Tschan, F., Keller, S., et al. (2014) · BMJ Quality & Safety · doi:10.1136/bmjqs-2014-002860 · Cited by 37
Reliability of Center of Pressure Measures for Assessing the Development of Sitting Postural Control
Kyvelidou, A., Harbourne, R. T., Stuberg, W. A., et al. (2009) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2009.01.031 · Cited by 37
Ramasundarahettige, C. F., Donner, A. P., & Zou, G. (2009) · Statistics in Medicine · doi:10.1002/sim.3523 · Cited by 37
Marcon, M., Ciritsis, B. D., Laux, C. J., et al. (2014) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.24777 · Cited by 37
A reliable, non-invasive measurement tool for anisotropy in normal skin and scar tissue
Verhaegen, P. D. H. M., Res, E. M., van Engelen, A., et al. (2010) · Skin Research and Technology · doi:10.1111/j.1600-0846.2010.00436.x · Cited by 37
Ralston, P. A. S., & Bays, C. L. (2015) · American Journal of Engineering Education (AJEE) · doi:10.19030/ajee.v6i2.9504 · Cited by 37
A comparison of normal and osteoarthritic humeral head size and morphology
Knowles, N. K., Carroll, M. J., Keener, J. D., et al. (2015) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2015.08.047 · Cited by 37
Schwaiger, B. J., Gersing, A. S., Lee, S., et al. (2016) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2016.04.001 · Cited by 37
Chen, W., Le, L. H., & Lou, E. H. M. (2016) · European Spine Journal · doi:10.1007/s00586-016-4492-6 · Cited by 37
Keefe, J. R., Webb, C. A., & DeRubeis, R. J. (2016) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000071 · Cited by 37
Linek, P., Saulicz, E., Wolny, T., et al. (2014) · Journal of Ultrasound in Medicine · doi:10.7863/ultra.33.6.1049 · Cited by 37
Hiwatashi, A., Togao, O., Yamashita, K., et al. (2016) · European Radiology · doi:10.1007/s00330-016-4406-3 · Cited by 37
Paes, E. C., Bittermann, G. K. P., Bittermann, D., et al. (2016) · Plastic & Reconstructive Surgery · doi:10.1097/01.prs.0000475769.06773.86 · Cited by 37
Güneyli, S., Ward, E. V. M., Peng, Y., et al. (2016) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.25418 · Cited by 37
Patellar height assessment in total knee arthroplasty: a new method
Caton, J. H., Prudhon, J. L., Aslanian, T., et al. (2016) · International Orthopaedics · doi:10.1007/s00264-016-3256-6 · Cited by 37
Henderson, M., Miles, A., Holgate, V., Peryman, S., & Allen, J. E. (2016) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2016.07.050 · Cited by 36
Morris, W. Z., Napora, J. K., Conry, K. T., et al. (2016) · Journal of Pediatric Orthopaedics · doi:10.1097/bpo.0000000000000881 · Cited by 37
Huo, Y., Asman, A. J., Plassard, A. J., & Landman, B. A. (2016) · Human Brain Mapping · doi:10.1002/hbm.23432 · Cited by 37
Costa, Y. M., Morita‐Neto, O., de Araújo-Júnior, E. N. S., Sampaio, F. A., Conti, P. C. R., & Bonjardim, L. R. (2016) · Journal of Oral Rehabilitation · doi:10.1111/joor.12477 · Cited by 37
Parent-Child Agreement on Parent-to-Child Maltreatment
Block, L. C.-D., Alink, L. R. A., Linting, M., van den Berg, L. J. M., Elzinga, B. M., Voorthuis, A., Tollenaar, M. S., & Bakermans‐Kranenburg, M. J. (2016) · Journal of Family Violence · doi:10.1007/s10896-016-9902-3 · Cited by 37
Validation of a Dish-Based Semiquantitative Food Questionnaire in Rural Bangladesh
Lin, P. D., Bromage, S., Mostofa, M. G., Allen, J. G., Oken, E., Kile, M. L., & Christiani, D. C. (2017) · Nutrients · doi:10.3390/nu9010049 · Cited by 37
Content analysis of business communication: introducing a German dictionary
Bannier, C., Pauls, T., & Walter, A. (2018) · Journal of Business Economics · doi:10.1007/s11573-018-0914-8 · Cited by 29
Electronic versus paper-pencil methods for assessing chemotherapy-induced peripheral neuropathy
Knoerl, R., Gray, E., Stricker, C. T., Mitchell, S. A., Kippe, K., Smith, G., Dudley, W. N., & Smith, E. (2017) · Supportive Care in Cancer · doi:10.1007/s00520-017-3764-y · Cited by 37
Kvasnicka, H. M., Orazi, A., Thiele, J., et al. (2017) · American Journal of Hematology · doi:10.1002/ajh.24837 · Cited by 37
Sigmundsson, H., Eriksen, A. D., Ofteland, G. S., & Haga, M. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01539 · Cited by 42
Heidenreich, M. J., Sanders, T. L., Hevesi, M., et al. (2017) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-017-4752-y · Cited by 37
Gansen-Ammann, D.-N., Meurs, J. A., Wihler, A., & Blickle, G. (2017) · Group & Organization Management · doi:10.1177/1059601117747487 · Cited by 37
MacLean, S., Geddes, F., Kelly, M., & Della, P. R. (2017) · Nurse Education Today · doi:10.1016/j.nedt.2017.12.024 · Cited by 37
Wu, Z., Mazzola, C. A., Catania, L., Owoeye, O., Yaramothu, C., Alvarez, T. L., Gao, Y., & Li, X. (2018) · CNS Neuroscience & Therapeutics · doi:10.1111/cns.12811 · Cited by 36
Spierling, S. R., Kreisler, A. D., Williams, C. A., Fang, S. Y., Pucci, S. N., Kines, K. T., & Zorrilla, E. P. (2018) · Physiology & Behavior · doi:10.1016/j.physbeh.2018.04.001 · Cited by 36
CHOROIDAL THICKNESS, VASCULAR FACTORS, AND AGE-RELATED MACULAR DEGENERATION
Gattoussi, S., Cougnard‐Grégoire, A., Korobelnik, J., et al. (2018) · Retina · doi:10.1097/iae.0000000000002237 · Cited by 37
Age-associated increase in mnemonic strategy use is linked to prefrontal cortex development
Yu, Q., McCall, D. M., Homayouni, R., Tang, L., Chen, Z., Schoff, D., Nishimura, M., Raz, S., & Ofen, N. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.07.008 · Cited by 38
Morrison, M. A., Payabvash, S., Chen, Y., Avadiappan, S., Shah, M. M., Zou, X., Hess, C. P., & Lupo, J. (2018) · NeuroImage Clinical · doi:10.1016/j.nicl.2018.08.002 · Cited by 37
Translation and validation of the Arabic version of the 5-item Oral Health Impact Profile: OHIP5-Ar
Alhajj, M. N., Halboub, E. S., Khalifa, N., Amran, A. G., Reissmann, D. R., Abdullah, A. G., Assad, M., Al-Basmi, A. A., & Al-Ghabri, F. A. (2018) · Health and Quality of Life Outcomes · doi:10.1186/s12955-018-1046-0 · Cited by 37
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
