The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 1,701–1,800 of 23,883
Fermanian, J. (2005) · Annales de Réadaptation et de Médecine Physique · doi:10.1016/j.annrmp.2005.04.004 · Cited by 150
Flansbjer, U., Blom, J., & Brogårdh, C. B. (2012) · PM&R · doi:10.1016/j.pmrj.2011.11.004 · Cited by 158
Caligiuri, M. E., Perrotta, P., Augimeri, A., Rocca, F., Quattrone, A., & Cherubini, A. (2015) · Neuroinformatics · doi:10.1007/s12021-015-9260-y · Cited by 151
Intra- and Interrater Reliability of Ultrasound Assessment of Gastric Volume
Kruisselbrink, R., Arzola, C., Endersby, R. V. W., Tse, C. C. H., Chan, V. W. S., & Perlas, A. (2014) · Anesthesiology · doi:10.1097/aln.0000000000000193 · Cited by 152
Two aspects of the therapeutic alliance: Differential relations with depressive symptom change.
Webb, C. A., DeRubeis, R. J., Amsterdam, J. D., Shelton, R. C., Hollon, S. D., & Dimidjian, S. (2011) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0023252 · Cited by 150
Estimating clinical stage of amyotrophic lateral sclerosis from the ALS Functional Rating Scale
Balendra, R., Jones, A., Jivraj, N. K., Knights, C., Ellis, C. M., Burman, R., Turner, M. R., Leigh, P. N., Shaw, C. E., & Al‐Chalabi, A. (2014) · Amyotrophic Lateral Sclerosis and Frontotemporal Degeneration · doi:10.3109/21678421.2014.897357 · Cited by 149
Tagami, T., Kushimoto, S., Yamamoto, Y., et al. (2010) · Critical Care · doi:10.1186/cc9250 · Cited by 151
Ottenbacher, K. J., Taylor, E. T., Msall, M. E., Braun, S., Lane, S. J., Granger, C. V., Lyons, N., & Duffy, L. C. (1996) · Developmental Medicine & Child Neurology · doi:10.1111/j.1469-8749.1996.tb15047.x · Cited by 151
Gaston, L., Thompson, L. W., Gallagher-Thompson, D. E., Cournoyer, L., & Gagnon, R. (1998) · Psychotherapy Research · doi:10.1093/ptr/8.2.190 · Cited by 149
Systematic review of the quality of randomized controlled trials for patellofemoral pain syndrome.
Bizzini, M., Childs, J. D., Piva, S., & Delitto, A. (2003) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2003.33.1.4 · Cited by 165
Bouchard, S., Sabourin, S., Lussier, Y., & Villeneuve, É. (2009) · Journal of Marital and Family Therapy · doi:10.1111/j.1752-0606.2009.00151.x · Cited by 152
Age differences in the reliability of the psychiatric interview of the child.
EDELBROCK, C. S., Costello, A. J., Dulcan, M. K., Kalas, R., & Conover, N. C. (1985) · PubMed · Cited by 151
Can chimpanzee (Pan troglodytes) happiness be estimated by human raters?
King, J. E., & Landau, V. I. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(02)00527-5 · Cited by 153
Shin, S. J., Morgeson, F. P., & Campion, M. A. (2006) · Journal of International Business Studies · doi:10.1057/palgrave.jibs.8400247 · Cited by 150
Stern, M., & Working Group on Serologic Screening for Celiac Disease (2000) · Journal of Pediatric Gastroenterology and Nutrition · doi:10.1097/00005176-200011000-00012 · Cited by 151
The effect of L1 use on pronunciation in Quichua–Spanish bilinguals
Guion, S. G., Flege, J. E., & Loftin, J. D. (2000) · Journal of Phonetics · doi:10.1006/jpho.2000.0104 · Cited by 150
Niemiec, S. L. S., Goldberg, A., & Wallace, M. (2009) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2009.06.003 · Cited by 150
An automated pipeline for constructing personalized virtual brains from multimodal neuroimaging data
Schirner, M., Rothmeier, S., Jirsa, V., McIntosh, A. R., & Ritter, P. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.03.055 · Cited by 147
Accuracy of the clinical examination to predict radiographic instability of the lumbar spine
Fritz, J. M., Piva, S. R., & Childs, J. D. (2005) · European Spine Journal · doi:10.1007/s00586-004-0803-4 · Cited by 155
Weber, W. G., Unterrainer, C., & Schmid, B. E. (2009) · Journal of Organizational Behavior · doi:10.1002/job.615 · Cited by 161
Schwarzenthal, M., Schachner, M. K., Juang, L. P., & van de Vijver, F. J. R. (2019) · European Journal of Social Psychology · doi:10.1002/ejsp.2617 · Cited by 151
A Randomized, Controlled Clinical Trial of a Treatment for Shoulder Pain
Ginn, K. A., Herbert, R., Khouw, W., & Lee, R. (1997) · Physical Therapy · doi:10.1093/ptj/77.8.802 · Cited by 149
Age differences in hippocampal subfield volumes from childhood to late adulthood
Daugherty, A. M., Bender, A. R., Raz, N., & Ofen, N. (2015) · Hippocampus · doi:10.1002/hipo.22517 · Cited by 149
Test-retest and alternate forms reliability of the assisting hand assessment
Holmefur, M., Aarts, P. B. M., Hoare, B., & Krumlinde‐Sundholm, L. (2009) · Journal of Rehabilitation Medicine · doi:10.2340/16501977-0448 · Cited by 152
Radiologic Criteria for the Diagnosis of Spinal Stenosis: Results of a Delphi Survey
Mamisch-Saupe, N., Brumann, M., Hodler, J., Held, U., Brunner, F., & Steurer, J. (2012) · Radiology · doi:10.1148/radiol.12111930 · Cited by 147
Scale- and Item-Level Factor Analyses of the VIA Inventory of Strengths
McGrath, R. E. (2012) · Assessment · doi:10.1177/1073191112450612 · Cited by 149
Acceleration-based gait test for healthy subjects: Reliability and reference data
Senden, R. H. J., Grimm, B. P., Heyligers, I. C., et al. (2009) · Gait & Posture · doi:10.1016/j.gaitpost.2009.04.008 · Cited by 149
ImPact Test-Retest Reliability: Reliably Unreliable?
Resch, J. E., Driscoll, A., McCaffrey, N., Brown, C. N., Ferrara, M. S., Macciocchi, S. N., Baumgartner, T. A., & Walpert, K. P. (2013) · Journal of Athletic Training · doi:10.4085/1062-6050-48.3.09 · Cited by 149
Retest reliability of event‐related potentials: E vidence from a variety of paradigms
Cassidy, S. M., Robertson, I. H., & O’Connell, R. G. (2012) · Psychophysiology · doi:10.1111/j.1469-8986.2011.01349.x · Cited by 149
Learning With Weblogs: Enhancing Cognitive and Social Knowledge Construction
Du, H. S., & Wagner, C. (2007) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2006.890848 · Cited by 149
McPoil, T. G., Vicenzino, B., Cornwall, M. W., Collins, N. J., & Warren, M. (2009) · Journal of Foot and Ankle Research · doi:10.1186/1757-1146-2-6 · Cited by 149
The impact of an exergame-based intervention on children's fundamental motor skills
Vernadakıs, N., Παπαστεργίου, Μ., Ζetou, E., & Antonıou, P. (2015) · Computers & Education · doi:10.1016/j.compedu.2015.01.001 · Cited by 149
Eliasson, A., Ullenhag, A., Wahlström, U., & Krumlinde‐Sundholm, L. (2016) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.13162 · Cited by 149
A meta-analysis of different HR-enhancing practices and performance of small and medium sized firms
Rauch, A., & Hatak, I. (2016) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2016.05.005 · Cited by 159
Reliability of the electrocortical response to gains and losses in the doors task
Levinson, A. R., Speed, B. C., Infantolino, Z. P., & Hajcak, G. (2017) · Psychophysiology · doi:10.1111/psyp.12813 · Cited by 155
The accuracy of the TriTrac-R3D accelerometer to estimate energy expenditure
Jakicic, J. M., Winters, C., Lagally, K. M., HO, J., Robertson, R. J., & Wing, R. R. (1999) · Medicine & Science in Sports & Exercise · doi:10.1097/00005768-199905000-00020 · Cited by 149
Maele, D. V., & Houtte, M. V. (2010) · Social Indicators Research · doi:10.1007/s11205-010-9605-8 · Cited by 182
McMahon, C. A., Ungerer, J. A., Tennant, C., & Saunders, D. M. (1997) · Fertility and Sterility · doi:10.1016/s0015-0282(97)00230-6 · Cited by 146
The REM sleep behavior disorder screening questionnaire: validation study of a Japanese version.
Miyamoto, T., Miyamoto, M., Iwanami, M., Kobayashi, M., Nakamura, M., Inoue, Y., Ando, C., & Hirata, K. (2009) · Sleep Medicine · doi:10.1016/j.sleep.2009.05.007 · Cited by 142
Reliability of clinical balance outcome measures in the elderly
Giorgetti, M., Harris, B. A., & Jette, A. M. (1998) · Physiotherapy Research International · doi:10.1002/pri.150 · Cited by 149
Solving Everyday Problems Creatively: The Role of Problem Construction and Personality Type
Reiter‐Palmon, R., Mumford, M. D., & Threlfall, K. V. (1998) · Creativity Research Journal · doi:10.1207/s15326934crj1103_1 · Cited by 149
Evaluating neurological outcome measures: the bare essentials.
Hobart, J., Lamping, D. L., & Thompson, A. J. (1996) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp.60.2.127 · Cited by 149
Knee flexion during stair ambulation is altered in individuals with patellofemoral pain
Crossley, K. M., Cowan, S., Bennell, K. L., & McConnell, J. (2003) · Journal of Orthopaedic Research® · doi:10.1016/j.orthres.2003.08.014 · Cited by 146
Corriveau, H., Hébert, R., Prince, F., & Raı̂che, M. (2001) · Archives of Physical Medicine and Rehabilitation · doi:10.1053/apmr.2001.18678 · Cited by 149
Alberta Infant Motor Scale: Reliability and Validity When Used on Preterm Infants in Taiwan
Jeng, S., Yau, K. T., Chen, L., & Hsiao, S.-F. (2000) · Physical Therapy · doi:10.1093/ptj/80.2.168 · Cited by 149
The Ripple Effect: Emotional Contagion in GroupsPreprint
Barsade, S. G. (2001) · SSRN Electronic Journal · doi:10.2139/ssrn.250894 · Cited by 150
Mikami, A. Y., Griggs, M. S., Lerner, M. D., Emeh, C. C., Reuland, M. M., Jack, A., & Anthony, M. R. (2013) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0029654 · Cited by 148
Wenger, E., Mårtensson, J., Noack, H., Bodammer, N. C., Kühn, S., Schaefer, S., Heinze, H., Düzel, E., Bäckman, L., Lindenberger, U., et al. (2014) · Human Brain Mapping · doi:10.1002/hbm.22473 · Cited by 151
Liu, N.-C., Troconis, E. L., Kalmar, L., Price, D. J., Wright, H. E., Adams, V. J., Sargan, D. R., & Ladlow, J. F. (2017) · PLOS ONE · doi:10.1371/journal.pone.0181928 · Cited by 148
Scoring hip osteoarthritis with MRI (SHOMRI): A whole joint osteoarthritis evaluation system
Lee, S., Nardo, L., Kumar, D., Wyatt, C. R., Souza, R. B., Lynch, J. A., McCulloch, C. E., Majumdar, S., Lane, N. E., & Link, T. M. (2014) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.24722 · Cited by 150
McFadyen, A. K., Webster, V. S., & Maclaren, W. M. (2006) · Journal of Interprofessional Care · doi:10.1080/13561820600991181 · Cited by 157
Paiva, C. E., Barroso, E. M., Carneseca, E. C., Souza, C. D. P., dos Santos, F. T., López, R. V. M., & Paiva, S. B. R. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-8 · Cited by 148
Moock, J., & Kohlmann, T. (2008) · Quality of Life Research · doi:10.1007/s11136-008-9317-6 · Cited by 150
Psychometric properties of the EQ-5D-5L in the general population of South Korea
Kim, T. H., Jo, M., Lee, S., Kim, S., & Chung, S. M. (2012) · Quality of Life Research · doi:10.1007/s11136-012-0331-3 · Cited by 150
Social Innovation: Enhancing Creative Performance Through Causal Analysis
Marcy, R. T., & Mumford, M. D. (2007) · Creativity Research Journal · doi:10.1080/10400410701397180 · Cited by 148
Joseph, G. B., Baum, T. H., Alizai, H., Carballido‐Gamio, J., Nardo, L., Virayavanich, W., Lynch, J. A., Nevitt, M. C., McCulloch, C. E., Majumdar, S., et al. (2012) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2012.04.003 · Cited by 148
Reliability of Clinical Measurements of Forward Bending Using the Modified Fingertip-to-Floor Method
Gauvin, M. G., Riddle, D. L., & Rothstein, J. M. (1990) · Physical Therapy · doi:10.1093/ptj/70.7.443 · Cited by 148
Carotid Artery Anatomy and Geometry as Risk Factors for Carotid Atherosclerotic Disease
Phan, T. G., Beare, R. J., Jolley, D., Das, G., Ren, M., Wong, K., Chong, W., Sinnott, M. D., Hilton, J. E., & Srikanth, V. K. (2012) · Stroke · doi:10.1161/strokeaha.111.645499 · Cited by 148
Minimally important difference estimates and methods: a protocol
Johnston, B. C., Ebrahim, S., Carrasco‐Labra, A., Furukawa, T. A., Patrick, D. L., Crawford, M. W., Hemmelgarn, B. R., Schünemann, H. J., Guyatt, G. H., & NESRALLAH, G. E. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-007953 · Cited by 154
Lineage tracing of human B cells reveals the in vivo landscape of human antibody class switching
Horns, F., Vollmers, C., Croote, D., Mackey, S. F., Swan, G. E., Dekker, C. L., Davis, M. M., & Quake, S. R. (2016) · eLife · doi:10.7554/elife.16578 · Cited by 146
King, G. A., Orchard, C., Khalili, H., & Avery, L. M. (2016) · Journal of Continuing Education in the Health Professions · doi:10.1097/ceh.0000000000000082 · Cited by 148
Venugopal, J. P., Rao, H. L., Weinreb, R. N., et al. (2017) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2017-310637 · Cited by 148
Association Between Insight and Outcome of Psychotherapy: Systematic Review and Meta-Analysis
Jennissen, S., Huber, J., Ehrenthal, J. C., Schauenburg, H., & Dinger, U. (2018) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.17080847 · Cited by 169
Zheng, B.-H., Liu, L., Zhang, Z.-Z., Shi, J.-Y., Dong, L., Tian, L., Ding, Z., Ji, Y., Rao, S., Zhou, J., et al. (2018) · BMC Cancer · doi:10.1186/s12885-018-5024-z · Cited by 148
Strand, L. I., Ljunggren, A. E., Bogen, B. E., Ask, T., & Johnsen, T. B. (2008) · European Journal of Pain · doi:10.1016/j.ejpain.2007.12.013 · Cited by 146
Sabari, J. S., Maltzev, I., Lubarsky, D., et al. (1998) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(98)90038-7 · Cited by 148
Cottone, P., Sabino, V., Steardo, L., & Zorrilla, E. P. (2007) · Neuropsychopharmacology · doi:10.1038/sj.npp.1301430 · Cited by 148
Reliability of centre of pressure summary measures of postural steadiness in healthy young adults
Santos, B. R., Delisle, A., Larivière, C., Plamondon, A., & Imbeau, D. (2007) · Gait & Posture · doi:10.1016/j.gaitpost.2007.05.008 · Cited by 147
Iverson, B. D., Gossman, M. R., Shaddeau, S. A., & Turner, M. E. (1990) · Physical Therapy · doi:10.1093/ptj/70.6.348 · Cited by 148
MacKay, J. W., Low, S. B. L., Smith, T. O., Toms, A. P., McCaskie, A. W., & Gilbert, F. J. (2018) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2017.11.018 · Cited by 148
Method and reporting quality in health professions education research: a systematic review
Cook, D. A., Levinson, A. J., & Garside, S. (2011) · Medical Education · doi:10.1111/j.1365-2923.2010.03890.x · Cited by 147
Test-Retest Reliability of the Women's Health Initiative Physical Activity Questionnaire
Meyer, A., Evenson, K. R., Morimoto, L. M., Siscovick, D. S., & White, E. (2009) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0b013e31818ace55 · Cited by 148
Validation of the EQ-5D in a general population sample in urban China
Wang, H.-M., Patrick, D. L., Edwards, T. C., Skalicky, A. M., Zeng, H.-Y., & Gu, W.-W. (2011) · Quality of Life Research · doi:10.1007/s11136-011-9915-6 · Cited by 147
Hsueh, I., Hsu, M., Sheu, C., Lee, S., Hsieh, C., & Lin, J. (2008) · Neurorehabilitation and neural repair · doi:10.1177/1545968308315999 · Cited by 147
Regueiro, M. D., Kip, K. E., Schraut, W. H., Baidoo, L., Sepulveda, A. R., Pesci, M., El‐Hachem, S., Harrison, J., & Binion, D. G. (2010) · Inflammatory Bowel Diseases · doi:10.1002/ibd.21355 · Cited by 146
O’Neill, M. L., Lidz, V., & Heilbrun, K. (2003) · Law and Human Behavior · doi:10.1023/a:1023435924569 · Cited by 147
Stemland, I., Ingebrigtsen, J., Christiansen, C. S., et al. (2015) · Ergonomics · doi:10.1080/00140139.2014.998724 · Cited by 147
Assessment of Hip Abductor Muscle Strength. A Validity and Reliability Study
Widler, K., Item‐Glatthorn, J. F., Bizzini, M., Impellizzeri, F. M., Munzinger, U., Leunig, M., & Maffiuletti, N. A. (2009) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.h.01119 · Cited by 147
Fjørtoft, I., Pedersen, A. V., Sigmundsson, H., et al. (2011) · Physical Therapy · doi:10.2522/ptj.20090350 · Cited by 147
Ağırdağ, O., Avermaet, P. V., & Houtte, M. V. (2013) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/016146811311500305 · Cited by 153
Termenón, M., Jaillard, A. S., Delon‐Martin, C., & Achard, S. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.05.062 · Cited by 154
Psychological responses to natural patterns in architecture
Coburn, A., Kardan, O., Kotabe, H., Steinberg, J., Hout, M. C., Robbins, A., MacDonald, J. A., Hayn-Leichsenring, G. U., & Berman, M. G. (2019) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2019.02.007 · Cited by 148
Tesluk, P. E., Farr, J. L., Mathieu, J. E., & Vance, R. J. (1995) · Personnel Psychology · doi:10.1111/j.1744-6570.1995.tb01773.x · Cited by 147
Gifford, R., Hine, D. W., Müller-Clemm, W. J., Reynolds, D. J., & Shaw, K. T. (2000) · Environment and Behavior · doi:10.1177/00139160021972487 · Cited by 147
Wang, H.-H., Liao, H., & Hsieh, C. (2006) · Physical Therapy · doi:10.2522/ptj.20050259 · Cited by 147
Bone Marrow and Recombinant Human Bone Morphogenetic Protein-2 in Osseous Repair
Lane, J. M., Yasko, A. W., Tomin, E., Cole, B. J., Waller, S., Browne, M., Turek, T. J., & Gross, J. (1999) · Clinical Orthopaedics and Related Research · doi:10.1097/00003086-199904000-00028 · Cited by 146
Suh, J., Eigsti, I.-M., Naigles, L., Barton, M., Kelley, E., & Fein, D. (2014) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-014-2042-9 · Cited by 146
Lenhard, F., Andersson, E., Mataix-Cols, D., Rück, C., Vigerland, S., Högström, J., Hillborg, M., Brander, G., Ljungström, M., Ljótsson, B., et al. (2017) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2016.09.515 · Cited by 146
The reliability of isokinetic testing of the ankle joint and a heel‐raise test for endurance
Möller, M., LIND, K., STYF, J. R., & Karlsson, J. (2003) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-003-0441-0 · Cited by 147
Swift, R. H., Harrigan, E. P., Cappelleri, J. C., Kramer, D., & Chandler, L. P. (2002) · Journal of Psychiatric Research · doi:10.1016/s0022-3956(01)00052-8 · Cited by 146
Gentner, R., & Claßen, J. (2008) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2008.11.005 · Cited by 146
Hannah, S. T., Walumbwa, F. O., & Fry, L. W. (2011) · Personnel Psychology · doi:10.1111/j.1744-6570.2011.01225.x · Cited by 158
Strauman, T. J., Vieth, A. Z., Eddington, K. M., Kolden, G. G., Woods, T. E., Klein, M. H., Papadakis, A. A., Schneider, K. L., & Kwapil, L. (2006) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.74.2.367 · Cited by 151
Massey, K., Barnes, M. J. D., Villines, D. C., Goldstein, J., Pierson, A. L. H., Scherer, C., Laan, B. V., & Summerfelt, W. T. (2015) · BMC Palliative Care · doi:10.1186/s12904-015-0008-0 · Cited by 146
Mannion, A. F., Pulkovski, N., Gubler, D., Gorelick, M., O’Riordan, D., Loupas, T., Schenk, P., Gerber, H., & Sprott, H. (2008) · European Spine Journal · doi:10.1007/s00586-008-0589-x · Cited by 146
Nam, S. Y., Choi, I. J., Ryu, K. H., Park, B. J., Kim, H. B., & Nam, B. (2010) · Gastroenterology · doi:10.1053/j.gastro.2010.08.019 · Cited by 147
Validity and Reliability of a Pediatric Reach Test
Bartlett, D. J., & Birmingham, T. B. (2003) · Pediatric Physical Therapy · doi:10.1097/01.pep.0000067885.63909.5c · Cited by 146
Development of the 25-item Cardiff Visual Ability Questionnaire for Children (CVAQC)
Khadka, J., Ryan, B., Margrain, T. H., et al. (2010) · British Journal of Ophthalmology · doi:10.1136/bjo.2009.171181 · Cited by 146
An Examination of Judge Reliability at a major U.S. Wine Competition
Hodgson, R. T. (2008) · Journal of Wine Economics · doi:10.1017/s1931436100001152 · Cited by 146
Test-retest reliability of transcranial magnetic stimulation EEG evoked potentials
Kerwin, L. J., Keller, C. J., Wu, W., Narayan, M., & Etkin, A. (2017) · Brain stimulation · doi:10.1016/j.brs.2017.12.010 · Cited by 142
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
