The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,342
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 10,701–10,800 of 23,881
Çevikaslan, A., Evans, D. W., Dedeoğlu, C., Kalaça, S., & Yazgan, Y. (2013) · Child Psychiatry & Human Development · doi:10.1007/s10578-013-0417-3 · Cited by 21
Madjid Tavana, D., Smither, J. W., & Anderson, R. V. (2005) · Computers & Operations Research · doi:10.1016/j.cor.2005.06.020 · Cited by 20
Biederman, G. B., Stepaniuk, S., Davey, V. A., et al. (1999) · Down Syndrome Research and Practice · doi:10.3104/reports.93 · Cited by 20
McIntosh, V. V. W., Jordan, J., McKenzie, J. M., Luty, S. E., Carter, F. A., Carter, J. D., FRAMPTON, C. M. A., & Joyce, P. R. (2005) · Psychotherapy Research · doi:10.1080/10503300500091124 · Cited by 20
Borsel, J. V., & Eeckhout, H. (2008) · Journal of Fluency Disorders · doi:10.1016/j.jfludis.2008.06.004 · Cited by 20
Merino-Marbán, R., Rodríguez, E. F., & Mayorga-Vega, D. (2014) · Research in Sports Medicine · doi:10.1080/15438627.2013.852089 · Cited by 20
Job requirements for control room jobs in nuclear power plants
Schumacher, S., Kleinmann, M., & Melchers, K. G. (2010) · Safety Science · doi:10.1016/j.ssci.2010.10.002 · Cited by 20
Evaluation of a training program intended to calibrate examiners of temporomandibular disorders
Dahlström, L., Keeling, S. D., Fricton, J., et al. (1994) · Acta Odontologica Scandinavica · doi:10.3109/00016359409029054 · Cited by 20
Digital stereoscopic analysis of the optic disc
Sheen, N. J. L., Morgan, J. E., Poulsen, J. L., & North, R. V. (2004) · Ophthalmology · doi:10.1016/j.ophtha.2004.04.018 · Cited by 22
The Terms Occupation and Activity Over the History of Official Occupational Therapy Publications
Bauerschmidt, B., & Nelson, D. L. (2011) · American Journal of Occupational Therapy · doi:10.5014/ajot.2011.000869 · Cited by 20
Mazaheri, M., Fardipour, S., Salavati, M., et al. (2010) · Disability and Rehabilitation · doi:10.3109/09638288.2010.544838 · Cited by 20
Eudell‐Simmons, E. M., Stein, M. B., DeFife, J. A., & Hilsenroth, M. J. (2005) · Journal of Personality Assessment · doi:10.1207/s15327752jpa8503_09 · Cited by 23
The four square step test in children with Down syndrome: Reliability and concurrent validity
Samuel, A. J., Verma, A., & Aranha, V. P. (2014) · Journal of Pediatric Neurosciences · doi:10.4103/1817-1745.147573 · Cited by 21
Kavanagh, E. C., Koulouris, G. C., Parker, L., et al. (2007) · American Journal of Roentgenology · doi:10.2214/ajr.07.2716 · Cited by 20
Parental assessment of adolescent quality of life: can it replace self-assessment?
Taylor, R. M., Grieve, A. P., Gibson, F., Dhawan, A. K., & Franck, L. S. (2011) · Quality of Life Research · doi:10.1007/s11136-011-9904-9 · Cited by 20
McGrath, D., Doheny, E. P., Walsh, L. C., McKeown, D. J., Cunningham, C. J., Crosby, L., Kenny, R. A., Stergiou, N., Caulfield, B., & Greene, B. R. (2012) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2012.6346669 · Cited by 20
Functional outcomes post-radial head arthroplasty: a systematic review of literature
Kaur, M. N., MacDermid, J. C., Grewal, R. S., et al. (2014) · Shoulder & Elbow · doi:10.1177/1758573214524934 · Cited by 20
Reproducibility of the short-latency reflex inhibition to loading of human inspiratory muscles
Murray, N., McKenzie, D. K., Gorman, R. B., Gandevia, S. C., & Butler, J. E. (2008) · Respiratory Physiology & Neurobiology · doi:10.1016/j.resp.2008.07.007 · Cited by 20
Determination of Language Dominance: Wada Test and fMRI Compared Using a Novel Sentence Task
Gutbrod, K., Spring, D., Degonda, N., Heinemann, D., Nirkko, A. C., Hauf, M., Ozdoba, C., Schnider, A., Schroth, G., & Wiest, R. (2011) · Journal of Neuroimaging · doi:10.1111/j.1552-6569.2011.00646.x · Cited by 20
Simulation of Liver Lesions for Pediatric CT
Hoe, C. L., Samei, E., Frush, D. P., et al. (2006) · Radiology · doi:10.1148/radiol.2381050477 · Cited by 20
Iverson, S. M., Feuer, W. J., Shi, W., et al. (2014) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2013-303877 · Cited by 20
Demand and withdraw behaviors in couples with a history of infidelity.
Balderrama‐Durbin, C. M., Allen, E. S., & RHOADES, G. K. (2011) · Journal of Family Psychology · doi:10.1037/a0026756 · Cited by 21
Melby, J. N., Hoyt, W. T., & Bryant, C. M. (2003) · Journal of Social and Personal Relationships · doi:10.1177/0265407503020002003 · Cited by 20
Sibutramine Treatment in Obesity: Predictors of Weight Loss Including Rorschach Personality Data
Elfhag, K., Rössner, S., Carlsson, A. M., & Barkeling, B. (2003) · Obesity Research · doi:10.1038/oby.2003.188 · Cited by 20
Gait Recognition Using Joint Moments, Joint Angles, and Segment Angles
Larsen, P. K., Lynnerup, N., Henriksen, M., et al. (2010) · Journal of Forensic Biomechanics · doi:10.4303/jfb/f100302 · Cited by 20
Syversveen, T., Brabrand, K., Midtvedt, K., Strøm, E. H., Hartmann, A., & Berstad, A. E. (2011) · Acta Radiologica · doi:10.1258/ar.2011.110215 · Cited by 20
Ringen, P. A., Melle, I. S., Berg, A. O., Agartz, I., Spigset, O., Simonsen, C., Sundet, K. S., & Andreassen, O. A. (2012) · Schizophrenia Research · doi:10.1016/j.schres.2012.10.021 · Cited by 20
Reliability of proximal migration measurements in shoulder arthroplasty
Nagels, J., Verweij, J., Stokdijk, M., et al. (2008) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2007.07.011 · Cited by 20
Drug-Related Problems and Hospital Admissions in Cardiac Transplant Recipients
Repp, K. L., Hayes, C. H., Woods, T. M., Allen, K. B., Kennedy, K. F., & Borkon, M. (2012) · Annals of Pharmacotherapy · doi:10.1345/aph.1r094 · Cited by 20
Walsh, J., Barrett, A., Murray, D., Ryan, J., Moroney, J. T., & Shannon, M. (2010) · Disability and Rehabilitation · doi:10.3109/09638280903171576 · Cited by 19
Greco, A., Petretta, M. P., Larobina, M., Gargiulo, S., Panico, M., Nekolla, S. G., Esposito, G., Petretta, M., Brunetti, A., & Cuocolo, A. (2012) · Journal of Nuclear Cardiology · doi:10.1007/s12350-012-9538-5 · Cited by 19
What Are We Aligning Tests to When We Report Test Alignment to the CEFR?
Harsch, C., & Hartig, J. (2015) · Language Assessment Quarterly · doi:10.1080/15434303.2015.1092545 · Cited by 20
Hafez, A. S., Bizzarro, R. L. G., Rivard, M., et al. (2003) · Ophthalmic surgery, lasers & imaging retina · doi:10.3928/1542-8877-20030901-18 · Cited by 20
Assessing Residents’ Written Learning Goals and Goal Writing Skill
Lockspeiser, T. M., Schmitter, P. A., Lane, J. L., Hanson, J. L., Rosenberg, A. A., & Park, Y. S. (2013) · Academic Medicine · doi:10.1097/acm.0b013e3182a352e6 · Cited by 20
Interrater Agreement of Computed Tomography Infarct Measurement
Pullicino, P. M., Snyder, W., Munschauer, F. E., et al. (1996) · Journal of Neuroimaging · doi:10.1111/jon19966116 · Cited by 20
Effect of Hepatic Perfusion on Microwave Ablation Zones in an Ex Vivo Porcine Liver Model
Siriwardana, P. N., Singh, S., Johnston, E. W., et al. (2016) · Journal of Vascular and Interventional Radiology · doi:10.1016/j.jvir.2016.03.006 · Cited by 20
Prostate size, nocturia and the digital rectal examination: a cohort study of 30 500 men
Stone, B. V., Shoag, J. E., Halpern, J. A., et al. (2016) · British Journal of Urology · doi:10.1111/bju.13613 · Cited by 20
Integrating multilevel analyses and occupational stress theory
Bliese, P. D., Jex, S. M., & Halverson, R. R. (2002) · Research in occupational stress and well being · doi:10.1016/s1479-3555(02)02006-1 · Cited by 20
Galbrun, L., & Kitapçi, K. (2016) · Applied Acoustics · doi:10.1016/j.apacoust.2016.07.003 · Cited by 20
Chiang, G. C.-Y., Galla, N., Ferraro, R., et al. (2016) · Journal of Neuroimaging · doi:10.1111/jon.12386 · Cited by 20
Gerke, O., Vilstrup, M. H., Segtnan, E. A., Halekoh, U., & Høilund‐Carlsen, P. F. (2016) · BMC Medical Imaging · doi:10.1186/s12880-016-0159-3 · Cited by 20
In pursuit of an environmentally friendly convention industry
Park, H.-Y., & Kim, D. (2017) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-06-2016-0333 · Cited by 20
Buteau, S., Hatzopoulou, M., Crouse, D. L., Smargiassi, A., Burnett, R. T., Logan, T., Cavellin, L. D., & Goldberg, M. S. (2017) · Environmental Research · doi:10.1016/j.envres.2017.03.017 · Cited by 21
Rehnitz, C., Kuni, B., Wuennemann, F., et al. (2017) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.25731 · Cited by 20
Gagnon, C., Lessard, I., Brais, B., Côté, I., Lavoie, C., Synofzik, M., & Mathieu, J. (2018) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2018.01.026 · Cited by 20
Augustin, M., Wechdorn, M., Pfeiffenberger, U., Himmel, T., Fialová, S., Werkmeister, R. M., Hitzenberger, C. K. F., Glösmann, M., & Baumann, B. (2018) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.17-23208 · Cited by 19
Malavasi, V. L., Pettorelli, D., Fantecchi, E., Zoccali, C., Laronga, G., Trenti, T., Lip, G. Y. H., & Boriani, G. (2018) · Internal and Emergency Medicine · doi:10.1007/s11739-018-1857-3 · Cited by 20
Multimodal stimuli regulate reproductive behavior and physiology in male túngara frogs
Still, M. B., Lea, A. M., Hofmann, H. A., et al. (2019) · Hormones and Behavior · doi:10.1016/j.yhbeh.2019.06.010 · Cited by 20
Ma, Y., & Corter, J. E. (2019) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2019.100587 · Cited by 23
Reliabilität der statischen Posturografie bei älteren Personen
Bauer, C. M., Gröger, I., Rupprecht, R., et al. (2009) · Zeitschrift für Gerontologie und Geriatrie · doi:10.1007/s00391-009-0052-6 · Cited by 20
Interpersonal Narratives in Cognitive and Interpersonal Psychotherapies
Crits-Christoph, P. F., Connolly, M., Shappell, S., Elkin, I., KRUPNICK, J. L., & Sotsky, S. M. (1999) · Psychotherapy Research · doi:10.1080/10503309912331332571 · Cited by 20
The Reliability of Simulated Clients' Quality-of-Care Ratings
León, F. R., Quiroz, G., & Brazzoduro, A. (1994) · Studies in Family Planning · doi:10.2307/2137944 · Cited by 20
Pyöriä, O., Talvitie, U., & Villberg, J. J. (2005) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2004.01.024 · Cited by 20
Actions of the Scalene Muscles for Rotation of the Cervical Spine in Macaque and Human
Buford, J. A., Yoder, S. M., Givens, D. L., et al. (2002) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2002.32.10.488 · Cited by 20
Staying with the feeling: An anger event in psychodynamic-interpersonal therapy.
Mackay, H. C., Barkham, M., & Stiles, W. B. (1998) · Journal of Counseling Psychology · doi:10.1037/0022-0167.45.3.279 · Cited by 20
Daleiden, E. L., Chorpita, B. F., Kollins, S. H., & Drabman, R. S. (1999) · Journal of Clinical Child Psychology · doi:10.1207/s15374424jccp280312 · Cited by 20
Hafkenscheid, A., & Rouckhout, D. (2009) · Journal of Personality Assessment · doi:10.1080/00223890802634340 · Cited by 18
Autonomy and Relatedness in Inner-City Families of Substance Abusing Adolescents
Samuolis, J., Hogue, A., Dauber, S. E., & Liddle, H. A. (2006) · Journal of Child & Adolescent Substance Abuse · doi:10.1300/j029v15n02_04 · Cited by 21
Chambless, D. L., Floyd, F. J., Rodebaugh, T. L., & Steketee, G. S. (2007) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.116.4.754 · Cited by 20
Foreman, S. A., Gibbins, J. A., Grienenberger, J. F., et al. (2000) · Psychotherapy Research · doi:10.1093/ptr/10.4.450 · Cited by 20
Social Dysfunction and Psychiatric Disorder in Mothers and Their Children
Mufson, L. H., Aidala, A. A., & Warner, V. (1994) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199411000-00006 · Cited by 20
Coding system of Therapeutic Focus on Action and Insight.
Samoilov, A., Goldfried, M. R., & Shapiro, D. A. (2000) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.68.3.513 · Cited by 20
Lee, D. L., & Tracey, T. J. G. (2008) · Psychotherapy · doi:10.1037/a0014336 · Cited by 20
Manber, R., Allen, J. J. B., Burton, K. W., & Kaszniak, A. W. (2000) · Psychophysiology · doi:10.1111/1469-8986.3750683 · Cited by 20
Heitman, R. J., Kovaleski, J. E., & Pugh, S. F. (2009) · Journal of Athletic Training · doi:10.4085/1062-6050-44.1.48 · Cited by 20
Reliability of Sequential Naloxone Challenge Tests
Rosen, M. I., McMahon, T. J., Margolin, A., et al. (1995) · The American Journal of Drug and Alcohol Abuse · doi:10.3109/00952999509002710 · Cited by 20
Development and Test of a Contingency Framework of Coupling
Beekun, R. I., & Glick, W. H. (2001) · The Journal of Applied Behavioral Science · doi:10.1177/0021886301374001 · Cited by 20
Tirsi, A., Bruehl, H., Sweat, V., Tsui, W., Reddy, S., Javier, E., Lee, C. M., & Convit, A. J. (2009) · Ophthalmology · doi:10.1016/j.ophtha.2008.12.046 · Cited by 20
WHAT IS ADOLESCENT ALCOHOL MISUSE IN THE UNITED STATES ACCORDING TO THE EXPERTS?
Hays, R. D., & Ellickson, P. L. (1996) · Alcohol and Alcoholism · doi:10.1093/oxfordjournals.alcalc.a008150 · Cited by 20
Forensic aspects of medical student abuse: a Canadian perspective.
Margittai, K. J., Moscarello, R., & Rossi, M. (1996) · PubMed · Cited by 20
Couples' Co-Construction of the Story of Their Child's Birth: Associations With Marital Adaptation
Oppenheim, D., Wamboldt, F. S., Gavin, L. A., et al. (1996) · Journal of Narrative and Life History · doi:10.1075/jnlh.6.1.01cou · Cited by 20
Intraclass Correlation for Two-by-Two Tables under Three Sampling Designs
Bodian, C. (1994) · Biometrics · doi:10.2307/2533208 · Cited by 20
Gibbons, M. B. C., Crits-Christoph, P. F., Levinson, J., Gladis, M., Siqueland, L., Barber, J. P., & Elkin, I. (2002) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.2002.56.1.3 · Cited by 20
Age Differences in the Reliability of the Psychiatric Interview of the Child
EDELBROCK, C. S., Costello, A. J., Dulean, M. K., et al. (1985) · Child Development · doi:10.1111/j.1467-8624.1985.tb00104.x · Cited by 20
Peer validation and idiographic analysis of social skill deficits
Dow, M. G. T. (1985) · Behavior Therapy · doi:10.1016/s0005-7894(85)80057-5 · Cited by 20
Sources of Unreliability of DSM-III Axis IV
Rey, J. M., Stewar, G. W., Plapp, J. M., Bashir, M. R., & Richards, I. (1987) · Australian & New Zealand Journal of Psychiatry · doi:10.3109/00048678709160902 · Cited by 20
Stringer, D. A., Cairns, R., Poskitt, K. J., et al. (1994) · Pediatric Radiology · doi:10.1007/bf02017647 · Cited by 20
Treatment of Socially Anxious College Men Using Cognitive Counseling and Skills Training.
Gormally, J., Varvil-Weld, D. C., Raphael, R., & Sipps, G. J. (1981) · Journal of Counseling Psychology · doi:10.1037//0022-0167.28.2.177 · Cited by 20
Reliability of Ratings for Multiple Judges: Intraclass Correlation and Metric Scales
Fagot, R. F. (1991) · Applied Psychological Measurement · doi:10.1177/014662169101500101 · Cited by 20
Measurement Variation in Method Comparison Studies: An Empirical Examination
Ottenbacher, K. J., & Tomchek, S. D. (1994) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(21)01611-7 · Cited by 20
Reliability of Absolute and Ratio Data In Assessment of Knee Extensor and Flexor Strength
Kramer, J. F., Ingham-Tupper, S., Walters-Stansbury, K., et al. (1994) · Isokinetics and Exercise Science · doi:10.3233/ies-1994-4201 · Cited by 20
Personalized Gaussian Processes for Future Prediction of Alzheimer's Disease ProgressionPreprint
Peterson, K., Rudovic, O. O., Guerrero, R., & Picard, R. W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00181 · Cited by 20
Observer Reliability and Agreement
de Vet, H. C. W. (2014) · Wiley eBooks · doi:10.1002/9781118445112.stat04910 · Cited by 21
Alzahrani, S. (2021) · PDXScholar (Portland State University) · doi:10.15760/etd.7527 · Cited by 22
Production Ergonomics: Identifying and Managing Risk in the Design of High Performance Work Systems
Neumann, P. (2022) · doi:10.32920/ryerson.14638713.v2 · Cited by 20
Brøndbo, H., Mathiassen, B., Martinussen, M., Heiervang, E., Eriksen, M., & Kvernmo, S. (2012) · Clinical Practice & Epidemiology in Mental Health · doi:10.2174/1745017901208010016 · Cited by 19
Khan, M. T., & Ullah, S. (2025) · International Journal of Innovation Science · doi:10.1108/ijis-07-2024-0206 · Cited by 19
How adaptive social robots influence cognitive, emotional, and self-regulated learning
Ackermann, H., Lange, A. L., Hafner, V. V., & Lazarides, R. (2025) · Scientific Reports · doi:10.1038/s41598-025-91236-0 · Cited by 27
Motivational Computing: Transformer-Based Automation of Implicit Motive Coding
Pang, J. S., Ring, H., Huang, K. Y., & Khoo, A. X. (2026) · Journal of Personality Assessment · doi:10.1080/00223891.2026.2630930 · Cited by 19
Neshati, R. (2017) · The Journal of High Technology Management Research · doi:10.1016/j.hitech.2017.04.004 · Cited by 20
Rusby, J. C., Smolkowski, K., Marquez, B., & Taylor, T. Κ. (2008) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2008.08.003 · Cited by 21
Chung, H. T.-K., Shakespeare, T. P., Wynne, C. J., et al. (2004) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2004.01.001 · Cited by 19
Evaluation of a novel feather scoring system for monitoring feather damaging behaviour in parrots
van Zeeland, Y. R. A., Bergers, M. J., van der Valk, L., Schoemaker, N. J., & Lumeij, J. T. (2012) · The Veterinary Journal · doi:10.1016/j.tvjl.2012.08.020 · Cited by 19
Reliability of a new virtual reality test to measure cervicocephalic kinaesthesia
Kramer, M., Honold, M., Hohl, K., Bockholt, U., Rettig, A., Elbel, M., & Dehner, C. (2008) · Journal of Electromyography and Kinesiology · doi:10.1016/j.jelekin.2008.05.005 · Cited by 19
Forsberg, A., Juréus, A., Cselényi, Z., Eriksdotter, M., Freund‐Levi, Y., Jeppsson, F., Swahn, B., Sandell, J., Julin, P., Schou, M., et al. (2013) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-012-2322-6 · Cited by 19
Celio, M. A., Vetter-O’Hagen, C. S., Lisman, S. A., Johansen, G. E., & SPEAR, L. P. (2011) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2011.05.035 · Cited by 19
Yen, A. W. (2015) · Journal of Interventional Gastroenterology · doi:10.7178/jig.183 · Cited by 19
Campagna, G., Zampetti, S., Gallozzi, A., Giansanti, S., Chiesa, C., Pacifico, L., & Buzzetti, R. (2016) · PLoS ONE · doi:10.1371/journal.pone.0156646 · Cited by 20
Stern‐ellran, K., Zilcha‐Mano, S., Sebba, R., & Levit‐Binnun, N. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01661 · Cited by 28
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
