The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,342
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 7,701–7,800 of 23,881
Validation of image analysis techniques to measure skin aging features from facial photographs
Hamer, M. A., Jacobs, L. C., Lall, J. S., Wollstein, A., Hollestein, L. M., Rae, A., Gossage, K. W., Albert Hofman, D., Liu, F., Kayser, M., et al. (2015) · Skin Research and Technology · doi:10.1111/srt.12205 · Cited by 34
Draper, C. E., Chew, K. T. L., Wang, R. Y., et al. (2011) · PM&R · doi:10.1016/j.pmrj.2010.10.020 · Cited by 34
Kolber, M. J., & Hanney, W. J. (2010) · PubMed · Cited by 34
ACCURACY OF WEIGHTBEARING ESTIMATION BY STROKE VERSUS HEALTHY SUBJECTS
Bohannon, R. W., & Tinti-Wald, D. (1991) · Perceptual and Motor Skills · doi:10.2466/pms.1991.72.3.935 · Cited by 34
Effects of Military Deployment on Cognitive Functioning
Vincent, A. S., Roebuck‐Spencer, T., Lopez, M. S., Twillie, D. A., Logan, B. W., Grate, S. J., Friedl, K. E., Schlegel, R. E., & Gilliland, K. (2012) · Military Medicine · doi:10.7205/milmed-d-11-00156 · Cited by 36
Gentile, S., Jouve, É., Dussol, B., et al. (2008) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-6-78 · Cited by 34
Cruz, J., Marques, A. S. P. D. D., Barbosa, A., de Figueiredo, D. M. P., & de Sousa, L. X. M. (2013) · American Journal of Alzheimer s Disease & Other Dementias® · doi:10.1177/1533317512473194 · Cited by 34
Cross-cultural validation of the UCLA prostate cancer index
Karakiewicz, P. I., Kattan, M. W., Tanguay, S., ELHILALI, M. M., Bazinet, M., Scardino, P. T., & Aprikian, A. G. (2003) · Urology · doi:10.1016/s0090-4295(02)02245-8 · Cited by 34
Adjusted Chi-Square Statistics: Application to Clustered Binary Data in Primary Care
REED, J. F. (2004) · The Annals of Family Medicine · doi:10.1370/afm.41 · Cited by 34
Transition to Home Care: Quality of Mental Health, Pharmacy, and Medical History Information
Brown, E. L., Raue, P. J., Mlodzianowski, A. E., et al. (2006) · The International Journal of Psychiatry in Medicine · doi:10.2190/6n5p-5cxh-l750-a8hv · Cited by 34
Cadenhead, S. L., McEwen, I. R., & Thompson, D. M. (2002) · Physical Therapy · doi:10.1093/ptj/82.7.658 · Cited by 34
Who’s bringing the donuts: The role of affective patterns in group decision making
Emich, K. J. (2014) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2014.03.001 · Cited by 35
Cruz, B. D. P., Hernández-Mocholí, M. A., del Pozo-Cruz, J., et al. (2014) · Journal of Back and Musculoskeletal Rehabilitation · doi:10.3233/bmr-140460 · Cited by 34
Improving methodological robustness in cross-cultural organizational research
van de Vijver, F. J. R., & Fischer, R. D. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511581151.019 · Cited by 34
Clinical Evaluation of the Shoulder Shrug Sign
Jia, X., Ji, J., Petersen, S. A., Keefer, J., & McFarland, E. G. (2008) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-008-0331-3 · Cited by 36
Reliability of cervical lordosis measurement techniques on long-cassette radiographs
Janusz, P., Tyrakowski, M., Yu, H., & Siemionow, K. B. (2015) · European Spine Journal · doi:10.1007/s00586-015-4345-8 · Cited by 36
Interactions Between Socioeconomic Status and Components of Variation in Cognitive Ability
Turkheimer, E. N., & Horn, E. E. (2013) · Springer eBooks · doi:10.1007/978-1-4614-7447-0_2 · Cited by 34
Siemionow, K. B., Janusz, P., & Główka, P. (2016) · European Spine Journal · doi:10.1007/s00586-016-4430-7 · Cited by 34
O’Neill, T., & Allen, N. J. (2014) · Group Dynamics Theory Research and Practice · doi:10.1037/gdn0000004 · Cited by 39
Emergent life events and their impact on service delivery: Should we expect the unexpected?
Chorpita, B. F., Korathu-Larson, P. A., Knowles, L. M., & Guan, K. (2014) · Professional Psychology Research and Practice · doi:10.1037/a0037746 · Cited by 35
Filatotchev, I. V., Su, Z., & Bruton, G. D. (2016) · Management and Organization Review · doi:10.1017/mor.2016.31 · Cited by 34
McKevitt, C. (2016) · Journal of University Teaching and Learning Practice · doi:10.53761/1.13.1.2 · Cited by 34
Leung, H. B., Yen, C. H., & Tse, P. Y. T. (2004) · PubMed · Cited by 34
The Development and Application of a Process-oriented "Thermometer" of IT Business Value
Tallon, P. P., & Kraemer, K. L. (2006) · Communications of the Association for Information Systems · doi:10.17705/1cais.01745 · Cited by 34
Piper, W. E., Ogrodniczuk, J. S., McCallum, M., Joyce, A. S., & Rosie, J. S. (2003) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.71.4.664 · Cited by 34
An Assessment of Self-Reported Disease Classification in Epidemiological Studies of Dry Eye
Nichols, J. J., Mitchell, G. L., & Nichols, K. K. (2004) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.04-0468 · Cited by 34
Sexton, H. C. (1993) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.61.1.128 · Cited by 34
Behavioral Aggression Is Associated with the 2D:4D Ratio in Men but Not in Women
Kuepper, Y., & Hennig, J. (2007) · Journal of Individual Differences · doi:10.1027/1614-0001.28.2.64 · Cited by 34
Emotional awareness, gender, and suspiciousness
Boden, M. T., & Berenbaum, H. (2007) · Cognition & Emotion · doi:10.1080/02699930600593412 · Cited by 38
Jacobs, R. R., Hofmann, D. A., & Kriska, S. D. (1990) · Human Performance · doi:10.1207/s15327043hup0302_3 · Cited by 34
Expectant adolescent couples' relations and subsequent parenting behavior
Florsheim, P., & Smith, A. (2005) · Infant Mental Health Journal · doi:10.1002/imhj.20076 · Cited by 34
Measures of Short-Term Test–Retest Reliability of Computerized Neurobehavioral Tests
Farahat, F. M., Rohlman, D. S., Storzbach, D. M., Ammerman, T. A., & Anger, W. K. (2003) · NeuroToxicology · doi:10.1016/s0161-813x(03)00079-2 · Cited by 34
Reliability analysis of event‐related brain potentials to olfactory stimuli
Thesen, T., & Murphy, C. (2002) · Psychophysiology · doi:10.1111/1469-8986.3960733 · Cited by 35
Behavioral Confirmation of the Loneliness Stereotype
Rotenberg, K. J., Gruman, J. A., & Ariganello, M. (2002) · Basic and Applied Social Psychology · doi:10.1207/s15324834basp2402_1 · Cited by 34
Finger-Movement Tracking Scores in Healthy Subjects
Carey, J. R., Bogard, C. L., King, B. A., & Suman, V. J. (1994) · Perceptual and Motor Skills · doi:10.2466/pms.1994.79.1.563 · Cited by 34
The Influence of Site of Stimulation, Age, and Gender on Pain Threshold in Healthy Children
Hogeweg, J. A., Kuis, W., Oostendorp, R. A. B., et al. (1996) · Physical Therapy · doi:10.1093/ptj/76.12.1331 · Cited by 34
Development of a disease-specific disability instrument for Pompe disease
Haley, S. M., Fragala, M. A., Aseltine, R. H., et al. (2003) · Developmental Neurorehabilitation · doi:10.1080/1363849031000139298 · Cited by 34
Television and Movie Representations of Salespeople: Beyond Willy Loman
Hartman, K. B. (2006) · Journal of Personal Selling and Sales Management · doi:10.2753/pss0885-3134260303 · Cited by 34
Resource use data by patient report or hospital records: Do they agree?
Kennedy, A. D. M., Brown, A., Torgerson, D. J., Campbell, J., & Grant, A. (2002) · BMC Health Services Research · doi:10.1186/1472-6963-2-2 · Cited by 33
Falkai, P., Schneider-Axmann, T., Honer, W. G., et al. (2003) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-003-0414-9 · Cited by 34
A radiological measure of shoulder subluxation in hemiplegia: its reliability and validity.
Ea, B., Goudreau, L., O'Riain, M. D., et al. (1993) · PubMed · Cited by 34
Reliability of reported age at onset for Parkinson's disease
Reider, C. R., Halter, C. A., Castelluccio, P., Oakes, D., Nichols, W. C., Foroud, T., & and the Parkinson Study Group (2002) · Movement Disorders · doi:10.1002/mds.10391 · Cited by 34
Recall of Childhood Psychopathology More Than 10 Years Later
Masia, C. L., Storch, E. A., Dent, H. C., Adams, P., Verdeli, H., Davies, M., & Weissman, M. M. (2003) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-200301000-00005 · Cited by 34
Member Ratings of Health Care Provided by 48 Physician Groups
Hays, R. D., Brown, J. A., Spritzer, K. L., Dixon, W., & Brook, R. H. (1998) · Archives of Internal Medicine · doi:10.1001/archinte.158.7.785 · Cited by 34
Spinath, F. M., & Angleitner, A. (1998) · Personality and Individual Differences · doi:10.1016/s0191-8869(98)00097-x · Cited by 34
Elliott, S., Taylor, S. M., Cameron, R., & Schabas, R. (1998) · Health Education Research · doi:10.1093/her/13.4.607 · Cited by 34
Play and Language in Developmentally Disordered Preschoolers
Sherman, M., Shapiro, T., & Glassman, M. B. (1983) · Journal of the American Academy of Child Psychiatry · doi:10.1097/00004583-198311000-00001 · Cited by 34
Fundamental Psychometric Considerations in Assessment
Wasserman, J. D., & Bracken, B. A. (2012) · Wiley eBooks · doi:10.1002/9781118133880.hop210003 · Cited by 35
Holman, C. D. J., Matz, L. R., Finlay‐Jones, L. R., et al. (1983) · Histopathology · doi:10.1111/j.1365-2559.1983.tb02253.x · Cited by 34
Teigen, L. M., Kuchnia, A. J., Nagel, E., Deuth, C., Vock, D. M., Mulasi, U., & Earthman, C. P. (2017) · Journal of Parenteral and Enteral Nutrition · doi:10.1002/jpen.1036 · Cited by 35
Hasin, D., Shmulewitz, D., Stohl, M., Greenstein, E., Roncone, S., Aharonovich, E., & Wall, M. (2020) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2020.108294 · Cited by 31
Song, S., Nerur, S., & Teng, J. T. C. (2007) · ACM SIGMIS Database: the DATABASE for Advances in Information Systems · doi:10.1145/1240616.1240620 · Cited by 50
When firms adopt sustainable human resource management: A fuzzy‐set analysis
Jia, J., Yuan, S., Wei, L., & Tang, G. (2023) · Human Resource Management · doi:10.1002/hrm.22164 · Cited by 34
Luca, N. J. C., Stinson, J., Feldman, B. M., Benseler, S. M., Beaton, D., Campillo, S., LeBlanc, C. M. A., van Wyk, M., & Bayoumi, A. M. (2017) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2017.7375 · Cited by 34
Makvandi, K., Hockings, P. D., Jensen, G., Unnerstall, T., Leonhardt, H., Jarl, L., Englund, C., Francis, S., Sundgren, A. K., Hulthe, J., et al. (2022) · Clinical Kidney Journal · doi:10.1093/ckj/sfac054 · Cited by 34
The role of work-to-venture role conflict on hybrid entrepreneurs’ transition into entrepreneurship
Carr, J. C., Marshall, D. R., Michaelis, T. L., Pollack, J. M., & Sheats, L. (2022) · Journal of Small Business Management · doi:10.1080/00472778.2022.2082455 · Cited by 33
Guan, S., Wan, D., Zhao, R., Canario, E., Meng, C., & Biswal, B. B. (2022) · Human Brain Mapping · doi:10.1002/hbm.26129 · Cited by 29
de Chastelaine, M., Srokova, S., Hou, M., Kidwai, A., Kafafi, S. S., Racenstein, M. L., & Rugg, M. D. (2023) · Cerebral Cortex · doi:10.1093/cercor/bhac518 · Cited by 35
Multi-institutional PET/CT image segmentation using federated deep transformer learning
Shiri, I., Razeghi, B., Sadr, A. V., Amini, M., Salimi, Y., Ferdowsi, S., Boor, P., Gündüz, D., Voloshynovskiy, S., & Zaidi, H. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107706 · Cited by 35
Contentment and Self-acceptance: Wellbeing Beyond Happiness
Cordaro, D. T., Bai, Y., Bradley, C. M., Zhu, F., Han, R., Keltner, D. J., Gatchpazian, A., & Zhao, Y. (2024) · Journal of Happiness Studies · doi:10.1007/s10902-024-00729-8 · Cited by 45
Bodroža, B., Dinić, B. M., & Bojić, L. (2024) · Royal Society Open Science · doi:10.1098/rsos.240180 · Cited by 61
Pasquali, L., Bruschini, C., Benetello, F., et al. (2025) · Current Biology · doi:10.1016/j.cub.2025.02.048 · Cited by 33
Vachhrajani, S. H., Sen, A. N., Satyan, K. B., Kulkarni, A. V., Birchansky, S. B., & Jea, A. H. (2014) · Journal of Neurosurgery Pediatrics · doi:10.3171/2014.7.peds13591 · Cited by 32
Seel, R. T., Corrigan, J. D., Dijkers, M. P. J. M., Barrett, R. S., Bogner, J. A., Smout, R. J., Garmoe, W. S., & Horn, S. D. (2015) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2014.10.027 · Cited by 33
Using IQ Discrepancy Scores To Examine the Neural Correlates of Specific Cognitive Abilities
Margolis, A. E., Bansal, R., Hao, X., Algermissen, M., Erickson, C., Klahr, K., Naglieri, J. A., & Peterson, B. S. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.0775-13.2013 · Cited by 35
Substantia Nigra Hyperechogenicity as a Marker for Parkinson's Disease: A Population-Based Study
Mahlknecht, P., Seppi, K., Stockner, H., Nocker, M., Scherfler, C., Kiechl, S. J., Willeit, J., Schmidauer, C., Gasperi, A., Rungger, G., et al. (2013) · Neurodegenerative Diseases · doi:10.1159/000348595 · Cited by 33
Levin, L. K., Henderson, H. A., & Ehrenreich, J. T. (2012) · Psychotherapy · doi:10.1037/a0028265 · Cited by 39
Achtenhagen, F., & Winther, E. (2014) · Journal of Vocational Education and Training · doi:10.1080/13636820.2014.916740 · Cited by 33
Llobet, R., Pollán, M., Antón, J., et al. (2014) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2014.01.021 · Cited by 33
Sonographic cervical length measurement in pregnant women with a cervical pessary
Goya, M., Pratcorona, L., Higueras, T., Pérez‐Hoyos, S., Carreras, E., & Cabero, L. (2011) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.8960 · Cited by 32
Can, S. S., Can, A. G., & Günendi, Z. (2011) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2011.02.001 · Cited by 33
Amorim, M. M., Borini, C. B., Lopes, S. L. P. D. C., et al. (2009) · International Journal of Morphology · doi:10.4067/s0717-95022009000400044 · Cited by 33
Moberg-Mogren, E., & Nelson, D. L. (2006) · American Journal of Occupational Therapy · doi:10.5014/ajot.60.2.226 · Cited by 33
Development and Reliability of the Lifetime Interview on Smoking Trajectories
Colby, S. M., Clark, M. A., Rogers, M. L., Ramsey, S. E., Graham, A. L., Boergers, J., Kahler, C. W., Papandonatos, G. D., Buka, S. L., NIAURA, R. S., et al. (2011) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntr212 · Cited by 33
Reliability of Force-Platform Measures of Postural Sway and Expertise-Related Differences
Meshkati, Z., Namazizadeh, M., Salavati, M., et al. (2011) · Journal of Sport Rehabilitation · doi:10.1123/jsr.20.4.442 · Cited by 33
Development of HomeSTEAD’s physical activity and screen time physical environment inventory
Hales, D. P., Vaughn, A., Mazzucca, S., Bryant, M., Tabak, R. G., McWilliams, C., Stevens, J., & Ward, D. S. (2013) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/1479-5868-10-132 · Cited by 33
Pineau, V., Lebel, B., Gouzy, S., Dutheil, J.-J., & Vielpeau, C. (2010) · Orthopaedics & Traumatology Surgery & Research · doi:10.1016/j.otsr.2010.04.007 · Cited by 33
van Rijswijk, L., Rooks, G., & Haans, A. (2015) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2015.11.006 · Cited by 35
Bailliez, A., Lairez, O., Merlin, C., et al. (2016) · Journal of Nuclear Medicine · doi:10.2967/jnumed.115.168575 · Cited by 33
EATI Island – A virtual-reality-based elder abuse and neglect educational intervention
Pickering, C. E. Z., Ridenour, K., Salaysay, Z., Reyes‐Gastelum, D., & Pierce, S. J. (2016) · Gerontology & Geriatrics Education · doi:10.1080/02701960.2016.1203310 · Cited by 33
Decreasing pelvic incidence is associated with greater risk of cam morphology
Morris, W. Z., Fowers, C. A., Yuh, R. T., et al. (2016) · Bone and Joint Research · doi:10.1302/2046-3758.59.bjr-2016-0028.r1 · Cited by 33
Duijnisveld, B. J., Henseler, J. F., Reijnierse, M., et al. (2016) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2016.10.020 · Cited by 33
Cruitt, P. J., Boudreaux, M. J., King, H. R., Oltmanns, J. R., & Oltmanns, T. F. (2018) · Personality Disorders Theory Research and Treatment · doi:10.1037/per0000321 · Cited by 33
Hussain, N., Hussain, F. N., Sermer, C., et al. (2017) · Canadian Journal of Surgery · doi:10.1503/cjs.000616 · Cited by 33
Sharma, A. M., Rieth, G. E., Tanenbaum, J. E., et al. (2017) · Journal of neurosurgery · doi:10.3171/2017.2.jns162114 · Cited by 33
Shortening of an existing generic online health‐related quality of life instrument for dogs
Reid, J., Wiseman‐Orr, L., & Scott, M. (2017) · Journal of Small Animal Practice · doi:10.1111/jsap.12772 · Cited by 34
Salavati, M., Akhbari, B., Ebrahimi, E., et al. (2017) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2017.06.003 · Cited by 33
Gisselman, A. S., D'Amico, M., & Smoliga, J. M. (2017) · The Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000002258 · Cited by 33
Ferri‐Morales, A., Ferreira, M. V. N., Vlachopoulos, D., Ubago‐Guisado, E., Torres-Costoso, A. I., de Moraes, A. C. F., BARKER, A. R., Moreno, L. A., Martínez‐Vizcaíno, V., & Gracia‐Marco, L. (2018) · Pediatric Exercise Science · doi:10.1123/pes.2017-0171 · Cited by 33
Suss, J., & Ward, P. (2018) · Cognition Technology & Work · doi:10.1007/s10111-018-0493-z · Cited by 34
Test-retest variability of resting-state networks in healthy aging and prodromal Alzheimer's disease
Conwell, K., von Reutern, B., Richter, N., Kukolja, J., Fink, G. R., & Onur, O. A. (2018) · NeuroImage Clinical · doi:10.1016/j.nicl.2018.06.016 · Cited by 32
French validation of the modified-falls efficacy scale (M-FES Fr)
Perrot, A., Castanier, C., Maillot, P., et al. (2018) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2018.07.001 · Cited by 33
Czakó, C. N., Sándor, G. L., Ecsedy, M., Récsán, Z., Horváth, H., Szepessy, Z., Nagy, Z. Z., & Kovács, I. (2018) · Scientific Reports · doi:10.1038/s41598-018-28994-7 · Cited by 33
Asymptotic variability of (multilevel) multirater kappa coefficients
Vanbelle, S. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218794733 · Cited by 33
Nagpaul, T., & Chen, J. (2019) · Children and Youth Services Review · doi:10.1016/j.childyouth.2019.02.015 · Cited by 39
Yokoyama, S., Ishigami, M., Honda, T., et al. (2019) · Hepatology Research · doi:10.1111/hepr.13381 · Cited by 33
Vrieze, T. D., Gebruers, N., Nevelsteen, I., et al. (2019) · Lymphatic Research and Biology · doi:10.1089/lrb.2019.0013 · Cited by 33
Joint Representation and Estimator Learning for Facial Action Unit Intensity Estimation
Zhang, Y., Wu, B., Dong, W., Li, Z., Liu, W., Hu, B.-G., & Ji, Q. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00357 · Cited by 33
Authenticity and inauthenticity in narrative identity
Wilt, J. A., Thomas, S., & McAdams, D. P. (2019) · Heliyon · doi:10.1016/j.heliyon.2019.e02178 · Cited by 33
Lim, H. H. R., Tang, Z. Y., Hashim, M. A. B. M., Yang, M., Koh, Y. L. E., & Koh, K. H. (2019) · Spine · doi:10.1097/brs.0000000000003325 · Cited by 33
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
