The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,342
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 9,701–9,800 of 23,881
Understanding the implications of social translucence for systems supporting communication at work
Szostek, A. M., Karapanos, E., Eggen, B. H., & Holenderski, M. (2008) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1460563.1460664 · Cited by 24
Test–retest reliability of the energy expenditure index in adolescents with cerebral palsy
Pritchard, L., & Darrah, J. (1999) · Developmental Medicine & Child Neurology · doi:10.1017/s0012162299001450 · Cited by 24
Moore, A. A., Karno, M. P., Ray, L. A., Ramirez, K. D., Barenstein, V., Portillo, M. J., Rizo, P., Borok, J. F., Liao, D. H., Barron, J., et al. (2015) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2015.11.003 · Cited by 25
Aging and spectral characteristics of the nonharmonic component of 24-h heart rate variability
Sakata, S., Hayano, J., Mukai, S., Okada, A., & Fujinami, T. (1999) · American Journal of Physiology-Regulatory, Integrative and Comparative Physiology · doi:10.1152/ajpregu.1999.276.6.r1724 · Cited by 25
Revanna, G. K., Kunjunju, A., & Sehgal, A. (2016) · Progress in Pediatric Cardiology · doi:10.1016/j.ppedcard.2016.12.003 · Cited by 24
Validity and reliability of smartphone orientation measurement to quantify dynamic balance function
Kuznetsov, N. A., Robins, R. K., Long, B. L., Jakiela, J. T., Haran, F. J., Ross, S. E., Wright, W. G., & Rhea, C. K. (2017) · Physiological Measurement · doi:10.1088/1361-6579/aaa3c2 · Cited by 30
Boerner, S., & Renz, S. (2008) · Cited by 24
Henggeler, S. W., Borduin, C. M., & Mann, B. J. (1987) · Journal of Applied Developmental Psychology · doi:10.1016/0193-3973(87)90023-2 · Cited by 24
Evaluation of students' clinical competence.
Dienst, E. R., & Armstrong, P. M. (1988) · Professional Psychology Research and Practice · doi:10.1037//0735-7028.19.3.339 · Cited by 24
Oakley, S. P., Portek, I., Szomor, Z., Turnbull, A., Murrell, G. A. C., Kirkham, B. W., & Lassere, M. N. (2002) · Annals of the Rheumatic Diseases · doi:10.1136/ard.61.6.540 · Cited by 23
Colman, R. J., Hudson, J. C., Barden, H. S., & Kemnitz, J. W. (1999) · Obesity Research · doi:10.1002/j.1550-8528.1999.tb00395.x · Cited by 24
Effects of a Short-term Dynamic Balance Training Program in Healthy Older Women
Bellew, J. W., Fenter, P. C., Chelette, B., et al. (2005) · Journal of Geriatric Physical Therapy · doi:10.1519/00139143-200504000-00001 · Cited by 24
Reliability Generalization of the Psychopathy Checklist Applied in Youthful Samples
Campbell, J. S., Pulos, S. M., Hogan, M., et al. (2005) · Educational and Psychological Measurement · doi:10.1177/0013164405275666 · Cited by 24
Chor, D. (1998) · Arquivos Brasileiros de Cardiologia · doi:10.1590/s0066-782x1998001100003 · Cited by 24
Wiser, S. L., & Goldfried, M. R. (1996) · Psychotherapy Research · doi:10.1080/10503309612331331828 · Cited by 24
Zakaria, D., Kramer, J. F., & Harburn, K. L. (1996) · Journal of Electromyography and Kinesiology · doi:10.1016/1050-6411(95)00025-9 · Cited by 24
Close encounters of the architectural kind
Heylighen, A., & Verstijnen, I. M. (2003) · Design Studies · doi:10.1016/s0142-694x(02)00040-6 · Cited by 24
Treatment of major depression in the context of panic disorder
Woody, S. R., McLean, P. D., Taylor, S. J., & Koch, W. J. (1999) · Journal of Affective Disorders · doi:10.1016/s0165-0327(98)00117-7 · Cited by 24
Pélissolo, A., Veysseyre, O., & Lépine, J. (1997) · Psychiatry Research · doi:10.1016/s0165-1781(97)00099-1 · Cited by 24
Localio, A. R., Margolis, D. J., Kagan, S. H., et al. (2006) · Wound Repair and Regeneration · doi:10.1111/j.1743-6109.2006.00146.x · Cited by 24
Muran, J. C., & Segal, Z. V. (1992) · Psychotherapy · doi:10.1037/0033-3204.29.4.524 · Cited by 24
Digital image analysis of cigarette filter staining to estimate smoke exposure
O’Connor, R. J., Kozlowski, L. T., Hammond, D. G., et al. (2007) · Nicotine & Tobacco Research · doi:10.1080/14622200701485026 · Cited by 24
Malt, U. F., Nystad, R., Bache, T., Noren, O., Sjaastad, M. C., Solberg, K. O., T⊘nseth, S., Zachariassen, P., & Mæhlum, E. (1995) · The British Journal of Psychiatry · doi:10.1192/bjp.166.3.374 · Cited by 24
Psychometrics of Neurological Soft Signs
Pine, D. S., SCOTT, M. R., BUSNER, C., et al. (1996) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199604000-00017 · Cited by 24
Patterns of dysmorphic features in schizophrenia
Scutt, L. E., Chow, E. W. C., Weksberg, R., et al. (2001) · American Journal of Medical Genetics · doi:10.1002/ajmg.1612 · Cited by 24
Disturbances of Affect Expression in Failure-to-Thrive
Polan, H. J., Leon, A. C., KAPLAN, M. D., et al. (1991) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199111000-00006 · Cited by 24
Sheikh, N. J. (2000) · doi:10.15760/etd.978 · Cited by 24
An fMRI Dataset for Concept Representation with Semantic Feature Annotations
Wang, S., Zhang, Y., Zhang, X., Sun, J., Lin, N., Zhang, J., & Zong, C. (2022) · Scientific Data · doi:10.1038/s41597-022-01840-2 · Cited by 24
Hirsch, C., Hirsch, M., John, M. T., & Bock, J. J. (2007) · Journal of Orofacial Orthopedics / Fortschritte der Kieferorthopädie · doi:10.1007/s00056-007-0708-z · Cited by 23
Inter-rater Reliability of a Clinical Test of Standing Function
Triolo, R. J., Bevelheimer, T., Eisenhower, G., & Wormser, D. (1995) · The Journal of Spinal Cord Medicine · doi:10.1080/10790268.1995.11719375 · Cited by 23
Dunleavy, K., Neil, J., Tallon, A., & Adamo, D. E. (2015) · Journal of Manual & Manipulative Therapy · doi:10.1179/2042618614y.0000000070 · Cited by 23
Li, W., & Zhang, F. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.612088 · Cited by 26
Why Task? Task as a Unit of Analysis for Language Education
Sasayama, S. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108868327.004 · Cited by 24
Cho, S., de Boeck, P. A. L., Naveiras, M., & Ervin, H. (2022) · Behavior Research Methods · doi:10.3758/s13428-021-01709-z · Cited by 23
Roth, L., Jordan, V., Schwarz, S., Willmes, K., Nuerk, H., van Dijck, J., & Cipora, K. (2024) · Cognition · doi:10.1016/j.cognition.2024.105781 · Cited by 31
Connectome-based fingerprinting: reproducibility, precision, and behavioral prediction
Ramduny, J., & Kelly, C. A. M. (2024) · Neuropsychopharmacology · doi:10.1038/s41386-024-01962-8 · Cited by 24
Lü, Y., Boudiffa, M., Dall’Ara, E., et al. (2015) · Medical Engineering & Physics · doi:10.1016/j.medengphy.2015.08.017 · Cited by 23
Reliability of plastic cups to measure breast volume
Hansson, E., Manjer, J., & Ringberg, A. (2013) · Journal of Plastic Surgery and Hand Surgery · doi:10.3109/2000656x.2013.870908 · Cited by 23
Azevedo‐Santos, I. F., Alves, I. G. N., Badauê‐Passos, D., Santana‐Filho, V. J., & DeSantana, J. M. (2015) · Pain Practice · doi:10.1111/papr.12287 · Cited by 23
Carasso, S., Mutlak, D., Lessick, J., et al. (2014) · Journal of the American Society of Echocardiography · doi:10.1016/j.echo.2014.09.006 · Cited by 23
The Use of Random Coefficient Modeling for Understanding and Predicting Job Performance Ratings
O’Neill, T., Goffin, R. D., & Gellatly, I. R. (2012) · Organizational Research Methods · doi:10.1177/1094428112438699 · Cited by 27
Peer assessment based on ratings in a social media course
Vozniuk, A., Holzer, A., & Gillet, D. (2014) · International Conference on Learning Analytics and Knowledge · doi:10.1145/2567574.2567608 · Cited by 27
Petralia, G., Summers, P. E., Viotti, S., Montefrancesco, R., Raimondi, S., & Bellomi, M. (2012) · Radiology · doi:10.1148/radiol.12111232 · Cited by 23
Cracking the Code: Using Personal Narratives in Research
Haden, C., & Hoffman, P. C. (2013) · Journal of Cognition and Development · doi:10.1080/15248372.2013.805135 · Cited by 20
Münzer, S., & Zadeh, M. V. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.06.047 · Cited by 23
Menéndez, L. P. (2016) · Journal of Forensic Sciences · doi:10.1111/1556-4029.13301 · Cited by 23
The relationship between lateral displacement of the mandible and scoliosis
Nakashima, A., Nakano, H., Yamada, T., et al. (2016) · Oral and Maxillofacial Surgery · doi:10.1007/s10006-016-0607-9 · Cited by 23
Jenkins, N. D. M., & Cramer, J. T. (2017) · Journal of the American Geriatrics Society · doi:10.1111/jgs.14769 · Cited by 23
Pöhlmann, S. T. L., Harkness, E. F., Taylor, C., et al. (2017) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2017.04.005 · Cited by 23
Incremental Validity of Self- and Informant Report of Personality Disorders in Later Life
Cruitt, P. J., & Oltmanns, T. F. (2017) · Assessment · doi:10.1177/1073191117706020 · Cited by 26
Colino, E., Sánchez‐Sánchez, J., García‐Unanue, J., et al. (2017) · Polymer Testing · doi:10.1016/j.polymertesting.2017.06.011 · Cited by 23
How do cynical employees serve their customers? A multi-method study
Demerouti, E., Xanthopoulou, D., & Bakker, A. B. (2017) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2017.1358165 · Cited by 23
Bussey, M. D., Aldabe, D., Adhia, D. B., et al. (2017) · Musculoskeletal Science and Practice · doi:10.1016/j.msksp.2017.09.004 · Cited by 23
Chain of events analysis for a scuba diving fatality
Divers Alert Network (DAN) Asia-Pacific, A., Lippmann, J. M., Stevenson, C., Taylor, D. M., Williams, J. W., & Mohebbi, M. (2017) · Diving and Hyperbaric Medicine Journal · doi:10.28920/dhm47.3.144-154 · Cited by 23
Standing, R. J., & Maulder, P. S. (2017) · PubMed Central · Cited by 23
Gender Gaps in Letter-Sound Knowledge Persist Across the First School Year
Sigmundsson, H., Eriksen, A. D., Ofteland, G. S., & Haga, M. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00301 · Cited by 27
Hoşbay, Z., Özkan, S., Tanrıverdi, M., et al. (2018) · Journal of Hand Therapy · doi:10.1016/j.jht.2017.10.006 · Cited by 23
Validation of French upper limb Erasmus modified Nottingham Sensory Assessment in stroke
Villepinte, C., Catella, E., Martin, M., et al. (2018) · Annals of Physical and Rehabilitation Medicine · doi:10.1016/j.rehab.2018.03.004 · Cited by 23
Burge, R., & Chaparro, A. (2018) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2018.05.033 · Cited by 23
Konradsen, H., Dieperink, K. B., Lauridsen, J. T., Sorknæs, A. D., & Østergaard, B. (2018) · Scandinavian Journal of Caring Sciences · doi:10.1111/scs.12591 · Cited by 23
Fauth, B., Decristan, J., Rieser, S., Klieme, E., & Büttner, G. (2018) · Social Psychology of Education · doi:10.1007/s11218-018-9462-x · Cited by 27
A pilot study of neuromuscular ultrasound as a biomarker for amyotrophic lateral sclerosis
Pathak, S., Caress, J. B., Wosiski‐Kuhn, M., Milligan, C., Williams, D., & Cartwright, M. S. (2018) · Muscle & Nerve · doi:10.1002/mus.26360 · Cited by 23
Tibubos, A. N., Köber, C., Habermas, T., & Rohrmann, S. (2019) · Journal of Research in Personality · doi:10.1016/j.jrp.2019.01.003 · Cited by 30
Physiologic widening of the medial clear space: What's normal?
Gibson, P. D., Ippolito, J. A., Hwang, J. S., et al. (2019) · Journal of Clinical Orthopaedics and Trauma · doi:10.1016/j.jcot.2019.04.016 · Cited by 23
Sahakyan, K., Li, X., Lodge, M. A., Werner, R. A., Bundschuh, R. A., Bundschuh, L., Kulkarni, H., Schuchardt, C., Baum, R. P., Pienta, K. J., et al. (2019) · Molecular Imaging and Biology · doi:10.1007/s11307-019-01376-9 · Cited by 23
Lizaur-Utrilla, A., Miralles-Muñoz, F. A., González-Parreño, S., & Lopez‐Prats, F. A. (2019) · Journal of Orthopaedic Research® · doi:10.1002/jor.24386 · Cited by 24
Dappled light disrupts prey detection by masking movement
Matchette, S. R., Cuthill, I. C., & Scott‐Samuel, N. E. (2019) · Animal Behaviour · doi:10.1016/j.anbehav.2019.07.006 · Cited by 23
Psycholinguistic norms for more than 300 lexical signs in German Sign Language (DGS)
Trettenbrein, P. C., Pendzich, N.-K., Cramer, J.-M., Steinbach, M., & Zaccarella, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-020-01524-y · Cited by 27
Reliability of Physiological Responses Induced by Basic Emotions: A Pilot Study
Jang, E.-H., Byun, S., Park, M., & Sohn, J. (2019) · Journal of PHYSIOLOGICAL ANTHROPOLOGY · doi:10.1186/s40101-019-0209-y · Cited by 24
do Nascimento, J. D. S., Alburquerque‐Sendín, F., Vigolvino, L. P., de Oliveira, W. F., & Sousa, C. D. O. (2020) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2019.04.002 · Cited by 23
Goals as identities: Boosting perceptions of healthy-eater identity for easier goal pursuit
Dominick, J. K., & Cole, S. (2020) · Motivation and Emotion · doi:10.1007/s11031-020-09824-8 · Cited by 25
Kittelson, A. J., Christensen, J. C., Loyd, B. J., Burrows, K. L., Iannitto, J., & Stevens‐Lapsley, J. E. (2020) · Disability and Rehabilitation · doi:10.1080/09638288.2020.1730454 · Cited by 25
Stability of mismatch negativity event‐related potentials in a multisite study
Roach, B. J., Hamilton, H., Bachman, P., Belger, A., Carrión, R. E., Duncan, E., Johannesen, J., Kenney, J. G., Light, G. A., Niznikiewicz, M. A., et al. (2020) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1819 · Cited by 23
A new method measurement for finger range of motion using a smartphone
Miyake, K., Mori, H., Matsuma, S., Kimura, C., Izumoto, M., Nakaoka, H., & Sayama, K. (2020) · Journal of Plastic Surgery and Hand Surgery · doi:10.1080/2000656x.2020.1755296 · Cited by 23
A comparison of automated and manual co-registration for magnetoencephalography
Houck, J. M., & Claus, E. D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0232100 · Cited by 24
Kuusela, P., Wennerholm, U., Fadl, H., Wesström, J., Lindgren, P., Hagberg, H., Jacobsson, B., & Valentin, L. (2020) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.13895 · Cited by 25
Jacobs, K., Rigdon, J., Chan, F. P., Cheng, J. Y., Alley, M. T., Vasanawala, S. S., & Maskatia, S. A. (2020) · Journal of Cardiovascular Magnetic Resonance · doi:10.1186/s12968-020-00612-4 · Cited by 22
Cox, R. M., Sholder, D. S., Stoll, L. E., et al. (2020) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2020.04.014 · Cited by 23
Attitudes, beliefs and knowledge of mathematics teachers regarding peer tutoring
Miravet, L. M., & Alegre, F. (2020) · European Journal of Teacher Education · doi:10.1080/02619768.2020.1803271 · Cited by 23
Test–retest reliability of the Friedreich’s ataxia rating scale
Rummey, C., Zesiewicz, T. A., Perez‐Lloret, S., Farmer, J. M., Pandolfo, M., & Lynch, D. R. (2020) · Annals of Clinical and Translational Neurology · doi:10.1002/acn3.51118 · Cited by 24
Bağcıer, F., Külcü, D. G., Yorulmaz, E., & Altunok, E. Ç. (2020) · Archives of Rheumatology · doi:10.46497/archrheumatol.2020.7515 · Cited by 23
Csécs, I., Pashakhanloo, F., Paskavitz, A., Jang, J., Al-Otaibi, T. K., Neisius, U., Manning, W. J., & Nezafat, R. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.120.016797 · Cited by 23
Wagner, D. R., & Teramoto, M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0244019 · Cited by 23
Self-Debriefing After Virtual Simulation: Measuring Depth of Reflection
MacKenna, V. A., Díaz, D. A., Chase, S. K., et al. (2021) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2021.01.002 · Cited by 23
McDermott, T. J., Kirlić, N., Akeman, E., Touthang, J., Clausen, A. N., Kuplicki, R., & Aupperle, R. L. (2021) · Human Brain Mapping · doi:10.1002/hbm.25371 · Cited by 24
Multimorbidity, ageing and mortality: normative data and cohort study in an American population
Rocca, W. A., Grossardt, B. R., Boyd, C. M., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-042633 · Cited by 23
Chen, S.-H., Miles, K. A., Taylor, S. A., Ganeshan, B., Rodriguez‐Justo, M., Fraioli, F., Wan, S., Afaq, A., Shortman, R. I., Walls, D., et al. (2021) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-021-05318-y · Cited by 23
Human resource management practices and organizational injury rates
Turner, N., Barling, J. I., Dawson, J., Deng, C., Parker, S. K., Patterson, M. G., & Stride, C. (2021) · Journal of Safety Research · doi:10.1016/j.jsr.2021.06.003 · Cited by 31
Wigsten, E., Hajj, A. A., Jonasson, P., EndoReCo, & Kvist, T. (2021) · International Endodontic Journal · doi:10.1111/iej.13548 · Cited by 23
Buimer, E. E. L., Schnack, H. G., Caspi, Y., van Haren, N. E. M., Milchenko, M. V., Pas, P., Initiative, A. D. N., Pol, H. E. H., & Brouwer, R. M. (2021) · Human Brain Mapping · doi:10.1002/hbm.25459 · Cited by 24
Healthy decisions in the cued-attribute food choice paradigm have high test-retest reliability
Barakchian, Z., Beharelle, A. R., & Hare, T. A. (2021) · Scientific Reports · doi:10.1038/s41598-021-91933-6 · Cited by 23
Kommers, I. O., Bouget, D., Pedersen, A., Eijgelaar, R. S., Ardon, H., Barkhof, F., Bello, L., Berger, M. S., Nibali, M. C., Furtner, J., et al. (2021) · Cancers · doi:10.3390/cancers13122854 · Cited by 23
Reliability on multiband diffusion NODDI models: A test retest study on children and adults
Lucignani, M., Breschi, L., Rossi‐Espagnet, M. C., Longo, D., Talamanca, L. F., Placidi, E., & Napolitano, A. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118234 · Cited by 23
Liu, C., Bian, Y., Meng, Y., Liu, F., Cao, K., Zhang, H., Fang, X., Li, J., Yu, J., Feng, X., et al. (2021) · Academic Radiology · doi:10.1016/j.acra.2021.05.017 · Cited by 24
The instability of functional connectomes across the first year of life
Dufford, A. J., Noble, S., Gao, S., & Scheinost, D. (2021) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2021.101007 · Cited by 24
Integrating Dietary Data into Microbiome Studies: A Step Forward for Nutri-Metaomics
Yáñez, F., Soler, Z., Oliero, M., Xie, Z., Oyarzun, I., Serrano-Gómez, G., & Manichanh, C. (2021) · Nutrients · doi:10.3390/nu13092978 · Cited by 22
Shaw, J. W., Bennett, B. L., Trigg, A., DeRosa, M., Taylor, F. C., Kiff, C., Ntais, D., Noon, K., King, M. T., & Cocks, K. (2021) · Value in Health · doi:10.1016/j.jval.2021.05.022 · Cited by 23
Rören, A., Mazarguil, A., Vaquero-Ramos, D., et al. (2022) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2021.782740 · Cited by 23
Veterans Crisis Line Call Outcomes: Distress, Suicidal Ideation, and Suicidal Urgency.
Britton, P. C., Karras, E., Stecker, T., Klein, J., Crasta, D., Brenner, L., & Pigeon, W. (2022) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2021.11.013 · Cited by 20
Pourmotabbed, H., Curry, A. L. D. J., Clarke, D., Tyler‐Kabara, E. C., & Abbas Babajani-feremi, A. (2022) · Human Brain Mapping · doi:10.1002/hbm.25726 · Cited by 23
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review