The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,342
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 7,601–7,700 of 23,881
McCarthy, K. S., Keefe, J. R., & Barber, J. P. (2014) · Psychotherapy Research · doi:10.1080/10503307.2014.973921 · Cited by 34
Hetsroni, I., Nyska, M., Ben‐Sira, D., et al. (2010) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0b013e3181dbe5f7 · Cited by 34
MR findings associated with positive distraction of the hip joint achieved by axial traction
Suter, A., Dietrich, T. J., Maier, M., et al. (2015) · Skeletal Radiology · doi:10.1007/s00256-015-2099-3 · Cited by 34
Kim, S. H., Chung, Y. S., Ropper, A. E., et al. (2015) · European Spine Journal · doi:10.1007/s00586-015-3849-6 · Cited by 34
Listener effort for highly intelligible tracheoesophageal speech
Nagle, K. F., & Eadie, T. L. (2012) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2012.01.001 · Cited by 34
Reproducibility of sodium MRI measures of articular cartilage of the knee in osteoarthritis
Newbould, R. D., Miller, S. R., Tielbeek, J. A. W., Toms, L., Rao, A. W., Gold, G. E., Strachan, R. K., Taylor, P. C., Matthews, P. M., & Brown, A. P. (2011) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2011.10.007 · Cited by 34
Wang, Q., & Yuan, Y. (2013) · Neurocomputing · doi:10.1016/j.neucom.2013.10.007 · Cited by 34
González‐Domínguez, E., Martins, R. B., Ponte, E. M. D., et al. (2014) · European Journal of Plant Pathology · doi:10.1007/s10658-014-0400-2 · Cited by 34
Beckstead, J. W., Pezzo, M. V., Beckie, T. M., Shahraki, F., Kentner, A. C., & Grace, S. L. (2013) · Medical Decision Making · doi:10.1177/0272989x13492017 · Cited by 34
Analyses of Wine-Tasting Data: A Tutorial
Olkin, I., Lou, Y., Stokes, L., et al. (2014) · Journal of Wine Economics · doi:10.1017/jwe.2014.26 · Cited by 34
Reliability analysis for continuous measurements: Equivalence test for agreement
Yi, Q., Wang, P., & He, Y. (2007) · Statistics in Medicine · doi:10.1002/sim.3110 · Cited by 35
Lelliott, P. T., Williams, R. H., Mears, A., et al. (2008) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.107.041681 · Cited by 34
SNODGRASS, W. T., Ziada, A. M., Yücel, S., et al. (2008) · Journal of Pediatric Urology · doi:10.1016/j.jpurol.2007.12.007 · Cited by 34
Toll, B. A., Martino, S., Latimer‐Cheung, A. E., Salovey, P., O’Malley, S. S., Carlin-Menter, S. M., Hopkins, J., Wu, R., Celestino, P. B., & Cummings, K. M. (2010) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djp468 · Cited by 35
Effects of trunk rotation on scapular kinematics and muscle activity during humeral elevation
Nagai, K., Tateuchi, H., Takashima, S., et al. (2013) · Journal of Electromyography and Kinesiology · doi:10.1016/j.jelekin.2013.01.012 · Cited by 34
Rolland, Y., Vérin, M., Payan, C., et al. (2011) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp.2010.214890 · Cited by 34
Tober, G., Clyne, W., Finnegan, O., Farrin, A. J., & Ian Russell Team (2008) · Alcohol and Alcoholism · doi:10.1093/alcalc/agn064 · Cited by 34
Modelling Events through Memory-based, Open-IE Patterns for Abstractive Summarization
Pighin, D., Cornolti, M., Alfonseca, E., et al. (2014) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/v1/p14-1084 · Cited by 34
Anhedonic symptoms of depression are linked to reduced motivation to obtain a reward
Franzén, J., & Brinkmann, K. (2015) · Motivation and Emotion · doi:10.1007/s11031-015-9529-3 · Cited by 34
Extracting Human Temporal Orientation from Facebook Language
Schwartz, H. A., Park, G., Sap, M., Weingarten, E., Eichstaedt, J., Kern, M., Stillwell, D., Kosinski, M., Berger, J., Seligman, M., et al. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1044 · Cited by 13
Mikami, A. Y., Szwedo, D. E., Ahmad, S. I., Samuels, A. S., & Hinshaw, S. P. (2015) · Journal of Abnormal Psychology · doi:10.1037/abn0000053 · Cited by 38
Knobloch-Fedders, L. M., Critchfield, K. L., Boisson, T., Woods, N., Bitman, R. L., & Durbin, C. E. (2014) · Journal of Counseling Psychology · doi:10.1037/a0035241 · Cited by 39
de la Fuente, J. S., Díaz, A. S., Orbis, I. C., Ramila, E., Izquierdo-García, E., Alba, C. E., & Rodríguez, I. E. (2016) · Annals of Pharmacotherapy · doi:10.1177/1060028016656385 · Cited by 34
Santani, D., Biel, J.-I., Labhart, F., Truong, J., Landolt, S., Kuntsche, E. N., & Gática-Pérez, D. (2016) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2971648.2971713 · Cited by 34
Inter-rater reliability of a reflective rubric to assess pharmacy students’ reflective thinking
Lucas, C., Bosnic-Anticevich, S. Z., Schneider, C. R., Bartimote-Aufflick, K., McEntee, M. F., & Smith, L. D. (2017) · Currents in Pharmacy Teaching and Learning · doi:10.1016/j.cptl.2017.07.025 · Cited by 34
Devreese, K. M. J., Poncet, A., Lindhoff‐Last, E., Musiał, J., de Moerloose, P., & Fontana, P. (2016) · Journal of Thrombosis and Haemostasis · doi:10.1111/jth.13560 · Cited by 35
Edmunds, S. R., Rozga, A., Li, Y., Karp, E. A., Ibañez, L. V., Rehg, J. M., & Stone, W. L. (2017) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-016-3002-3 · Cited by 34
Woolley, J. D., Chuang, B., Fussell, C., Scherer, S. J., Biagianti, B., Fulford, D., Mathalon, D. H., & Vinogradov, S. (2017) · Psychological Medicine · doi:10.1017/s0033291716003433 · Cited by 36
Lee, J., Min, J., & Lee, H. (2017) · Journal of Knowledge Management · doi:10.1108/jkm-04-2016-0163 · Cited by 40
Assessment of metabolic phenotypic variability in children’s urine using 1H NMR spectroscopy
Maître, L., Lau, C. E., Vizcaíno, E., Robinson, O., Casas, M., Siskos, A., Want, E. J., Athersuch, T. J., Slama, R., Vrijheid, M., et al. (2017) · Scientific Reports · doi:10.1038/srep46082 · Cited by 35
Potential Predictors of Injury among Pre-Professional Ballet and Contemporary Dancers
Yau, R. K., Golightly, Y. M., Richardson, D., Runfola, C. D., Waller, A. E., & Marshall, S. W. (2017) · Journal of Dance Medicine & Science · doi:10.12678/1089-313x.21.2.53 · Cited by 34
Pelvic retroversion: a compensatory mechanism for lumbar stenosis
Pourtaheri, S., Sharma, A. M., Savage, J. W., Kalfas, I. H., Mroz, T. E., Benzel, E., & Steinmetz, M. P. (2017) · Journal of Neurosurgery Spine · doi:10.3171/2017.2.spine16963 · Cited by 35
Nickless, A., Rayner, P. J., Engelbrecht, F. A., et al. (2018) · Atmospheric chemistry and physics · doi:10.5194/acp-18-4765-2018 · Cited by 34
Kozák, L. R., van Graan, L. A., Chaudhary, U. J., Szabó, Á., & Lemieux, L. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.09.014 · Cited by 33
Thresholding functional connectomes by means of mixture modeling
Bielczyk, N., Walocha, F., Ebel, P., Haak, K. V., Llera, A., Buitelaar, J. K., Glennon, J. C., & Beckmann, C. F. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.01.003 · Cited by 36
Togao, O., Hiwatashi, A., Yamashita, K., et al. (2018) · British Journal of Radiology · doi:10.1259/bjr.20170912 · Cited by 34
Trombetta, B. A., Carlyle, B. C., Koenig, A. M., Shaw, L. M. J., Trojanowski, J. Q., Wolk, D. A., Locascio, J. J., & Arnold, S. E. (2018) · PLoS ONE · doi:10.1371/journal.pone.0193707 · Cited by 34
Family communication with teens at clinical high-risk for psychosis or bipolar disorder.
Salinger, J. M., O’Brien, M. P., Miklowitz, D. J., Marvin, S. E., & Cannon, T. D. (2018) · Journal of Family Psychology · doi:10.1037/fam0000393 · Cited by 35
Towards a reliable repeated-measures beads task for assessing the jumping to conclusions bias
McLean, B. F., Mattiske, J. K., & Balzan, R. P. (2018) · Psychiatry Research · doi:10.1016/j.psychres.2018.04.043 · Cited by 35
Pamukoff, D. N., Montgomery, M. M., Holmes, S. C., et al. (2018) · Gait & Posture · doi:10.1016/j.gaitpost.2018.07.174 · Cited by 34
Measuring Team Effectiveness in the Health Care Setting: An Inventory of Survey Tools
Kash, B. A., Cheon, O., Halzack, N. M., & Miller, T. R. (2018) · Health Services Insights · doi:10.1177/1178632918796230 · Cited by 36
Lennartz, S., Laukamp, K. R., Neuhaus, V., et al. (2018) · European Journal of Radiology · doi:10.1016/j.ejrad.2018.09.010 · Cited by 34
Kanko, L., Birmingham, T. B., Bryant, D., et al. (2018) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2018.11.012 · Cited by 34
Boukadi, M., Marcotte, K., Bedetti, C., Houde, J., Désautels, A., Deslauriers‐Gauthier, S., Chapleau, M., Boré, A., Descoteaux, M., & Brambati, S. M. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.01055 · Cited by 34
One-year pilot study on the effects of nitisinone on melanin in patients with OCA-1B
Adams, D. R., Menezes, S. M., Jauregui, R., Valivullah, Z. M., Power, B., Abraham, M., Jeffrey, B. G., Garced, A., Alur, R. P., Cunningham, D., et al. (2019) · JCI Insight · doi:10.1172/jci.insight.124387 · Cited by 34
Lee, S. J., Jeong, J. H., Lee, C., et al. (2019) · Arthritis & Rheumatology · doi:10.1002/art.40860 · Cited by 34
Rezk‐Hanna, M., Mosenifar, Z., Benowitz, N. L., et al. (2019) · Circulation · doi:10.1161/circulationaha.118.037375 · Cited by 34
Automatic analysis of pre‐ejection period during sleep using impedance cardiogram
Forouzanfar, M., Baker, F. C., Colrain, I. M., Goldstone, A., & de Zambotti, M. (2019) · Psychophysiology · doi:10.1111/psyp.13355 · Cited by 34
Measuring Saccade Latency Using Smartphone Cameras
Lai, H.-Y., Saavedra-Peña, G., Sodini, C. G., Sze, V., & Heldt, T. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2913846 · Cited by 34
Ketharanathan, V., Torgersen, G. R., Petrovski, B. É., & Preus, H. R. (2019) · BMC Oral Health · doi:10.1186/s12903-019-0769-6 · Cited by 35
Validity and reliability of the EQ-5D-5 L in family caregivers of leukemia patients
Li, L., Liu, C., Cai, X., Yu, H., Zeng, X., Sui, M., Zheng, E., Li, Y., Xu, J., Zhou, J., et al. (2019) · BMC Cancer · doi:10.1186/s12885-019-5721-2 · Cited by 34
Hope, Religiosity, and Mental Health in U.S. Veterans and Active Duty Military with PTSD Symptoms
Koenig, H. G., Youssef, N. A., Smothers, Z., Oliver, J. P., Boucher, N. A., Ames, D., Volk, F. A., Teng, E. J., & Haynes, K. (2019) · Military Medicine · doi:10.1093/milmed/usz146 · Cited by 44
Emotion regulation strategy use and eating disorder symptoms in daily life
Mikhail, M. E., & Kring, A. M. (2019) · Eating Behaviors · doi:10.1016/j.eatbeh.2019.101315 · Cited by 39
Leafblad, N. D., Larson, D. R., Fleisig, G. S., et al. (2019) · Sports Health A Multidisciplinary Approach · doi:10.1177/1941738119869945 · Cited by 34
Kampa, M., Schick, A., Sebastian, A., Wessa, M., Tüscher, O., Kalisch, R., & Yuen, K. S. L. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116223 · Cited by 31
Słomka, K. J., Sobota, G., Skowronek, T., et al. (2017) · PubMed · doi:10.5277/abb-00572-2016-05 · Cited by 34
The role of the inferior parietal lobule in writer’s cramp
Merchant, S. H. I., Frangos, E., Parker, J. A., Bradson, M. L., Wu, T., Felipe Vial, U., Leodori, G., Bushnell, M. C., Horovitz, S. G., Hallett, M., et al. (2020) · Brain · doi:10.1093/brain/awaa138 · Cited by 35
Kang, X., Xi, Y., Liu, T., et al. (2020) · BMC Medical Imaging · doi:10.1186/s12880-020-00450-x · Cited by 34
The Concerns About Pain (CAP) Scale: A Patient-Reported Outcome Measure of Pain Catastrophizing
Amtmann, D., Bamer, A. M., Liljenquist, K. S., Cowan, P., Salem, R., Turk, D. C., & Jensen, M. P. (2020) · Journal of Pain · doi:10.1016/j.jpain.2020.03.004 · Cited by 35
Haddad, M., Hermassi, S., Aganović, Z., Dalansi, F., Kharbach, M., Mohamed, A. O., & Bibi, K. W. (2020) · Frontiers in Physiology · doi:10.3389/fphys.2020.00957 · Cited by 33
A Systematic Review of Surveys for Measuring Patient-centered Care in the Hospital Setting
Handley, S. C., Bell, S., & Nembhard, I. M. (2020) · Medical Care · doi:10.1097/mlr.0000000000001474 · Cited by 36
Mallah, N., Rodríguez‐Cano, R., Figueiras, A., & Takkouche, B. (2020) · Scientific Reports · doi:10.1038/s41598-020-77769-6 · Cited by 41
The morphokinetic signature of mosaic embryos: evidence in support of their own genetic identity
Martín, Á., Rodrigo, L., Beltrán, D., et al. (2021) · Fertility and Sterility · doi:10.1016/j.fertnstert.2020.12.031 · Cited by 34
Telomere dynamics across the early life course: Findings from a longitudinal study in children
Cowell, W., Tang, D. L., Yu, J., Guo, J., Wang, S., Baccarelli, A., Perera, F. P., & Herbstman, J. B. (2021) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2021.105270 · Cited by 35
Buchanan, C. R., Maniega, S. M., Hernández, M. D. C. V., Ballerini, L., Barclay, G., Taylor, A. M., Russ, T. C., Tucker–Drob, E. M., Wardlaw, J. M., Deary, I. J., et al. (2021) · Human Brain Mapping · doi:10.1002/hbm.25473 · Cited by 33
Workplace bullying as an organisational issue: Aligning climate and leadership
Plimmer, G., Nguyen, D. T. N., Teo, S., & Tuckey, M. R. (2021) · Work & Stress · doi:10.1080/02678373.2021.1969479 · Cited by 37
Wang, X., Ding, F., Chang, C., He, J., Wang, X., & Yao, S. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.816051 · Cited by 34
Deep learning based detection of enlarged perivascular spaces on brain MRI
Rashid, T., Liu, H., Ware, J. B., Li, K., Romero, J. R., Fadaee, E., Nasrallah, I. M., Hilal, S., Bryan, R. N., Hughes, T. M., et al. (2023) · Neuroimage Reports · doi:10.1016/j.ynirp.2023.100162 · Cited by 39
Development of the child abuse and neglect reporting self‐efficacy questionnaire for nurses
Lee, P.-Y., Dunne, M. P., Chou, F., et al. (2011) · The Kaohsiung Journal of Medical Sciences · doi:10.1016/j.kjms.2011.10.032 · Cited by 34
Bentman, S., O’Sullivan, C., & Stokes, M. J. (2010) · Clinical Physiology and Functional Imaging · doi:10.1111/j.1475-097x.2010.00960.x · Cited by 34
Pérez‐Cruzado, D., González‐Sánchez, M., & Cuesta‐Vargas, A. (2014) · BioMedical Engineering OnLine · doi:10.1186/1475-925x-13-127 · Cited by 34
Bracmort, K. S., Engel, B. A., & Frankenberger, J. R. (2004) · Journal of Soil and Water Conservation · doi:10.1080/00224561.2004.12435745 · Cited by 34
Ticlopidine versus oral anticoagulation for coronary stenting
Cosmi, B., Rubboli, A., Castelvetri, C., & Milandri, M. (2001) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002133 · Cited by 34
Wear Particles and Osteolysis in Patients With Total Wrist Arthroplasty
Boeckstyns, M. E. H., Toxværd, A., Bansal, M., et al. (2014) · The Journal Of Hand Surgery · doi:10.1016/j.jhsa.2014.07.046 · Cited by 34
Herda, T. J., Weir, J. P., Ryan, E. D., et al. (2009) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2009.01.028 · Cited by 34
Long‐term course and outcome in delusional disorder
Opjordsmoen, S. (1988) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1988.tb06387.x · Cited by 34
The Swedish National Adult Reading Test (NART‐SWE): A test of premorbid IQ
Rolstad, S., Nordlund, A., Gustavsson, M. H., Eckerström, C., KLANG, O., Hansen, S., & Wallin, A. (2008) · Scandinavian Journal of Psychology · doi:10.1111/j.1467-9450.2008.00677.x · Cited by 35
Jetha, M. K., Schmidt, L. A., & Goldberg, J. O. (2008) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2008.12.011 · Cited by 34
Stemplewski, R., Maciaszek, J., Osiński, W., et al. (2011) · Journal of Human Kinetics · doi:10.2478/v10078-011-0018-9 · Cited by 34
Reproducibility of Structural, Resting-State BOLD and DTI Data between Identical Scanners
Huang, L., Wang, X., Baliki, M. N., Wang, L., Apkarian, A. V., & Parrish, T. B. (2012) · PLoS ONE · doi:10.1371/journal.pone.0047684 · Cited by 34
A study of the noninstrumented physical examination of the knee found high observer variability
Wood, L. R. J., Peat, G., Wilkie, R., Hay, E. M., Thomas, E., & Sim, J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.11.004 · Cited by 34
Johnsen, M. B., Eitzen, I., Moksnes, H., et al. (2013) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-013-2771-x · Cited by 34
Porcerelli, J. H., Shahar, G., Blatt, S. J., et al. (2006) · Personality and Individual Differences · doi:10.1016/j.paid.2005.10.027 · Cited by 34
A multi-joint lower-limb tracking-trajectory test for the assessment of motor coordination
Maffiuletti, N. A., Bizzini, M., Schatt, S., & Munzinger, U. (2005) · Neuroscience Letters · doi:10.1016/j.neulet.2005.04.064 · Cited by 34
Vogel, L. C., Samdani, A. F., Chafetz, R. S., et al. (2009) · Spinal Cord · doi:10.1038/sc.2008.180 · Cited by 34
Mood patterns and variations associated with personality disorder pathology.
Farmer, R. F., Nash, H. M., & Dance, D. (2004) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2004.03.009 · Cited by 26
Hall, R. J., Rogers, R. G., Saiz, L., et al. (2007) · International Urogynecology Journal · doi:10.1007/s00192-006-0254-z · Cited by 34
Prior, S., Mitchell, T., Whiteley, R. J., et al. (2014) · BMC Sports Science Medicine and Rehabilitation · doi:10.1186/2052-1847-6-13 · Cited by 34
Sato, M. (2014) · System · doi:10.1016/j.system.2014.05.004 · Cited by 37
Developing an estimate of daily cumulative loading for the knee: Examining test–retest reliability
Robbins, S. M. K., Birmingham, T. B., Jones, G. R., Callaghan, J. P., & Maly, M. R. (2009) · Gait & Posture · doi:10.1016/j.gaitpost.2009.07.118 · Cited by 34
The Rorschach in Forensic Practice
Gacono, C. B., Evans, F. B., & Viglione, D. J. (2002) · Journal of Forensic Psychology Practice · doi:10.1300/j158v02n03_03 · Cited by 34
Assessment of Affect Integration: Validation of the Affect Consciousness Construct
Solbakken, O. A., Hansen, R. S., Havik, O. E., & Monsen, J. T. (2011) · Journal of Personality Assessment · doi:10.1080/00223891.2011.558874 · Cited by 34
The Personality Organization Diagnostic Form
Gamache, D., Laverdière, O., Diguer, L., Hébert, É., Larochelle, S., & Descôteaux, J. (2009) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0b013e3181a20897 · Cited by 34
Newborns discriminate schematic faces from scrambled faces.
Easterbrook, M. A., Kisilevsky, B. S., Muir, D. W., & Laplante, D. P. (1999) · Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale · doi:10.1037/h0087312 · Cited by 35
Madjar, N. Y., & Oldham, G. R. (2002) · Creativity Research Journal · doi:10.1207/s15326934crj1402_10 · Cited by 34
A new instrument for assessing the quality of studies on prevalence
Giannakopoulos, N. N., Rammelsberg, P., Eberhard, L., & Schmitter, M. (2011) · Clinical Oral Investigations · doi:10.1007/s00784-011-0557-4 · Cited by 35
Friedman, L., Turner, J. A., Stern, H. S., Mathalon, D. H., Trondsen, L. C., & Potkin, S. G. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.12.040 · Cited by 34
Brunner, H. I., Higgins, G. C., Klein‐Gitelman, M. S., Lapidus, S., Olson, J. C., Onel, K. B., Punaro, M., Ying, J., & Giannini, E. H. (2010) · Arthritis Care & Research · doi:10.1002/acr.20154 · Cited by 34
Criterion validity of 3D trunk accelerations to assess external work and power in able-bodied gait
Meichtry, A., Romkes, J., Gobelet, C., Brunner, R., & Müller, R. (2006) · Gait & Posture · doi:10.1016/j.gaitpost.2005.12.016 · Cited by 34
Antecedents for achievement of alignment in organizations
Beehr, T. A., Glazer, S., Fischer, R. D., Linton, L. L., & Hansen, C. P. (2008) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317908x310247 · Cited by 34
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
