The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,342
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 10,601–10,700 of 23,881
Nattrass, A., Kellett, S., Hardy, G. E., & Ricketts, T. (2014) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s135246581400006x · Cited by 20
Schurr, K., Sherrington, C., Wallbank, G., et al. (2012) · Clinical Rehabilitation · doi:10.1177/0269215511427323 · Cited by 20
Özdemir, H., Çandır, M. O., Karbuz, A., et al. (2011) · The Turkish Journal of Pediatrics · doi:10.24953/turkjpediatr.2011.1831 · Cited by 20
Zhu, H. W., McGrath, C. P. J., McMillan, A. S., et al. (2007) · Community Dentistry And Oral Epidemiology · doi:10.1111/j.1600-0528.2006.00370.x · Cited by 20
Achilli, F., Malafronte, C., Maggiolini, S., Lenatti, L., Squadroni, L., Gibelli, G., Capogrossi, M. C., Dadone, V., Gentile, F., Bassetti, B., et al. (2014) · Heart · doi:10.1136/heartjnl-2013-304955 · Cited by 20
The MDS Challenging Behavior Profile for long-term care
Gerritsen, D. L., Achterberg, W. P., Steverink, N., Pot, A. M., Frijters, D. H. M., & Ribbe, M. W. (2008) · Aging & Mental Health · doi:10.1080/13607860701529882 · Cited by 20
Velghe, K., Vermaire, J. C., & Gregory‐Eaves, I. (2012) · Freshwater Biology · doi:10.1111/fwb.12017 · Cited by 20
Recurrence of MDD: A prospective study of personality pathology and cognitive distortions.
Craighead, W. E., Sheets, E. S., Craighead, L. W., & Madsen, J. W. (2010) · Personality Disorders Theory Research and Treatment · doi:10.1037/a0020456 · Cited by 20
Stein, M. B., Slavin‐Mulford, J. M., Siefert, C. J., et al. (2015) · Rorschachiana Journal of the International Society for the Rorschach · doi:10.1027/1192-5604/a000057 · Cited by 20
Perceived barriers to medication adherence in pediatric and adolescent solid organ transplantation
Danziger‐Isakov, L. A., Frazier, T. W., Worley, S. E., Williams, N. M., Shellmer, D. A., Dharnidharka, V. R., Gupta, N. A., Iklé, D. N., & Sweet, S. C. (2015) · Pediatric Transplantation · doi:10.1111/petr.12648 · Cited by 20
Brogårdh, C. B., & Lexell, J. E. (2015) · PM&R · doi:10.1016/j.pmrj.2015.09.023 · Cited by 20
Effect of saddle height on skin temperature measured in different days of cycling
Priego‐Quesada, J. I., Carpes, F. P., Palmer, R. S., Pérez‐Soriano, P., & de Anda, R. M. C. O. (2016) · SpringerPlus · doi:10.1186/s40064-016-1843-z · Cited by 25
Predator experience homogenizes consistent individual differences in predator avoidance
Sommer‐Trembo, C., Zimmer, C., Jourdan, J., Bierbach, D., & Plath, M. W. (2016) · Journal of Ethology · doi:10.1007/s10164-016-0460-1 · Cited by 20
Hoen, L. ’., Korfage, I. J., Verhallen, J. T. C. M., van Ledden‐Klok, M. J., van den Hoek, J., Blok, B. F. M., & Scheepe, J. R. (2016) · The Journal of Urology · doi:10.1016/j.juro.2016.02.2973 · Cited by 20
Reliability and Validity of Modified Algometer in Abdominal Examination
Ko, S., Kim, H., Kim, S.-K., et al. (2016) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2016/3052954 · Cited by 20
Caughy, M. O., Peredo, T. N., Owen, M. T., & Mills, B. A. (2016) · Developmental Psychology · doi:10.1037/a0040075 · Cited by 24
Imaging of gliomas at 1.5 and 3 Tesla - A comparative study
Tselikas, L., Souillard-Scémama, R., Naggara, O. N., Mellerio, C., Varlet, P., Dezamis, E., Dômont, J., Dhermain, F. G., Devaux, B. C., Chrétien, F. C., et al. (2014) · Neuro-Oncology · doi:10.1093/neuonc/nou332 · Cited by 20
Pitman, S. R., Slavin‐Mulford, J. M., & Hilsenroth, M. J. (2014) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0000000000000137 · Cited by 20
Postural seated balance in children can be assessed with good reliability
Pedersen, L. K., Martinkevich, P., Ege, S., et al. (2016) · Gait & Posture · doi:10.1016/j.gaitpost.2016.04.007 · Cited by 20
Weck, F., Grikscheit, F., Höfling, V., Kordt, A., Hamm, A. O., Gerlach, A. L., Alpers, G. W., Arolt, V., Kircher, T. T. J., Pauli, P., et al. (2016) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2016.05.007 · Cited by 23
Jenkins, N. D. M., Housh, T. J., Buckner, S. L., Bergstrom, H. C., Smith, C. M., Cochrane, K. C., Hill, E. C., Miramonti, A. A., Schmidt, R. J., Johnson, G. O., et al. (2016) · PubMed · Cited by 22
The estimation of platelet count from a blood smear on the basis of the red cell: platelet ratio.
Brahimi, M., Osmani, S., Arabi, A., Entasoltane, B., Taghezout, Z., Elkahili, B. S., & Bekadja, M. A. (2009) · PubMed · Cited by 20
COMT Val158Met Polymorphism Modulates Huntington's Disease Progression
de Diego‐Balaguer, R., Schramm, C., Cournu‐Rebeix, I., Emmanuel Dupoux, A., Dürr, A., Brice, A., Charles, P., de Langavant, L. C., Youssov, K., Verny, C., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0161106 · Cited by 18
Baque, E., Barber, L. A., Sakzewski, L., & Boyd, R. N. (2016) · Brain Injury · doi:10.1080/02699052.2016.1201594 · Cited by 20
Yang, H., Kim, D.-E., Jang, W.-H., et al. (2016) · Toxicology in Vitro · doi:10.1016/j.tiv.2016.11.010 · Cited by 20
Watkins, S. C., Anders, S. H., Clebone, A., Hughes, E., Patel, V., Zeigler, L. N., Shi, Y., Shotwell, M. S., McEvoy, M. D., & Weinger, M. B. (2016) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000191 · Cited by 20
Reliability of infarct volumetry: Its relevance and the improvement by a software-assisted approach
Friedländer, F., Bohmann, F. O., Brunkhorst, M., Chae, J., Devraj, K., Köhler, Y., Kraft, P., Kuhn, H., Lucaciu, A., Luger, S., et al. (2016) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x16681311 · Cited by 22
Personality dimensions of the captive California sea lion (Zalophus californianus).
Ciardelli, L. E., Weiß, A., Powell, D., et al. (2017) · Journal of comparative psychology · doi:10.1037/com0000054 · Cited by 20
Reliability of a novel ultrasound system for gray‐scale analysis of muscle
OʼBrien, T. G., Gonzalez, M. L. C., Ghosh, P. S., Mandrekar, J. N., & Boon, A. J. (2016) · Muscle & Nerve · doi:10.1002/mus.25513 · Cited by 21
Could a woman be superman? Gender and the embodiment of power postures
Bailey, A. H., LaFrance, M., & Dovidio, J. F. (2017) · Comprehensive Results in Social Psychology · doi:10.1080/23743603.2016.1248079 · Cited by 25
Mayall, P. R., Pilbrow, V. C., & Bitadze, L. (2017) · PLoS ONE · doi:10.1371/journal.pone.0171064 · Cited by 20
Gür, G., Yakut, Y., & Grivas, T. B. (2017) · Prosthetics and Orthotics International · doi:10.1177/0309364617690393 · Cited by 20
Comparison of the HUI3 and the EQ-5D-3L in a nursing home setting
Lung, T. W.-C., Howard, K., Etherton‐Beer, C., Sim, M., Lewin, G. F., & Arendts, G. (2017) · PLoS ONE · doi:10.1371/journal.pone.0172796 · Cited by 20
Zaki, H. F. M., Faisal Shafait, B., & Mian, A. S. (2017) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2017.02.008 · Cited by 20
Lessard, I., Lavoie, C., Côté, I., Mathieu, J., Brais, B., & Gagnon, C. (2017) · Journal of the Neurological Sciences · doi:10.1016/j.jns.2017.03.046 · Cited by 20
Meeremans, P., Yochum, N., Kochzius, M., Ampe, B., Tuyttens, F., & Uhlmann, S. S. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179092 · Cited by 20
Aiyegbusi, O. L., Kyte, D. G., Cockwell, P., Marshall, T. P., Dutton, M., Slade, A. L., Marklew, N., Price, G. W., Verdi, R., Waters, J., et al. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-016687 · Cited by 20
A research and development decision model for pharmaceutical industry: case of China
Chan, L., & Daim, T. U. (2017) · R and D Management · doi:10.1111/radm.12285 · Cited by 20
Palmer, T. B., Pineda, J. G., & Durham, R. M. (2017) · Journal of Applied Biomechanics · doi:10.1123/jab.2017-0213 · Cited by 20
Najas-García, A., Carmona, V. R., & Gómez‐Benito, J. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00063 · Cited by 25
A semi-automated algorithm for hypothalamus volumetry in 3 Tesla magnetic resonance images
Wolff, J., Schindler, S., Lucas, C., Binninger, A.-S., Weinrich, L., Schreiber, J., Hegerl, U., Möller, H. E., Leitzke, M., Geyer, S., et al. (2018) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2018.04.007 · Cited by 21
Torres-Costoso, A. I., Vlachopoulos, D., Ubago‐Guisado, E., Ferri‐Morales, A., Cavero‐Redondo, I., Martínez‐Vizcaíno, V., & Gracia‐Marco, L. (2018) · Pediatric Exercise Science · doi:10.1123/pes.2017-0217 · Cited by 20
Perceptual and Quantitative Assessment of Dysphonia Across Vowel Categories
Anand, S., Skowronski, M. D., Shrivastav, R., et al. (2018) · Journal of Voice · doi:10.1016/j.jvoice.2017.12.018 · Cited by 20
Xiong, X., Zhou, Z., Figini, M., Shangguan, J., Zhang, Z., & Chen, W. (2018) · PubMed Central · Cited by 20
Knapik, D. M., Cumsky, J. L., Tanenbaum, J. E., et al. (2018) · HSS Journal® The Musculoskeletal Journal of Hospital for Special Surgery · doi:10.1007/s11420-018-9618-4 · Cited by 20
Mohamadi, S., Dadgoo, M., Ebrahimi, E., Salavati, M., Saeedi, A., & Valiollahi, B. (2018) · Disability and Rehabilitation · doi:10.1080/09638288.2018.1452053 · Cited by 20
Psychometric Properties of Figure Rating Scales in Children: The Impact of Figure Ordering
Zitzmann, J., & Warschburger, P. (2018) · Journal of Personality Assessment · doi:10.1080/00223891.2018.1505731 · Cited by 26
Z Score Reference Ranges of Fetal Cardiothoracic Diameter Ratio
Sirilert, S., Tongprasert, F., Srisupundit, K., Tongsong, T., & Luewan, S. (2018) · Journal of Ultrasound in Medicine · doi:10.1002/jum.14786 · Cited by 20
Fujiwara, T., Nishizawa, M., Hoshide, S., Kanegae, H., & Kario, K. (2018) · Journal of Clinical Hypertension · doi:10.1111/jch.13407 · Cited by 19
Sethi, S. K., Kisch, S. J., Ghassaban, K., Rajput, A. H., Rajput, A. H., Babyn, P., Liu, S., Szkup, P., & Haacke, E. M. (2018) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2018.10.001 · Cited by 21
Development and assessment of the Korean Author Recognition Test
Lee, H., Seong, E., Choi, W., & Lowder, M. W. (2018) · Quarterly Journal of Experimental Psychology · doi:10.1177/1747021818814461 · Cited by 20
Accuracy and bias of automatic hippocampal segmentation in children and adolescents
Herten, A., Konrad, K., Krinzinger, H., Seitz, J., & von Polier, G. G. (2018) · Brain Structure and Function · doi:10.1007/s00429-018-1802-2 · Cited by 23
van der Straaten, R., Bruijnes, A. K. B. D., Vanwanseele, B., Jonkers, I., Baets, L. D., & Timmermans, A. A. A. (2019) · Sensors · doi:10.3390/s19010141 · Cited by 20
The female sexual function index: reliability and validity in Spanish postmenopausal women
Herrezuelo, I. P., Hita‐Contreras, F., Martínez‐Amat, A., Aibar‐Almazán, A., Cruz‐Díaz, D., Wangensteen, R., Achalandabaso‐Ochoa, A., & Díaz-Mohedo, E. (2019) · Menopause The Journal of The North American Menopause Society · doi:10.1097/gme.0000000000001305 · Cited by 20
Kwan, B. C. H., Jugé, L., Gandevia, S. C., & Bilston, L. E. (2019) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2018.12.003 · Cited by 20
Paolillo, A., da Silva, S. A., Carvalho, H., & Pasini, M. (2020) · Journal of Safety Research · doi:10.1016/j.jsr.2019.12.009 · Cited by 20
On Precision of Code Clone Detection Tools
Farmahinifarahani, F., Saini, V., Yang, D., Sajnani, H., & Lopes, C. V. (2019) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8668015 · Cited by 20
Maddock, C. L., Noor, S. M. N., Kothari, A., et al. (2019) · Journal of Children s Orthopaedics · doi:10.1302/1863-2548.13.190015 · Cited by 20
Pilot Evaluation of Aggregate Plate Waste as a Measure of Students’ School Lunch Consumption
Chapman, L. E., Richardson, S. A., McLeod, L., et al. (2019) · Journal of the Academy of Nutrition and Dietetics · doi:10.1016/j.jand.2019.04.001 · Cited by 20
TEST-RETEST RELIABILITY OF ELECTRONIC HAND DYNAMOMETER IN HEALTHY ADULTS.
Mani, P. (2019) · International Journal of Advanced Research · doi:10.21474/ijar01/9042 · Cited by 20
Validated Assessment Scales for Cellulite Dimples on the Buttocks and Thighs in Female Patients
Hexsel, D. M., Fabi, S. G., Sattler, G., et al. (2019) · Dermatologic Surgery · doi:10.1097/dss.0000000000001993 · Cited by 20
Validation of Optical Coherence Tomography Retinal Segmentation in Neurodegenerative Disease
Wong, B. M., Cheng, R., Mandelcorn, E. D., et al. (2019) · Translational Vision Science & Technology · doi:10.1167/tvst.8.5.6 · Cited by 20
del Corral, T., Sánchez, A., & López‐de‐Uralde‐Villanueva, I. (2019) · Journal of Cystic Fibrosis · doi:10.1016/j.jcf.2019.10.007 · Cited by 20
Sigmundsson, H., Haga, M., Ofteland, G. S., & Solstad, T. (2019) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2019.100756 · Cited by 26
Vaughn, A., Hales, D. P., Neshteruk, C. D., & Ward, D. S. (2019) · PLoS ONE · doi:10.1371/journal.pone.0226984 · Cited by 22
Grigoriadis, A., Morsbach, F., Voulgarakis, N., Said, K., Bergquist, A. M., & Kartalis, N. (2020) · Scandinavian Journal of Gastroenterology · doi:10.1080/00365521.2020.1720281 · Cited by 20
Binkley, J. M., Weiler, M. J., Frank, N. D., Bober, L., Dixon, J. B., & Stratford, P. W. (2020) · Physical Therapy · doi:10.1093/ptj/pzz175 · Cited by 20
Choose your words wisely: Optimizing impacts on standardized performance testing
Chua, L.-K., Wulf, G., & Lewthwaite, R. (2020) · Gait & Posture · doi:10.1016/j.gaitpost.2020.05.001 · Cited by 21
Environmental Factors Shape Sleep EEG Connectivity During Early Adolescence
Markovic, A., Kaess, M., & Tarokh, L. (2020) · Cerebral Cortex · doi:10.1093/cercor/bhaa151 · Cited by 18
Cross-classified models in I/O psychology
Claus, A. M., Arend, M. G., Burk, C. L., Kiefer, C., & Wiese, B. S. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2020.103447 · Cited by 22
Wang, X., Liang, D., Lu, C., Shi, Y., Zhang, J., Cao, Y., Fang, C., Huang, D., & Jin, Y. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01041-8 · Cited by 20
Talbot, C. F., Garner, J. P., Maness, A. C., McCowan, B., Capitanio, J. P., & Parker, K. J. (2020) · Autism Research · doi:10.1002/aur.2335 · Cited by 19
Xu, C., Yin, M., & Mo, W. (2020) · Global Spine Journal · doi:10.1177/2192568220947049 · Cited by 20
Is the Political Slant of Psychology Research Related to Scientific Replicability?
Reinero, D. A., Wills, J. A., Brady, W. J., Mende‐Siedlecki, P., Crawford, J. T., & Bavel, J. J. V. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620924463 · Cited by 21
Grierson, L., Winemaker, S., Taniguchi, A., Howard, M., Marshall, D., & Zazulak, J. (2020) · Perspectives on Medical Education · doi:10.1007/s40037-020-00611-2 · Cited by 20
Measurement of Lumbar Lordosis: A Comparison of 2 Alternatives to the Cobb Angle
Russell, B. S., Muhlenkamp-Wermert, K. A., & Hoiriis, K. T. (2020) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2020.05.009 · Cited by 20
Hinks, A., Davidson, B., Akagi, R., & Power, G. A. (2020) · Scandinavian Journal of Medicine and Science in Sports · doi:10.1111/sms.13842 · Cited by 20
Palmer, T. B., Blinch, J., Farrow, A. C., et al. (2020) · Physiological Measurement · doi:10.1088/1361-6579/abc40b · Cited by 20
Prabhu, R., Miller, S. R., Simpson, T. W., & Meisel, N. A. (2020) · Journal of Engineering Design · doi:10.1080/09544828.2020.1851661 · Cited by 21
Hermassi, S., Chelly, M. S., Michalsik, L. B., Sanal‐Hayes, N. E. M., Hayes, L. D., & Cadenas‐Sánchez, C. (2021) · PLoS ONE · doi:10.1371/journal.pone.0246476 · Cited by 22
Roossien, C. C., Hodselmans, A. P., Heus, R., Reneman, M. F., & Verkerke, G. J. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18094896 · Cited by 20
Gielis, K., Abeele, M.-E. V., Croon, R. D., Dierick, P., Ferreira-Brito, F., Assche, L. V., Verbert, K., Tournoy, J., & Abeele, V. V. (2021) · JMIR Serious Games · doi:10.2196/18359 · Cited by 25
Correa-Rojas, J. (2021) · Ciencias Psicológicas · doi:10.22235/cp.v15i2.2318 · Cited by 20
Wallenberg, R. B., Belzer, M. L., Ramsey, D. C., et al. (2022) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2021.12.037 · Cited by 20
Corrado, P. A., Wentland, A. L., Starekova, J., Dhyani, A., Goss, K. N., & Wieben, O. E. (2022) · European Radiology · doi:10.1007/s00330-022-08616-7 · Cited by 20
Guha, D., Ibrahim, G. M., Kertzer, J. D., et al. (2014) · Journal of neurosurgery · doi:10.3171/2014.7.jns132141 · Cited by 20
THE EFFECT OF ACUPUNCTURE ON ALPHA-MOTONEURON EXCITABILITY
Chan, A.-C., Vujnovich, A. L., & Bradnam, L. V. (2004) · Acupuncture & Electro-Therapeutics Research · doi:10.3727/036012904815901506 · Cited by 20
Observer reliability of arteriovenous malformations grading scales using current imaging modalities
Griessenauer, C. J., Miller, J. H., Agee, B. S., Fisher, W. S., Curé, J. K., Chapman, P. R., Foreman, P. M., Fisher, W. A. M., Witcher, A. C., & Walters, B. C. (2014) · Journal of neurosurgery · doi:10.3171/2014.2.jns131262 · Cited by 20
The Selection of Key Informants in IB Cross-Cultural Studies
Lenartowicz, T., & Roth, K. (2004) · Management International Review · Cited by 20
Russo, C. A., Jaubert, M., Jin, Z., et al. (2012) · Echocardiography · doi:10.1111/j.1540-8175.2011.01655.x · Cited by 20
Preliminary Evaluation of an Observational Measure of Group Cohesion for Group Psychotherapy
Lerner, M. D., McLeod, B. D., & Mikami, A. Y. (2012) · Journal of Clinical Psychology · doi:10.1002/jclp.21933 · Cited by 20
de Vries, M., de Koning, P. J., de Haan, M. W., et al. (2005) · Investigative Radiology · doi:10.1097/01.rli.0000163004.65460.8b · Cited by 20
Velocity discrimination: Reliability and construct validity in older adults
Westlake, K. P., Wu, Y., & Culham, E. G. (2007) · Human Movement Science · doi:10.1016/j.humov.2006.12.002 · Cited by 19
Subgingival microbial profiles of Sudanese patients with aggressive periodontitis
Elabdeen, H. R. Z., Mustafa, M., Hastürk, H., et al. (2014) · Journal of Periodontal Research · doi:10.1111/jre.12250 · Cited by 20
Wodchis, W. P., Maxwell, C. J., Venturini, A., Walker, J. D., Zhang, J., Hogan, D. B., & Feeny, D. F. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.08.009 · Cited by 20
Gutman, R. E., Pannu, H. K., Cundiff, G. W., et al. (2005) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2004.11.028 · Cited by 20
Effects of mode of presentation on ratings of empathic communication in medical interviews
Nicolai, J., Demmel, R., & Farsch, K. (2009) · Patient Education and Counseling · doi:10.1016/j.pec.2009.10.014 · Cited by 20
Assessing Visual-Spatial Creativity in Youth on the Autism Spectrum
Diener, E., Wright, C., Smith, K., & Wright, S. D. (2014) · Creativity Research Journal · doi:10.1080/10400419.2014.929421 · Cited by 22
Lang, E. V., Sood, A., Anderson, B., Kettenmann, E., & Armstrong, E. (2005) · Academic Radiology · doi:10.1016/j.acra.2005.03.064 · Cited by 20
Hannula, J. A., Lahtela, K., Järvikoski, A., Salminen, J. K., & Mäkelä, P. (2006) · Nordic Journal of Psychiatry · doi:10.1080/08039480600937140 · Cited by 20
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
