The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,342
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 8,501–8,600 of 23,881
Berke, L., Huenerfauth, M., & Patel, K. (2019) · ACM Transactions on Accessible Computing · doi:10.1145/3314205 · Cited by 29
Clinical validation of three-dimensional ultrasound for abdominal aortic aneurysm
Ghulam, Q. M., Kilaru, S., Ou, S., & Sillesen, H. H. (2019) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2019.03.066 · Cited by 28
Lee, T. T., Jiang, W., Cheng, C. L. K., et al. (2019) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2019.05.031 · Cited by 29
Smith, M. L., Kakuhikire, B., Baguma, C., et al. (2020) · PubMed · doi:10.7189/jogh.10.010412 · Cited by 29
Lee, J. S. (2020) · Australasian Journal of Educational Technology · doi:10.14742/ajet.5319 · Cited by 24
Szentes, B. L., Schultz, K., Nowak, D., Schüler, M., & Schwarzkopf, L. (2020) · BMC Pulmonary Medicine · doi:10.1186/s12890-020-01205-8 · Cited by 29
Hackert, M. Q. N., van Exel, J. N. A., & Brouwer, W. B. F. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113109 · Cited by 29
Brewe, A. M., Mazefsky, C. A., & White, S. W. (2020) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-020-04623-z · Cited by 30
Maldaner, N., Sosnova, M., Zeitlberger, A. M., et al. (2020) · Journal of Neurosurgery Spine · doi:10.3171/2020.5.spine20547 · Cited by 29
Uher, J. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2020.110205 · Cited by 29
Byun, D. J., Gorovets, D. J., Jacobs, L., et al. (2020) · Radiation Oncology · doi:10.1186/s13014-020-01681-6 · Cited by 29
Guinart, D., de Filippis, R., Rosson, S., Patil, B., Prizgint, L., Talasazan, N., Meltzer, H., Kane, J. M., & Gibbons, R. D. (2020) · Schizophrenia Bulletin · doi:10.1093/schbul/sbaa168 · Cited by 28
Duda, J. M., Moser, A. D., Zuo, C. S., Du, F., Chen, X., Perlo, S., Richards, C. E., Nascimento, N. F., Ironside, M., Crowley, D. J., et al. (2020) · Magnetic Resonance in Medicine · doi:10.1002/mrm.28587 · Cited by 30
Fraeulin, L., Holzgreve, F., Brinkbäumer, M., Dziuba, A., Friebe, D., Klemz, S., Schmitt, M., Anna-Lena Theis, A., Tenberg, S., van Mark, A., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0243646 · Cited by 36
Varghese, B. A., Hwang, D., Cen, S., Lei, X., Levy, J. R., Desai, B., Goodenough, D. J., & Duddalwar, V. A. (2021) · Journal of Applied Clinical Medical Physics · doi:10.1002/acm2.13162 · Cited by 29
Getting the Picture Too Late: Handoffs and the Effectiveness of Idea Implementation in Creative Work
Berg, J. M., & Yu, A. (2021) · Academy of Management Journal · doi:10.5465/amj.2019.1330 · Cited by 23
Cole, M., Murray, K. D., St‐Onge, E., Risk, B. B., Zhong, J., Schifitto, G., Descoteaux, M., & Zhang, Z. (2021) · Human Brain Mapping · doi:10.1002/hbm.25447 · Cited by 30
Test–retest reliability and validity of body composition methods in adults
de Zepetnek, J. O. T., Lee, J. J., Boateng, T., Plastina, S., Cleary, S. R., Huang, L., Kucab, M., Paterakis, S., Brett, N. R., & Bellissimo, N. (2021) · Clinical Physiology and Functional Imaging · doi:10.1111/cpf.12716 · Cited by 31
Cretenoud, A. F., Grzeczkowski, L., Kunchulia, M., & Herzog, M. H. (2021) · Journal of Vision · doi:10.1167/jov.21.5.26 · Cited by 27
Yi, Y., Chen, Y., & He, X. (2021) · Management and Organization Review · doi:10.1017/mor.2021.10 · Cited by 35
Intra-session test-retest reliability of functional connectivity in infants
Wang, Y., Hinds, W. A., Duarte, C. S., Lee, S., Monk, C. E., Wall, M. M., CANINO, G. J., Milani, A. C. C., Jackowski, A. P., Mamin, M. G., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118284 · Cited by 32
Examining Emotion Perception Agreement in Live Music Performance
Yang, S., Reed, C. N., Chew, E., & Barthet, M. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3093787 · Cited by 32
Wion, R. K., Hill, N. L., Bell, T. R., Mogle, J. A., Yates, J., & Bhang, I. (2021) · Journal of Geriatric Psychiatry and Neurology · doi:10.1177/08919887211023591 · Cited by 29
Fijačko, N., Creber, R. M., Gosak, L., Štiglic, G., Egan, D., Chaka, B., Debeljak, N., Strnad, M., & Skok, P. (2021) · JMIR mhealth and uhealth · doi:10.2196/25437 · Cited by 33
D’hondt, F., Maene, C., Vervaet, R., Houtte, M. V., & Stevens, P. A. J. (2021) · Social Psychology of Education · doi:10.1007/s11218-021-09651-w · Cited by 41
Jiang, L., Xia, Y., Wang, L., et al. (2021) · Atmospheric chemistry and physics · doi:10.5194/acp-21-16985-2021 · Cited by 29
Test–retest reliability of the Epworth Sleepiness Scale in clinical trial settings
Rosenberg, R. P., Babson, K. A., Menno, D. M., Morris, S. J., Baladi, M. G., Hyman, D. L., & Black, J. E. (2021) · Journal of Sleep Research · doi:10.1111/jsr.13476 · Cited by 29
Landová, E., Janovcová, M., Štolhoferová, I., Rádlová, S., Frýdlová, P., Sedláčková, K., & Frynta, D. (2021) · PLoS ONE · doi:10.1371/journal.pone.0257726 · Cited by 34
Çelik, Y., Stuart, S., Woo, W. L., & Godfrey, A. (2021) · Sensors · doi:10.3390/s21196476 · Cited by 29
Estimated Pao 2: A Continuous and Noninvasive Method to Estimate Pao 2 and Oxygenation Index
Sauthier, M., Tuli, G., Jouvet, P. A., Brownstein, J. S., & Randolph, A. G. (2021) · Critical Care Explorations · doi:10.1097/cce.0000000000000546 · Cited by 34
Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion
Zhang, S., Ding, Y., Wei, Z., & Guan, C. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00397 · Cited by 30
Pop-out search instigates beta-gated feature selectivity enhancement across V4 layers
Westerberg, J. A., Sigworth, E. A., Schall, J. D., & Maier, A. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2103702118 · Cited by 31
Graff, K., Tansey, R. C., Ip, A., Rohr, C. S., Dimond, D., Dewey, D., & Bray, S. (2022) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2022.101087 · Cited by 27
Kirlić, N., Cohen, Z. P., Tsuchiyagaito, A., Misaki, M., McDermott, T. J., Aupperle, R. L., Stewart, J. L., Singh, M. K., Paulus, M. P., & Bodurka, J. (2022) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-022-00991-4 · Cited by 32
Pronk, T., Hirst, R. J., Wiers, R. W. H. J., & Murre, J. M. J. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01885-6 · Cited by 27
Othman, E., Werner, P., Saxen, F., Fiedler, M.-A., & Al-Hamadi, A. K. (2022) · Sensors · doi:10.3390/s22134992 · Cited by 31
Eremenco, S. L., Chen, W.-H., Blum, S. I., et al. (2022) · Quality of Life Research · doi:10.1007/s11136-022-03180-5 · Cited by 29
Klingelhöfer-Jens, M., Ehlers, M. R., Kuhn, M., Keyaniyan, V., & Lonsdorf, T. B. (2022) · eLife · doi:10.7554/elife.78717 · Cited by 33
Forthmann, B., & Doebler, P. (2022) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/aca0000518 · Cited by 29
Solís-Lemus, J. A., Baptiste, T. M. G., Barrows, R. K., Sillett, C., Gharaviri, A., Raffaele, G., Razeghi, O., Strocchi, M., Sim, I., Kotadia, I., et al. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107009 · Cited by 29
Koçak, B., Yüzkan, S., Mutlu, S., Karagülle, M., Kala, A., Kadıoğlu, M., Solak, S., Sunman, Ş., Temiz, Z. H., & Ganiyusufoğlu, A. K. (2023) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2023.232543 · Cited by 29
Thunberg, C., Wiker, T., Bundt, C., & Huster, R. J. (2024) · Cortex · doi:10.1016/j.cortex.2024.02.008 · Cited by 31
Levin, C., Kagan, T., Rosen, S., & Saban, M. (2024) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2024.104771 · Cited by 33
Repeatability of Cone Spacing Measures in Eyes With Inherited Retinal Degenerations
Zayit‐Soudry, S., Sippl-Swezey, N., Porco, T. C., Lynch, S. K., Syed, R., Ratnam, K., Menghini, M., Roorda, A., & Duncan, J. L. (2015) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.15-17010 · Cited by 30
Jones, T., & Schlegel, C. S. (2013) · Research in Nursing & Health · doi:10.1002/nur.21578 · Cited by 31
Styck, K. M., & Walsh, S. M. (2015) · Psychological Assessment · doi:10.1037/pas0000157 · Cited by 29
Clifton, A. D., & Kuper, L. E. (2010) · Journal of Personality · doi:10.1111/j.1467-6494.2010.00686.x · Cited by 30
Medina‐Rosas, J., Gladman, D. D., Su, J., Sabapathy, A., Urowitz, M. B., & Touma, Z. (2015) · Arthritis Research & Therapy · doi:10.1186/s13075-015-0808-x · Cited by 29
Friendship Quality Scale: Conceptualization, Development and Validation.
Thien, L. M., Razak, N. B. A., & Jamil, H. (2012) · Cited by 29
Reliability and Predictive Validity of Inlow’s 60-Second Diabetic Foot Screen Tool
Murphy, C. A., Laforet, K., Rosa, P. D., et al. (2012) · Advances in Skin & Wound Care · doi:10.1097/01.asw.0000415343.45178.91 · Cited by 29
Reliability of hip internal rotation range of motion measurement using a digital inclinometer
Krause, D. A., Hollman, J. H., Krych, A. J., et al. (2014) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-014-3096-0 · Cited by 29
Assessing the reliability of student evaluations of teaching: choosing the right coefficient
Morley, D. D. (2013) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2013.796508 · Cited by 31
Geaney, L. E., Brenneman, D. J., Cote, M. P., et al. (2010) · Orthopedics · doi:10.3928/01477447-20100429-10 · Cited by 29
Stefanic, N., & Randles, C. (2014) · Music Education Research · doi:10.1080/14613808.2014.909398 · Cited by 28
Kelsey, R. M., Alpert, B. S., Dahmer, M. K., Krushkal, J., & Quasney, M. W. (2011) · Psychophysiology · doi:10.1111/j.1469-8986.2011.01319.x · Cited by 29
Reliability of the Brazilian version of HCR-20 Assessing Risk for Violence
Telles, L. E. D. B., Day, V. P., Folino, J. Ó., et al. (2009) · Brazilian Journal of Psychiatry · doi:10.1590/s1516-44462009005000001 · Cited by 29
King, J. T., DiLuna, M. L., Cicchetti, D. V., et al. (2006) · Neurosurgery · doi:10.1227/01.neu.0000232666.67779.41 · Cited by 29
Nguyen, B.-V., Vincent, J., Hamm, J. B., Abalain, J.-H., Carré, J., Nowak, E., Ould-Ahmed, M., Arvieux, C. C., & Guéret, G. (2009) · Anesthesia & Analgesia · doi:10.1213/ane.0b013e3181b62664 · Cited by 28
Raykov, T. (2011) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2011.534319 · Cited by 29
Miekisiak, G. J., Kołłątaj, M., Dobrogowski, J., Kloc, W., Libionka, W., Banach, M., Łątka, D., Sobolewski, T., Sulewski, A., Nowakowski, A., et al. (2012) · European Spine Journal · doi:10.1007/s00586-012-2607-2 · Cited by 29
What do we know about the relationship between source monitoring deficits and executive dysfunction?
Haj, M. E., & Allain, P. (2012) · Neuropsychological Rehabilitation · doi:10.1080/09602011.2012.658267 · Cited by 32
Fischer, M. A., & Shrout, P. E. (2006) · Environment and Behavior · doi:10.1177/0013916505280083 · Cited by 29
Self-citationFratkin, J. L., Sinn, D. L., Thomas, S., Hilliard, S., Olson, Z., & Gosling, S. D. (2014) · Behavioural Processes · doi:10.1016/j.beproc.2014.09.028 · Cited by 29
Hilsenroth, M. J., Eudell‐Simmons, E. M., DeFife, J. A., & Charnas, J. W. (2007) · International Journal of Testing · doi:10.1080/15305050701438033 · Cited by 29
Falorni, A., Bini, V., Betterle, C., Brozzetti, A., Castaño, L. A., Fichna, M., Kämpe, O., Mellgren, G., Peterson, P., Chen, S., et al. (2015) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2014-1106 · Cited by 29
Chen, W., Song, X., Zhang, Y., Darvesh, S., Zhang, N., D’Arcy, R. C. N., Black, S. E., & Rockwood, K. J. (2010) · Dementia and Geriatric Cognitive Disorders · doi:10.1159/000319537 · Cited by 29
von Borstel, U. K., Pirsich, W., Gauly, M., & Bruns, E. (2012) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2012.04.007 · Cited by 29
Reliability of a New Instrument for Measuring Plantarflexor Muscle Strength
Örtqvist, M., Gutierrez-Farewik, E. M., Farewik, M., Jansson, A., Bartonek, Å., & Broström, E. W. (2007) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2007.05.028 · Cited by 28
Roach, A. T., McGrath, D., Wixson, C. S., et al. (2010) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.2009.00167.x · Cited by 29
SCORS–G Stimulus Characteristics of Select Thematic Apperception Test Cards
Stein, M. B., Slavin‐Mulford, J. M., Siefert, C. J., Sinclair, S. J., Renna, M. E., Malone, J. C., Bello, I., & Blais, M. A. (2013) · Journal of Personality Assessment · doi:10.1080/00223891.2013.823440 · Cited by 29
Test–retest variability in structural parameters measured with glaucoma imaging devices
Araie, M. (2012) · Japanese Journal of Ophthalmology · doi:10.1007/s10384-012-0181-0 · Cited by 29
Therapists' experience of difficulty in practice
Schröder, T., & Davis, J. D. (2004) · Psychotherapy Research · doi:10.1093/ptr/kph028 · Cited by 29
Southerst, D., Stupar, M., Côté, P., Mior, S. A., & Stern, P. J. (2013) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2013.05.023 · Cited by 28
Hoffart, A., Borge, F., Sexton, H. C., & Clark, D. M. (2008) · Psychotherapy Research · doi:10.1080/10503300802369343 · Cited by 29
Psychometric Properties of the Hooper Visual Organization Test
Lopez, M. N., Lazar, M. D., & Oh, S. (2003) · Assessment · doi:10.1177/1073191102250183 · Cited by 29
Identifying Gravitational Insecurity in Children: A Pilot Study
May-Benson, T. A., & Koomar, J. A. (2007) · American Journal of Occupational Therapy · doi:10.5014/ajot.61.2.142 · Cited by 29
Sato, N., Nunome, H., & Ikegami, Y. (2014) · Journal of Applied Biomechanics · doi:10.1123/jab.2014-0027 · Cited by 29
Thiébaut, É., Adrien, J., Blanc, R., et al. (2010) · Autism Research and Treatment · doi:10.1155/2010/875037 · Cited by 29
Yamao, Y., Yamakado, K., Takaki, H., et al. (2010) · Radiology · doi:10.1148/radiol.09090733 · Cited by 29
Police and Military as Good Strangers
Klein, G. A., Klein, H. A., Lande, B., Borders, J., & Whitacre, J. C. (2015) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12110 · Cited by 37
Retest Reliability of Force-Time Variables of Neck Muscles Under Isometric Conditions
Almosnino, S., Pelland, L., & Stevenson, J. M. (2010) · Journal of Athletic Training · doi:10.4085/1062-6050-45.5.453 · Cited by 29
The reliability and variation of measurements of the os calcis angles in children
Clint, S. A., Morris, T. P., Shaw, O. M., et al. (2010) · Journal of Bone and Joint Surgery - British Volume · doi:10.1302/0301-620x.92b4.22565 · Cited by 29
Putnam, K. M., & Kring, A. M. (2007) · Psychiatry Research · doi:10.1016/j.psychres.2006.09.010 · Cited by 29
Social support from fellow group members triggers additional effort in groups
Hüffmeier, J., Wessolowski, K., van Randenborgh, A., Bothin, J., Schmid‐Loertzer, N., & Hertel, G. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2021 · Cited by 29
Does Instruction in Semantic Clustering and Switching Enhance Verbal Fluency in Children?
Hurks, P. P. M. (2012) · The Clinical Neuropsychologist · doi:10.1080/13854046.2012.708361 · Cited by 30
Anthropometric Measurements from Photographic Images
Hung, P. C. K., Witana, C. P., & Goonetilleke, R. S. (2004) · Cited by 29
Rábago, C. A., Dingwell, J. B., & Wilken, J. M. (2015) · PLoS ONE · doi:10.1371/journal.pone.0142083 · Cited by 29
Interrater Reliability of an Oral Case Presentation Rating Tool in a Pediatric Clerkship
Lewin, L. O., Beraho, L., Dolan, S., et al. (2013) · Teaching and Learning in Medicine · doi:10.1080/10401334.2012.741537 · Cited by 29
Parallel Processes: Using Motivational Interviewing as an Implementation Coaching Strategy
Hettema, J. E., Ernst, D., Williams, J. R., & Miller, K. J. (2013) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-013-9381-8 · Cited by 29
Whitehurst, D. G. T., Norman, R., Brazier, J. E., & Viney, R. C. (2014) · Value in Health · doi:10.1016/j.jval.2014.03.1720 · Cited by 29
Reproducibility of 3-dimensional ultrasound readings of volume of carotid atherosclerotic plaque
Ludwig, M., Zieliński, T., Schremmer, D., & Stumpe, K. O. (2008) · Cardiovascular Ultrasound · doi:10.1186/1476-7120-6-42 · Cited by 30
Inter- and intra-observer reliability of a smartphone application for measuring hallux valgus angles
WALTER, R. D., Kosy, J. D., & Cove, R. (2012) · Foot and Ankle Surgery · doi:10.1016/j.fas.2012.08.004 · Cited by 29
Vasanth, L., Foo, L. F., Potter, H. G., et al. (2010) · The Journal of Rheumatology · doi:10.3899/jrheum.090063 · Cited by 29
Horstman, H. K., Maliski, R., Hays, A., Cox, J., Enderle, A., & Nelson, L. R. (2015) · Communication Monographs · doi:10.1080/03637751.2015.1068945 · Cited by 29
Nealis, L. J., Sherry, S., Lee‐Baggley, D., Stewart, S. H., & Macneil, M. A. (2016) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-016-9537-y · Cited by 30
Gomez, R. J., Barrowman, N., Elia, S., Manias, E., Royle, J., & Harrison, D. M. (2013) · Pain Research and Management · doi:10.1155/2013/897104 · Cited by 29
Kwon, S. R., Restrepo‐Kennedy, N., Dawson, D. V., Hernandez, M., Denehy, G. E., Blanchette, D. R., Gratton, D. G., Aquilino, S. A., & Armstrong, S. R. (2014) · Journal of Dental Education · doi:10.1002/j.0022-0337.2014.78.12.tb05844.x · Cited by 29
Schierz, O., & Reissmann, D. R. (2016) · Journal of Prosthodontic Research · doi:10.1016/j.jpor.2016.03.005 · Cited by 29
Pilot Testing the Debriefing for Meaningful Learning Evaluation Scale
Bradley, C. S., & Dreifuerst, K. T. (2016) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2016.01.008 · Cited by 29
Viljoen, J. L., Gray, A. L., Shaffer, C. S., Latzman, N. E., Scalora, M. J., & Ullman, D. L. (2015) · Sexual Abuse · doi:10.1177/1079063215595404 · Cited by 37
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
