The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,342
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 9,101–9,200 of 23,881
Bató, A., Brodszky, V., Gergely, L. H., Gáspár, K., Wikonkál, N. M., Kinyó, Á., Szabó, Á., Beretzky, Z., Szegedi, A., Remenyik, É., et al. (2021) · Quality of Life Research · doi:10.1007/s11136-020-02732-x · Cited by 26
Müller, P. L., Liefers, B. J., Treis, T., Rodrigues, F. G., Olvera‐Barrios, A., Paul, B., Dhingra, N., Lotery, A. J., Bailey, C., Taylor, P. M., et al. (2021) · Translational Vision Science & Technology · doi:10.1167/tvst.10.3.4 · Cited by 26
Rapid validity testing at the front end of innovation
Häufler, B. P., Globocnik, D., Saldías, P. L., & Salomo, S. (2021) · Journal of Product Innovation Management · doi:10.1111/jpim.12585 · Cited by 29
LinguaPix database: A megastudy of picture-naming norms
Krautz, A. E., & Keuleers, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01651-0 · Cited by 28
de Freitas, K. M. S., Massaro, C. D. S., Miranda, F., de Freitas, M. R., Janson, G., & Garib, D. G. (2021) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2020.05.027 · Cited by 27
Sunoqrot, M. R. S., Selnæs, K. M., Sandsmark, E., Langørgen, S., Bertilsson, H., Bathen, T. F., & Elschot, M. (2021) · Diagnostics · doi:10.3390/diagnostics11091690 · Cited by 26
Locus Coeruleus magnetic resonance imaging in cognitively intact elderly subjects
Giorgi, F. S., Lombardo, F. P., Galgani, A., Hlavata, H., Latta, D. D., Martini, N., Pavese, N., Ghicopulos, I., Baldacci, F., Coi, A., et al. (2021) · Brain Imaging and Behavior · doi:10.1007/s11682-021-00562-0 · Cited by 27
Kassinopoulos, M., & Mitsis, G. D. (2021) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2021.10.028 · Cited by 24
Picken, S., Summers, H., & Al‐Dadah, O. (2021) · Skeletal Radiology · doi:10.1007/s00256-021-03937-y · Cited by 26
Martin, J. L., Lippi, C. A., Ibarra, A. M. S., Ayala, E. B., Mordecai, E. A., Sippy, R. J., Heras, F., Blackburn, J. K., & Ryan, S. J. (2021) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0009931 · Cited by 30
Habla conmigo, daddy! Fathers’ language input in North American bilingual Latinx families
Ramírez, N. F., Hippe, D. S., Correa, L., Andert, J., & Baralt, M. (2022) · Infancy · doi:10.1111/infa.12450 · Cited by 30
Robustness of κ‐type coefficients for clinical agreement
Vanacore, A., & Pellegrino, M. S. (2022) · Statistics in Medicine · doi:10.1002/sim.9341 · Cited by 26
Lee, J., Ha, S., Kim, R. E. Y., Lee, M., Kim, D., & Lim, H. K. (2022) · Diagnostics · doi:10.3390/diagnostics12030623 · Cited by 26
Zech, H., Gable, P. A., DIJK, W. W. V., & van Dillen, L. F. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01920-6 · Cited by 26
A Machine Learning Perspective on fNIRS Signal Quality Control Approaches
Bizzego, A., Neoh, M. J. Y., Gabrieli, G., & Esposito, G. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3198110 · Cited by 29
Feinauer, S., Schuller, L., Groh, I., Huestegge, L., & Petzoldt, T. (2022) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2022.08.009 · Cited by 35
Brain age vector: A measure of brain aging with enhanced neurodegenerative disorder specificity
Chen, R., Yang, Y., Ye, C., Lv, H., & Ma, T. (2022) · Human Brain Mapping · doi:10.1002/hbm.26066 · Cited by 25
Klein, S. D., Collins, P. F., & Luciana, M. M. (2022) · Cognitive Psychology · doi:10.1016/j.cogpsych.2022.101518 · Cited by 31
Lei, D., Li, W., Qin, K., Ai, Y., Tallman, M. J., Patino, L. R., Welge, J. A., Blom, T. J., Klein, C. C., Fleck, D. E., et al. (2022) · Neuropsychopharmacology · doi:10.1038/s41386-022-01463-6 · Cited by 26
Kowalska-Duplaga, K., & Duplaga, M. (2023) · BMC Public Health · doi:10.1186/s12889-023-15603-0 · Cited by 27
Improving alcohol health literacy and reducing alcohol consumption: recommendations for Germany
Manthey, J., Kokole, D., Riedel-Heller, S. G., Rowlands, G., Schäfer, I., Schomerus, G., Soellner, R., & Kilian, C. (2023) · Addiction Science & Clinical Practice · doi:10.1186/s13722-023-00383-0 · Cited by 26
Yin, G., Li, T., Jin, S., Wang, N., Li, J., Wu, C., He, H., & Wang, J. (2023) · Cerebral Cortex · doi:10.1093/cercor/bhad178 · Cited by 26
Stark, B. C., Alexander, J. M., Hittson, A. K., Doub, A., Igleheart, M., Streander, T., & Jewell, E. (2023) · Journal of Speech Language and Hearing Research · doi:10.1044/2023_jslhr-22-00266 · Cited by 31
Measuring Tree Diameter with Photogrammetry Using Mobile Phone Cameras
Ahamed, A., Foye, J., Poudel, S., et al. (2023) · Forests · doi:10.3390/f14102027 · Cited by 26
Reliability and generalizability of neural speech tracking in younger and older adults
Panela, R. A., Copelli, F., & Herrmann, B. (2023) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2023.11.007 · Cited by 26
Sedie, A. D., Terslev, L., Bruyn, G. A. W., et al. (2024) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2024.152406 · Cited by 26
Cardiopulmonary Exercise Testing in a Prospective Multicenter Cohort of Older Adults
Wolf, C., Blackwell, T., Johnson, E., Glynn, N. W., Nicklas, B. J., Kritchevsky, S. B., Carnero, E. Á., Cawthon, P. M., Cummings, S. R., Toledo, F. G. S., et al. (2024) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000003444 · Cited by 31
Haworth, C. M. A., Dale, P. S., & Plomin, R. J. (2009) · Child Development · doi:10.1111/j.1467-8624.2009.01289.x · Cited by 26
Hards, A., Davies, S., Salman, A., et al. (2012) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/s12630-012-9752-2 · Cited by 26
Responsiveness of Two Methods for Measuring Foot and Ankle Volume
Henschke, N., Boland, R. A., & Adams, R. D. (2006) · Foot & Ankle International · doi:10.1177/107110070602701013 · Cited by 26
The Relationship of Teacher Use of Critical Sociocultural Practices with Student Achievement
Teemant, A., & Hausman, C. (2012) · Critical education · doi:10.14288/ce.v4i4.182434 · Cited by 26
Zhang, X., Yang, L., Gao, F., et al. (2015) · Medical Science Monitor · doi:10.12659/msm.894406 · Cited by 26
Rashidian, A., & Nooraie, R. Y. (2011) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2011.01649.x · Cited by 25
Luévano‐Contreras, C., Durkin, T., Pauls, M., & Chapman‐Novakofski, K. (2013) · International Journal of Food Sciences and Nutrition · doi:10.3109/09637486.2013.816939 · Cited by 26
Mackay, H. C., Barkham, M., Stiles, W. B., & Goldfried, M. R. (2002) · Journal of Counseling Psychology · doi:10.1037/0022-0167.49.3.376 · Cited by 26
Reliability and Internal Consistency Findings from the C-SEI
Adamson, K. A., Parsons, M. E., Hawkins, K., et al. (2011) · Journal of Nursing Education · doi:10.3928/01484834-20110715-02 · Cited by 26
Gutiérrez-Becker, B., Cosı́o, F. A., Huerta, M. E. G., Benavides-Serralde, J. A., Camargo-Marín, L., & Medina-Bañuelos, V. (2013) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-013-1082-1 · Cited by 26
Strand, P. S. (2002) · Journal of Clinical Child & Adolescent Psychology · doi:10.1207/s15374424jccp3101_03 · Cited by 26
Characterization of a normal control group: Are they healthy?
Aine, C. J., Sanfratello, L., Adair, J. C., Knoefel, J. E., Qualls, C., Lundy, S. L., Caprihan, A., Stone, D., & Stephen, J. M. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.09.025 · Cited by 26
Maternal antecedents of delay ability in young children
SILVERMAN, I. W., & Ippolito, M. F. (1995) · Journal of Applied Developmental Psychology · doi:10.1016/0193-3973(95)90005-5 · Cited by 28
Ramirez, J., Scott, C. J. M., McNeely, A. A., Berezuk, C., Gao, F., Szilagyi, G. M., & Black, S. E. (2014) · Journal of Visualized Experiments · doi:10.3791/50887 · Cited by 26
Endoscopic Assessment of Vocal Fold Movements during Cough
Britton, D., Yorkston, K. M., Eadie, T. L., et al. (2012) · Annals of Otology Rhinology & Laryngology · doi:10.1177/000348941212100105 · Cited by 26
Nacewicz, B. M., Angelos, L., Dalton, K. M., Fischer, R., Anderle, M. J., Alexander, A. L., & Davidson, R. J. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.08.090 · Cited by 27
Zelaya, F. O., Zois, E., Muller-Pollard, C., Lythgoe, D. J., Lee, S., Andrews, C., Smart, T. S., Conrod, P., Vennart, W., Williams, S., et al. (2011) · Magnetic Resonance Materials in Physics Biology and Medicine · doi:10.1007/s10334-011-0293-4 · Cited by 26
The reliability of physical performance measures in peripheral neuropathy
Manor, B. D., Doherty, A., & Li, L. (2008) · Gait & Posture · doi:10.1016/j.gaitpost.2008.01.004 · Cited by 26
Incremental validity of the Structured Interview for the Five‐Factor Model of Personality (SIFFM)
Stepp, S. D., Trull, T. J., Burr, R. M., Wolfenstein, M., & Vieth, A. Z. (2005) · European Journal of Personality · doi:10.1002/per.565 · Cited by 26
Jenkins, N. D. M., Buckner, S. L., Cochrane, K. C., Bergstrom, H. C., Palmer, T. B., Johnson, G. O., Schmidt, R. J., Housh, T. J., & Cramer, J. T. (2014) · Experimental Gerontology · doi:10.1016/j.exger.2014.04.015 · Cited by 30
Stability and change in adult personality: genetic and environmental components
Pedersen, N. L., & Reynolds, C. A. (1998) · European Journal of Personality · doi:10.1002/(sici)1099-0984(1998090)12:5 · Cited by 86
Bonin, P., Guillemard, D., & Méot, A. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0279-9 · Cited by 26
McGrath, C. P. J., McMillan, A. S., ZHU, H. W., & Li, L. S. W. (2009) · Journal of Oral Rehabilitation · doi:10.1111/j.1365-2842.2009.01941.x · Cited by 26
Carbonaro, S., Villines, T. C., Hausleiter, J., et al. (2009) · Journal of cardiovascular computed tomography · doi:10.1016/j.jcct.2009.05.010 · Cited by 26
Hays, R. T., Jacobs, J. W., Prince, C., et al. (1992) · International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0202_5 · Cited by 26
Human alteration of the rural landscape: Variations in visual perception
Ballester, V. C., Torres-Sibille, A. D. C., Cloquell-Ballester, V.-A., et al. (2011) · Environmental Impact Assessment Review · doi:10.1016/j.eiar.2011.03.002 · Cited by 26
A Novel Sonographic Method of Measuring Patellar Tendon Length
Gellhorn, A. C., Morgenroth, D. C., & Goldstein, B. (2012) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2012.01.020 · Cited by 26
Self-monitoring and personality: A behavioural-genetic study
Wolf, H., Spinath, F. M., Riemann, R., & Angleitner, A. (2009) · Personality and Individual Differences · doi:10.1016/j.paid.2009.01.040 · Cited by 27
Reliability of Lumbar Lordosis Measurement in Patients With Spondylolisthesis
Hwang, J.-H., Modi, H. N., Suh, S.-W., et al. (2010) · Spine · doi:10.1097/brs.0b013e3181c9a75f · Cited by 26
van der Mei, I. A. F., Blizzard, L., Stankovich, J. M., Ponsonby, A., & Dwyer, T. (2002) · Journal of Photochemistry and Photobiology B Biology · doi:10.1016/s1011-1344(02)00331-7 · Cited by 26
Hart, D. L., Deutscher, D., Crane, P. K., & Wang, Y.-C. (2009) · Quality of Life Research · doi:10.1007/s11136-009-9517-8 · Cited by 26
Hutzler, Y. S., Rodríguez, B. L., Láiz, N. M., Díez, I., & Barak, S. (2013) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2013.09.015 · Cited by 25
Sigurðardóttir, Þ., Steingrímsdóttir, Þ., Árnason, Á., & Bø, K. (2009) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1080/00016340903093567 · Cited by 26
The Eriksonian Life Story: Developmental Scripts and Psychosocial Adaptation
Wilt, J. A., Cox, K. S., & McAdams, D. P. (2010) · Journal of Adult Development · doi:10.1007/s10804-010-9093-8 · Cited by 26
Reliable scar scoring system to assess photographs of burn patients
Mecott, G. A., Finnerty, C. C., Herndon, D. N., et al. (2014) · Journal of Surgical Research · doi:10.1016/j.jss.2014.10.055 · Cited by 26
The Impact of Baseline Trend Control on Visual Analysis of Single-Case Data
Mercer, S. H., & Sterling, H. E. (2011) · Journal of School Psychology · doi:10.1016/j.jsp.2011.11.004 · Cited by 27
Fulford, D., Pearson, R., Stuart, B. K., Fisher, M., Mathalon, D. H., Vinogradov, S., & Loewy, R. L. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.07.047 · Cited by 26
Exploring the Link Between Rating Congruence and Managerial Effectiveness
Brutus, S., Fleenor, J. W., & Tisak, J. (2009) · Canadian Journal of Administrative Sciences / Revue Canadienne des Sciences de l Administration · doi:10.1111/j.1936-4490.1999.tb00691.x · Cited by 25
Assessment of aerobic capacity and walking economy of unilateral transfemoral amputees
Gjøvaag, T. F., Starholm, I. M., Mirtaheri, P., Hegge, F. W., & Skjetne, K. (2013) · Prosthetics and Orthotics International · doi:10.1177/0309364613490444 · Cited by 26
Basanta, M., Ibrahim, B., Douce, D. S., et al. (2012) · Journal of Breath Research · doi:10.1088/1752-7155/6/2/026002 · Cited by 26
Patel, S., Toms, A. P., Rehman, J. M., et al. (2011) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.j.00709 · Cited by 26
MacDermid, J. C., Arumugam, V., Vincent, J. I., et al. (2015) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-015-0578-2 · Cited by 26
Beppu, T., Fujiwara, S., Nishimoto, H., Koeda, A., Narumi, S., Mori, K., Ogasawara, K., & Sasaki, M. (2012) · Journal of Neurology · doi:10.1007/s00415-011-6402-5 · Cited by 25
Cardio-ventilatory coupling in young healthy resting subjects
Friedman, L., Dick, T. E., Jacono, F. J., Loparo, K. A., Yeganeh, A., Fishman, M., Wilson, C. G., & Strohl, K. P. (2012) · Journal of Applied Physiology · doi:10.1152/japplphysiol.01424.2010 · Cited by 28
Hoffart, A., Sexton, H. C., Nordahl, H. M., & Stiles, T. C. (2005) · Psychotherapy Research · doi:10.1080/10503300500091702 · Cited by 26
Baumann, M., Baumann, C., Bihan, E. L., et al. (2008) · BMC Health Services Research · doi:10.1186/1472-6963-8-244 · Cited by 26
Sakız, H., Sart, Z. H., Börkan, B., Korkmaz, B., & Babür, N. (2015) · Learning Disabilities Research and Practice · doi:10.1111/ldrp.12060 · Cited by 32
Durmuş, B., Ay, L., Hokken‐Koelega, A. C. S., Raat, H., Albert Hofman, D., Steegers, E. A. P., & Jaddoe, V. W. V. (2011) · European Journal of Epidemiology · doi:10.1007/s10654-010-9544-3 · Cited by 27
School Site Visits for Community-Based Participatory Research on Healthy Eating
Patel, A. I., Bogart, L. M., Uyeda, K. E., Martı́nez, H., Knizewski, R., Ryan, G. W., & Schuster, M. A. (2009) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2009.08.009 · Cited by 26
Davis, A. M., Palaganas, M., Badley, E. M., et al. (2011) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2010.149211 · Cited by 26
Barone, L., Giudice, M. D., Fossati, A., Manaresi, F., Perinetti, B. A., Colle, L., & Veglia, F. (2009) · International Journal of Behavioral Development · doi:10.1177/0165025409103134 · Cited by 26
Now You Hear Me, Now You Don't: Eyelid Closures as an Indicator of Auditory Task Disengagement
Ong, J. L., Asplund, C. L., Chia, T. T. Y., & Chee, M. W. L. (2013) · SLEEP · doi:10.5665/sleep.3218 · Cited by 25
Chan, S. G., Ho, S. C., Kreiger, N., Darlington, G., Adlaf, E. M., So, K. F., & Chong, P. Y. Y. (2008) · Journal of Nutrition · doi:10.1093/jn/138.3.567 · Cited by 26
Smith, D. C., Lanesskog, D., Cleeland, L., Motl, R. W., Weikert, M., & Dlugonski, D. (2012) · Health & Social Work · doi:10.1093/hsw/hls011 · Cited by 26
White‐Traut, R. C., Wink, T., Minehart, T., & Holditch‐Davis, D. (2012) · Newborn and Infant Nursing Reviews · doi:10.1053/j.nainr.2012.06.005 · Cited by 28
Herrmann‐Werner, A., Holderried, F., Schäffeler, N., Weyrich, P., Rießen, R., Zipfel, S., & Celebi, N. (2013) · BMC Medical Education · doi:10.1186/1472-6920-13-15 · Cited by 28
A Social Network Approach to Peer Assessment: Improving Predictive Validity
Luria, G., & Kalish, Y. (2013) · Human Resource Management · doi:10.1002/hrm.21541 · Cited by 26
Pancani, S., Rowson, J., Tindale, W. B., Heron, N., Langley, J., McCarthy, A. D., Quinn, A., Reed, H., Stanton, A., Shaw, P. J., et al. (2015) · Clinical Biomechanics · doi:10.1016/j.clinbiomech.2015.11.010 · Cited by 26
More emotional facial expressions during episodic than during semantic autobiographical retrieval
Haj, M. E., Antoine, P., & Nandrino, J. (2015) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-015-0397-9 · Cited by 30
Bioelectrical impedance spectroscopy to estimate fluid balance in critically ill patients
Dewitte, A., Carles, P., Joannès-Boyau, O., Fleureau, C., Rozé, H., Combe, C., & Ouattara, A. (2015) · Journal of Clinical Monitoring and Computing · doi:10.1007/s10877-015-9706-7 · Cited by 26
Talasan, H., Scheiman, M. M., Li, X., et al. (2016) · Journal of Vision · doi:10.1167/16.1.7 · Cited by 26
Developing Symbolic Interpretation Through Literary Argumentation
Sosa, T., Hall, A., Goldman, S. R., & Lee, C. D. (2015) · Journal of the Learning Sciences · doi:10.1080/10508406.2015.1124040 · Cited by 32
Validation of the Dutch version of the critical‐care pain observation tool
Stilma, W., Rijkenberg, S., Feijen, H. M., Maaskant, J. M., & Endeman, H. (2015) · Nursing in Critical Care · doi:10.1111/nicc.12225 · Cited by 26
Ambulatory assessment of shoulder abduction strength curve using a single wearable inertial sensor
Picerno, P., Viero, V., Donati, M., Triossi, T., Tancredi, V., & Melchiorri, G. (2015) · The Journal of Rehabilitation Research and Development · doi:10.1682/jrrd.2014.06.0146 · Cited by 26
Gorney, A. M., Blau, S. R., Dohse, C. S., Griffith, E. H., Williams, K. D., Lim, J., Kňazovický, D., Lascelles, B. D. X., & Olby, N. J. (2016) · Journal of Veterinary Internal Medicine · doi:10.1111/jvim.13913 · Cited by 26
CT Perfusion in Acute Stroke Predicts Final Infarct Volume- Inter-observer Study
Shankar, J., Langlands, G., Doucette, S. P., et al. (2016) · Canadian Journal of Neurological Sciences / Journal Canadien des Sciences Neurologiques · doi:10.1017/cjn.2015.349 · Cited by 26
Burwell, S. J., Malone, S. M., & Iacono, W. G. (2016) · Psychophysiology · doi:10.1111/psyp.12646 · Cited by 25
Terada, T., Loehr, S., Guigard, E., McCargar, L. J., Bell, G. J., Senior, P. A., & Boulé, N. G. (2014) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2013.0355 · Cited by 26
The Triangle Model of Congenital Cervical Stenosis
Jenkins, T. J., Mai, H. T., Burgmeier, R. J., et al. (2015) · Spine · doi:10.1097/brs.0000000000001227 · Cited by 26
Using Multi-Informant Designs to Address Key Informant and Common Method Bias
Homburg, C., Klarmann, M., & Totzek, D. (2012) · Gabler Verlag eBooks · doi:10.1007/978-3-8349-3722-3_4 · Cited by 26
Dagnone, D., Hall, A. K., Sebok‐Syer, S. S., Klinger, D. A., Woolfrey, K. G. H., Davison, C., Ross, J., McNeil, G., & Moore, S. W. (2016) · Canadian Medical Education Journal · doi:10.36834/cmej.36682 · Cited by 25
Evaluating a visual-acoustic simulation for wind park assessment
Manyoky, M., Hayek, U. W., Pieren, R., et al. (2016) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2016.03.013 · Cited by 26
McLeod, B. D., Southam‐Gerow, M. A., Rodríguez, A., Quinoy, A. M., Arnold, C. C., Kendall, P. C., & Weisz, J. R. (2016) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2016.1253018 · Cited by 26
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
