The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 3,001–3,100 of 23,883
Lurie, J. D., Tosteson, A. N. A., Tosteson, T. D., et al. (2008) · Spine · doi:10.1097/brs.0b013e31816c8379 · Cited by 95
Cross Cultural Management Research
Gerhart, B. A. (2008) · International Journal of Cross Cultural Management · doi:10.1177/1470595808096669 · Cited by 94
Nybakk, E., Crespell, P., Hansen, E., & Lunnan, A. (2008) · Forest Ecology and Management · doi:10.1016/j.foreco.2008.09.040 · Cited by 95
Couvelard, A., Deschamps, L., Ravaud, P., Baron, G., Sauvanet, A., Hentic, O., Colnot, N., Paradis, V., Belghiti, J., Bédossa, P., et al. (2008) · Modern Pathology · doi:10.1038/modpathol.2008.177 · Cited by 92
Andreas Schäfer, A., Forstmann, B. U., Neumann, J., Wharton, S., Mietke, A., Bowtell, R. W., & Turner, R. (2011) · Human Brain Mapping · doi:10.1002/hbm.21404 · Cited by 96
Harris, K. K., Dawson, J., Jones, L. D., Beard, D. J., & Price, A. J. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003365 · Cited by 98
Falla, D., Campbell, C. D., Fagan, A., et al. (2003) · Manual Therapy · doi:10.1016/s1356-689x(03)00008-0 · Cited by 95
Imprinting Effects of Founding Core Teams on HR Values in New Ventures
Leung, A., der Foo, M., & Chaturvedi, S. (2013) · Entrepreneurship Theory and Practice · doi:10.1111/j.1540-6520.2012.00532.x · Cited by 95
Palosaari, K., Vuotila, J., Takalo, R., et al. (2006) · Lara D. Veeken · doi:10.1093/rheumatology/kel137 · Cited by 95
Jokubkiene, L., Sladkevicius, P., & Valentin, L. (2007) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.3922 · Cited by 94
Validation of bioelectrical impedance analysis in patients with amyotrophic lateral sclerosis
Desport, J., Preux, P., Bouteloup‐Demange, C., Clavelou, P., Beaufrère, B., Bonnet, C., & Couratier, P. (2003) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/77.5.1179 · Cited by 95
Schubert, T. W., Zickfeld, J., Seibt, B., & Fiske, A. P. (2016) · Cognition & Emotion · doi:10.1080/02699931.2016.1268998 · Cited by 102
Evaluation of the uniformity of fit of general outcome prediction models
Moreno, R. P., Apolone, G., & Miranda, D. R. (1998) · Intensive Care Medicine · doi:10.1007/s001340050513 · Cited by 92
A multidimensional approach to the genetics of schizophrenia
Dworkin, R. H., Lenzenweger, M. F., Moldin, S. O., et al. (1988) · American Journal of Psychiatry · doi:10.1176/ajp.145.9.1077 · Cited by 95
Pexsters, A., Luts, J., Van Schoubroeck, D., Bottomley, C., Van Calster, B., Van Huffel, S., Abdallah, Y., D'Hooghe, T., Lees, C., Timmerman, D., et al. (2011) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.8884 · Cited by 96
Adair, W. L., Hideg, I., & Spence, J. R. (2013) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022113492894 · Cited by 119
Turnover in health care: the mediating effects of employee engagement
Collini, S. A., Guidroz, A. M., & Perez, L. M. (2013) · Journal of Nursing Management · doi:10.1111/jonm.12109 · Cited by 138
TODDLERS' CONCENTRATION: DOES MATERNAL DEPRESSION MAKE A DIFFERENCE?
Breznitz, Z., & Friedman, S. L. (1988) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.1988.tb00715.x · Cited by 94
Warming, S., Precht, D. H., Suadicani, P., et al. (2008) · Applied Ergonomics · doi:10.1016/j.apergo.2008.04.021 · Cited by 94
Morton, F. B., Lee, P. C., Buchanan‐Smith, H. M., Brosnan, S. F., Thierry, B., Paukner, A., de Waal, F. Β. Μ., Widness, J., Essler, J. L., & Weiß, A. (2013) · Journal of comparative psychology · doi:10.1037/a0031723 · Cited by 94
de Nijs, P. F. A., Ferdinand, R. F., de Bruin, E. I., Dekker, M. C. J., van Duijn, C. M., & Verhulst, D. C. (2004) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-004-0405-z · Cited by 94
Weinstein, T. A. R., & Capitanio, J. P. (2008) · Animal Behaviour · doi:10.1016/j.anbehav.2008.01.024 · Cited by 94
Haugen, I. K., Lillegraven, S., Slatkowsky-Christensen, B., Haavardsholm, E. A., Sesseng, S., Kvien, T. K., van der Heijde, D. M. F. M., & Bøyesen, P. (2011) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2010.144527 · Cited by 96
Sources of variation in emotional awareness: Age, gender, and socioeconomic status
Mankus, A. M., Boden, M. T., & Thompson, R. J. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2015.09.043 · Cited by 109
Nath, P., & Ramanathan, R. (2015) · International Journal of Production Economics · doi:10.1016/j.ijpe.2015.09.040 · Cited by 94
Lantian, A., Bagneux, V., Delouvée, S., & Gauvrit, N. (2021) · Applied Cognitive Psychology · doi:10.1002/acp.3790 · Cited by 124
Latreche, A., Kelaiaia, R., Chemori, A., & Kerboua, A. (2023) · Measurement · doi:10.1016/j.measurement.2023.112826 · Cited by 111
Cagol, A., Benkert, P., Melie‐García, L., Schaedelin, S. A., Leber, S., Tsagkas, C., Baraković, M., Galbusera, R., Lu, P., Weigel, M., et al. (2023) · Neurology · doi:10.1212/wnl.0000000000207768 · Cited by 102
Radiographic Fracture Assessments: Which Ones Can We Reliably Make?
Martin, J., Marsh, J. L., Nepola, J. V., et al. (2000) · Journal of Orthopaedic Trauma · doi:10.1097/00005131-200008000-00001 · Cited by 94
Katz, I. M., & Campbell, J. D. (1994) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.67.3.513 · Cited by 94
Social desirability and consensual validity of personality traits
Konstabel, K., Aavik, T., & Allïk, J. (2006) · European Journal of Personality · doi:10.1002/per.593 · Cited by 94
Infiltrants Inhibit Progression of Natural Caries Lesions in vitro
Paris, S., & Meyer‐Lueckel, H. (2010) · Journal of Dental Research · doi:10.1177/0022034510376040 · Cited by 94
Methodological pitfalls in early detection studies – the NAPE Lecture 2002
Friis, S., Larsen, T. K., Melle, I. S., Opjordsmoen, S., Johannessen, J. O., Haahr, U., Simonsen, E. H., Rund, B. R., Vaglum, P., & McGlashan, T. H. (2003) · Acta Psychiatrica Scandinavica · doi:10.1034/j.1600-0447.2003.02600.x · Cited by 94
Cognitive Differences in Proficient and Nonproficient Essay Scorers
Wolfe, E. W., KAO, C.-W., & Ranney, M. A. (1998) · Written Communication · doi:10.1177/0741088398015004002 · Cited by 94
Sexual Predator Civil Commitment: A Comparison of Selected and Released Offenders
Levenson, J. S. (2004) · International Journal of Offender Therapy and Comparative Criminology · doi:10.1177/0306624x04265089 · Cited by 94
The Development of an Assessment To Evaluate Behavioral Factors Associated With Falling
Clemson, L. M., Cumming, R. G., & Heard, R. C. (2003) · American Journal of Occupational Therapy · doi:10.5014/ajot.57.4.380 · Cited by 94
Mueller, T. L., Stauber, M., Kohler, T., Eckstein, F., Müller, R., & van Lenthe, H. (2008) · Bone · doi:10.1016/j.bone.2008.10.045 · Cited by 94
Sung, J. K., Jee, W.-H., Jung, J., et al. (2014) · Radiology · doi:10.1148/radiol.13130399 · Cited by 94
Smeets, R., Hijdra, H. J., Kester, A. D. M., Hitters, M. W. M. G. C., & Knottnerus, J. A. (2006) · Clinical Rehabilitation · doi:10.1177/0269215506070698 · Cited by 94
Are Mexicans more or less sociable than Americans? Insights from a naturalistic observation study
Ramírez‐Esparza, N., Mehl, M. R., Bermúdez, J. Á., & Pennebaker, J. W. (2008) · Journal of Research in Personality · doi:10.1016/j.jrp.2008.09.002 · Cited by 94
Reliability of Three Single-Leg Hop Tests
Booher, L. D., Hench, K. M., Worrell, T. W., & Stikeleather, J. (1993) · Journal of Sport Rehabilitation · doi:10.1123/jsr.2.3.165 · Cited by 93
Behavioral Observation and Coding
Heyman, R. E., Lorber, M. F., Eddy, J. M., & West, T. V. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9780511996481.018 · Cited by 100
Fatima, R.-S., Kelly, S., Drake, K. M., et al. (1998) · Community Dentistry And Oral Epidemiology · doi:10.1111/j.1600-0528.1999.tb02023.x · Cited by 94
Reliability of Physical Performance Measures in Nursing Home Residents With Alzheimer's Disease
Tappen, R. M., Roach, K. E., Büchner, D. M., Barry, C. D., & Edelstein, J. (1997) · The Journals of Gerontology Series A · doi:10.1093/gerona/52a.1.m52 · Cited by 94
Lee, J. A., Halpern, E. M., Lovblom, L. E., Yeung, E. F. Y., Bril, V., & Perkins, B. A. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086515 · Cited by 94
Google Glass for Documentation of Medical Findings: Evaluation in Forensic Medicine
Albrecht, U., von Jan, U., Kuebler, J. F., Zoeller, C., Lacher, M., Muensterer, O. J., Ettinger, M., Klintschar, M., & Hagemeier, L. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3225 · Cited by 93
Brand, R. J., Shallcross, W. L., Sabatos‐DeVito, M., & Massie, K. P. (2007) · Infancy · doi:10.1111/j.1532-7078.2007.tb00223.x · Cited by 93
Assessing collaboration in a wiki: The reliability of university students' peer assessment
Wever, B. D., Keer, H. V., Schellens, T., & Valcke, M. (2011) · The Internet and Higher Education · doi:10.1016/j.iheduc.2011.07.003 · Cited by 93
The Reliability and Vatidity of Pain Threshold Measurements in Osteoarthritis of the Knee
Wessel, J. (1995) · Scandinavian Journal of Rheumatology · doi:10.3109/03009749509100881 · Cited by 93
Marin, M. M., Lampatz, A., Wandl, M., & Leder, H. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00536 · Cited by 100
Schmidt, M. F., Storrs, J. M., Freeman, K. B., Jack, C. R., Turner, S. T., Griswold, M. E., & Mosley, T. H. (2018) · Human Brain Mapping · doi:10.1002/hbm.24017 · Cited by 98
Weighted Stochastic Block Models of the Human Connectome across the Life Span
Faskowitz, J., Yan, X., Zuo, X., & Sporns, O. (2018) · Scientific Reports · doi:10.1038/s41598-018-31202-1 · Cited by 95
Xue, J., Guo, H., Gao, Y., Wang, X., Cui, H., Chen, Z., Wang, B., & Xiang, J. (2019) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2019.00326 · Cited by 94
How do you feel? Using natural language processing to automatically rate emotion in psychotherapy
Tanana, M., Soma, C. S., Kuo, P. B., Bertagnolli, N. M., Dembe, A., Pace, B. T., Srikumar, V., Atkins, D. C., & Imel, Z. E. (2021) · Behavior Research Methods · doi:10.3758/s13428-020-01531-z · Cited by 102
Interrater Reliability of Scoring of Pain Drawings in a Self-Report Health Survey
Lacey, R. J., Lewis, M., Jordan, K. P., Jinks, C., & Sim, J. (2005) · Spine · doi:10.1097/01.brs.0000174274.38485.ee · Cited by 93
A Reliability Study of Measurement Techniques to Determine Static Scapular Position
Gibson, M. H., Goebel, G. V., Jordan, T., et al. (1995) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1995.21.2.100 · Cited by 93
Wideband Middle Ear Power Measurement in Infants and Children
Hunter, L. L., Tubaugh, L., Jackson, A., et al. (2008) · Journal of the American Academy of Audiology · doi:10.3766/jaaa.19.4.4 · Cited by 93
Finlayson, O., John, V., Yeung, R., et al. (2012) · Stroke · doi:10.1161/strokeaha.112.665208 · Cited by 93
Sheets, E. S., & Craighead, W. E. (2014) · Behaviour Research and Therapy · doi:10.1016/j.brat.2014.09.001 · Cited by 93
Spalding, S. J., Kwoh, C. K., Boudreau, R. M., et al. (2008) · Arthritis Research & Therapy · doi:10.1186/ar2360 · Cited by 93
Teacher practices as predictors of children's classroom social preference
Mikami, A. Y., Griggs, M. S., Reuland, M. M., & Gregory, A. (2011) · Journal of School Psychology · doi:10.1016/j.jsp.2011.08.002 · Cited by 94
Use of the EQ-5D among patients suffering from dementia
Ankri, J., Beaufils, B., Novella, J., Morrone, I., Guillemin, F., Jolly, D., Ploton, L., & Blanchard, F. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00175-6 · Cited by 94
Stolarova, M., Wolf, C., Rinker, T., & Brielmann, A. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00509 · Cited by 93
Parent-Child Agreement on Ratings of Anxiety in Children
Engel, N., Rodrigue, J., & Geffken, G. (1994) · Psychological Reports · doi:10.2466/pr0.1994.75.3.1251 · Cited by 105
Anxiety-like behavior in Rett syndrome: characteristics and assessment by anxiety scales
Barnes, K. V., Coughlin, F., O’Leary, H. M., Bruck, N. M., Bazin, G. A., Beinecke, E. B., Walco, A. C., Cantwell, N. G., & Kaufmann, W. E. (2015) · Journal of Neurodevelopmental Disorders · doi:10.1186/s11689-015-9127-4 · Cited by 95
Reliability of the Life Chart Schedule for assessment of the long-term course of schizophrenia
Susser, E., Finnerty, M. T., Mojtabai, R., Yale, S. A., Conover, S., Goetz, R. R., & AMADOR, X. F. (2000) · Schizophrenia Research · doi:10.1016/s0920-9964(99)00088-2 · Cited by 93
Ottosson, H., Bodlund, O., Ekselius, L., Grann, M., von Knorring, L., Kullgren, G., Lindström, E., & Söderberg, S. (1998) · European Psychiatry · doi:10.1016/s0924-9338(98)80013-8 · Cited by 92
Boiteau, M., Malouin, F., & Richards, C. L. (1995) · Physical Therapy · doi:10.1093/ptj/75.9.796 · Cited by 93
Kupper, Z., Ramseyer, F., Hoffmann, H., Kalbermatten, S., & Tschacher, W. (2010) · Schizophrenia Research · doi:10.1016/j.schres.2010.03.032 · Cited by 103
Haywood, K. L. (2002) · British journal of rheumatology · doi:10.1093/rheumatology/41.11.1295 · Cited by 93
Reliability of Measuring Anterior Laxity of the Knee Joint Using a Knee Ligament Arthrometer
Hanten, W. P., & Pace, M. B. (1987) · Physical Therapy · doi:10.1093/ptj/67.3.357 · Cited by 93
Huth, K. C., Neuhaus, K. W., Gygax, M., Bücher, K., Crispin, A., Paschos, E., Hickel, R., & Lussi, A. (2008) · Journal of Dentistry · doi:10.1016/j.jdent.2008.08.013 · Cited by 93
Measurement of passive ankle dorsiflexion: Procedure and reliability
Moseley, A. M., & Adams, R. D. (1991) · Australian Journal of Physiotherapy · doi:10.1016/s0004-9514(14)60540-7 · Cited by 93
Does hunger influence judgments of female physical attractiveness?
Swami, V., & Tovée, M. J. (2006) · British Journal of Psychology · doi:10.1348/000712605x80713 · Cited by 93
Optimizing the Deployment of Public Access Defibrillators
Chan, T. C. Y., Demirtas, D., & Kwon, R. H. (2016) · Management Science · doi:10.1287/mnsc.2015.2312 · Cited by 96
Antisocial personality disorder, conduct disorder, and substance abuse in schizophrenia.
Mueser, K. T., Drake, R. E., Ackerson, T. H., Alterman, A. I., Miles, K. M., & Noordsy, D. L. (1997) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.106.3.473 · Cited by 93
Reliability of EMG spectral parameters in repeated measurements of back muscle fatigue
Elfving, B., Németh, G., Arvidsson, I., et al. (1999) · Journal of Electromyography and Kinesiology · doi:10.1016/s1050-6411(98)00049-2 · Cited by 93
Biaggi, P., Jedrzkiewicz, S., Gruner, C., Meineri, M., Karski, J. M., Vegas, A., Tanner, F. C., Rakowski, H., Ivanov, J., David, T. E., et al. (2012) · Journal of the American Society of Echocardiography · doi:10.1016/j.echo.2012.03.010 · Cited by 89
An examination of the Motivational Interviewing Treatment Integrity code
Pierson, H. M., Hayes, S. C., Gifford, E. V., Roget, N. A., Padilla, M., Bissett, R. T., Berry, K., Kohlenberg, B. S., Rhode, R., & L.Fisher, G. (2006) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2006.07.001 · Cited by 92
Kemp, J. L., Schache, A. G., Makdissi, M., Sims, K., & Crossley, K. M. (2012) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2012.11.887 · Cited by 92
Naylor, J., Hayen, A. D., Davidson, E. R., Hackett, D., Harris, I. A., Kamalasena, G., & Mittal, R. (2014) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-15-235 · Cited by 93
Nestler, S., Egloff, B., Küfner, A. C. P., & Back, M. D. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0029461 · Cited by 93
Петитта, Л., Probst, T. M., Barbaranelli, C., & Ghezzi, V. (2016) · Accident Analysis & Prevention · doi:10.1016/j.aap.2016.11.012 · Cited by 92
Personalized Automatic Estimation of Self-Reported Pain Intensity from Facial Expressions
Lopez-Martinez, D., Rudovic, O. O., & Picard, R. W. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.286 · Cited by 99
Nagai, K., Miyamato, T., Okamae, A., Tamaki, A., Fujioka, H., Wada, Y., Uchiyama, Y., Shinmura, K., & Domen, K. (2018) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2018.02.005 · Cited by 93
Mapping population-based structural connectomes
Zhang, Z., Descoteaux, M., Zhang, J., Girard, G., Chamberland, M., Dunson, D. B., Srivastava, A., & Hongtu Zhu, B. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.12.064 · Cited by 93
Iwata, M., Yamamoto, A., Matsuo, S., Hatano, G., Miyazaki, M., Fukaya, T., Fujiwara, M., Asai, Y., & Suzuki, S. (2019) · Europe PMC (PubMed Central) · Cited by 92
Teachers’ first year in the profession: the power of high-quality support
Thomas, L., Tuytens, M., Moolenaar, N. M., Devos, G., Kelchtermans, G., & Vanderlinde, R. (2019) · Teachers and Teaching · doi:10.1080/13540602.2018.1562440 · Cited by 121
Simpson, E. L., Eckert, L., Gadkari, A. S., Mallya, U. G., Yang, M., Nelson, L. M., Brown, T. M., Reaney, M., Mahajan, P. S., Guillemin, I., et al. (2019) · BMC Dermatology · doi:10.1186/s12895-019-0095-3 · Cited by 91
Incidence and Location of Atelectasis Developed During Bronchoscopy Under General Anesthesia
Sagar, A.-E. S., Sabath, B. F., Eapen, G. A., et al. (2020) · CHEST Journal · doi:10.1016/j.chest.2020.05.565 · Cited by 92
You Are Known by How You Vlog: Personality Impressions and Nonverbal Behavior in YouTube
Biel, J.-I., Aran, O., & Gática-Pérez, D. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14160 · Cited by 92
Wood, A. J., Raynes‐Greenow, C. H., Carberry, A. E., & Jeffery, H. E. (2013) · Journal of Paediatrics and Child Health · doi:10.1111/jpc.12119 · Cited by 92
Boyd, R. N., Sakzewski, L., Ziviani, J. M., Abbott, D. F., Badawy, R. A. B., Gilmore, R., Provan, K., Tournier, J., Macdonell, R. A. L., & Jackson, G. D. (2010) · BMC Neurology · doi:10.1186/1471-2377-10-4 · Cited by 90
Adoption of free/libre open source software in public organizations: factors of impact
Rossi, B., Russo, B., & Succi, G. (2012) · Information Technology and People · doi:10.1108/09593841211232677 · Cited by 92
Exploring the Effects of Value Diversity on Team Effectiveness
Woehr, D. J., Arciniega, L. M., & Poling, T. L. (2012) · Journal of Business and Psychology · doi:10.1007/s10869-012-9267-4 · Cited by 92
Social Category Salience Moderates the Effect of Diversity Faultlines on Information Elaboration
Meyer, B. M., Shemla, M., & Schermuly, C. C. (2011) · Small Group Research · doi:10.1177/1046496411398396 · Cited by 92
Kracht, D., Shroff, R. C., Baig, S., Doyon, A., Jacobi, C., Zeller, R., Querfeld, U., Schaefer, F. S., Wühl, E., Schmidt, B. M. W., et al. (2011) · American Journal of Hypertension · doi:10.1038/ajh.2011.147 · Cited by 91
ARCHITECTS PREDICT LAY EVALUATIONS OF LARGE CONTEMPORARY BUILDINGS: WHOSE CONCEPTUAL PROPERTIES?
Brown, G., & Gifford, R. (2001) · Journal of Environmental Psychology · doi:10.1006/jevp.2000.0176 · Cited by 92
Observational Measures of Parenting in Anxious and Nonanxious Mothers: Does Type of Task Matter?
Ginsburg, G. S., Grover, R. L., Cord, J. J., & Ialongo, N. (2006) · Journal of Clinical Child & Adolescent Psychology · doi:10.1207/s15374424jccp3502_16 · Cited by 93
Affective and social-behavioral correlates of physical and social anhedonia in schizophrenia.
Blanchard, J. J., Bellack, A. S., & Mueser, K. T. (1994) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.103.4.719 · Cited by 92
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
