The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
The job demands-resources model of burnout.
Demerouti, E., Bakker, A. B., Nachreiner, F., & Schaufeli, W. B. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.3.499 · Cited by 13,643
Design and validation of a histological scoring system for nonalcoholic fatty liver disease†
Kleiner, D. E., Brunt, E. M., Van Natta, M., Behling, C., Contos, M. J., Cummings, O. W., Ferrell, L. D., Liu, Y., Torbenson, M. S., Unalp‐Arida, A., et al. (2005) · Hepatology · doi:10.1002/hep.20701 · Cited by 10,791
Yushkevich, P. A., Piven, J., Hazlett, H. C., Smith, R. G., Ho, S., Gee, J. C., & Gerig, G. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.01.015 · Cited by 8,868
Cicchetti, D. V. (1994) · Psychological Assessment · doi:10.1037/1040-3590.6.4.284 · Cited by 9,349
Social Relationships and Mortality Risk: A Meta-analytic Review
Holt-Lunstad, J., Smith, T. B., & Layton, J. B. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000316 · Cited by 7,272
“13) and for continuous variables (mean intraclass correlation [26] = 0.”
BackgroundASSESSING THE WORK ENVIRONMENT FOR CREATIVITY.
AMABILE, T. M., CONTI, R., COON, H., LAZENBY, J., & HERRON, M. (1996) · Academy of Management Journal · doi:10.2307/256995 · Cited by 6,029
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
QUANTIFYING TEST-RETEST RELIABILITY USING THE INTRACLASS CORRELATION COEFFICIENT AND THE SEM
Weir, J. P. (2005) · The Journal of Strength and Conditioning Research · doi:10.1519/15184.1 · Cited by 4,864
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
Measures of Reliability in Sports Medicine and Science
Hopkins, W. G. (2000) · Sports Medicine · doi:10.2165/00007256-200030010-00001 · Cited by 4,635
Zwanenburg, A., Vallières, M., Abdalah, M. A., Aerts, H. J. W. L., Andrearczyk, V., Apte, A., Ashrafinia, S., Bakas, S., Beukinga, R. J., Boellaard, R., et al. (2020) · Radiology · doi:10.1148/radiol.2020191145 · Cited by 4,177
Roos, E. M., Roos, H. P., Lohmander, S. L., Ekdahl, C. S., & Beynnon, B. D. (1998) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1998.28.2.88 · Cited by 3,989
DPABI: Data Processing & Analysis for (Resting-State) Brain Imaging
Yan, C., Wang, X., Zuo, X., & Zang, Y. (2016) · Neuroinformatics · doi:10.1007/s12021-016-9299-4 · Cited by 3,935
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608
Aggregate Confusion: The Divergence of ESG Ratings
Berg, F., Kölbel, J. F., & Rigobon, R. (2022) · Review of Finance · doi:10.1093/rof/rfac033 · Cited by 3,444
The ripple effect: emotional contagion and its influence on group behavior
Barsade, S. G. (2002) · Administrative Science Quarterly · doi:10.2307/3094912 · Cited by 3,361
Predicting the Probability for Falls in Community-Dwelling Older Adults Using the Timed Up & Go Test
Shumway‐Cook, A., Brauer, S., & Woollacott, M. H. (2000) · Physical Therapy · doi:10.1093/ptj/80.9.896 · Cited by 3,355
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
Hallgren, K. A. (2012) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.08.1.p023 · Cited by 3,884
Regional Brain Changes in Aging Healthy Adults: General Trends, Individual Differences and Modifiers
Raz, N., Lindenberger, U., Rodrigue, K. M., Kennedy, K. M., Head, D., Williamson, A., Dahle, C., Gerstorf, D., & Acker, J. D. (2005) · Cerebral Cortex · doi:10.1093/cercor/bhi044 · Cited by 2,875
Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool
Taha, A. A., & Hanbury, A. (2015) · BMC Medical Imaging · doi:10.1186/s12880-015-0068-x · Cited by 3,247
A penetration-aspiration scale
Rosenbek, J. C., Robbins, J. A., Roecker, E. B., Coyle, J. L., & Wood, J. L. (1996) · Dysphagia · doi:10.1007/bf00417897 · Cited by 2,803
Repeatability for Gaussian and non-Gaussian data: a practical guide for biologists.
Nakagawa, S., & Schielzeth, H. (2010) · Biological Reviews · doi:10.1111/j.1469-185x.2010.00141.x · Cited by 2,584
Mobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233
Within-subject template estimation for unbiased longitudinal image analysis
Reuter, M., Schmansky, N. J., Rosas, H. D., & Fischl, B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.02.084 · Cited by 2,356
Presentation and validation of the Radboud Faces Database
Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H. J., Hawk, S. T., & van Knippenberg, A. (2010) · Cognition & Emotion · doi:10.1080/02699930903485076 · Cited by 2,431
The grateful disposition: A conceptual and empirical topography.
McCullough, M. E., Emmons, R. A., & Tsang, J.-A. (2002) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.82.1.112 · Cited by 3,664
The Multidimensional Anxiety Scale for Children (MASC): factor structure, reliability, and validity
MARCH, J. S., PARKER, J. D. A., SULLIVAN, K., STALLINGS, P., & CONNERS, C. K. (1997) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199704000-00019 · Cited by 2,236
rwg: an assessment of within-group interrater agreement
James, L. R., Demaree, R. G., & Wolf, G. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.2.306 · Cited by 2,221
Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed
Kottner, J., Audigé, L., Brorson, S., Donner, A. P., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M. M., & Streiner, D. L. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.002 · Cited by 2,314
Eliasson, A., Krumlinde‐Sundholm, L., Rösblad, B., Beckung, E., Arner, M., Öhrvall, A., & Rosenbaum, P. (2006) · Developmental Medicine & Child Neurology · doi:10.1017/s0012162206001162 · Cited by 2,197
The yale global tic severity scale: initial testing of a clinician-rated scale of tic severity
LECKMAN, J. F., RIDDLE, M. A., HARDIN, M. T., ORT, S. I., SWARTZ, K. L., STEVENSON, J., & COHEN, D. J. (1989) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-198907000-00015 · Cited by 2,146
Development and validation of the international knee documentation committee subjective knee form
Irrgang, J. J., Anderson, A. F., Boland, A. L., Harner, C. D., Kurosaka, M., Neyret, P., Richmond, J. C., & Shelborne, K. D. (2001) · The American Journal of Sports Medicine · doi:10.1177/03635465010290051301 · Cited by 2,204
Cook, D. A., Hatala, R., Brydges, R., Zendejas, B., Szostek, J. H., Wang, A. T., Erwin, P. J., & Hamstra, S. J. (2011) · JAMA · doi:10.1001/jama.2011.1234 · Cited by 2,170
Children's Yale-Brown Obsessive Compulsive Scale: Reliability and Validity
SCAHILL, L., RIDDLE, M. A., McSWIGGIN-HARDIN, M., ORT, S. I., KING, R. A., GOODMAN, W. K., CICCHETTI, D., & LECKMAN, J. F. (1997) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199706000-00023 · Cited by 2,045
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Livingstone, S. R., & Russo, F. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196391 · Cited by 2,084
Sharma, S. (2000) · Academy of Management Journal · doi:10.2307/1556361 · Cited by 1,919
Weathers, F. W., Bovin, M. J., Lee, D. J., Sloan, D. M., Schnurr, P. P., Kaloupek, D. G., Keane, T. M., & Marx, B. P. (2018) · Psychological Assessment · doi:10.1037/pas0000486 · Cited by 1,799
Consensus measurement in Delphi studies
von der Gracht, H. A. (2012) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2012.04.013 · Cited by 1,831
Sangha, O., Stucki, G., Liang, M. H., Fossel, A. H., & Katz, J. N. (2003) · Arthritis Care & Research · doi:10.1002/art.10993 · Cited by 1,841
Daperno, M., D’Haens, G., Assche, G. V., Baert, F., Bulois, P., Maunoury, V., Sostegni, R., Rocca, R., Pera, A., Gevers, A., et al. (2004) · Gastrointestinal Endoscopy · doi:10.1016/s0016-5107(04)01878-4 · Cited by 1,779
rptR: repeatability estimation and variance decomposition by generalized linear mixed‐effects models
Stoffel, M. A., Nakagawa, S., & Schielzeth, H. (2017) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12797 · Cited by 1,779
Bae, T. J., Qian, S., Miao, C., & Fiet, J. O. (2014) · Entrepreneurship Theory and Practice · doi:10.1111/etap.12095 · Cited by 2,033
Predicting relapse: A meta-analysis of sexual offender recidivism studies.
Hanson, R. K., & Bussière, M. T. (1998) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.66.2.348 · Cited by 2,375
Linking service climate and customer perceptions of service quality: Tests of a causal model.
Schneider, B. N., White, S. S., & Paul, M. C. (1998) · Journal of Applied Psychology · doi:10.1037/0021-9010.83.2.150 · Cited by 1,754
Defining and developing measures of lean production
Shah, R., & Ward, P. T. (2007) · Journal of Operations Management · doi:10.1016/j.jom.2007.01.019 · Cited by 2,399
Keller, M. B. (1987) · Archives of General Psychiatry · doi:10.1001/archpsyc.1987.01800180050009 · Cited by 1,643
On the fit of models to covariances and methodology to the Bulletin.
Bentler, P. M. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.112.3.400 · Cited by 1,658
TAKING CHARGE AT WORK: EXTRAROLE EFFORTS TO INITIATE WORKPLACE CHANGE.
Morrison, E. W., & Phelps, C. (1999) · Academy of Management Journal · doi:10.2307/257011 · Cited by 1,722
Aggregation bias in estimates of perceptual agreement.
James, L. R. (1982) · Journal of Applied Psychology · doi:10.1037/0021-9010.67.2.219 · Cited by 2,139
Adaptability in the workplace: Development of a taxonomy of adaptive performance.
Pulakos, E. D., Arad, S., Donovan, M. A., & Plamondon, K. E. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.4.612 · Cited by 1,632
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457
Foreign Accent, Comprehensibility, and Intelligibility in the Speech of Second Language Learners
Munro, M. J., & Derwing, T. M. (1995) · Language Learning · doi:10.1111/j.1467-1770.1995.tb00963.x · Cited by 1,554
Yan, C.-G., Cheung, B., Kelly, C., Colcombe, S., Craddock, R. C., Di Martino, A., Li, Q., Zuo, X.-N., Castellanos, F. X., & Milham, M. P. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.03.004 · Cited by 1,461
Internet-Based Learning in the Health Professions
Cook, D., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2008) · JAMA · doi:10.1001/jama.300.10.1181 · Cited by 1,603
Zohar, D. M. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.4.587 · Cited by 1,559
Coding of facial expressions of pain in the laboratory mouse
Langford, D. J., Bailey, A. L., Chanda, M. L., Clarke, S. E. D., Drummond, T. E., Echols, S., Glick, S., Ingrao, J. C., Klassen-Ross, T., LaCroix‐Fralish, M. L., et al. (2010) · Nature Methods · doi:10.1038/nmeth.1455 · Cited by 1,570
(2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.1.18 · Cited by 1,446
Responsiveness of the Numeric Pain Rating Scale in Patients with Low Back Pain
Childs, J. D., Piva, S. R., & Fritz, J. M. (2005) · Spine · doi:10.1097/01.brs.0000164099.92112.29 · Cited by 1,481
Responsive and Proactive Market Orientation and New‐Product Success*
Narver, J. C., Slater, S. F., & MacLachlan, D. L. (2004) · Journal of Product Innovation Management · doi:10.1111/j.0737-6782.2004.00086.x · Cited by 1,466
The oscillating brain: Complex and reliable
Zuo, X., Martino, A. D., Kelly, C. A. M., Shehzad, Z. E., Gee, D. G., Klein, D. F., Castellanos, F. X., Biswal, B. B., & Milham, M. P. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.09.037 · Cited by 1,466
Binkley, J. M., Stratford, P. W., Lott, S., & Riddle, D. L. (1999) · PubMed · Cited by 1,435
Smith, J. B., & Barclay, D. W. (1997) · Journal of Marketing · doi:10.2307/1252186 · Cited by 1,442
Subjective well-being and adaptation to life events: A meta-analysis.
Luhmann, M., Hofmann, W., Eid, M., & Lucas, R. E. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0025948 · Cited by 1,410
Blagojevic, M., Jinks, C., Jeffery, A., & Jordan, K. P. (2009) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2009.08.010 · Cited by 1,471
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331
Common methods bias: does common methods variance really bias results?
Doty, D. H., & Glick, W. H. (1998) · Organizational Research Methods · doi:10.1177/109442819814002 · Cited by 1,380
Nock, M. K., Holmberg, E. B., Photos, V. I., & Michel, B. D. (2007) · Psychological Assessment · doi:10.1037/1040-3590.19.3.309 · Cited by 1,352
Monitoring Depression Treatment Outcomes With the Patient Health Questionnaire-9
L??we, B., Un??tzer, J., Callahan, C. M., Perkins, A. J., & Kroenke, K. (2004) · Medical Care · doi:10.1097/00005650-200412000-00006 · Cited by 1,354
Collins, C. J., & Clark, K. D. (2003) · Academy of Management Journal · doi:10.2307/30040665 · Cited by 1,360
Development and validation of the Neuropathic Pain Symptom Inventory
Bouhassira, D., Attal, N., Fermanian, J., Alchaar, H., Gautron, M., Masquelier, E., Rostaing, S., Lanteri-Minet, M., Collin, E., Grisart, J., et al. (2004) · Pain · doi:10.1016/j.pain.2003.12.024 · Cited by 1,329
Gamification for health and wellbeing: A systematic review of the literature
Johnson, D., Deterding, S., Kuhn, K.-A., Staneva, A., Stoyanov, S., & Hides, L. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.10.002 · Cited by 1,237
Measuring the whole or the parts?
Beaton, D., Katz, J. N., Fossel, A. H., Wright, J. G., Tarasuk, V. S., & Bombardier, C. (2001) · Journal of Hand Therapy · doi:10.1016/s0894-1130(01)80043-0 · Cited by 1,306
Buckner, R. L., Head, D. M., Parker, J. B., Fotenos, A. F., Marcus, D. S., Morris, J. C., & Snyder, A. Z. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.06.018 · Cited by 1,325
Glick, W. H. (1985) · The Academy of Management Review · doi:10.5465/amr.1985.4279045 · Cited by 1,286
The Patient and Observer Scar Assessment Scale: A Reliable and Feasible Tool for Scar Evaluation
Draaijers, L. J., Tempelman, F. R. H., Botman, Y. A. M., Tuinebreijer, W. E., Middelkoop, E., Kreis, R. W., & van Zuijlen, P. P. M. (2004) · Plastic & Reconstructive Surgery · doi:10.1097/01.prs.0000122207.28773.56 · Cited by 1,279
Culturally adapted mental health intervention: a meta-analytic review
Griner, D., & Smith, T. B. (2006) · Psychotherapy: Theory, Research, Practice, Training · doi:10.1037/0033-3204.43.4.531 · Cited by 1,236
RELIABILITY OF GAIT PERFORMANCE TESTS IN MEN AND WOMEN WITH HEMIPARESIS AFTER STROKE
Lexell, J. E., Flansbjer, U., Holmbäck, A. M., Downham, D. Y., & Patten, C. (2004) · Journal of Rehabilitation Medicine · doi:10.1080/16501970410017215 · Cited by 1,248
Raz, N. (1997) · Cerebral Cortex · doi:10.1093/cercor/7.3.268 · Cited by 1,292
Patterson, M., West, M. A., Shackleton, V., Dawson, J., Lawthom, R., Maitlis, S., Robinson, D. L., & Wallace, A. M. (2005) · Journal of Organizational Behavior · doi:10.1002/job.312 · Cited by 1,318
Stam, W., & Elfring, T. (2008) · Academy of Management Journal · doi:10.5465/amj.2008.30744031 · Cited by 1,218
Measurement reliability and agreement in psychiatry
Shrout, P. E. (1998) · Statistical Methods in Medical Research · doi:10.1177/096228029800700306 · Cited by 1,211
Self-citationCollins, J. J., & Luca, C. J. D. (1993) · Experimental Brain Research · doi:10.1007/bf00229788 · Cited by 1,208
Construction of a 3D probabilistic atlas of human cortical structures
Shattuck, D. W., Mirza, M., Adisetiyo, V., Hojatkashani, C., Salamon, G., Narr, K. L., Poldrack, R. A., Bilder, R. M., & Toga, A. W. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2007.09.031 · Cited by 1,208
Hare Psychopathy Checklist-Revised
Hollerbach, P., Mokros, A., Nitschke, J., & Habermeyer, E. (2018) · Forensische Psychiatrie Psychologie Kriminologie · doi:10.1007/s11757-018-0478-z · Cited by 1,208
FIT, EQUIFINALITY, AND ORGANIZATIONAL EFFECTIVENESS: A TEST OF TWO CONFIGURATIONAL THEORIES.
DOTY, D. H., GLICK, W. H., & HUBER, G. P. (1993) · Academy of Management Journal · doi:10.5465/256810 · Cited by 1,198
Morosini, P. L., Magliano, L., Brambilla, L., Ugolini, S., & Pioli, R. (2000) · PubMed · Cited by 1,168
The reliability of three-dimensional kinematic gait measurements: A systematic review
McGinley, J. L., Baker, R. J., Wolfe, R. S., & Morris, M. E. (2008) · Gait & Posture · doi:10.1016/j.gaitpost.2008.09.003 · Cited by 1,155
Smith, T. B., McCullough, M. E., & Poll, J. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.4.614 · Cited by 1,150
Development of a Shoulder Pain and Disability Index
Roach, K. E., Budiman‐Mak, E., Songsiridej, N., & Lertratanakul, Y. (1991) · Arthritis & Rheumatism · doi:10.1002/art.1790040403 · Cited by 1,151
Evidence of Validity for the Foot and Ankle Ability Measure (FAAM)
Martin, R. L., Irrgang, J. J., Burdett, R. G., Conti, S. F., & Swearingen, J. M. V. (2005) · Foot & Ankle International · doi:10.1177/107110070502601113 · Cited by 1,148
Tugade, M. M., Fredrickson, B. L., & Feldman Barrett, L. (2004) · Journal of Personality · doi:10.1111/j.1467-6494.2004.00294.x · Cited by 1,610
Individual Variability in Functional Connectivity Architecture of the Human Brain
Mueller, S., Wang, D., Fox, M., Yeo, B. T. T., Sepulcre, J., Sabuncu, M. R., Shafee, R., Lu, J., & Liu, H. (2013) · Neuron · doi:10.1016/j.neuron.2012.12.028 · Cited by 1,168
Handbook of Religion and Health
Koenig, H. G., VanderWeele, T. J., & Peteet, J. R. (2023) · doi:10.1093/oso/9780190088859.001.0001 · Cited by 1,160
Interpersonal facilitation and job dedication as separate facets of contextual performance.
Scotter, J. R. V., & Motowidlo, S. J. (1996) · Journal of Applied Psychology · doi:10.1037/0021-9010.81.5.525 · Cited by 1,144
The Effectiveness of Creativity Training: A Quantitative Review
Scott, G., Leritz, L. E., & Mumford, M. D. (2004) · Creativity Research Journal · doi:10.1080/10400410409534549 · Cited by 1,132
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
