Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Inter-rater agreement was high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation = 0.80, SD = .14).
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
Intercoder reliability: Shrout-Fleiss single ICC score agreement of .998.
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (ICC(2,1) and ICC(3,1)).
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
A sample of 11% of video data was double coded by the researcher and an experienced coder, and inter-rater reliability resulted in good to excellent reliability across CHIPPA subscales.
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
Interrater agreement for adherence ratings was high (intraclass correlation [2, 1] = 0.85).
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
The PDSS interrater reliability of pre- and posttreatment assessments was excellent (ICC = 0.97 and 0.96, respectively).
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency).
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of dataset difficulty (variance between tasks) and agent consistency (variance within tasks).
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
Reproducibility of final placenta, fetus and amniotic fluid volumes was assessed using ICC(2,1) for inter-observer repeatability between the semi-automated Random Walker results and the newly generated FCNN outputs.
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931 · see citing works
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137 · see citing works
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012 · see citing works
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573 · see citing works
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625 · see citing works
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373 · see citing works
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206 · see citing works
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,100 · see citing works
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953 · see citing works
Estimation of the reliability of ratings.
Ebel, R. L. (1951) · Psychometrika · doi:10.1007/bf02288803 · Cited by 976 · see citing works
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.5.762 · Cited by 818 · see citing works
Unbiased Estimation of Certain Correlation Coefficients
Olkin, I., & Pratt, J. W. (1958) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177706717 · Cited by 491 · see citing works
Intraclass Correlation and the Analysis of Variance
Trese, T., & Haggard, E. A. (1958) · The American Catholic Sociological Review · doi:10.2307/3708305 · Cited by 403 · see citing works
The Approximate Sampling Distribution of Kuder-Richardson Reliability Coefficient Twenty
Feldt, L. S. (1965) · Psychometrika · doi:10.1007/bf02289499 · Cited by 296 · see citing works
Approximate Interval Estimation for a Certain Intraclass Correlation Coefficient
Fleiss, J. L., & Shrout, P. E. (1978) · Psychometrika · doi:10.1007/bf02293867 · Cited by 148 · see citing works
Self-citationBehavioral cues of interpersonal warmth.
Bayés, M. (1972) · Journal of Consulting and Clinical Psychology · doi:10.1037/h0033367 · Cited by 142 · see citing works
Intraclass Correlation and the Analysis of Variance.
Sitgreaves, R., & Haggard, E. A. (1960) · Journal of the American Statistical Association · doi:10.2307/2281758 · Cited by 116 · see citing works
Reliability Formulas for Independent Decision Data When Reliability Data are Matched
Rajaratnam, N. (1960) · Psychometrika · doi:10.1007/bf02289730 · Cited by 45 · see citing works
Comment on Bartko's "On various intraclass correlation reliability coefficients."
Algina, J. J. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.135 · Cited by 33 · see citing works
Bartko, J. J. (1978) · Psychological Bulletin · doi:10.1037//0033-2909.85.1.139 · Cited by 13 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 29,639 · see citing works
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582 · see citing works
The job demands-resources model of burnout.
Demerouti, E., Bakker, A. B., Nachreiner, F., & Schaufeli, W. B. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.3.499 · Cited by 13,643 · see citing works
Design and validation of a histological scoring system for nonalcoholic fatty liver disease†
Kleiner, D. E., Brunt, E. M., Van Natta, M., Behling, C., Contos, M. J., Cummings, O. W., Ferrell, L. D., Liu, Y., Torbenson, M. S., Unalp‐Arida, A., et al. (2005) · Hepatology · doi:10.1002/hep.20701 · Cited by 10,791 · see citing works
Yushkevich, P. A., Piven, J., Hazlett, H. C., Smith, R. G., Ho, S., Gee, J. C., & Gerig, G. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.01.015 · Cited by 8,868 · see citing works
Cicchetti, D. V. (1994) · Psychological Assessment · doi:10.1037/1040-3590.6.4.284 · Cited by 9,349 · see citing works
Social Relationships and Mortality Risk: A Meta-analytic Review
Holt-Lunstad, J., Smith, T. B., & Layton, J. B. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000316 · Cited by 7,272 · see citing works
“13) and for continuous variables (mean intraclass correlation [26] = 0.”
BackgroundASSESSING THE WORK ENVIRONMENT FOR CREATIVITY.
AMABILE, T. M., CONTI, R., COON, H., LAZENBY, J., & HERRON, M. (1996) · Academy of Management Journal · doi:10.2307/256995 · Cited by 6,029 · see citing works
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903 · see citing works
QUANTIFYING TEST-RETEST RELIABILITY USING THE INTRACLASS CORRELATION COEFFICIENT AND THE SEM
Weir, J. P. (2005) · The Journal of Strength and Conditioning Research · doi:10.1519/15184.1 · Cited by 4,864 · see citing works
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774 · see citing works
Measures of Reliability in Sports Medicine and Science
Hopkins, W. G. (2000) · Sports Medicine · doi:10.2165/00007256-200030010-00001 · Cited by 4,635 · see citing works
Zwanenburg, A., Vallières, M., Abdalah, M. A., Aerts, H. J. W. L., Andrearczyk, V., Apte, A., Ashrafinia, S., Bakas, S., Beukinga, R. J., Boellaard, R., et al. (2020) · Radiology · doi:10.1148/radiol.2020191145 · Cited by 4,177 · see citing works
Roos, E. M., Roos, H. P., Lohmander, S. L., Ekdahl, C. S., & Beynnon, B. D. (1998) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1998.28.2.88 · Cited by 3,913 · see citing works
DPABI: Data Processing & Analysis for (Resting-State) Brain Imaging
Yan, C., Wang, X., Zuo, X., & Zang, Y. (2016) · Neuroinformatics · doi:10.1007/s12021-016-9299-4 · Cited by 3,906 · see citing works
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608 · see citing works
Aggregate Confusion: The Divergence of ESG Ratings
Berg, F., Kölbel, J. F., & Rigobon, R. (2022) · Review of Finance · doi:10.1093/rof/rfac033 · Cited by 3,444 · see citing works
The ripple effect: emotional contagion and its influence on group behavior
Barsade, S. G. (2002) · Administrative Science Quarterly · doi:10.2307/3094912 · Cited by 3,361 · see citing works
Predicting the Probability for Falls in Community-Dwelling Older Adults Using the Timed Up & Go Test
Shumway‐Cook, A., Brauer, S., & Woollacott, M. H. (2000) · Physical Therapy · doi:10.1093/ptj/80.9.896 · Cited by 3,355 · see citing works
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
Hallgren, K. A. (2012) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.08.1.p023 · Cited by 3,884 · see citing works
Regional Brain Changes in Aging Healthy Adults: General Trends, Individual Differences and Modifiers
Raz, N., Lindenberger, U., Rodrigue, K. M., Kennedy, K. M., Head, D., Williamson, A., Dahle, C., Gerstorf, D., & Acker, J. D. (2005) · Cerebral Cortex · doi:10.1093/cercor/bhi044 · Cited by 2,875 · see citing works
Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool
Taha, A. A., & Hanbury, A. (2015) · BMC Medical Imaging · doi:10.1186/s12880-015-0068-x · Cited by 3,022 · see citing works
A penetration-aspiration scale
Rosenbek, J. C., Robbins, J. A., Roecker, E. B., Coyle, J. L., & Wood, J. L. (1996) · Dysphagia · doi:10.1007/bf00417897 · Cited by 2,803 · see citing works
Repeatability for Gaussian and non-Gaussian data: a practical guide for biologists.
Nakagawa, S., & Schielzeth, H. (2010) · Biological Reviews · doi:10.1111/j.1469-185x.2010.00141.x · Cited by 2,584 · see citing works
Mobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233 · see citing works
Within-subject template estimation for unbiased longitudinal image analysis
Reuter, M., Schmansky, N. J., Rosas, H. D., & Fischl, B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.02.084 · Cited by 2,356 · see citing works
Presentation and validation of the Radboud Faces Database
Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H. J., Hawk, S. T., & van Knippenberg, A. (2010) · Cognition & Emotion · doi:10.1080/02699930903485076 · Cited by 2,431 · see citing works
The grateful disposition: A conceptual and empirical topography.
McCullough, M. E., Emmons, R. A., & Tsang, J.-A. (2002) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.82.1.112 · Cited by 3,664 · see citing works
The Multidimensional Anxiety Scale for Children (MASC): factor structure, reliability, and validity
MARCH, J. S., PARKER, J. D. A., SULLIVAN, K., STALLINGS, P., & CONNERS, C. K. (1997) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199704000-00019 · Cited by 2,236 · see citing works
rwg: an assessment of within-group interrater agreement
James, L. R., Demaree, R. G., & Wolf, G. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.2.306 · Cited by 2,221 · see citing works
Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed
Kottner, J., Audigé, L., Brorson, S., Donner, A. P., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M. M., & Streiner, D. L. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.002 · Cited by 2,239 · see citing works
Eliasson, A., Krumlinde‐Sundholm, L., Rösblad, B., Beckung, E., Arner, M., Öhrvall, A., & Rosenbaum, P. (2006) · Developmental Medicine & Child Neurology · doi:10.1017/s0012162206001162 · Cited by 2,197 · see citing works
The yale global tic severity scale: initial testing of a clinician-rated scale of tic severity
LECKMAN, J. F., RIDDLE, M. A., HARDIN, M. T., ORT, S. I., SWARTZ, K. L., STEVENSON, J., & COHEN, D. J. (1989) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-198907000-00015 · Cited by 2,146 · see citing works
Development and Validation of the International Knee Documentation Committee Subjective Knee Form
Irrgang, J. J., Anderson, A. F., Boland, A. L., Harner, C. D., Kurosaka, M., Neyret, P., Richmond, J. C., & Shelbourne, K. D. (2001) · The American Journal of Sports Medicine · doi:10.1177/03635465010290051301 · Cited by 2,184 · see citing works
Cook, D. A., Hatala, R., Brydges, R., Zendejas, B., Szostek, J. H., Wang, A. T., Erwin, P. J., & Hamstra, S. J. (2011) · JAMA · doi:10.1001/jama.2011.1234 · Cited by 2,170 · see citing works
Children's Yale-Brown Obsessive Compulsive Scale: Reliability and Validity
SCAHILL, L., RIDDLE, M. A., McSWIGGIN-HARDIN, M., ORT, S. I., KING, R. A., GOODMAN, W. K., CICCHETTI, D., & LECKMAN, J. F. (1997) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199706000-00023 · Cited by 2,045 · see citing works
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053 · see citing works
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338 · see citing works
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · see citing works
Livingstone, S. R., & Russo, F. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196391 · Cited by 1,955 · see citing works
Sharma, S. (2000) · Academy of Management Journal · doi:10.2307/1556361 · Cited by 1,875 · see citing works
Weathers, F. W., Bovin, M. J., Lee, D. J., Sloan, D. M., Schnurr, P. P., Kaloupek, D. G., Keane, T. M., & Marx, B. P. (2018) · Psychological Assessment · doi:10.1037/pas0000486 · Cited by 1,799 · see citing works
Consensus measurement in Delphi studies
von der Gracht, H. A. (2012) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2012.04.013 · Cited by 1,831 · see citing works
Sangha, O., Stucki, G., Liang, M. H., Fossel, A. H., & Katz, J. N. (2003) · Arthritis Care & Research · doi:10.1002/art.10993 · Cited by 1,799 · see citing works
Daperno, M., D’Haens, G., Assche, G. V., Baert, F., Bulois, P., Maunoury, V., Sostegni, R., Rocca, R., Pera, A., Gevers, A., et al. (2004) · Gastrointestinal Endoscopy · doi:10.1016/s0016-5107(04)01878-4 · Cited by 1,779 · see citing works
rptR: repeatability estimation and variance decomposition by generalized linear mixed‐effects models
Stoffel, M. A., Nakagawa, S., & Schielzeth, H. (2017) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12797 · Cited by 1,779 · see citing works
Bae, T. J., Qian, S., Miao, C., & Fiet, J. O. (2014) · Entrepreneurship Theory and Practice · doi:10.1111/etap.12095 · Cited by 1,873 · see citing works
Predicting relapse: A meta-analysis of sexual offender recidivism studies.
Hanson, R. K., & Bussière, M. T. (1998) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.66.2.348 · Cited by 2,375 · see citing works
Linking service climate and customer perceptions of service quality: Tests of a causal model.
Schneider, B. N., White, S. S., & Paul, M. C. (1998) · Journal of Applied Psychology · doi:10.1037/0021-9010.83.2.150 · Cited by 1,720 · see citing works
Defining and developing measures of lean production
Shah, R., & Ward, P. T. (2007) · Journal of Operations Management · doi:10.1016/j.jom.2007.01.019 · Cited by 2,399 · see citing works
Keller, M. B. (1987) · Archives of General Psychiatry · doi:10.1001/archpsyc.1987.01800180050009 · Cited by 1,643 · see citing works
On the fit of models to covariances and methodology to the Bulletin.
Bentler, P. M. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.112.3.400 · Cited by 1,658 · see citing works
TAKING CHARGE AT WORK: EXTRAROLE EFFORTS TO INITIATE WORKPLACE CHANGE.
Morrison, E. W., & Phelps, C. (1999) · Academy of Management Journal · doi:10.2307/257011 · Cited by 1,636 · see citing works
Aggregation bias in estimates of perceptual agreement.
James, L. R. (1982) · Journal of Applied Psychology · doi:10.1037/0021-9010.67.2.219 · Cited by 2,139 · see citing works
Adaptability in the workplace: Development of a taxonomy of adaptive performance.
Pulakos, E. D., Arad, S., Donovan, M. A., & Plamondon, K. E. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.4.612 · Cited by 1,632 · see citing works
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457 · see citing works
Foreign Accent, Comprehensibility, and Intelligibility in the Speech of Second Language Learners
Munro, M. J., & Derwing, T. M. (1995) · Language Learning · doi:10.1111/j.1467-1770.1995.tb00963.x · Cited by 1,554 · see citing works
Yan, C.-G., Cheung, B., Kelly, C., Colcombe, S., Craddock, R. C., Di Martino, A., Li, Q., Zuo, X.-N., Castellanos, F. X., & Milham, M. P. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.03.004 · Cited by 1,461 · see citing works
Internet-Based Learning in the Health Professions
Cook, D., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2008) · JAMA · doi:10.1001/jama.300.10.1181 · Cited by 1,552 · see citing works
Zohar, D. M. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.4.587 · Cited by 1,506 · see citing works
Coding of facial expressions of pain in the laboratory mouse
Langford, D. J., Bailey, A. L., Chanda, M. L., Clarke, S. E. D., Drummond, T. E., Echols, S., Glick, S., Ingrao, J. C., Klassen-Ross, T., LaCroix‐Fralish, M. L., et al. (2010) · Nature Methods · doi:10.1038/nmeth.1455 · Cited by 1,498 · see citing works
(2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.1.18 · Cited by 1,446 · see citing works
Responsiveness of the Numeric Pain Rating Scale in Patients with Low Back Pain
Childs, J. D., Piva, S. R., & Fritz, J. M. (2005) · Spine · doi:10.1097/01.brs.0000164099.92112.29 · Cited by 1,481 · see citing works
Responsive and Proactive Market Orientation and New‐Product Success*
Narver, J. C., Slater, S. F., & MacLachlan, D. L. (2004) · Journal of Product Innovation Management · doi:10.1111/j.0737-6782.2004.00086.x · Cited by 1,466 · see citing works
The oscillating brain: Complex and reliable
Zuo, X., Martino, A. D., Kelly, C. A. M., Shehzad, Z. E., Gee, D. G., Klein, D. F., Castellanos, F. X., Biswal, B. B., & Milham, M. P. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.09.037 · Cited by 1,460 · see citing works
Binkley, J. M., Stratford, P. W., Lott, S., & Riddle, D. L. (1999) · PubMed · Cited by 1,435 · see citing works
Smith, J. B., & Barclay, D. W. (1997) · Journal of Marketing · doi:10.2307/1252186 · Cited by 1,406 · see citing works
Subjective well-being and adaptation to life events: A meta-analysis.
Luhmann, M., Hofmann, W., Eid, M., & Lucas, R. E. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0025948 · Cited by 1,410 · see citing works
Blagojevic, M., Jinks, C., Jeffery, A., & Jordan, K. P. (2009) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2009.08.010 · Cited by 1,399 · see citing works
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · see citing works
Common methods bias: does common methods variance really bias results?
Doty, D. H., & Glick, W. H. (1998) · Organizational Research Methods · doi:10.1177/109442819814002 · Cited by 1,380 · see citing works
Nock, M. K., Holmberg, E. B., Photos, V. I., & Michel, B. D. (2007) · Psychological Assessment · doi:10.1037/1040-3590.19.3.309 · Cited by 1,352 · see citing works
Monitoring Depression Treatment Outcomes With the Patient Health Questionnaire-9
L??we, B., Un??tzer, J., Callahan, C. M., Perkins, A. J., & Kroenke, K. (2004) · Medical Care · doi:10.1097/00005650-200412000-00006 · Cited by 1,354 · see citing works
Collins, C. J., & Clark, K. D. (2003) · Academy of Management Journal · doi:10.2307/30040665 · Cited by 1,360 · see citing works
Development and validation of the Neuropathic Pain Symptom Inventory
Bouhassira, D., Attal, N., Fermanian, J., Alchaar, H., Gautron, M., Masquelier, E., Rostaing, S., Lanteri-Minet, M., Collin, E., Grisart, J., et al. (2004) · Pain · doi:10.1016/j.pain.2003.12.024 · Cited by 1,329 · see citing works
Gamification for health and wellbeing: A systematic review of the literature
Johnson, D., Deterding, S., Kuhn, K.-A., Staneva, A., Stoyanov, S., & Hides, L. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.10.002 · Cited by 1,237 · see citing works
Measuring the whole or the parts?
Beaton, D., Katz, J. N., Fossel, A. H., Wright, J. G., Tarasuk, V. S., & Bombardier, C. (2001) · Journal of Hand Therapy · doi:10.1016/s0894-1130(01)80043-0 · Cited by 1,305 · see citing works
Buckner, R. L., Head, D. M., Parker, J. B., Fotenos, A. F., Marcus, D. S., Morris, J. C., & Snyder, A. Z. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.06.018 · Cited by 1,290 · see citing works
Glick, W. H. (1985) · The Academy of Management Review · doi:10.5465/amr.1985.4279045 · Cited by 1,286 · see citing works
The Patient and Observer Scar Assessment Scale: A Reliable and Feasible Tool for Scar Evaluation
Draaijers, L. J., Tempelman, F. R. H., Botman, Y. A. M., Tuinebreijer, W. E., Middelkoop, E., Kreis, R. W., & van Zuijlen, P. P. M. (2004) · Plastic & Reconstructive Surgery · doi:10.1097/01.prs.0000122207.28773.56 · Cited by 1,279 · see citing works
Culturally adapted mental health intervention: a meta-analytic review
Griner, D., & Smith, T. B. (2006) · Psychotherapy: Theory, Research, Practice, Training · doi:10.1037/0033-3204.43.4.531 · Cited by 1,236 · see citing works
RELIABILITY OF GAIT PERFORMANCE TESTS IN MEN AND WOMEN WITH HEMIPARESIS AFTER STROKE
Lexell, J. E., Flansbjer, U., Holmbäck, A. M., Downham, D. Y., & Patten, C. (2004) · Journal of Rehabilitation Medicine · doi:10.1080/16501970410017215 · Cited by 1,248 · see citing works
Raz, N. (1997) · Cerebral Cortex · doi:10.1093/cercor/7.3.268 · Cited by 1,239 · see citing works
Patterson, M. G., West, M. A., Shackleton, V. J., Dawson, J., Lawthom, R., Maitlis, S., Robinson, D. L., & Wallace, A. M. R. (2005) · Journal of Organizational Behavior · doi:10.1002/job.312 · Cited by 1,262 · see citing works
Stam, W., & Elfring, T. (2008) · Academy of Management Journal · doi:10.5465/amj.2008.30744031 · Cited by 1,218 · see citing works
Measurement reliability and agreement in psychiatry
Shrout, P. E. (1998) · Statistical Methods in Medical Research · doi:10.1177/096228029800700306 · Cited by 1,211 · see citing works
Self-citationCollins, J. J., & Luca, C. J. D. (1993) · Experimental Brain Research · doi:10.1007/bf00229788 · Cited by 1,208 · see citing works
Construction of a 3D probabilistic atlas of human cortical structures
Shattuck, D. W., Mirza, M., Adisetiyo, V., Hojatkashani, C., Salamon, G., Narr, K. L., Poldrack, R. A., Bilder, R. M., & Toga, A. W. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2007.09.031 · Cited by 1,208 · see citing works
Hare Psychopathy Checklist-Revised
Hollerbach, P., Mokros, A., Nitschke, J., & Habermeyer, E. (2018) · Forensische Psychiatrie Psychologie Kriminologie · doi:10.1007/s11757-018-0478-z · Cited by 1,208 · see citing works
FIT, EQUIFINALITY, AND ORGANIZATIONAL EFFECTIVENESS: A TEST OF TWO CONFIGURATIONAL THEORIES.
DOTY, D. H., GLICK, W. H., & HUBER, G. P. (1993) · Academy of Management Journal · doi:10.5465/256810 · Cited by 1,198 · see citing works
Morosini, P. L., Magliano, L., Brambilla, L., Ugolini, S., & Pioli, R. (2000) · PubMed · Cited by 1,168 · see citing works
The reliability of three-dimensional kinematic gait measurements: A systematic review
McGinley, J. L., Baker, R. J., Wolfe, R. S., & Morris, M. E. (2008) · Gait & Posture · doi:10.1016/j.gaitpost.2008.09.003 · Cited by 1,155 · see citing works
Smith, T. B., McCullough, M. E., & Poll, J. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.4.614 · Cited by 1,150 · see citing works
Development of a Shoulder Pain and Disability Index
Roach, K. E., Budiman‐Mak, E., Songsiridej, N., & Lertratanakul, Y. (1991) · Arthritis & Rheumatism · doi:10.1002/art.1790040403 · Cited by 1,151 · see citing works
Evidence of Validity for the Foot and Ankle Ability Measure (FAAM)
Martin, R. L., Irrgang, J. J., Burdett, R. G., Conti, S. F., & Swearingen, J. M. V. (2005) · Foot & Ankle International · doi:10.1177/107110070502601113 · Cited by 1,148 · see citing works
Tugade, M. M., Fredrickson, B. L., & Feldman Barrett, L. (2004) · Journal of Personality · doi:10.1111/j.1467-6494.2004.00294.x · Cited by 1,610 · see citing works
Individual Variability in Functional Connectivity Architecture of the Human Brain
Mueller, S., Wang, D., Fox, M., Yeo, B. T. T., Sepulcre, J., Sabuncu, M. R., Shafee, R., Lu, J., & Liu, H. (2013) · Neuron · doi:10.1016/j.neuron.2012.12.028 · Cited by 1,142 · see citing works
Handbook of Religion and Health
Koenig, H. G., VanderWeele, T. J., & Peteet, J. R. (2023) · doi:10.1093/oso/9780190088859.001.0001 · Cited by 1,150 · see citing works
Interpersonal facilitation and job dedication as separate facets of contextual performance.
Scotter, J. R. V., & Motowidlo, S. J. (1996) · Journal of Applied Psychology · doi:10.1037/0021-9010.81.5.525 · Cited by 1,144 · see citing works
The Effectiveness of Creativity Training: A Quantitative Review
Scott, G., Leritz, L. E., & Mumford, M. D. (2004) · Creativity Research Journal · doi:10.1080/10400410409534549 · Cited by 1,132 · see citing works
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 23,433
- References
- 20
- Citations by year
- 2026: 870 · 2025: 878 · 2024: 858 · 2023: 945 · 2022: 1028 · 2021: 1297 · 2020: 1128 · 2019: 1066 · 2018: 1127 · 2017: 1070 · 2016: 1075 · 2015: 1034 · 2014: 1157 · 2013: 1038 · 2012: 903
- FWCI
- 33.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
