Forming inferences about some intraclass correlation coefficients.

McGraw, Kenneth O.; Wong, S. P.

doi:10.1037/1082-989x.1.1.30

Cited by 6,903Reliability and Agreement in MeasurementPsychometric Methodologies and TestingStatistical Methods and Bayesian Inference

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Forming inferences about some intraclass correlation coefficients.

McGraw, Kenneth O., Wong, S. P.·1996·Psychological Methods

DOI 10.1037/1082-989x.1.1.30

Open this paper in CoChat — free account

Abstract

Although intraclass correlation coefficients (ICCs) are commonly used in behavioral measurement, psychometrics, and behavioral genetics, procedures available for forming inferences about ICCs are not widely known. Following a review of the distinction between various forms of the ICC, this article p

Integrity signals

Registry

Crossref · 10.1037/1082-989x.1.1.30

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency).
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as McGraw and Wong 1996
Retrieval Accuracy showed strong agreement among annotators, with Percent Agreement ranging from 93% (GPT-4o) to 98% (GLM) and ICC values ranging from 78.1% for GPT-4o to 96.9% for GLM.
“Retrieval Accuracy showed strong agreement among annotators. Percent Agreement ranged from 93% (GPT-4o) to 98% (GLM). The ICC values, which measure absolute agreement between raters [39], range from 78.1% for GPT-4o to 96.9% for GLM.”
From CAIM: Development and Evaluation of a Cognitive AI Memory Framework for Long-Term Interaction with Intelligent Agents (2026) · cites this paper as [39]

Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373

Flagged first, then by citations

Sorted by citations

A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Quality criteria were proposed for measurement properties of health status questionnaires
Terwee, C. B., Bot, S. D. M., de Boer, M. R., van der Windt, D. A. W. M., Knol, D. L., Dekker, J., Bouter, L. M., & de Vet, H. C. W. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.012 · Cited by 10,764
Validation of the theoretical domains framework for use in behaviour change and implementation research
Cane, J., O’Connor, D., & Michie, S. (2012) · Implementation Science · doi:10.1186/1748-5908-7-37 · Cited by 4,888
“To examine the agreement of these construct x domain ratings and construct assignment across participants, two-way intraclass correlation coefficient (ICC) measures of consistency were used within each domain [43].”
Method
The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping
Zwanenburg, A., Vallières, M., Abdalah, M. A., Aerts, H. J. W. L., Andrearczyk, V., Apte, A., Ashrafinia, S., Bakas, S., Beukinga, R. J., Boellaard, R., et al. (2020) · Radiology · doi:10.1148/radiol.2020191145 · Cited by 4,177
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608
What Is the Male-to-Female Ratio in Autism Spectrum Disorder? A Systematic Review and Meta-Analysis
Loomes, R., Hull, L., & Mandy, W. P. L. (2017) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2017.03.013 · Cited by 2,804
Repeatability for Gaussian and non-Gaussian data: a practical guide for biologists.
Nakagawa, S., & Schielzeth, H. (2010) · Biological Reviews · doi:10.1111/j.1469-185x.2010.00141.x · Cited by 2,584
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053
Prevalence of co-occurring mental health diagnoses in the autism population: a systematic review and meta-analysis
Lai, M.-C., Kassee, C., Besney, R., Bonato, S., Hull, L., Mandy, W., Szatmari, P., & Ameis, S. H. (2019) · The Lancet Psychiatry · doi:10.1016/s2215-0366(19)30289-5 · Cited by 1,892
Dynamic reconfiguration of human brain networks during learning
Bassett, D. S., Wymbs, N. F., Porter, M. A., Mucha, P. J., Carlson, J. M., & Grafton, S. T. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1018985108 · Cited by 1,705
Are leader stereotypes masculine? A meta-analysis of three research paradigms
Koenig, A. M., Eagly, A. H., Mitchell, A. A., & Ristikari, T. (2011) · Psychological Bulletin · doi:10.1037/a0023557 · Cited by 1,748
The society for vascular surgery lower extremity threatened limb classification system: risk stratification based on wound, ischemia, and foot infection (WIfI)
Mills, J. L., Conte, M. S., Armstrong, D. G., Pomposelli, F. B., Schanzer, A., Sidawy, A. N., & Andros, G. (2014) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2013.08.003 · Cited by 1,718
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457
Aging and autobiographical memory: dissociating episodic from semantic retrieval
Levine, B., Svoboda, E., Hay, J. F., Winocur, G., & Moscovitch, M. (2002) · Psychology and Aging · doi:10.1037/0882-7974.17.4.677 · Cited by 1,521
A safe haven: An attachment theory perspective on support seeking and caregiving in intimate relationships
Collins, N. L., & Feeney, B. C. (2000) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.78.6.1053 · Cited by 1,303
How young can children reliably and validly self-report their health-related quality of life?: An analysis of 8,591 children across age subgroups with the PedsQL™ 4.0 Generic Core Scales
Varni, J. W., Limbers, C. A., & Burwinkle, T. M. (2007) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-5-1 · Cited by 1,075
Sample size requirements for estimating intraclass correlations with desired precision
Bonett, D. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1108 · Cited by 987
The social media disorder scale
van den Eijnden, R. J. J. M., Lemmens, J. S., & Valkenburg, P. M. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.03.038 · Cited by 948
“In addition, the intra-class correlation coef ﬁ cient (ICC) was established, using a two factor mixed effects model and type consistency (McGraw & Wong,1996).”
Interrater agreement and interrater reliability: Key concepts, approaches, and applications
Gisev, N., Bell, J. S., & Chen, T. F. (2013) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2012.04.004 · Cited by 924
Interactions Within Groups and Subgroups: The Effects of Demographic Faultlines
Lau, D. C., & Murnighan, J. K. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.17843943 · Cited by 831
The effectiveness of early head start for 3-year-old children and their parents: lessons for policy and programs
Love, J. M., Kisker, E. E., Ross, C., Raikes, H., Constantine, J., Boller, K., Brooks-Gunn, J., Chazan-Cohen, R., Tarullo, L. B., Brady-Smith, C., et al. (2005) · Developmental Psychology · doi:10.1037/0012-1649.41.6.885 · Cited by 800
Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review
Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008) · Value in Health · doi:10.1111/j.1524-4733.2007.00231.x · Cited by 623
Stability and Course of Neuropsychological Deficits in Schizophrenia
Heaton, R. K., Gladsjo, J. A., Palmer, B. W., Kuck, J., Marcotte, T. D., & Jeste, D. V. (2001) · Archives of General Psychiatry · doi:10.1001/archpsyc.58.1.24 · Cited by 618
COSMIN Risk of Bias tool to assess the quality of studies on reliability or measurement error of outcome measurement instruments: a Delphi study
Mokkink, L. B., Boers, M., van der Vleuten, C. P. M., Bouter, L. M., Alonso, J., Patrick, D. L., de Vet, H. C. W., & Terwee, C. B. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01179-5 · Cited by 588
Development and validation of the camouflaging autistic traits questionnaire (CAT-Q)
Hull, L., Mandy, W., Lai, M.-C., Baron-Cohen, S., Allison, C., Smith, P., & Petrides, K. V. (2019) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-018-3792-6 · Cited by 496
ISway: a sensitive, valid and reliable measure of postural control
Mancini, M., Salarian, A., Carlson-Kuhta, P., Zampieri, C., King, L., Chiari, L., & Horak, F. B. (2012) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-9-59 · Cited by 499
iTUG, a Sensitive and Reliable Measure of Mobility
Salarian, A., Horak, F. B., Zampieri, C., Carlson-Kuhta, P., Nutt, J. G., & Aminian, K. (2010) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2010.2047606 · Cited by 489
Multimedia instructions and cognitive load theory: Effects of modality and cueing
Tabbers, H. K., Martens, R. L., & Van Merriënboer, J. J. G. (2004) · British Journal of Educational Psychology · doi:10.1348/000709904322848824 · Cited by 478
Activity Achievement Emotions and Academic Performance: A Meta-analysis
Camacho-Morles, J., Slemp, G. R., Pekrun, R., Loderer, K., Hou, H., & Oades, L. G. (2021) · Educational Psychology Review · doi:10.1007/s10648-020-09585-3 · Cited by 475
Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives.
Cho, K., Schunn, C. D., & Wilson, R. W. (2006) · Journal of Educational Psychology · doi:10.1037/0022-0663.98.4.891 · Cited by 355
A Scale to Measure Nonattachment: A Buddhist Complement to Western Research on Attachment and Adaptive Functioning
Sahdra, B. K., Shaver, P. R., & Brown, K. W. (2010) · Journal of Personality Assessment · doi:10.1080/00223890903425960 · Cited by 352
Appetitive traits and relationships with BMI in adults: Development of the Adult Eating Behaviour Questionnaire
Hunot, C., Fildes, A., Croker, H., Llewellyn, C. H., Wardle, J., & Beeken, R. J. (2016) · Appetite · doi:10.1016/j.appet.2016.05.024 · Cited by 340
Language‐association cortex asymmetry in autism and specific language impairment
De Fossé, L., Hodge, S. M., Makris, N., Kennedy, D. N., Caviness, V. S., McGrath, L., Steele, S., Ziegler, D. A., Herbert, M. R., Frazier, J. A., et al. (2004) · Annals of Neurology · doi:10.1002/ana.20275 · Cited by 339
How school leadership influences student learning: A test of “The four paths model”
Leithwood, K., Sun, J., & Schumacker, R. (2020) · Educational Administration Quarterly · doi:10.1177/0013161x19878772 · Cited by 329
Nurses’ Shift Length and Overtime Working in 12 European Countries: The Association With Perceived Quality of Care and Patient Safety
Griffiths, P., Dall’Ora, C., Simon, M., Ball, J., Lindqvist, R., Rafferty, A.-M., Schoonhoven, L., Tishelman, C., & Aiken, L. H. (2014) · Medical Care · doi:10.1097/mlr.0000000000000233 · Cited by 316
“The ICC 1 measures the degree of similarity between individuals within a cluster.(29) It also indicates the proportion of variance in the outcome that can be attributed to variation between groups (wards, hospitals, countries) as opposed to between individuals.”
Background
Glioblastoma Multiforme: Exploratory Radiogenomic Analysis by Using Quantitative Image Features
Gevaert, O., Mitchell, L. A., Achrol, A. S., Xu, J., Echegaray, S., Steinberg, G. K., Cheshier, S. H., Napel, S., Zaharchuk, G., & Plevritis, S. K. (2014) · Radiology · doi:10.1148/radiol.14131731 · Cited by 319
Effects of caffeine on the human circadian clock in vivo and in vitro
Burke, T. M., Markwald, R. R., McHill, A. W., Chinoy, E. D., Snider, J. A., Bessman, S. C., Jung, C. M., O’Neill, J. S., & Wright, K. P. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aac5125 · Cited by 283
Qualitative Attributes and Measurement Properties of Physical Activity Questionnaires: A Checklist
Terwee, C. B., Mokkink, L. B., van Poppel, M. N. M., Chinapaw, M. J. M., van Mechelen, W., & de Vet, H. C. W. (2010) · Sports Medicine · doi:10.2165/11531370-000000000-00000 · Cited by 269
Measuring fMRI reliability with the intra-class correlation coefficient
Caceres, A., Hall, D. L., Zelaya, F. O., Williams, S. C. R., & Mehta, M. A. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.12.035 · Cited by 255
Puberty influences medial temporal lobe and cortical gray matter maturation differently in boys than girls matched for sexual maturity
Bramen, J. E., Hranilovich, J. A., Dahl, R. E., Forbes, E. E., Chen, J., Toga, A. W., Dinov, I. D., Worthman, C. M., & Sowell, E. R. (2011) · Cerebral Cortex · doi:10.1093/cercor/bhq137 · Cited by 266
Executive Function in Preschool Children: Test–Retest Reliability
Beck, D. M., Schaefer, C., Pang, K., & Carlson, S. M. (2011) · Journal of Cognition and Development · doi:10.1080/15248372.2011.563485 · Cited by 262
Confidence Limits for Intraclass Reliability Coefficients
Baumgartner, T. A., & Chung, H. (2001) · Measurement in Physical Education and Exercise Science · doi:10.1207/s15327841mpee0503_4 · Cited by 249
One-year test-retest reliability of intrinsic connectivity network fMRI in older adults
Guo, C. C., Kurth, F., Zhou, J., Mayer, E. A., Eickhoff, S. B., Kramer, J. H., & Seeley, W. W. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.03.027 · Cited by 252
Self-efficacy and resource allocation: Support for a nonmonotonic, discontinuous model.
Vancouver, J. B., More, K. M., & Yoder, R. J. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.1.35 · Cited by 234
Research Note —A Model of Conflict, Leadership, and Performance in Virtual Teams
Wakefield, R. L., Leidner, D. E., & Garrison, G. (2008) · Information Systems Research · doi:10.1287/isre.1070.0149 · Cited by 220
DSM-5 field trials in the United States and Canada, part I: study design, sampling strategy, implementation, and analytic approaches
Clarke, D. E., Narrow, W. E., Regier, D. A., Kuramoto, S. J., Kupfer, D. J., Kuhl, E. A., Greiner, L., & Kraemer, H. C. (2013) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.12070998 · Cited by 201
PORTAAL: A Classroom Observation Tool Assessing Evidence-Based Teaching Practices for Active Learning in Large Science, Technology, Engineering, and Mathematics Classes
Eddy, S. L., Converse, M., & Wenderoth, M. P. (2015) · CBE—Life Sciences Education · doi:10.1187/cbe.14-06-0095 · Cited by 182
“The resulting ICCs were all > 0.8, and thus all were in the “excellent” range (Supplemental Table 2; Cicchetti 1994).”
MethodInfluential
Transformational and Transactional Leadership in Virtual and Physical Environments
Hoyt, C. L., & Blascovich, J. (2003) · Small Group Research · doi:10.1177/1046496403257527 · Cited by 181
A meta-analysis of autonomous and controlled forms of teacher motivation
Slemp, G. R., Field, J. G., & Cho, A. S. H. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2020.103459 · Cited by 179
State-dependent variability of dynamic functional connectivity between frontoparietal and default networks relates to cognitive flexibility
Douw, L., Wakeman, D. G., Tanaka, N., Liu, H., & Stufflebeam, S. M. (2016) · Neuroscience · doi:10.1016/j.neuroscience.2016.09.034 · Cited by 162
A scale for rating the quality of psychological trials for pain
Yates, S. L., Morley, S., Eccleston, C., & de C. Williams, A. C. (2005) · Pain · doi:10.1016/j.pain.2005.06.018 · Cited by 150
Defining the optimal method for measuring baseline metabolic tumour volume in diffuse large B cell lymphoma
Ilyas, H., Mikhaeel, N. G., Dunn, J. T., Rahman, F., Møller, H., Smith, D., & Barrington, S. F. (2018) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-018-3953-z · Cited by 150
Memory reactivation in healthy aging: evidence of stimulus-specific dedifferentiation
St-Laurent, M., Abdi, H., Bondad, A., & Buchsbaum, B. R. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.3054-13.2014 · Cited by 126
Direct comparison of US and MR elastography for staging liver fibrosis in patients with nonalcoholic fatty liver disease
Imajo, K., Honda, Y., Kobayashi, T., Nagai, K., Ozaki, A., Iwaki, M., Kessoku, T., Ogawa, Y., Takahashi, H., Saigusa, Y., et al. (2022) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2020.12.016 · Cited by 140
Are GAF scores reliable in routine clinical use?
Vatnaland, T., Vatnaland, J., Friis, S., & Opjordsmoen, S. (2007) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.2006.00925.x · Cited by 137
Reliability and Validity of the Multidimensional Dyspnea Profile
Meek, P. M., Banzett, R., Parsall, M. B., Gracely, R. H., Schwartzstein, R. M., & Lansing, R. (2012) · Chest · doi:10.1378/chest.11-1087 · Cited by 127
Initial validation of a computer-administered Addiction Severity Index: The ASI–MV.
Bultler, S. F., Budman, S. H., Goldman, R. J., Newman, F. J., Beckley, K. E., Trottier, D., & Cacciola, J. S. (2001) · Psychology of Addictive Behaviors · doi:10.1037/0893-164x.15.1.4 · Cited by 127
Stop Signal and Conners’ Continuous Performance Tasks: Test—Retest Reliability of Two Inhibition Measures in ADHD Children
Soreni, N., Crosbie, J., Ickowicz, A., & Schachar, R. (2009) · Journal of Attention Disorders · doi:10.1177/1087054708326110 · Cited by 121
Symptom Dimensions of the Psychotic Symptom Rating Scales in Psychosis: A Multisite Study
Woodward, T. S., Jung, K., Hwang, H., Yin, J., Taylor, L., Menon, M., Peters, E., Kuipers, E., Waters, F., Lecomte, T., et al. (2014) · Schizophrenia Bulletin · doi:10.1093/schbul/sbu014 · Cited by 116
The SENSE study: Post intervention effects of a randomized controlled trial of a cognitive-behavioral and mindfulness-based group sleep improvement intervention among at-risk adolescents
Blake, M., Waloszek, J. M., Schwartz, O., Raniti, M., Simmons, J. G., Blake, L., Murray, G., Dahl, R. E., Bootzin, R., Dudgeon, P., et al. (2016) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000142 · Cited by 114
Ventromedial prefrontal damage causes a pervasive impairment of episodic memory and future thinking
Bertossi, E., Tesini, C., Cappelli, A., & Ciaramelli, E. (2016) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2016.01.034 · Cited by 101
Developments in the Criterion–related Validation of Selection Procedures: A Critical Review and Recommendations for Practice
VAN IDDEKINGE, C. H., & PLOYHART, R. E. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00133.x · Cited by 101
A randomized controlled trial of cognitive behavioral therapy (CBT) for adjusting to multiple sclerosis (the saMS trial): Does CBT work and for whom does it work?
Moss-Morris, R., Dennison, L., Landau, S., Yardley, L., Silber, E., & Chalder, T. (2013) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0029132 · Cited by 97
Bladder neck mobility in continent nulliparous women
Peschers, U. M., Fanger, G., Schaer, G. N., Vodusek, D. B., DeLancey, J. O. L., & Schuessler, B. (2001) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2001.00066.x · Cited by 96
Implicit and explicit measures of alcohol and smoking cognitions.
McCarthy, D. M., & Thompsen, D. M. (2006) · Psychology of Addictive Behaviors · doi:10.1037/0893-164x.20.4.436 · Cited by 91
Between-seasons test-retest reliability of clinically measured reaction time in National Collegiate Athletic Association Division I athletes.
Eckner, J. T., Kutcher, J. S., & Richardson, J. K. (2011) · Journal of Athletic Training · doi:10.4085/1062-6050-46.4.409 · Cited by 89
The process of change in cognitive therapy for depression when combined with antidepressant medication: predictors of early intersession symptom gains
Strunk, D. R., Cooper, A. A., Ryan, E. T., DeRubeis, R. J., & Hollon, S. D. (2012) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0029281 · Cited by 77
Quality of systematic reviews in pediatric oncology – A systematic review
Lundh, A., Knijnenburg, S. L., Jørgensen, A. W., van Dalen, E. C., & Kremer, L. C. M. (2009) · Cancer Treatment Reviews · doi:10.1016/j.ctrv.2009.08.010 · Cited by 78
Gaining Insight Into Human Nature: A Review of Literature Classroom Intervention Studies
Schrijvers, M., Janssen, T., Fialho, O., & Rijlaarsdam, G. (2019) · Review of Educational Research · doi:10.3102/0034654318812914 · Cited by 76
Examining psychometric properties and measurement invariance of a Chinese version of the Self-Compassion Scale – Short Form (SCS-SF) in nursing students and medical workers
Meng, R., Yu, Y., Chai, S., Luo, X., Gong, B., Liu, B., Hu, Y., Luo, Y., & Yu, C. (2019) · Psychology Research and Behavior Management · doi:10.2147/prbm.s216411 · Cited by 72
Psychometric properties of the NIH Toolbox cognition battery in healthy older adults: reliability, validity, and agreement with standard neuropsychological tests
Scott, E. P., Sorrell, A., & Benitez, A. (2019) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617719000614 · Cited by 69
The OCI-12: A syndromally valid modification of the obsessive-compulsive inventory-revised
Abramovitch, A., Abramowitz, J. S., & McKay, D. (2021) · Psychiatry Research · doi:10.1016/j.psychres.2021.113808 · Cited by 68
Validity and inter-rater reliability of inertial gait measurements in Parkinson's disease: A pilot study
Esser, P., Dawes, H., Collett, J., Feltham, M. G., & Howells, K. (2012) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2012.01.005 · Cited by 67
Arginine vasopressin in cerebrospinal fluid is a marker of sociality in nonhuman primates
Parker, K. J., Garner, J. P., Oztan, O., Tarara, E. R., Li, J., Sclafani, V., Del Rosso, L. A., Chun, K., Berquist, S. W., Chez, M. G., et al. (2018) · Science Translational Medicine · doi:10.1126/scitranslmed.aam9100 · Cited by 64
Application of Generalizability Theory to the Big Five Inventory
Arterberry, B. J., Martens, M. P., Cadigan, J. M., & Rohrer, D. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.05.015 · Cited by 65
Tracking Evidence Based Practice with Youth: Validity of the MATCH and Standard Manual Consultation Records
Ward, A. M., Regan, J., Chorpita, B. F., Starace, N., Rodriguez, A., Okamura, K., Daleiden, E. L., Bearman, S. K., Weisz, J. R., & The Research Network on Youth Mental Health (2013) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2012.700505 · Cited by 62
Depression and Empathic Accuracy in Couples: An Interpersonal Model of Gender Differences in Depression
Gadassi, R., Mor, N., & Rafaeli, E. (2011) · Psychological Science · doi:10.1177/0956797611414728 · Cited by 56
Data Quality and Reliability Assessment of Wearable EMG and IMU Sensor for Construction Activity Recognition
Bangaru, S. S., Wang, C., & Aghazadeh, F. (2020) · Sensors · doi:10.3390/s20185264 · Cited by 58
The influence of follower mood on leader mood and task performance: An affective, follower-centric perspective of leadership
Tee, E. (. J., Ashkanasy, N. M., & Paulsen, N. (2013) · The Leadership Quarterly · doi:10.1016/j.leaqua.2013.03.005 · Cited by 59
Transdiagnostic versus Diagnosis-Specific Group Cognitive Behavioral Therapy for Anxiety Disorders and Depression: A Randomized Controlled Trial
Reinholt, N., Hvenegaard, M., Christensen, A. B., Eskildsen, A., Hjorthøj, C., Poulsen, S., Arendt, M. B., Rosenberg, N. K., Gryesten, J. R., Aharoni, R. N., et al. (2022) · Psychotherapy and Psychosomatics · doi:10.1159/000516380 · Cited by 55
Moment-to-Moment Brain Signal Variability Reliably Predicts Psychiatric Treatment Outcome
Månsson, K. N. T., Waschke, L., Manzouri, A., Furmark, T., Fischer, H., & Garrett, D. D. (2022) · Biological Psychiatry · doi:10.1016/j.biopsych.2021.09.026 · Cited by 49
Distance and percentage distance from median BMI as alternatives to BMI z score
Freedman, D. S., Woo, J. G., Ogden, C. L., Xu, J. H., & Cole, T. J. (2020) · British Journal of Nutrition · doi:10.1017/s0007114519002046 · Cited by 49
Administering the MADRS by telephone or face-to-face: a validity study.
Hermens, M. L., Adèr, H. J., van Hout, H. P., Terluin, B., van Dyck, R., & de Haan, M. (2006) · Annals of General Psychiatry · doi:10.1186/1744-859x-5-3 · Cited by 47
Therapist Effects in Guided Internet-Delivered CBT for Anxiety Disorders
Almlöv, J., Carlbring, P., Källqvist, K., Paxling, B., Cuijpers, P., & Andersson, G. (2011) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s135246581000069x · Cited by 48
The psychosocial correlates of quality of life in the dialysis population: a systematic review and meta-regression analysis
Chan, R., Brooks, R., Steel, Z., Heung, T., Erlich, J., Chow, J., & Suranyi, M. (2012) · Quality of Life Research · doi:10.1007/s11136-011-9973-9 · Cited by 48
Voxel-based morphometry at ultra-high fields. A comparison of 7T and 3T MRI data
Seiger, R., Hahn, A., Hummer, A., Kranz, G. S., Ganger, S., Küblböck, M., Kraus, C., Sladky, R., Kasper, S., Windischberger, C., et al. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.03.019 · Cited by 48
Two-way imputation: a Bayesian method for estimating missing scores in tests and questionnaires, and an accurate approximation
Van Ginkel, J. R., Andries Van der Ark, L., Sijtsma, K., & Vermunt, J. K. (2007) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2006.12.022 · Cited by 46
Comparison of inter‐ and intraobserver agreement and reliability between three different types of placental volume measurement technique (XI VOCAL™, VOCAL™ and multiplanar) and validity in the in‐vitro setting
Cheong, K. B., Leung, K. Y., Li, T. K. T., Chan, H. Y., Lee, Y. P., & Tang, M. H. Y. (2010) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.7609 · Cited by 45
Reliability of Adherence and Competence Assessment in Cognitive Behavioral Therapy: Influence of Clinical Experience
Weck, F., Hilling, C., Schermelleh-Engel, K., Rudari, V., & Stangier, U. (2011) · Journal of Nervous & Mental Disease · doi:10.1097/nmd.0b013e3182124617 · Cited by 44
Occupational Prestige: The Status Component of Socioeconomic Status
Hughes, B. T., Srivastava, S., Leszko, M., & Condon, D. M. (2024) · Collabra: Psychology · doi:10.1525/collabra.92882 · Cited by 44
Test-retest reliability and sample size estimates after MRI scanner relocation
Melzer, T. R., Keenan, R. J., Leeper, G. J., Kingston-Smith, S., Felton, S. A., Green, S. K., Henderson, K. J., Palmer, N. J., Shoorangiz, R., Almuqbel, M. M., et al. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116608 · Cited by 43
Performing Multilingual Analysis With Linguistic Inquiry and Word Count 2015 (LIWC2015). An Equivalence Study of Four Languages
Dudău, D. P., & Sava, F. A. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.570568 · Cited by 36
Asymptotic variability of (multilevel) multirater kappa coefficients
Vanbelle, S. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218794733 · Cited by 33
Examiner training and reliability in two randomized clinical trials of adult dental caries.
Banting, D. W., Amaechi, B. T., Bader, J. D., Blanchard, P., Gilbert, G. H., Gullion, C. M., Holland, J. C., Makhija, S. K., Papas, A., Ritter, A. V., et al. (2011) · Journal of public health dentistry · doi:10.1111/j.1752-7325.2011.00278.x · Cited by 32
“These numerical caries scores were also used to determine an intra-class correlation coefficient (ICC) that provides an estimate of inter-examiner agreement.(4, 5).”
Method
An experimental manipulation of maternal perfectionistic anxious rearing behaviors with anxious and non-anxious children
Mitchell, J. H., Broeren, S., Newall, C., & Hudson, J. L. (2013) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2012.12.006 · Cited by 30
Measuring Quality of Delivery in a Substance Use Prevention Program
Giles, S., Jackson-Newsom, J., Pankratz, M. M., Hansen, W. B., Ringwalt, C. L., & Dusenbury, L. (2008) · The Journal of Primary Prevention · doi:10.1007/s10935-008-0155-7 · Cited by 29
When Zero May Not Be Zero: A Cautionary Note on the Use of Inter-Rater Reliability in Evaluating Grant Peer Review
Erosheva, E. A., Martinková, P., & Lee, C. J. (2021) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12681 · Cited by 28
“Inter- rater reliability is defined as the proportion of variance attributable to the object of measurement as in the intra- class correlation coefficient (McGraw & Wong, 1996).”
BackgroundMethod
Assessing fidelity to evidence-based practices in usual care: The example of family therapy for adolescent behavior problems
Hogue, A., & Dauber, S. (2013) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2012.12.001 · Cited by 23
Looking Inside the Black Box of Mindfulness Meditation: Investigating Attention and Awareness During Meditation Using the Mindful Awareness Task (MAT)
Hadash, Y., Ruimi, L., & Bernstein, A. (2021) · doi:10.31231/osf.io/dwzcj · Cited by 23
Looking inside the black box of mindfulness meditation: Investigating attention and awareness during meditation using the mindful awareness task (MAT).
Hadash, Y., Ruimi, L., & Bernstein, A. (2023) · Psychological Assessment · doi:10.1037/pas0001194 · Cited by 23

Authors: McGraw, K. O., & Wong, S. P.
Affiliations: University of Mississippi (US)
Venue: Psychological Methods
Publisher: American Psychological Association (APA)
Published: 1996-03
Volume: 1
Issue: 1
Pages: 30-46
ISSN: 1082-989X (ISSN-L)
1939-1463
DOI: 10.1037/1082-989x.1.1.30
OpenAlex: W2067724039
Cited by: 6,903
References: 3
Citations by year: 2026: 382 · 2025: 392 · 2024: 350 · 2023: 403 · 2022: 430 · 2021: 531 · 2020: 463 · 2019: 405 · 2018: 354 · 2017: 286 · 2016: 298 · 2015: 300 · 2014: 309 · 2013: 250 · 2012: 273
FWCI: 20.24
Type: journal article
Language: en
Update notices: correction (2006-07-10)
Open access: closed
Locations: Psychological Methods (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Keywords: intraclass correlation coefficient, statistical inference, behavioral genetics, behavioral assessment, psychometrics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Forming inferences about some intraclass correlation coefficients.

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Metadata

Record

About these checks