Forming inferences about some intraclass correlation coefficients.
Abstract
Although intraclass correlation coefficients (ICCs) are commonly used in behavioral measurement, psychometrics, and behavioral genetics, procedures available for forming inferences about ICCs are not widely known. Following a review of the distinction between various forms of the ICC, this article p
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency).
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as McGraw and Wong 1996
Retrieval Accuracy showed strong agreement among annotators, with Percent Agreement ranging from 93% (GPT-4o) to 98% (GLM) and ICC values ranging from 78.1% for GPT-4o to 96.9% for GLM.
“Retrieval Accuracy showed strong agreement among annotators. Percent Agreement ranged from 93% (GPT-4o) to 98% (GLM). The ICC values, which measure absolute agreement between raters [39], range from 78.1% for GPT-4o to 96.9% for GLM.”
From CAIM: Development and Evaluation of a Cognitive AI Memory Framework for Long-Term Interaction with Intelligent Agents (2026) · cites this paper as [39]
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931 · see citing works
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 29,639 · see citing works
Quality criteria were proposed for measurement properties of health status questionnaires
Terwee, C. B., Bot, S. D. M., de Boer, M. R., van der Windt, D. A. W. M., Knol, D. L., Dekker, J., Bouter, L. M., & de Vet, H. C. W. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.012 · Cited by 10,764 · see citing works
Cane, J., O’Connor, D., & Michie, S. (2012) · Implementation Science · doi:10.1186/1748-5908-7-37 · Cited by 4,888 · see citing works
“To examine the agreement of these construct x domain ratings and construct assignment across participants, two-way intraclass correlation coefficient (ICC) measures of consistency were used within each domain [43].”
MethodZwanenburg, A., Vallières, M., Abdalah, M. A., Aerts, H. J. W. L., Andrearczyk, V., Apte, A., Ashrafinia, S., Bakas, S., Beukinga, R. J., Boellaard, R., et al. (2020) · Radiology · doi:10.1148/radiol.2020191145 · Cited by 4,177 · see citing works
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608 · see citing works
What Is the Male-to-Female Ratio in Autism Spectrum Disorder? A Systematic Review and Meta-Analysis
Loomes, R., Hull, L., & Mandy, W. P. L. (2017) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2017.03.013 · Cited by 2,804 · see citing works
Repeatability for Gaussian and non-Gaussian data: a practical guide for biologists.
Nakagawa, S., & Schielzeth, H. (2010) · Biological Reviews · doi:10.1111/j.1469-185x.2010.00141.x · Cited by 2,584 · see citing works
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053 · see citing works
Lai, M.-C., Kassee, C., Besney, R., Bonato, S., Hull, L., Mandy, W., Szatmari, P., & Ameis, S. H. (2019) · The Lancet Psychiatry · doi:10.1016/s2215-0366(19)30289-5 · Cited by 1,892 · see citing works
Dynamic reconfiguration of human brain networks during learning
Bassett, D. S., Wymbs, N. F., Porter, M. A., Mucha, P. J., Carlson, J. M., & Grafton, S. T. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1018985108 · Cited by 1,705 · see citing works
Are leader stereotypes masculine? A meta-analysis of three research paradigms
Koenig, A. M., Eagly, A. H., Mitchell, A. A., & Ristikari, T. (2011) · Psychological Bulletin · doi:10.1037/a0023557 · Cited by 1,748 · see citing works
Mills, J. L., Conte, M. S., Armstrong, D. G., Pomposelli, F. B., Schanzer, A., Sidawy, A. N., & Andros, G. (2014) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2013.08.003 · Cited by 1,718 · see citing works
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457 · see citing works
Aging and autobiographical memory: dissociating episodic from semantic retrieval
Levine, B., Svoboda, E., Hay, J. F., Winocur, G., & Moscovitch, M. (2002) · Psychology and Aging · doi:10.1037/0882-7974.17.4.677 · Cited by 1,521 · see citing works
Collins, N. L., & Feeney, B. C. (2000) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.78.6.1053 · Cited by 1,303 · see citing works
Varni, J. W., Limbers, C. A., & Burwinkle, T. M. (2007) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-5-1 · Cited by 1,075 · see citing works
Sample size requirements for estimating intraclass correlations with desired precision
Bonett, D. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1108 · Cited by 987 · see citing works
The social media disorder scale
van den Eijnden, R. J. J. M., Lemmens, J. S., & Valkenburg, P. M. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.03.038 · Cited by 948 · see citing works
Interrater agreement and interrater reliability: Key concepts, approaches, and applications
Gisev, N., Bell, J. S., & Chen, T. F. (2013) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2012.04.004 · Cited by 924 · see citing works
Interactions Within Groups and Subgroups: The Effects of Demographic Faultlines
Lau, D. C., & Murnighan, J. K. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.17843943 · Cited by 831 · see citing works
Love, J. M., Kisker, E. E., Ross, C., Raikes, H., Constantine, J., Boller, K., Brooks-Gunn, J., Chazan-Cohen, R., Tarullo, L. B., Brady-Smith, C., et al. (2005) · Developmental Psychology · doi:10.1037/0012-1649.41.6.885 · Cited by 800 · see citing works
Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008) · Value in Health · doi:10.1111/j.1524-4733.2007.00231.x · Cited by 623 · see citing works
Stability and Course of Neuropsychological Deficits in Schizophrenia
Heaton, R. K., Gladsjo, J. A., Palmer, B. W., Kuck, J., Marcotte, T. D., & Jeste, D. V. (2001) · Archives of General Psychiatry · doi:10.1001/archpsyc.58.1.24 · Cited by 618 · see citing works
Mokkink, L. B., Boers, M., van der Vleuten, C. P. M., Bouter, L. M., Alonso, J., Patrick, D. L., de Vet, H. C. W., & Terwee, C. B. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01179-5 · Cited by 588 · see citing works
Development and validation of the camouflaging autistic traits questionnaire (CAT-Q)
Hull, L., Mandy, W., Lai, M.-C., Baron-Cohen, S., Allison, C., Smith, P., & Petrides, K. V. (2019) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-018-3792-6 · Cited by 496 · see citing works
ISway: a sensitive, valid and reliable measure of postural control
Mancini, M., Salarian, A., Carlson-Kuhta, P., Zampieri, C., King, L., Chiari, L., & Horak, F. B. (2012) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-9-59 · Cited by 499 · see citing works
iTUG, a Sensitive and Reliable Measure of Mobility
Salarian, A., Horak, F. B., Zampieri, C., Carlson-Kuhta, P., Nutt, J. G., & Aminian, K. (2010) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2010.2047606 · Cited by 489 · see citing works
Multimedia instructions and cognitive load theory: Effects of modality and cueing
Tabbers, H. K., Martens, R. L., & Van Merriënboer, J. J. G. (2004) · British Journal of Educational Psychology · doi:10.1348/000709904322848824 · Cited by 478 · see citing works
Activity Achievement Emotions and Academic Performance: A Meta-analysis
Camacho-Morles, J., Slemp, G. R., Pekrun, R., Loderer, K., Hou, H., & Oades, L. G. (2021) · Educational Psychology Review · doi:10.1007/s10648-020-09585-3 · Cited by 475 · see citing works
Cho, K., Schunn, C. D., & Wilson, R. W. (2006) · Journal of Educational Psychology · doi:10.1037/0022-0663.98.4.891 · Cited by 355 · see citing works
Sahdra, B. K., Shaver, P. R., & Brown, K. W. (2010) · Journal of Personality Assessment · doi:10.1080/00223890903425960 · Cited by 352 · see citing works
Hunot, C., Fildes, A., Croker, H., Llewellyn, C. H., Wardle, J., & Beeken, R. J. (2016) · Appetite · doi:10.1016/j.appet.2016.05.024 · Cited by 340 · see citing works
Language‐association cortex asymmetry in autism and specific language impairment
De Fossé, L., Hodge, S. M., Makris, N., Kennedy, D. N., Caviness, V. S., McGrath, L., Steele, S., Ziegler, D. A., Herbert, M. R., Frazier, J. A., et al. (2004) · Annals of Neurology · doi:10.1002/ana.20275 · Cited by 339 · see citing works
How school leadership influences student learning: A test of “The four paths model”
Leithwood, K., Sun, J., & Schumacker, R. (2020) · Educational Administration Quarterly · doi:10.1177/0013161x19878772 · Cited by 329 · see citing works
Griffiths, P., Dall’Ora, C., Simon, M., Ball, J., Lindqvist, R., Rafferty, A.-M., Schoonhoven, L., Tishelman, C., & Aiken, L. H. (2014) · Medical Care · doi:10.1097/mlr.0000000000000233 · Cited by 316 · see citing works
“The ICC 1 measures the degree of similarity between individuals within a cluster.(29) It also indicates the proportion of variance in the outcome that can be attributed to variation between groups (wards, hospitals, countries) as opposed to between individuals.”
BackgroundGlioblastoma Multiforme: Exploratory Radiogenomic Analysis by Using Quantitative Image Features
Gevaert, O., Mitchell, L. A., Achrol, A. S., Xu, J., Echegaray, S., Steinberg, G. K., Cheshier, S. H., Napel, S., Zaharchuk, G., & Plevritis, S. K. (2014) · Radiology · doi:10.1148/radiol.14131731 · Cited by 319 · see citing works
Effects of caffeine on the human circadian clock in vivo and in vitro
Burke, T. M., Markwald, R. R., McHill, A. W., Chinoy, E. D., Snider, J. A., Bessman, S. C., Jung, C. M., O’Neill, J. S., & Wright, K. P. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aac5125 · Cited by 283 · see citing works
Qualitative Attributes and Measurement Properties of Physical Activity Questionnaires: A Checklist
Terwee, C. B., Mokkink, L. B., van Poppel, M. N. M., Chinapaw, M. J. M., van Mechelen, W., & de Vet, H. C. W. (2010) · Sports Medicine · doi:10.2165/11531370-000000000-00000 · Cited by 269 · see citing works
Measuring fMRI reliability with the intra-class correlation coefficient
Caceres, A., Hall, D. L., Zelaya, F. O., Williams, S. C. R., & Mehta, M. A. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.12.035 · Cited by 255 · see citing works
Bramen, J. E., Hranilovich, J. A., Dahl, R. E., Forbes, E. E., Chen, J., Toga, A. W., Dinov, I. D., Worthman, C. M., & Sowell, E. R. (2011) · Cerebral Cortex · doi:10.1093/cercor/bhq137 · Cited by 266 · see citing works
Executive Function in Preschool Children: Test–Retest Reliability
Beck, D. M., Schaefer, C., Pang, K., & Carlson, S. M. (2011) · Journal of Cognition and Development · doi:10.1080/15248372.2011.563485 · Cited by 262 · see citing works
Confidence Limits for Intraclass Reliability Coefficients
Baumgartner, T. A., & Chung, H. (2001) · Measurement in Physical Education and Exercise Science · doi:10.1207/s15327841mpee0503_4 · Cited by 249 · see citing works
One-year test-retest reliability of intrinsic connectivity network fMRI in older adults
Guo, C. C., Kurth, F., Zhou, J., Mayer, E. A., Eickhoff, S. B., Kramer, J. H., & Seeley, W. W. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.03.027 · Cited by 252 · see citing works
Self-efficacy and resource allocation: Support for a nonmonotonic, discontinuous model.
Vancouver, J. B., More, K. M., & Yoder, R. J. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.1.35 · Cited by 234 · see citing works
<b>Research Note</b> —A Model of Conflict, Leadership, and Performance in Virtual Teams
Wakefield, R. L., Leidner, D. E., & Garrison, G. (2008) · Information Systems Research · doi:10.1287/isre.1070.0149 · Cited by 220 · see citing works
Clarke, D. E., Narrow, W. E., Regier, D. A., Kuramoto, S. J., Kupfer, D. J., Kuhl, E. A., Greiner, L., & Kraemer, H. C. (2013) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.12070998 · Cited by 201 · see citing works
Eddy, S. L., Converse, M., & Wenderoth, M. P. (2015) · CBE—Life Sciences Education · doi:10.1187/cbe.14-06-0095 · Cited by 182 · see citing works
“The resulting ICCs were all > 0.8, and thus all were in the “excellent” range (Supplemental Table 2; Cicchetti 1994).”
MethodInfluentialTransformational and Transactional Leadership in Virtual and Physical Environments
Hoyt, C. L., & Blascovich, J. (2003) · Small Group Research · doi:10.1177/1046496403257527 · Cited by 181 · see citing works
A meta-analysis of autonomous and controlled forms of teacher motivation
Slemp, G. R., Field, J. G., & Cho, A. S. H. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2020.103459 · Cited by 179 · see citing works
Douw, L., Wakeman, D. G., Tanaka, N., Liu, H., & Stufflebeam, S. M. (2016) · Neuroscience · doi:10.1016/j.neuroscience.2016.09.034 · Cited by 162 · see citing works
A scale for rating the quality of psychological trials for pain
Yates, S. L., Morley, S., Eccleston, C., & de C. Williams, A. C. (2005) · Pain · doi:10.1016/j.pain.2005.06.018 · Cited by 150 · see citing works
Ilyas, H., Mikhaeel, N. G., Dunn, J. T., Rahman, F., Møller, H., Smith, D., & Barrington, S. F. (2018) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-018-3953-z · Cited by 150 · see citing works
Memory reactivation in healthy aging: evidence of stimulus-specific dedifferentiation
St-Laurent, M., Abdi, H., Bondad, A., & Buchsbaum, B. R. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.3054-13.2014 · Cited by 126 · see citing works
Imajo, K., Honda, Y., Kobayashi, T., Nagai, K., Ozaki, A., Iwaki, M., Kessoku, T., Ogawa, Y., Takahashi, H., Saigusa, Y., et al. (2022) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2020.12.016 · Cited by 140 · see citing works
Are GAF scores reliable in routine clinical use?
Vatnaland, T., Vatnaland, J., Friis, S., & Opjordsmoen, S. (2007) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.2006.00925.x · Cited by 137 · see citing works
Reliability and Validity of the Multidimensional Dyspnea Profile
Meek, P. M., Banzett, R., Parsall, M. B., Gracely, R. H., Schwartzstein, R. M., & Lansing, R. (2012) · Chest · doi:10.1378/chest.11-1087 · Cited by 127 · see citing works
Initial validation of a computer-administered Addiction Severity Index: The ASI–MV.
Bultler, S. F., Budman, S. H., Goldman, R. J., Newman, F. J., Beckley, K. E., Trottier, D., & Cacciola, J. S. (2001) · Psychology of Addictive Behaviors · doi:10.1037/0893-164x.15.1.4 · Cited by 127 · see citing works
Soreni, N., Crosbie, J., Ickowicz, A., & Schachar, R. (2009) · Journal of Attention Disorders · doi:10.1177/1087054708326110 · Cited by 121 · see citing works
Symptom Dimensions of the Psychotic Symptom Rating Scales in Psychosis: A Multisite Study
Woodward, T. S., Jung, K., Hwang, H., Yin, J., Taylor, L., Menon, M., Peters, E., Kuipers, E., Waters, F., Lecomte, T., et al. (2014) · Schizophrenia Bulletin · doi:10.1093/schbul/sbu014 · Cited by 116 · see citing works
Blake, M., Waloszek, J. M., Schwartz, O., Raniti, M., Simmons, J. G., Blake, L., Murray, G., Dahl, R. E., Bootzin, R., Dudgeon, P., et al. (2016) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000142 · Cited by 114 · see citing works
Ventromedial prefrontal damage causes a pervasive impairment of episodic memory and future thinking
Bertossi, E., Tesini, C., Cappelli, A., & Ciaramelli, E. (2016) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2016.01.034 · Cited by 101 · see citing works
VAN IDDEKINGE, C. H., & PLOYHART, R. E. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00133.x · Cited by 101 · see citing works
Moss-Morris, R., Dennison, L., Landau, S., Yardley, L., Silber, E., & Chalder, T. (2013) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0029132 · Cited by 97 · see citing works
Bladder neck mobility in continent nulliparous women
Peschers, U. M., Fanger, G., Schaer, G. N., Vodusek, D. B., DeLancey, J. O. L., & Schuessler, B. (2001) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2001.00066.x · Cited by 96 · see citing works
Implicit and explicit measures of alcohol and smoking cognitions.
McCarthy, D. M., & Thompsen, D. M. (2006) · Psychology of Addictive Behaviors · doi:10.1037/0893-164x.20.4.436 · Cited by 91 · see citing works
Eckner, J. T., Kutcher, J. S., & Richardson, J. K. (2011) · Journal of Athletic Training · doi:10.4085/1062-6050-46.4.409 · Cited by 89 · see citing works
Strunk, D. R., Cooper, A. A., Ryan, E. T., DeRubeis, R. J., & Hollon, S. D. (2012) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0029281 · Cited by 77 · see citing works
Quality of systematic reviews in pediatric oncology – A systematic review
Lundh, A., Knijnenburg, S. L., Jørgensen, A. W., van Dalen, E. C., & Kremer, L. C. M. (2009) · Cancer Treatment Reviews · doi:10.1016/j.ctrv.2009.08.010 · Cited by 78 · see citing works
Gaining Insight Into Human Nature: A Review of Literature Classroom Intervention Studies
Schrijvers, M., Janssen, T., Fialho, O., & Rijlaarsdam, G. (2019) · Review of Educational Research · doi:10.3102/0034654318812914 · Cited by 76 · see citing works
Meng, R., Yu, Y., Chai, S., Luo, X., Gong, B., Liu, B., Hu, Y., Luo, Y., & Yu, C. (2019) · Psychology Research and Behavior Management · doi:10.2147/prbm.s216411 · Cited by 72 · see citing works
Scott, E. P., Sorrell, A., & Benitez, A. (2019) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617719000614 · Cited by 69 · see citing works
The OCI-12: A syndromally valid modification of the obsessive-compulsive inventory-revised
Abramovitch, A., Abramowitz, J. S., & McKay, D. (2021) · Psychiatry Research · doi:10.1016/j.psychres.2021.113808 · Cited by 68 · see citing works
Esser, P., Dawes, H., Collett, J., Feltham, M. G., & Howells, K. (2012) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2012.01.005 · Cited by 67 · see citing works
Arginine vasopressin in cerebrospinal fluid is a marker of sociality in nonhuman primates
Parker, K. J., Garner, J. P., Oztan, O., Tarara, E. R., Li, J., Sclafani, V., Del Rosso, L. A., Chun, K., Berquist, S. W., Chez, M. G., et al. (2018) · Science Translational Medicine · doi:10.1126/scitranslmed.aam9100 · Cited by 64 · see citing works
Application of Generalizability Theory to the Big Five Inventory
Arterberry, B. J., Martens, M. P., Cadigan, J. M., & Rohrer, D. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.05.015 · Cited by 65 · see citing works
Ward, A. M., Regan, J., Chorpita, B. F., Starace, N., Rodriguez, A., Okamura, K., Daleiden, E. L., Bearman, S. K., Weisz, J. R., & The Research Network on Youth Mental Health (2013) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2012.700505 · Cited by 62 · see citing works
Gadassi, R., Mor, N., & Rafaeli, E. (2011) · Psychological Science · doi:10.1177/0956797611414728 · Cited by 56 · see citing works
Bangaru, S. S., Wang, C., & Aghazadeh, F. (2020) · Sensors · doi:10.3390/s20185264 · Cited by 58 · see citing works
Tee, E. (. J., Ashkanasy, N. M., & Paulsen, N. (2013) · The Leadership Quarterly · doi:10.1016/j.leaqua.2013.03.005 · Cited by 59 · see citing works
Reinholt, N., Hvenegaard, M., Christensen, A. B., Eskildsen, A., Hjorthøj, C., Poulsen, S., Arendt, M. B., Rosenberg, N. K., Gryesten, J. R., Aharoni, R. N., et al. (2022) · Psychotherapy and Psychosomatics · doi:10.1159/000516380 · Cited by 55 · see citing works
Moment-to-Moment Brain Signal Variability Reliably Predicts Psychiatric Treatment Outcome
Månsson, K. N. T., Waschke, L., Manzouri, A., Furmark, T., Fischer, H., & Garrett, D. D. (2022) · Biological Psychiatry · doi:10.1016/j.biopsych.2021.09.026 · Cited by 49 · see citing works
Administering the MADRS by telephone or face-to-face: a validity study.
Hermens, M. L., Adèr, H. J., van Hout, H. P., Terluin, B., van Dyck, R., & de Haan, M. (2006) · Annals of General Psychiatry · doi:10.1186/1744-859x-5-3 · Cited by 47 · see citing works
Therapist Effects in Guided Internet-Delivered CBT for Anxiety Disorders
Almlöv, J., Carlbring, P., Källqvist, K., Paxling, B., Cuijpers, P., & Andersson, G. (2011) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s135246581000069x · Cited by 48 · see citing works
Chan, R., Brooks, R., Steel, Z., Heung, T., Erlich, J., Chow, J., & Suranyi, M. (2012) · Quality of Life Research · doi:10.1007/s11136-011-9973-9 · Cited by 48 · see citing works
Voxel-based morphometry at ultra-high fields. A comparison of 7T and 3T MRI data
Seiger, R., Hahn, A., Hummer, A., Kranz, G. S., Ganger, S., Küblböck, M., Kraus, C., Sladky, R., Kasper, S., Windischberger, C., et al. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.03.019 · Cited by 48 · see citing works
Van Ginkel, J. R., Andries Van der Ark, L., Sijtsma, K., & Vermunt, J. K. (2007) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2006.12.022 · Cited by 46 · see citing works
Cheong, K. B., Leung, K. Y., Li, T. K. T., Chan, H. Y., Lee, Y. P., & Tang, M. H. Y. (2010) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.7609 · Cited by 45 · see citing works
Weck, F., Hilling, C., Schermelleh-Engel, K., Rudari, V., & Stangier, U. (2011) · Journal of Nervous & Mental Disease · doi:10.1097/nmd.0b013e3182124617 · Cited by 44 · see citing works
Occupational Prestige: The Status Component of Socioeconomic Status
Hughes, B. T., Srivastava, S., Leszko, M., & Condon, D. M. (2024) · Collabra: Psychology · doi:10.1525/collabra.92882 · Cited by 44 · see citing works
Test-retest reliability and sample size estimates after MRI scanner relocation
Melzer, T. R., Keenan, R. J., Leeper, G. J., Kingston-Smith, S., Felton, S. A., Green, S. K., Henderson, K. J., Palmer, N. J., Shoorangiz, R., Almuqbel, M. M., et al. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116608 · Cited by 43 · see citing works
Dudău, D. P., & Sava, F. A. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.570568 · Cited by 36 · see citing works
Asymptotic variability of (multilevel) multirater kappa coefficients
Vanbelle, S. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218794733 · Cited by 33 · see citing works
Examiner training and reliability in two randomized clinical trials of adult dental caries.
Banting, D. W., Amaechi, B. T., Bader, J. D., Blanchard, P., Gilbert, G. H., Gullion, C. M., Holland, J. C., Makhija, S. K., Papas, A., Ritter, A. V., et al. (2011) · Journal of public health dentistry · doi:10.1111/j.1752-7325.2011.00278.x · Cited by 32 · see citing works
“These numerical caries scores were also used to determine an intra-class correlation coefficient (ICC) that provides an estimate of inter-examiner agreement.(4, 5).”
MethodMitchell, J. H., Broeren, S., Newall, C., & Hudson, J. L. (2013) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2012.12.006 · Cited by 30 · see citing works
Measuring Quality of Delivery in a Substance Use Prevention Program
Giles, S., Jackson-Newsom, J., Pankratz, M. M., Hansen, W. B., Ringwalt, C. L., & Dusenbury, L. (2008) · The Journal of Primary Prevention · doi:10.1007/s10935-008-0155-7 · Cited by 29 · see citing works
Erosheva, E. A., Martinková, P., & Lee, C. J. (2021) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12681 · Cited by 28 · see citing works
“Inter- rater reliability is defined as the proportion of variance attributable to the object of measurement as in the intra- class correlation coefficient (McGraw & Wong, 1996).”
BackgroundMethodHogue, A., & Dauber, S. (2013) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2012.12.001 · Cited by 23 · see citing works
Hadash, Y., Ruimi, L., & Bernstein, A. (2021) · doi:10.31231/osf.io/dwzcj · Cited by 23 · see citing works
Hadash, Y., Ruimi, L., & Bernstein, A. (2023) · Psychological Assessment · doi:10.1037/pas0001194 · Cited by 23 · see citing works
Speech-Language Dissociations, Distractibility, and Childhood Stuttering
Clark, C. E., Conture, E. G., Walden, T. A., & Lambert, W. E. (2015) · American Journal of Speech-Language Pathology · doi:10.1044/2015_ajslp-14-0198 · Cited by 22 · see citing works
- Authors
- McGraw, K. O., & Wong, S. P.
- Affiliations
- University of Mississippi (US)
- Venue
- Psychological Methods
- Published
- 1996-03
- Volume
- 1
- Issue
- 1
- Pages
- 30-46
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2067724039
- Cited by
- 6,903
- References
- 3
- Citations by year
- 2026: 382 · 2025: 392 · 2024: 350 · 2023: 403 · 2022: 430 · 2021: 531 · 2020: 463 · 2019: 405 · 2018: 354 · 2017: 286 · 2016: 298 · 2015: 300 · 2014: 309 · 2013: 250 · 2012: 273
- FWCI
- 20.24
- Type
- journal article
- Language
- en
- Update notices
- correction (2006-07-10)
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- intraclass correlation coefficient, statistical inference, behavioral genetics, behavioral assessment, psychometrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review