The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 4,701–4,800 of 23,883
Mall, N. A., Hardaker, W. M., Nunley, J. A., et al. (2006) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2006.04.021 · Cited by 62
Tragesser, S. L., Sher, K. J., Trull, T. J., & Park, A. (2007) · Experimental and Clinical Psychopharmacology · doi:10.1037/1064-1297.15.3.282 · Cited by 62
Problem-Solving Team Behaviors
Taggar, S., & Brown, T. C. (2001) · Small Group Research · doi:10.1177/104649640103200602 · Cited by 62
Müller, M. J., & Szegedi, A. (2002) · Journal of Clinical Psychopharmacology · doi:10.1097/00004714-200206000-00013 · Cited by 62
VALIDITY AND REPRODUCIBILITY OF HAND-HELD DYNAMOMETRY IN CHILDREN AGED 4–11 YEARS
van den Beld, W. A., van der Sanden, G., Sengers, R. C. A., Verbeek, A. L. M., & Gabreëls, F. (2005) · Journal of Rehabilitation Medicine · doi:10.1080/16501970510044043 · Cited by 63
Cross-study validation and combined analysis of gene expression microarray data
Garrett-Mayer, E. S., Parmigiani, G., Zhong, X., Cope, L., & Gabrielson, E. W. (2007) · Biostatistics · doi:10.1093/biostatistics/kxm033 · Cited by 60
A Case Meta-Analysis of Gainsharing Plans as Organization Development Interventions
Bullock, R. J., & Tubbs, M. E. (1990) · The Journal of Applied Behavioral Science · doi:10.1177/0021886390263011 · Cited by 62
Valla, J., Bergeron, L., Bidaut‐Russell, M., St‐Georges, M., & Gaudet, N. (1997) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.1997.tb01698.x · Cited by 62
Lora, A., Bai, G., Bianchi, S., Bolongaro, G., Civenti, G., Erlicher, A., Maresca, G., Monzani, E., Panetta, B., Morgen, D. V., et al. (2001) · Epidemiologia e Psichiatria Sociale · doi:10.1017/s1121189x00005339 · Cited by 65
Kues, J. M., Rothstein, J. M., & Lamb, R. L. (1992) · Physical Therapy · doi:10.1093/ptj/72.7.492 · Cited by 62
Parkinsonism and Related Disorders
Slow, E., & Lang, A. E. (2019) · DeckerMed Transitional Year Weekly Curriculum™ · doi:10.2310/tywc.1178 · Cited by 62
Observer Reliability and Agreement
de Vet, H. C. W. (2005) · Encyclopedia of Biostatistics · doi:10.1002/0470011815.b2a04033 · Cited by 62
Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning
Sachdeva, E., Agarwal, N., Chundi, S., Roelofs, S., Li, J., Kochenderfer, M., Choi, C., & Dariush, B. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00734 · Cited by 63
Li, L., Zeng, L., Lin, Z.-J., Cazzell, M., & Liu, H. (2015) · Journal of Biomedical Optics · doi:10.1117/1.jbo.20.5.050801 · Cited by 78
Marra, C. A., Lynd, L. D., Harvard, S. S., et al. (2011) · BMC Health Services Research · doi:10.1186/1472-6963-11-69 · Cited by 61
The Efficacy of Acute Electroconvulsive Therapy in Atypical Depression
Husain, M. M., McClintock, S. M., Rush, A. J., Knapp, R. G., Fink, M., Rummans, T. A., Rasmussen, K. G., Claassen, C. A., Petrides, G., Biggs, M. M., et al. (2008) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.v69n0310 · Cited by 61
López, F. (2012) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/016146811211400803 · Cited by 61
Hanley‐Cook, G., Tung, J. Y. A., Sattamini, I. F., et al. (2020) · Nutrients · doi:10.3390/nu12072039 · Cited by 61
Gupta, A. S., Patel, S., Premasiri, A. S., & Vieira, F. G. (2023) · Nature Communications · doi:10.1038/s41467-023-40917-3 · Cited by 61
Conscientiousness Is Not Always a Good Predictor of Performance: The Case of Creativity
Reiter‐Palmon, R., Illies, J. J., & Cross, L. K. (2009) · DigitalCommons@UNO · Cited by 61
de Winckel, A. V., Feys, H., van der Knaap, S., Messerli, R., Baronti, F., Lehmann, R., Hemelrijk, B. V., Panté, F., Perfetti, C., & Weerdt, W. D. (2006) · Clinical Rehabilitation · doi:10.1177/0269215506072181 · Cited by 62
Jans-Beken, L., Lataster, J., Leontjevas, R., & Jacobs, N. (2015) · Psychologica Belgica · doi:10.5334/pb.bd · Cited by 64
Hicks, G. E., George, S. Z., Nevitt, M. C., Cauley, J. A., & Vogt, M. T. (2006) · Journal of Spinal Disorders & Techniques · doi:10.1097/01.bsd.0000210116.94273.ad · Cited by 61
Test-Retest Reliability for Aerodynamic Measures of Voice
Awan, S. N., Novaleski, C. K., & Yingling, J. R. (2013) · Journal of Voice · doi:10.1016/j.jvoice.2013.07.002 · Cited by 61
Velocity Vector Imaging: Standard Tissue-Tracking Results Acquired in Normals—The VVI-STRAIN Study
Carasso, S., Biaggi, P., Rakowski, H., et al. (2012) · Journal of the American Society of Echocardiography · doi:10.1016/j.echo.2012.01.005 · Cited by 61
Music and Psychophysiological Recovery from Stress
Radstaak, M., Geurts, S. A. E., BROSSCHOT, J. F., & Kompier, M. A. J. (2014) · Psychosomatic Medicine · doi:10.1097/psy.0000000000000094 · Cited by 66
Automated Prediction of the Apnea-Hypopnea Index from Nocturnal Oximetry Recordings
Marcos, J. V., Hornero, R., Álvarez, D., Aboy, M., & del Campo, F. (2011) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2011.2167971 · Cited by 61
Bless, J. J., Westerhausen, R., Arciuli, J., Kompus, K., Gudmundsen, M., & Hugdahl, K. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00042 · Cited by 63
Tract Specific Reproducibility of Tractography Based Morphology and Diffusion Metrics
Besseling, R. M. H., Jansen, J. F. A., Overvliet, G. M., Vaessen, M. J., Braakman, H. M. H., Hofman, P. A. M., Aldenkamp, A. P., & Backes, W. H. (2012) · PLoS ONE · doi:10.1371/journal.pone.0034125 · Cited by 62
The measurement of quality management culture in schools: development and validation of the SQMCS
Detert, J. R., Schroeder, R. G., & Cudeck, R. A. (2003) · Journal of Operations Management · doi:10.1016/s0272-6963(02)00130-4 · Cited by 61
Klos, T. V. S., Harman, M. K., Habets, R. J. E., et al. (2000) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1053/jars.2000.4634 · Cited by 61
The Effects of Competition on Improvisers’ Motivation, Stress, and Creative Performance
Eisenberg, J., & Thompson, W. F. (2011) · Creativity Research Journal · doi:10.1080/10400419.2011.571185 · Cited by 72
Vrabel, K. R., Hoffart, A., Rø, Ø., Martinsen, E. W., & Rosenvinge, J. H. (2010) · Journal of Abnormal Psychology · doi:10.1037/a0019857 · Cited by 62
Farraher, S. W., Jara, H. J., Chang, K. J., et al. (2006) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.20758 · Cited by 61
Baum, T. H., Stehling, C., Joseph, G. B., Carballido‐Gamio, J., Schwaiger, B. J., Müller-Höcker, C., Nevitt, M. C., Lynch, J. A., McCulloch, C. E., & Link, T. M. (2011) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.22834 · Cited by 61
Timing phases of the sit-to-walk movement: Validity of a clinical test
Kerr, A., Rafferty, D., Kerr, K. M., et al. (2006) · Gait & Posture · doi:10.1016/j.gaitpost.2006.07.004 · Cited by 61
Perceptual Structure of Adductor Spasmodic Dysphonia and Its Acoustic Correlates
Cannito, M. P., Doiuchi, M., Murry, T., et al. (2012) · Journal of Voice · doi:10.1016/j.jvoice.2012.05.005 · Cited by 61
Effects of Reward on Self‐regulation, Intrinsic Motivation and Creativity
Selart, M., Nordström, T., Kuvaas, B., & Takemura, K. (2008) · Scandinavian Journal of Educational Research · doi:10.1080/00313830802346314 · Cited by 61
Furr-Holden, D., Campbell, K. D. M., Milam, A. J., Smart, M. J., Ialongo, N., & Leaf, P. J. (2010) · Evaluation Review · doi:10.1177/0193841x10368493 · Cited by 61
Test-retest and inter-rater reliability of a method to measure wrist and finger spasticity
Gäverth, J., Sandgren, M., Lindberg, P. G., et al. (2013) · Journal of Rehabilitation Medicine · doi:10.2340/16501977-1160 · Cited by 61
Discourse Complexity of College Writers With and Without Disabilities
Gregg, N., Coleman, C., Stennett, R. B., et al. (2002) · Journal of Learning Disabilities · doi:10.1177/002221940203500103 · Cited by 61
Low Kilovoltage CT of the Neck with 70 kVp: Comparison with a Standard Protocol
Gnannt, R., Winklehner, A., Goetti, R. P., Schmidt, B. T., Kollias, S., & Alkadhi, H. (2012) · American Journal of Neuroradiology · doi:10.3174/ajnr.a2910 · Cited by 61
Ditmyer, M. M., Dounis, G., Mobley, C. C., et al. (2011) · BMC Oral Health · doi:10.1186/1472-6831-11-12 · Cited by 61
McGill, D. A., van der Vleuten, C. P. M., & Clarke, M. J. (2011) · Advances in Health Sciences Education · doi:10.1007/s10459-011-9296-1 · Cited by 61
Visuospatial processing in early Alzheimer’s disease: A multimodal neuroimaging study
Jacobs, H. I. L., Gronenschild, E. H. B. M., Evers, E. A. T., Ramakers, I. H. G. B., Hofman, P. A. M., Backes, W. H., Jolles, J., Verhey, F. R. J., & van Boxtel, M. P. J. (2012) · Cortex · doi:10.1016/j.cortex.2012.01.005 · Cited by 64
A review of scales for assessing the risk of developing a pressure ulcer in individuals with SCI
Mortenson, W. B., & Miller, W. C. (2007) · Spinal Cord · doi:10.1038/sj.sc.3102129 · Cited by 61
Hayen, A. D., Dennis, R. J., & Finch, C. F. (2006) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2006.09.002 · Cited by 61
Houtte, M. V., & Maele, D. V. (2011) · Oxford Review of Education · doi:10.1080/03054985.2011.595552 · Cited by 61
Effect of pregnancy on trunk range of motion when sitting and standing
Gilleard, W., Crosbie, J., & Smith, R. M. (2002) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1034/j.1600-0412.2002.811104.x · Cited by 61
Classic articles in Psychology in the Science Citation Index Expanded : A bibliometric analysis
Ho, Y., & Hartley, J. R. (2015) · British Journal of Psychology · doi:10.1111/bjop.12163 · Cited by 61
Karno, M., Beutler, L., & Harwood, T. (2002) · Addictive Behaviours · doi:10.1016/s0306-4603(01)00209-x · Cited by 71
Cross-Cultural Register Differences in Infant-Directed Speech: An Initial Study
Farran, L. K., Lee, C.-C., Yoo, H., & Oller, D. K. (2016) · PLoS ONE · doi:10.1371/journal.pone.0151518 · Cited by 61
Alliance and outcome in varying imagery procedures for PTSD: A study of within-person processes.
Hoffart, A., Øktedalen, T., Langkaas, T. F., & Wampold, B. E. (2013) · Journal of Counseling Psychology · doi:10.1037/a0033604 · Cited by 60
Wolfe, K., Seaman, M. A., & Drasgow, E. (2016) · Behavior Modification · doi:10.1177/0145445516644699 · Cited by 61
Can, D., Marín, R. A., Georgiou, P., Imel, Z. E., Atkins, D. C., & Narayanan, S. S. (2016) · Journal of Counseling Psychology · doi:10.1037/cou0000111 · Cited by 61
Short- and Long-Term Sleep Stability in Insomniacs and Healthy Controls
Gaines, J., Vgontzas, A. N., Fernández‐Mendoza, J., Basta, M., Pejovic, S., He, F., & Bixler, E. O. (2015) · SLEEP · doi:10.5665/sleep.5152 · Cited by 62
Bjaastad, J. F., Haugland, B. S. M., Fjermestad, K. W., Torsheim, T., Havik, O. E., Heiervang, E. R., & Öst, L. (2015) · Psychological Assessment · doi:10.1037/pas0000230 · Cited by 61
Zuily, S., Domingues, V., Selton‐Suty, C., Eschwège, V., Bertoletti, L., Chaouat, A., Chabot, F., Regnault, V., Horn, E. M., Erkan, D., et al. (2017) · Autoimmunity Reviews · doi:10.1016/j.autrev.2017.04.003 · Cited by 62
Schwarzenthal, M., Juang, L. P., Schachner, M. K., van de Vijver, F. J. R., & Handrick, A. (2017) · Journal of Community & Applied Social Psychology · doi:10.1002/casp.2317 · Cited by 68
Real-time human respiration carbon dioxide measurement device for cardiorespiratory assessment
Singh, O. P., Howe, T. A., & Malarvili, M. B. (2017) · Journal of Breath Research · doi:10.1088/1752-7163/aa8dbd · Cited by 64
Kyriakidis, M., Majumdar, A., & Ochieng, W. Y. (2017) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2017.10.012 · Cited by 61
Hip Strength as a Predictor of Ankle Sprains in Male Soccer Players: A Prospective Study
Powers, C. M., Ghoddosi, N., Straub, R. K., & Khayambashi, K. (2017) · Journal of Athletic Training · doi:10.4085/1062-6050-52.11.18 · Cited by 61
Knowing left from right: asymmetric functional connectivity during resting state
Raemaekers, M. A. H., Schellekens, W., Petridou, N., & Ramsey, N. F. (2018) · Brain Structure and Function · doi:10.1007/s00429-017-1604-y · Cited by 61
Bologna, M., Corino, V., Montin, E., Messina, A., Calareso, G., Greco, F. G., Sdao, S., & Mainardi, L. T. (2018) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-018-0092-9 · Cited by 61
Ma, Y., Mithraratne, K., Wilson, N., Wang, X., Ma, Y., & Zhang, Y. (2019) · Sensors · doi:10.3390/s19071660 · Cited by 61
Lamina-specific cortical dynamics in human visual and sensorimotor cortices
Bonaiuto, J. J., Meyer, S. S., Little, S. J., Rossiter, H. E., Callaghan, M. F., Dick, F., Barnes, G. R., & Bestmann, S. (2018) · eLife · doi:10.7554/elife.33977 · Cited by 67
Accelerating functional gene discovery in osteoarthritis
Butterfield, N. C., Curry, K. F., Steinberg, J., Dewhurst, H. F., Komla-Ebri, D. S. K., Mannan, N. S., Adoum, A., Leitch, V. D., Logan, J. G., Waung, J. A., et al. (2021) · Nature Communications · doi:10.1038/s41467-020-20761-5 · Cited by 63
May, L., Burnham, R. S., & Steadward, R. D. (1997) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(97)90029-0 · Cited by 61
Interrater reliability in scoring the Wisconsin card sorting test
Axelrod, B. N., GOLDMAN, R. S., & Woodard, J. L. (1992) · The Clinical Neuropsychologist · doi:10.1080/13854049208401851 · Cited by 61
Kiel, D. P., Cali, C., Mercier, C. A., Dawson‐Hughes, B., Hannan, M. T., & Anderson, J. J. (1995) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.5650100719 · Cited by 61
Intrasubject variability of objective voice measures
Bough, I. D., Heuer, R. J., Sataloff, R. T., et al. (1996) · Journal of Voice · doi:10.1016/s0892-1997(96)80044-0 · Cited by 61
Hoffart, A., Versland, S., & Sexton, H. C. (2002) · Cognitive Therapy and Research · doi:10.1023/a:1014521819858 · Cited by 62
An improved method of analysis of observer variation between pathologists
Holman, C., JAMES, I. R., Heenan, P. J., Matz, L. R., Blackwell, J. B., Kelsall, G. R. H., SINGH, A., & Seldam, R. E. J. T. (1982) · Histopathology · doi:10.1111/j.1365-2559.1982.tb02752.x · Cited by 61
Chronic Liver Allograft Rejection
Demetris, A. J., Seaberg, E. C., Batts, K. P., et al. (1998) · The American Journal of Surgical Pathology · doi:10.1097/00000478-199801000-00004 · Cited by 61
Assessing periodontal disease risk
Persson, G. R., Mancl, L., Martin, J. A., et al. (2003) · The Journal of the American Dental Association · doi:10.14219/jada.archive.2003.0224 · Cited by 61
Reproducibility of anthropometric and body composition measurements: the HERITAGE Family Study
Wilmore, J., Stanforth, P., Domenick, M., Gagnon, J., Daw, E., Leon, A., Rao, D., Skinner, J., & Bouchard, C. (1997) · International Journal of Obesity · doi:10.1038/sj.ijo.0800404 · Cited by 61
Lin, Q., Dai, Z., Xia, M., Han, Z., Huang, R., Gong, G., Liu, C., Bi, Y., & Yong He, A. (2015) · Scientific Data · doi:10.1038/sdata.2015.56 · Cited by 61
Creagh, A. P., Hamy, V., Yuan, H., Mertes, G., Tomlinson, R., Chen, W.-H., Williams, R., Llop, C., Yee, C., Duh, M. S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01013-y · Cited by 64
Jenkins, N. D. M., Buckner, S. L., Bergstrom, H. C., Cochrane, K. C., Goldsmith, J. A., Housh, T. J., Johnson, G. O., Schmidt, R. J., & Cramer, J. T. (2014) · Experimental Gerontology · doi:10.1016/j.exger.2014.07.007 · Cited by 65
Critical Success Factors in SOA Implementation: An Exploratory Study
Lee, J. H., Shim, H.-J., & Kim, K. K. (2010) · Information Systems Management · doi:10.1080/10580531003685188 · Cited by 60
Timmings, C., Khan, S., Moore, J. E., Marquez, C., Pyka, K., & Straus, S. E. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0262-y · Cited by 70
van Everdingen, W. M., Zweerink, A., Nijveldt, R., et al. (2017) · International journal of cardiac imaging · doi:10.1007/s10554-017-1253-5 · Cited by 60
Study of an Extensive Set of Eye Movement Features: Extraction Methods and Statistical Analysis
Rigas, I., Friedman, L., & Komogortsev, O. V. (2018) · Journal of Eye Movement Research · doi:10.16910/jemr.11.1.3 · Cited by 60
Group-Based Emotions: Evidence for Emotion-Performance Relationships in Team Sports
Campo, M., Champely, S., Louvet, B., Rosnet, É., Ferrand, C., Pauketat, J. V. T., & Mackie, D. M. (2019) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.2018.1563274 · Cited by 60
CGA: a new feature selection model for visual human action recognition
Guha, R., Khan, A. H., Singh, P. K., Sarkar, R., & Bhattacharjee, D. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-05297-5 · Cited by 62
Living up to the expectations: The effect of design thinking on project success
Roth, K., Globocnik, D., Rau, C., & Neyer, A. (2020) · Creativity and Innovation Management · doi:10.1111/caim.12408 · Cited by 74
Kuhn, M., Gerlicher, A. M. V., & Lonsdorf, T. B. (2022) · Psychophysiology · doi:10.1111/psyp.14058 · Cited by 55
Bonobos and chimpanzees remember familiar conspecifics for decades
Lewis, L. S., Wessling, E. G., Kano, F., Stevens, J. M. G., Call, J., & Krupenye, C. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2304903120 · Cited by 61
Lefèvre‐Colau, M., Poiraudeau, S., Oberlin, C., Demaille, S., Fermanian, J., Rannou, F., & Revel, M. (2003) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(03)00128-x · Cited by 58
Agreement between physicians' and patients' ratings on the Montgomery–Åsberg Depression Rating Scale
Cunningham, J. L., Wernroth, L., von Knorring, L., Berglund, L., & Ekselius, L. (2011) · Journal of Affective Disorders · doi:10.1016/j.jad.2011.07.005 · Cited by 60
Mulcahey, M. J., Gaughan, J., Chafetz, R. S., et al. (2011) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2011.03.003 · Cited by 60
Sengupta, S. P., Amaki, M., Bansal, M., et al. (2014) · Journal of the American Society of Echocardiography · doi:10.1016/j.echo.2014.01.024 · Cited by 60
Thompson, N., Gordey, L., Bowles, H., et al. (2013) · Advances in Skin & Wound Care · doi:10.1097/01.asw.0000431329.50869.6f · Cited by 60
Burns, A., Herbert‐Read, J. E., Morrell, L. J., & Ward, A. J. W. (2012) · PLoS ONE · doi:10.1371/journal.pone.0036567 · Cited by 62
Bowen, M. T., Kevin, R. C., May, M. D., Staples, L., Hunt, G. E., & McGregor, I. S. (2013) · PLoS ONE · doi:10.1371/journal.pone.0068483 · Cited by 60
The Value of a Smile: Does Emotional Performance Matter More in Familiar or Unfamiliar Exchanges?
Gabriel, A. S., Acosta, J., & Grandey, A. A. (2013) · Journal of Business and Psychology · doi:10.1007/s10869-013-9329-2 · Cited by 61
Touboul, P., Vicaut, É., Labreuche, J., Belliard, J. P., Cohen, S., Kownator, S., & Pithois-Merli, I. (2004) · Cerebrovascular Diseases · doi:10.1159/000081913 · Cited by 58
Xiong, S., Goonetilleke, R. S., & Jiang, Z. (2011) · Ergonomics · doi:10.1080/00140139.2011.552736 · Cited by 60
Good test–retest reproducibility for an instrument to capture self-reported melanoma risk factors
Morze, C. J., Olsen, C. M., Perry, S. L., Jackman, L. M., Ranieri, B. A., O’Brien, S. M., Cicero, R. A., & Whiteman, D. C. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.014 · Cited by 60
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
