The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Intraclass correlations: Uses in assessing rater reliability.
Abstract
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, the inter-rater agreement was quite high for categorical variables (mean Cohen's kappa = 0.73, SD = 0.13) and for continuous variables (mean intraclass correlation [26] = 0.80, SD = .14).”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [26]
In the citing paper
“we used the methodology Shrout and Fleiss (1979) describe and computed the Shrout-Fleiss single ICC score agreement of .998, which is considered quite high (Koo and Li 2016).”
From Driving Brand Engagement Through Online Social Influencers: An Empirical Investigation of Sponsored Blogging Campaigns (2019) · cites this paper as Shrout and Fleiss (1979)
In the citing paper
“In contrast, Cohen's quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC (2,1) and ICC(3,1) coefficients), where the items and raters are the two factors (Carrasco, 2010;Fleiss & Cohen, 1973;Janson & Olsson, 2001;Schuster, 2004;Schuster & Smith, 2005;Shrout & Fleiss, 1979;Warrens, 2014).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“A sample of 11% of video data was double coded by the researcher and an experienced coder (the developer of CHIPPA). Inter‐rater reliability was based on intra‐class correlation (ICC) statistics from the mean rating for the two coders (k = 2) using a two‐way mixed effects consistency model (see Table 2 below) and resulted in good to excellent reliability across the CHIPPA subscales (Koo & Li, 2016”
From A plausible role of imagination in pretend play, counterfactual reasoning, and executive functions (2023) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“The results suggest that coders showed high agreement (intraclass correlation [2, 1] = 0.85; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“Interrater reliability of the pre- and posttreatment assessments was excellent (intraclass correlations [2, 1] = 0.97 and 0.96, respectively; Shrout & Fleiss, 1979).”
From Respiratory and cognitive mediators of treatment change in panic disorder: Evidence for intervention specificity. (2010) · cites this paper as Shrout & Fleiss, 1979
In the citing paper
“ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Shrout and Fleiss 1979
In the citing paper
“Reproducibility of the final placenta, fetus and amniotic fluid volume in millilitres was assessed using the ICC (2,1) [37], [38] for inter-observer repeatability between the semi-automated Random Walker results, regarded as the ground-truth used for performance evaluation, and the newly generated FCNN outputs.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [37]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 27,137
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,013
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
The theory of the estimation of test reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 1,953
Cited by
Sorted by citations
Items 4,901–5,000 of 23,883
Wu, S. P. W., & Rau, M. A. (2017) · Learning and Instruction · doi:10.1016/j.learninstruc.2017.09.010 · Cited by 61
A preliminary study of occupants’ use of manual lighting controls in private offices: A case study
Gilani, S., & O'Brien, W. (2018) · Energy and Buildings · doi:10.1016/j.enbuild.2017.11.055 · Cited by 50
Agreement in DNA methylation levels from the Illumina 450K array across batches, tissues, and time
Forest, M., O’Donnell, K. J., Voisin, G., Gaudreau, H., MacIsaac, J. L., McEwen, L. M., Silveira, P. P., Steiner, M. S., Kobor, M. S., Meaney, M. J., et al. (2018) · Epigenetics · doi:10.1080/15592294.2017.1411443 · Cited by 57
Zhang, Y., Dong, W., Hu, B.-G., & Ji, Q. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00246 · Cited by 59
Adolfsson, E., & Andershed, A. N. (2018) · JBRA · doi:10.5935/1518-0557.20180042 · Cited by 59
Total Ankle Arthroplasty Survival and Risk Factors for Failure
Escudero, M. I., Le, V., Barahona, M., et al. (2019) · Foot & Ankle International · doi:10.1177/1071100719849084 · Cited by 59
Nehrer, S., Ljuhar, R., Steindl, P., Simon, R., Maurer, D., Ljuhar, D., Bertalan, Z., Dimai, H. P., Goetz, C., & Paixão, T. (2019) · Cartilage · doi:10.1177/1947603519888793 · Cited by 59
Evangelista, M. C., & Steagall, P. V. M. (2021) · Scientific Reports · doi:10.1038/s41598-021-84696-7 · Cited by 59
Business Ethics in Brazil and the U.S.: A Comparative Investigation
Beekun, R. I., Stedham, Y. E., & Yamamura, J. H. (2003) · Journal of Business Ethics · doi:10.1023/a:1022200702149 · Cited by 57
Development of an adherence/competence rating scale for individual drug counseling
Barber, J. P., Mercer, D. E., Krakauer, I. D., & Calvo, N. (1996) · Drug and Alcohol Dependence · doi:10.1016/s0376-8716(96)01305-1 · Cited by 59
Pragmatic language skills alter closed head injury: Ability to negotiate requests
McDonald, S., & van Sommers, P. (1993) · Cognitive Neuropsychology · doi:10.1080/02643299308253466 · Cited by 59
Does questionnaire structure influence response in postal surveys?
Dunn, K. M., Jordan, K. P., & Croft, P. R. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00567-x · Cited by 61
Social networks and clinical outcomes of dually diagnosed homeless persons
Trumbetta, S., Mueser, K., Quimby, E., Bebout, R., & Teague, G. (1999) · Behavior Therapy · doi:10.1016/s0005-7894(99)80018-5 · Cited by 60
The relationship between formal thought disorder and executive functioning component processes.
Kerns, J. G., & Berenbaum, H. (2003) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.112.3.339 · Cited by 58
Vittengl, J. R., Clark, L. A., Kraft, D., & Jarrett, R. B. (2005) · Psychological Medicine · doi:10.1017/s0033291704004143 · Cited by 60
Developing a magnetic resonance imaging scoring system for peripheral psoriatic arthritis.
McQueen, F. M., Lassere, M. N., Bird, P., et al. (2007) · PubMed · Cited by 59
Lateralized effects of diazepam on frontal brain electrical asymmetries in rhesus monkeys
Davidson, R. J., Kalin, N. H., & Shelton, S. E. (1992) · Biological Psychiatry · doi:10.1016/0006-3223(92)90131-i · Cited by 59
Validity and reliability of undergraduate performance assessments in an anesthesia simulator
Morgan, P. J., Cleave‐Hogg, D. M., Guest, C. B., & Herold, J. (2001) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/bf03019750 · Cited by 59
Gross, M. T., Lapp, A. K., & Davis, J. M. (1991) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1991.13.1.11 · Cited by 59
CONSENSUS IN CRIME SERIOUSNESS: EMPIRICAL REALITY OR METHODOLOGICAL ARTIFACT?*
Cullen, F. T., Link, B. G., Travis, L. F., & Wozniak, J. F. (1985) · Criminology · doi:10.1111/j.1745-9125.1985.tb00328.x · Cited by 59
Editorial: Reliability and Reproducibility in Functional Connectomics
Zuo, X., Biswal, B. B., & Poldrack, R. A. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00117 · Cited by 57
What Makes A Hero? The Impact of Integrity on Admiration and Interpersonal Judgment
Schlenker, B. R., Weigold, M. F., & Schlenker, K. A. (2008) · Journal of Personality · doi:10.1111/j.1467-6494.2007.00488.x · Cited by 75
Latent trees for estimating intensity of Facial Action Units
Kaltwang, S., Todorović, S., & Pantić, M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298626 · Cited by 57
Driban, J. B., Lo, G. H., Lee, J. Y., et al. (2011) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-12-217 · Cited by 58
A pilot study of psychodynamic psychotherapy for adult survivors of childhood sexual abuse
Price, J. L., Hilsenroth, M. J., Callahan, K. L., Petretic-Jackson, P. A., & Bonge, D. R. (2004) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.421 · Cited by 58
Johnson, A. H., Miller, F. G., Chafouleas, S. M., Welsh, M. E., Riley‐Tillman, T. C., & Fabiano, G. A. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2015.10.001 · Cited by 58
Psychometric properties of the Arabic version of the Belief into Action Scale
Alakhdhair, S., Sheets, V. L., Geib, R. W., Alkhuwaildi, A., & Koenig, H. G. (2016) · Mental Health Religion & Culture · doi:10.1080/13674676.2016.1266471 · Cited by 58
Validation of the English and French versions of the Brief Health Care Climate Questionnaire
Czajkowska, Z., Wang, H., Hall, N. C., Sewitch, M. J., & Körner, A. (2017) · Health Psychology Open · doi:10.1177/2055102917730675 · Cited by 60
Early pregnancy exposure to metal mixture and birth outcomes – A prospective study in Project Viva
Rahman, M. L., Oken, E., Hivert, M., Rifas‐Shiman, S. L., Lin, P. D., Colicino, E., Wright, R. O., Amarasiriwardena, C. J., Henn, B. C., Gold, D. R., et al. (2021) · Environment International · doi:10.1016/j.envint.2021.106714 · Cited by 57
Gill, S. D., Hely, R., Page, R. S., Hely, A., Harrison, B., & Landers, S. (2022) · Physiotherapy Research International · doi:10.1002/pri.1957 · Cited by 57
Feasibility of Markerless Motion Capture for Three-Dimensional Gait Assessment in Community Settings
McGuirk, T. E., Perry, E. S., Sihanath, W. B., Riazati, S., & Patten, C. (2022) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2022.867485 · Cited by 56
Impairment of Dynamic Single‐Leg Balance Performance in Individuals With Hip Chondropathy
Hatton, A. L., Kemp, J. L., Brauer, S., Clark, R. A., & Crossley, K. M. (2013) · Arthritis Care & Research · doi:10.1002/acr.22193 · Cited by 58
Parietal lobe epilepsy: the great imitator among focal epilepsies
Ristić, A. J., Alexopoulos, A. V., So, N. K., Wong, C., & Najm, I. (2012) · Epileptic Disorders · doi:10.1684/epd.2012.0484 · Cited by 58
Test-retest reliability of resting-state magnetoencephalography power in sensor and source space
Martín-Buro, M. C., Garcés, P., & Maestú, F. (2015) · Human Brain Mapping · doi:10.1002/hbm.23027 · Cited by 62
One Small Step for Manuals: Computer-Assisted Training in Twelve-Step Facilitation
Sholomskas, D. E., & Carroll, K. M. (2006) · Journal of Studies on Alcohol · doi:10.15288/jsa.2006.67.939 · Cited by 58
Gastrointestinal Endoscopy Competency Assessment Tool: reliability and validity evidence
Walsh, C. M., Ling, S. C., Khanna, N. V., Grover, S. C., Yu, J. J., Cooper, M. A., Yong, E., Nguyen, G. C., May, G. R., Walters, T. D., et al. (2015) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2014.11.030 · Cited by 61
Olsen, E., Andersen, P. H., & Pfau, T. (2012) · Sensors · doi:10.3390/s120608145 · Cited by 58
Grand, A. P., Kuhar, C. W., Leighty, K. A., Bettinger, T. L., & Laudenslager, M. L. (2012) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2012.09.002 · Cited by 58
Development of the Comprehensive Cervical Dystonia Rating Scale: Methodology
Comella, C., Fox, S. H., Bhatia, K. P., et al. (2015) · Movement Disorders Clinical Practice · doi:10.1002/mdc3.12131 · Cited by 58
Rastle, K., Croot, K., Harrington, J. M., & Coltheart, M. (2005) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.31.5.1083 · Cited by 58
Hanssen-Bauer, K., Aalen, O., Ruud, T., & Heyerdahl, S. (2007) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-007-0134-y · Cited by 55
Andersson, M., Moberg, L., Svantesson, U. M., Sundbom, A., Johansson, H., & Emtner, M. (2011) · Primary Care Respiratory Journal · doi:10.4104/pcrj.2011.00082 · Cited by 59
Gerdtz, M. F., Collins, M. L., Chu, M., et al. (2008) · Emergency Medicine Australasia · doi:10.1111/j.1742-6723.2008.01089.x · Cited by 58
Stambaugh, L. F., Hogue, A., Johnson, S., Diamond, G. M., & Liddle, H. A. (2005) · Psychotherapy Research · doi:10.1080/10503300512331326994 · Cited by 58
Accuracy of Three Techniques to Determine Cell Viability in 3D Tissues or Scaffolds
Gantenbein, B., Potier, E., Zeiter, S., van der Werf, M., Sprecher, C. M., & Ito, K. (2008) · Tissue Engineering Part C Methods · doi:10.1089/ten.tec.2008.0313 · Cited by 58
van Duijn, G., Dijkxhoorn, Y. M., Noens, I. L. J., SCHOLTE, E. M., & van Berckelaer‐Onnes, I. (2009) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.282 · Cited by 58
Cerebral gray and white matter changes and clinical course in metachromatic leukodystrophy
Groeschel, S., Dali, C. Í., Clas, P., Böhringer, J., Dunø, M., Krarup, C., Kehrer, C., Wilke, M., & Krägeloh‐Mann, I. (2012) · Neurology · doi:10.1212/wnl.0b013e31826e9ad2 · Cited by 58
Youdas, J. W., Hollman, J. H., Aalbers, M. J., Ahrenholz, H. N., Aten, R. A., & Cremers, J. J. (2006) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2006.09.012 · Cited by 58
Graci, V., & Salsich, G. B. (2014) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2014.04.011 · Cited by 58
Engelmann, T., & Hesse, F. W. (2011) · Computers in Human Behavior · doi:10.1016/j.chb.2011.06.002 · Cited by 58
Rabin, L. A., Borgos, M. J., Saykin, A. J., Wishart, H. A., Crane, P. K., Nutter‐Upham, K. E., & Flashman, L. A. (2006) · Journal of Clinical and Experimental Neuropsychology · doi:10.1080/13825580601025908 · Cited by 59
Vegter, R. J. K., Lamoth, C. J. C., de Groot, S., Veeger, H. E. J., & van der Woude, L. H. V. (2013) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-10-9 · Cited by 57
Influence of knee position on the postural stability index registered by the Biodex Stability System
Pereira, H. M., de Campos, T. F., Santos, M. B., et al. (2008) · Gait & Posture · doi:10.1016/j.gaitpost.2008.05.003 · Cited by 58
Pauling, J. D., Shipley, J. A., Hart, D., et al. (2015) · The Journal of Rheumatology · doi:10.3899/jrheum.141437 · Cited by 58
Bruns, A., Künnecke, B., Risterucci, C., Moreau, J., & von Kienlin, M. (2009) · Magnetic Resonance in Medicine · doi:10.1002/mrm.21779 · Cited by 58
Chen, J. F., Ginn, K. A., & Herbert, R. (2009) · Australian Journal of Physiotherapy · doi:10.1016/s0004-9514(09)70056-x · Cited by 58
Gabel, C. P., Michener, L. A., Melloh, M., & Burkett, B. (2010) · Journal of Hand Therapy · doi:10.1016/j.jht.2009.09.007 · Cited by 58
Basner, M., Griefahn, B., Müller, U., Plath, G., & Samel, A. (2007) · Sleep · doi:10.1093/sleep/30.10.1349 · Cited by 70
Caudate Nucleus Volumes in Frontotemporal Lobar Degeneration: Differential Atrophy in Subtypes
Looi, J. C., Lindberg, O. E., Zandbelt, B. B., Östberg, P., Andersen, C., Botes, L., Svensson, L. A., & Wahlund, L. (2008) · American Journal of Neuroradiology · doi:10.3174/ajnr.a1168 · Cited by 58
Prevalence of Metatarsus Adductus in Patients Undergoing Hallux Valgus Surgery
Aiyer, A. A., Shariff, R., Ying, L., et al. (2014) · Foot & Ankle International · doi:10.1177/1071100714551022 · Cited by 58
Disagreement in Parental Reports of Father Involvement
Charles, P., Spielfogel, J. E., Gorman‐Smith, D., Schoeny, M. E., Henry, D. B., & Tolan, P. H. (2016) · Journal of Family Issues · doi:10.1177/0192513x16644639 · Cited by 59
Tracy, K., Adler, L. A., Rotrosen, J., et al. (1997) · PubMed · Cited by 58
Pretend play and creativity in preschool-age children: Associations and brief intervention.
Fehr, K. K., & Russ, S. W. (2016) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/aca0000054 · Cited by 74
Accurate shade image matching by using a smartphone camera
Tam, W.-K., & Lee, H.-J. (2016) · Journal of Prosthodontic Research · doi:10.1016/j.jpor.2016.07.004 · Cited by 62
Marini, J. C., Sendecki, J. A., Cornillie, F. J., et al. (2016) · The AAPS Journal · doi:10.1208/s12248-016-9981-3 · Cited by 58
Oerbekke, M. S., Stukstette, M. J. P. M., Schütte, K., de Bie, R. A., Pisters, M. F., & Vanwanseele, B. (2016) · Gait & Posture · doi:10.1016/j.gaitpost.2016.10.005 · Cited by 58
Galmiche, C., Bernhard, J., Yacoub, M., et al. (2016) · American Journal of Roentgenology · doi:10.2214/ajr.16.16832 · Cited by 58
Anterolateral Entorhinal Cortex Volume Predicted by Altered Intra-Item Configural Processing
Yeung, L., Olsen, R. K., Bild-Enkin, H. E. P., D’Angelo, M. C., Kacollja, A., McQuiggan, D. A., Keshabyan, A., Ryan, J. D., & Barense, M. D. (2017) · Journal of Neuroscience · doi:10.1523/jneurosci.3664-16.2017 · Cited by 61
Han, A., André, M. P., Deiranieh, L. H., Housman, E., Erdman, J. W., Loomba, R. S., Sirlin, C. B., & O’Brien, W. D. (2018) · Journal of Ultrasound in Medicine · doi:10.1002/jum.14537 · Cited by 59
When less is more: the downside of customer knowledge sharing in new product development teams
Tang, Y., & Marinova, D. (2019) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00646-w · Cited by 58
Nikkonen, S., Afara, I. O., Leppänen, T., & Töyräs, J. (2019) · Scientific Reports · doi:10.1038/s41598-019-49330-7 · Cited by 58
Investigating the effects of beat and deictic gestures of a lecturer in educational videos
Beege, M., Ninaus, M., Schneider, S., Nebel, S., Schlemmel, J., Weidenmüller, J., Moeller, K., & Rey, G. D. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103955 · Cited by 71
Florkow, M. C., Willemsen, K., Zijlstra, F., Foppen, W., van der Wal, B. C. H., van Zyp, J. R. N. V. D. V., Viergever, M. A., Castelein, R. M., Weinans, H., van Stralen, M., et al. (2021) · Journal of Orthopaedic Research® · doi:10.1002/jor.25127 · Cited by 80
Prolonged Speech and Modification of Stuttering
Packman, A., Onslow, M., & van Doorn, J. (1994) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3704.724 · Cited by 58
Interrater reliability of measurements of comorbid illness should be reported
Hall, S. F., Groome, P. A., Streiner, D. L., et al. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.02.006 · Cited by 58
Rothwell, P. M. (2000) · Journal of Neurology · doi:10.1007/s004150070068 · Cited by 58
Reproducibility of Physical Performance and Physiologic Assessments
Wolinsky, F. D., Miller, D. K., Andresen, E. M., et al. (2005) · Journal of Aging and Health · doi:10.1177/0898264304272784 · Cited by 58
Reliability of the self-awareness of deficits interview for adults with traumatic brain injury
Simmond, M., & Fleming, J. (2003) · Brain Injury · doi:10.1080/0269905021000013219 · Cited by 58
Mackenzie, C., & Lowit, A. (2007) · International Journal of Language & Communication Disorders · doi:10.1080/13682820600861776 · Cited by 58
Vaz, M. D., Kramer, J. F., Rorabeck, C. H., et al. (1993) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.1993.18.4.526 · Cited by 58
Eadie, T. L., & Doyle, P. C. (2002) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2002/087 · Cited by 58
Bushman, B. J., & Wells, G. L. (2001) · Personality and Social Psychology Bulletin · doi:10.1177/0146167201279005 · Cited by 58
Relationship of neuropsychological and MRI measures to age of onset of schizophreniaa
Jeste, D. V., McAdams, L. A., Palmer, B. W., BRAFF, D. L., Jernigan, T. L., Paulsen, J. S., Stout, J. C., Symonds, L. L., Bailey, A., & Heaton, R. K. (1998) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1998.tb10058.x · Cited by 58
Interactions of depressed college students with their roommates: Not necessarily negative.
Burchill, S. L., & Stiles, W. B. (1988) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.55.3.410 · Cited by 58
Consequences of Success: Do More Teeth Translate into More Disease and Utilization?
Joshi, A., Douglass, C. W., Feldman, H. A., et al. (1996) · Journal of Public Health Dentistry · doi:10.1111/j.1752-7325.1996.tb02435.x · Cited by 58
Increasing employment in ex-heroin addicts II: Methadone maintenance sample
Hall, S. M., Loeb, P. C., LeVois, M., et al. (1981) · Behavior Therapy · doi:10.1016/s0005-7894(81)80083-4 · Cited by 58
A cross‐sectional investigation of breast density and insulin‐like growth factor I
Maskarinec, G., Williams, A. E., & Kaaks, R. (2003) · International Journal of Cancer · doi:10.1002/ijc.11505 · Cited by 58
The great escape: The role of self-esteem and self-related cognition in terror management
Wisman, A., Heflick, N., & Goldenberg, J. L. (2015) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.05.006 · Cited by 65
Disrupted cortico‐limbic connectivity during reward processing in remitted bipolar I disorder
Dutra, S. J., Man, V. Y., Hedy Kober, A., Cunningham, W. A., & Gruber, J. (2017) · Bipolar Disorders · doi:10.1111/bdi.12560 · Cited by 58
Affect regulation training reduces symptom severity in depression – A randomized controlled trial
Berking, M., Eichler, E., Luhmann, M., Diedrich, A., Hiller, W., & Rief, W. (2019) · PLoS ONE · doi:10.1371/journal.pone.0220436 · Cited by 71
Dissociable neural systems for unconditioned acute and sustained fear
Hudson, M., Seppälä, K., Putkinen, V., Sun, L., Glerean, E., Karjalainen, T., Karlsson, H. K., Hirvonen, J., & Nummenmaa, L. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116522 · Cited by 58
Uncemented femoral stem orientation and position in total hip arthroplasty: A CT study
Belzunce, M. A., Henckel, J., Laura, A. D., et al. (2020) · Journal of Orthopaedic Research® · doi:10.1002/jor.24627 · Cited by 57
A concrete example of construct construction in natural language
Yeomans, M. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.10.008 · Cited by 61
A meta-analysis of Libet-style experiments
Braun, M. N., Wessler, J., & Friese, M. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.06.018 · Cited by 63
Fiscella, K. A., Epstein, R. M., Griggs, J. J., Marshall, M., & Shields, C. G. (2021) · PLoS ONE · doi:10.1371/journal.pone.0257794 · Cited by 56
Anokhin, A. P., Luciana, M. M., Banich, M. T., Barch, D. M., Bjork, J. M., Gonzalez, M. R., González, R., Haist, F., Jacobus, J., Lisdahl, K. M., et al. (2022) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2022.101078 · Cited by 57
Structural and functional connectivity reconstruction with CATO - A Connectivity Analysis TOolbox
de Lange, S. C., Helwegen, K., & van den Heuvel, M. P. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120108 · Cited by 54
Koenig, N., Tonidandel, S., Thompson, I. B., Albritton, B. H., Koohifar, F., Yankov, G. P., Speer, A. B., Hardy, J. H., Gibson, C., Frost, C., et al. (2023) · Personnel Psychology · doi:10.1111/peps.12608 · Cited by 78
The effects of long-term high-altitude exposure on cognition: A meta-analysis
Su, R., Jia, S., Zhang, N., Wang, Y., Li, H., Zhang, D., Ma, H., & Su, Y. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2024.105682 · Cited by 60
Halmburger, A., Baumert, A., & Schmitt, M. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2071 · Cited by 57
Metadata
- Authors
- Shrout, P. E., & Fleiss, J. L.
- Affiliations
- Columbia University (US)
- ORCID
- Shrout, Patrick E.
- Venue
- Psychological Bulletin
- Published
- 1979
- Volume
- 86
- Issue
- 2
- Pages
- 420-428
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2141403362
- Cited by
- 24,532
- References
- 20
- Citations by year
- 2026: 1452 · 2025: 1085 · 2024: 998 · 2023: 1023 · 2022: 1083 · 2021: 1379 · 2020: 1189 · 2019: 1116 · 2018: 1178 · 2017: 1113 · 2016: 1106 · 2015: 1057 · 2014: 1172 · 2013: 1044 · 2012: 914
- FWCI
- 13.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intraclass correlation coefficient, rater reliability, inter-rater reliability, reliability coefficients, confidence intervals, multiple raters
- MeSH terms
- Humans, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
