The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
QUANTIFYING TEST-RETEST RELIABILITY USING THE INTRACLASS CORRELATION COEFFICIENT AND THE SEM
Abstract
Reliability, the consistency of a test or measurement, is frequently quantified in the movement sciences literature. A common metric is the intraclass correlation coefficient (ICC). In addition, the SEM, which can be calculated from the ICC, is also frequently reported in reliability studies. However, there are several versions of the ICC, and confusion exists in the movement sciences regarding which ICC to use. Further, the utility of the SEM is not fully appreciated. In this review, the basics of classic reliability theory are addressed in the context of choosing and interpreting an ICC. The primary distinction between ICC equations is argued to be one concerning the inclusion (equations 2,1 and 2,k) or exclusion (equations 3,1 and 3,k) of systematic error in the denominator of the ICC equation. Inferential tests of mean differences, which are performed in the process of deriving the necessary variance components for the calculation of ICC values, are useful to determine if systematic error is present. If so, the measurement schedule should be modified (removing trials where learning and/or fatigue effects are present) to remove systematic error, and ICC equations that only consider random error may be safely used. The use of ICC values is discussed in the context of estimating the effects of measurement error on sample size, statistical power, and correlation attenuation. Finally, calculation and application of the SEM are discussed. It is shown how the SEM and its variants can be used to construct confidence intervals for individual scores and to determine the minimal difference needed to be exhibited for one to be confident that a true change in performance of an individual has occurred.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Measures of Reliability in Sports Medicine and Science
Hopkins, W. G. (2000) · Sports Medicine · doi:10.2165/00007256-200030010-00001 · Cited by 4,635
Reliability: what is it, and how is it measured?
Bruton, A., Conway, J. H., & Holgate, S. T. (2000) · Physiotherapy · doi:10.1016/s0031-9406(05)61211-4 · Cited by 763
Cited by
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018) · Frontiers in Public Health · doi:10.3389/fpubh.2018.00149 · Cited by 5,925
Development and validation of the user version of the Mobile application rating scale (uMARS)
Stoyanov, S. R., Hides, L., Kavanagh, D. J., & Wilson, H. (2016) · JMIR mhealth and uhealth · doi:10.2196/mhealth.5849 · Cited by 872
“Interclass correlation coefficients (ICCs) [16-18] were used, as they provide weighted values of rater agreement and assess proximity rather than equality of ratings.”
Zuo, X.-N., Kelly, C., Adelstein, J. S., Klein, D. F., Castellanos, F. X., & Milham, M. P. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.10.080 · Cited by 744
Perceived Stress Scale: Reliability and Validity Study in Brazil
Siqueira Reis, R., Ferreira Hino, A. A., & Romélio Rodriguez Añez, C. (2010) · Journal of Health Psychology · doi:10.1177/1359105309346343 · Cited by 698
Swami, V., & Barron, D. (2019) · Body Image · doi:10.1016/j.bodyim.2018.08.014 · Cited by 454
The reliability and validity of ultrasound to quantify muscles in older adults: A systematic review
Nijholt, W., Scafoglieri, A., Jager‐Wittenaar, H., Hobbelen, J. S. M., & van der Schans, C. P. (2017) · Journal of Cachexia Sarcopenia and Muscle · doi:10.1002/jcsm.12210 · Cited by 459
“It is important to report complete information on the type of ICC used in the study as this influences the results and is needed for an appropriate interpretation of the results.(46) Furthermore, in addition to type of ICC used in the study, data on the magnitude of the error, for example, standard error of the measur…”
MethodPuig-Diví, A., Escalona-Marfil, C., Padullés-Riu, J. M., Busquets, A., Padullés-Chando, X., & Marcos-Ruiz, D. (2019) · PLoS ONE · doi:10.1371/journal.pone.0216448 · Cited by 434
Empirical recommendations for improving the stability of the dot-probe task in clinical research.
Price, R. B., Kuckertz, J. M., Siegle, G. J., Ladouceur, C. D., Silk, J. S., Ryan, N. D., Dahl, R. E., & Amir, N. (2015) · Psychological Assessment · doi:10.1037/pas0000036 · Cited by 312
Supramaximal training and postexercise parasympathetic reactivation in adolescents
BUCHHEIT, M., MILLET, G. P., PARISY, A., POURCHEZ, S., LAURSEN, P. B., & AHMAIDI, S. (2008) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0b013e31815aa2ee · Cited by 261
Mangine, G. T., Hoffman, J. R., Gonzalez, A. M., Townsend, J. R., Wells, A. J., Jajtner, A. R., Beyer, K. S., Boone, C. H., Miramonti, A. A., Wang, R., et al. (2015) · Physiological Reports · doi:10.14814/phy2.12472 · Cited by 204
Clarke, D. E., Narrow, W. E., Regier, D. A., Kuramoto, S. J., Kupfer, D. J., Kuhl, E. A., Greiner, L., & Kraemer, H. C. (2013) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.12070998 · Cited by 201
Cook, J., Hislop, J., Adewuyi, T., Harrild, K., Altman, D., Ramsay, C., Fraser, C., Buckley, B., Fayers, P., Harvey, I., et al. (2014) · Health Technology Assessment · doi:10.3310/hta18280 · Cited by 194
Moradmand, H., Aghamiri, S. M. R., & Ghaderi, R. (2020) · Journal of Applied Clinical Medical Physics · doi:10.1002/acm2.12795 · Cited by 173
Test–retest measurements of dopamine D1-type receptors using simultaneous PET/MRI imaging
Kaller, S., Rullmann, M., Patt, M., Becker, G.-A., Luthardt, J., Girbardt, J., Meyer, P. M., Werner, P., Barthel, H., Bresch, A., et al. (2017) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-017-3645-0 · Cited by 152
Effects of Whey Isolate, Creatine, and Resistance Training on Muscle Hypertrophy
CRIBB, P. J., WILLIAMS, A. D., STATHIS, C. G., CAREY, M. F., & HAYES, A. (2007) · Medicine & Science in Sports & Exercise · doi:10.1249/01.mss.0000247002.32589.ef · Cited by 165
Clinician versus machine: Reliability and responsiveness of motor endpoints in Parkinson's disease
Heldman, D. A., Espay, A. J., LeWitt, P. A., & Giuffrida, J. P. (2014) · Parkinsonism & Related Disorders · doi:10.1016/j.parkreldis.2014.02.022 · Cited by 156
Possibilities and limitations of the polar RS800 in measuring heart rate variability at rest
Wallén, M. B., Hasson, D., Theorell, T., Canlon, B., & Osika, W. (2012) · European Journal of Applied Physiology · doi:10.1007/s00421-011-2079-9 · Cited by 135
Body composition and fitness during strength and/or endurance training in older men
SILLANPÄÄ, E., HÄKKINEN, A., NYMAN, K., MATTILA, M., CHENG, S., KARAVIRTA, L., LAAKSONEN, D. E., HUUHKA, N., KRAEMER, W. J., & HÄKKINEN, K. (2008) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0b013e318165c854 · Cited by 130
Psychometric Evaluation of the Fear of COVID-19 Scale Among Chinese Population
Chi, X., Chen, S., Chen, Y., Chen, D., Yu, Q., Guo, T., Cao, Q., Zheng, X., Huang, S., Hossain, M. M., et al. (2022) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-020-00441-7 · Cited by 137
Harden, R. N., Maihofner, C., Abousaad, E., Vatine, J.-J., Kirsling, A., Perez, R. S. G. M., Kuroda, M., Brunner, F., Stanton-Hicks, M., Marinus, J., et al. (2017) · Pain · doi:10.1097/j.pain.0000000000000927 · Cited by 102
Dolphens, M., Cagnie, B., Coorevits, P., Vanderstraeten, G., Cardon, G., Dʼhooge, R., & Danneels, L. (2012) · Spine · doi:10.1097/brs.0b013e3182408053 · Cited by 88
Test–retest reliability of five frequently used executive tasks in healthy adults
Soveri, A., Lehtonen, M., Karlsson, L. C., Lukasik, K., Antfolk, J., & Laine, M. (2018) · Applied Neuropsychology: Adult · doi:10.1080/23279095.2016.1263795 · Cited by 81
Schuler, M., Musekamp, G., Faller, H., Ehlebracht-König, I., Gutenbrunner, C., Kirchhof, R., Bengel, J., Nolte, S., Osborne, R. H., & Schwarze, M. (2013) · Quality of Life Research · doi:10.1007/s11136-012-0268-6 · Cited by 77
Li, L., Zeng, L., Lin, Z.-J., Cazzell, M., & Liu, H. (2015) · Journal of Biomedical Optics · doi:10.1117/1.jbo.20.5.050801 · Cited by 78
Validity and reliability of a food skills questionnaire
Kennedy, L. G., Kichler, E. J., Seabrook, J. A., Matthews, J. I., & Dworatzek, P. D. N. (2019) · Journal of Nutrition Education and Behavior · doi:10.1016/j.jneb.2019.02.003 · Cited by 72
Test-Retest Reliability of Barbell Velocity during the Free-Weight Bench-Press Exercise
Stock, M. S., Beck, T. W., DeFreitas, J. M., & Dillon, M. A. (2011) · Journal of Strength and Conditioning Research · doi:10.1519/jsc.0b013e318201bdf9 · Cited by 70
Meng, R., Yu, Y., Chai, S., Luo, X., Gong, B., Liu, B., Hu, Y., Luo, Y., & Yu, C. (2019) · Psychology Research and Behavior Management · doi:10.2147/prbm.s216411 · Cited by 72
Jenkins, N. D. M., Buckner, S. L., Bergstrom, H. C., Cochrane, K. C., Goldsmith, J. A., Housh, T. J., Johnson, G. O., Schmidt, R. J., & Cramer, J. T. (2014) · Experimental Gerontology · doi:10.1016/j.exger.2014.07.007 · Cited by 65
Tinsley, G. M., Moore, M. L., Benavides, M. L., Dellinger, J. R., & Adamson, B. T. (2020) · Clinical Nutrition · doi:10.1016/j.clnu.2020.02.008 · Cited by 66
Duc, C., Salvia, P., Lubansu, A., Feipel, V., & Aminian, K. (2014) · Medical Engineering & Physics · doi:10.1016/j.medengphy.2013.09.002 · Cited by 61
Creagh, A. P., Hamy, V., Yuan, H., Mertes, G., Tomlinson, R., Chen, W.-H., Williams, R., Llop, C., Yee, C., Duh, M. S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01013-y · Cited by 64
Aalbers, T., Baars, M. A. E., Olde Rikkert, M. G. M., & Kessels, R. P. C. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2860 · Cited by 59
Valamatos, M. J., Tavares, F., Santos, R. M., Veloso, A. P., & Mil-Homens, P. (2018) · European Journal of Applied Physiology · doi:10.1007/s00421-018-3932-x · Cited by 54
Voxel-based morphometry at ultra-high fields. A comparison of 7T and 3T MRI data
Seiger, R., Hahn, A., Hummer, A., Kranz, G. S., Ganger, S., Küblböck, M., Kraus, C., Sladky, R., Kasper, S., Windischberger, C., et al. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.03.019 · Cited by 48
Lockwood, C. M., Roberts, M. D., Dalbo, V. J., Smith-Ryan, A. E., Kendall, K. L., Moon, J. R., & Stout, J. R. (2017) · Journal of the American College of Nutrition · doi:10.1080/07315724.2016.1140094 · Cited by 47
Using Portable Transducers to Measure Tremor Severity
Elble, R., & McNames, J. (2016) · Tremor and Other Hyperkinetic Movements · doi:10.7916/d8dr2vcc · Cited by 43
Motor unit control strategies of endurance- versus resistance-trained individuals.
Herda, T. J., Siedlik, J. A., Trevino, M. A., Cooper, M. A., & Weir, J. P. (2015) · Muscle & Nerve · doi:10.1002/mus.24597 · Cited by 42
Self-citationCaen, K., Bourgois, J. G., Stassijns, E., & Boone, J. (2022) · European Journal of Applied Physiology · doi:10.1007/s00421-022-04942-2 · Cited by 41
Yogev, A., Arnold, J., Nelson, H., Clarke, D. C., Guenette, J. A., Sporer, B. C., & Koehle, M. S. (2023) · Frontiers in Sports and Active Living · doi:10.3389/fspor.2023.1143393 · Cited by 40
Lockwood, C. M., Moon, J. R., Tobkin, S. E., Walter, A. A., Smith, A. E., Dalbo, V. J., Cramer, J. T., & Stout, J. R. (2008) · Nutrition & Metabolism · doi:10.1186/1743-7075-5-11 · Cited by 38
Schlager, A., Ahlqvist, K., Rasmussen-Barr, E., Bjelland, E. K., Pingel, R., Olsson, C., Nilsson-Wikmar, L., & Kristiansson, P. (2018) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-018-2290-5 · Cited by 35
“The SEM is important since a smaller SEM indicates more reliable results [33].”
BackgroundAssessment of Head Tremor with Accelerometers Versus Gyroscopic Transducers
Elble, R. J., Hellriegel, H., Raethjen, J., & Deuschl, G. (2017) · Movement Disorders Clinical Practice · doi:10.1002/mdc3.12379 · Cited by 35
Niemeijer, V. M., Spee, R. F., Jansen, J. P., Buskermolen, A. B. C., van Dijk, T., Wijn, P. F. F., & Kemps, H. M. C. (2017) · Clinical Physiology and Functional Imaging · doi:10.1111/cpf.12269 · Cited by 33
Al‐Hashimi, K., Weerakkody, V., Elbanna, S., & Schwarz, G. (2022) · Public Administration Review · doi:10.1111/puar.13447 · Cited by 32
Beltran-Alacreu, H., López-de-Uralde-Villanueva, I., Paris-Alemany, A., Angulo-Díaz-Parreño, S., & La Touche, R. (2014) · Journal of Physical Therapy Science · doi:10.1589/jpts.26.915 · Cited by 30
Determination of Second Lactate Threshold Using Near-infrared Spectroscopy in Elite Cyclists
Salas-Montoro, J.-A., Mateo-March, M., Sánchez-Muñoz, C., & Zabala, M. (2022) · International Journal of Sports Medicine · doi:10.1055/a-1738-0252 · Cited by 27
Stout, J. R., Fukuda, D. H., Kendall, K. L., Smith-Ryan, A. E., Moon, J. R., & Hoffman, J. R. (2015) · Experimental Gerontology · doi:10.1016/j.exger.2015.02.012 · Cited by 25
Ferreira, S., Raimundo, A., del Pozo-Cruz, J., Leite, N., Pinto, A., & Marmeleira, J. (2024) · Aging Clinical and Experimental Research · doi:10.1007/s40520-024-02726-6 · Cited by 23
Factors influencing ultra-endurance athletes food choices: an adapted food choice questionnaire
Blennerhassett, C., McNaughton, L. R., & Sparks, S. A. (2019) · Research in Sports Medicine · doi:10.1080/15438627.2018.1530999 · Cited by 21
van Hooff, M., Meijer, E. J., Scheltinga, M. R. M., Savelberg, H. H. C. M., & Schep, G. (2022) · Clinical Physiology and Functional Imaging · doi:10.1111/cpf.12738 · Cited by 20
28 days of creatine nitrate supplementation is apparently safe in healthy individuals
Joy, J. M., Lowery, R. P., Falcone, P. H., Mosman, M. M., Vogel, R. M., Carson, L. R., Tai, C.-Y., Choate, D., Kimber, D., Ormes, J. A., et al. (2014) · Journal of the International Society of Sports Nutrition · doi:10.1186/s12970-014-0060-9 · Cited by 19
Xiong, P., Spira, A. P., & Hall, B. J. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17145219 · Cited by 19
Biomechanical measures in participants with shoulder pain: Intra-rater reliability
Michener, L. A., Elmore, K. A., Darter, B. J., & Timmons, M. K. (2016) · Manual Therapy · doi:10.1016/j.math.2015.10.011 · Cited by 18
(2018) · Europe PMC (PubMed Central) · Cited by 16
Koo, T. K., Cohen, J. H., & Zheng, Y. (2011) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2011.05.009 · Cited by 16
Cederberg, J. T., Weineland, S., Dahl, J., & Ljungman, G. (2018) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2018.09.004 · Cited by 13
Anders, J. P. V., Neltner, T. J., Smith, R. W., Keller, J. L., Housh, T. J., Daugherty, F. J., Tempesta, M. S., Dash, A. K., Munt, D. J., Schmidt, R. J., et al. (2021) · Journal of the International Society of Sports Nutrition · doi:10.1186/s12970-021-00456-y · Cited by 11
Fuerst, J., Lynn, R., & Kirkegaard, E. (2019) · Psych · doi:10.3390/psych1010004 · Cited by 10
“Applying the standard formula to correct the correlation for unreliability in both color and Wordsum [24], given the weighted ICCs noted in Section 3.”
MethodSendra-Pérez, C., Encarnación-Martínez, A., Oficial-Casado, F., Salvador-Palmer, R., & Priego-Quesada, J. I. (2023) · Physiological Measurement · doi:10.1088/1361-6579/ad1457 · Cited by 9
Arnold, J. I., Yogev, A., Nelson, H., van Hooff, M., & Koehle, M. S. (2024) · Frontiers in Physiology · doi:10.3389/fphys.2024.1449384 · Cited by 8
Learning and adherence to baby massage after two teaching strategies
Cruz, C. M., Caromano, F. A., Gonçalves, L. L., Machado, T. G., & Voos, M. C. (2014) · Journal for Specialists in Pediatric Nursing · doi:10.1111/jspn.12076 · Cited by 7
Mier, C. M., & Shapiro, B. S. (2013) · Journal of Strength and Conditioning Research · doi:10.1519/jsc.0b013e318257806d · Cited by 6
Eserhaut, D. A., DeLeo, J. M., Provost, J. A., Ackerman, K. E., & Fry, A. C. (2025) · Frontiers in Physiology · doi:10.3389/fphys.2025.1538465 · Cited by 4
Ettema, M., Brurok, B., & Baumgart, J. K. (2021) · Frontiers in Physiology · doi:10.3389/fphys.2021.749356 · Cited by 4
Metadata
- Authors
- WEIR, J. P.
- Venue
- Journal of Strength and Conditioning Research
- Published
- 2005-02
- Volume
- 19
- Issue
- 1
- Pages
- 231-240
- ISSN
- 1064-8011 (ISSN-L)
- OpenAlex
- W1980721081
- Cited by
- 4,842
- References
- 4
- FWCI
- 33.18
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
