Assessing intrarater, interrater, and test-retest reliability of continuous measurements
Abstract
In this paper we review the problem of defining and estimating intrarater, interrater and test-retest reliability of continuous measurements. We argue that the usual notion of product-moment correlation is well adapted in a test-retest situation, whereas the concept of intraclass correlation should be used for intrarater and interrater reliability. The key difference between these two approaches is the treatment of systematic error, which is often due to a learning effect for test-retest data. We also consider the reliability of a sum and a difference of variables and illustrate the effects on components. Further, we compare these approaches of reliability with the concept of limits of agreement proposed by Bland and Altman (for evaluating the agreement between two methods of clinical measurements) and show how product-moment correlation is related to it. We then propose new kinds of limits of agreement which are related to intraclass correlation. A test battery to study the development of neuro-motor functions in children and adolescents illustrates our purpose throughout the paper.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Inter-rater reliability for V_T' across eight brain regions ranged between 0.82 and 0.97.
“The interclass coefficient to assess inter-rater reliability (Rousson et al., 2002) ranged between 0.82 and 0.97 for the V T Ј measure across the eight brain regions.”
From Human Tobacco Smokers in Early Abstinence Have Higher Levels of β<sub>2</sub>* Nicotinic Acetylcholine Receptors than Nonsmokers (2006) · cites this paper as Rousson et al., 2002
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965 · see citing works
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
Beckett, A. H., Rowland, M., & TURNER, P. (1965) · The Lancet · doi:10.1016/s0140-6736(65)91037-8 · Cited by 16,917 · see citing works
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373 · see citing works
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206 · see citing works
Encyclopedia of Statistical Sciences.
Tanur, J. M., Casella, G., Dykstra, R. L., Finster, M., Gaver, D. P., Greenhouse, J. B., Iversen, G. R., Jasso, G., Kmenta, J., Press, S. J., et al. (1989) · Journal of the American Statistical Association · doi:10.2307/2289675 · Cited by 728 · see citing works
Bland, M., & Altman, D. G. (1990) · Computers in Biology and Medicine · doi:10.1016/0010-4825(90)90013-f · Cited by 689 · see citing works
Statistical evaluation of agreement between two methods for measuring a quantitative variable
Lee, J., Koh, D. S. Q., & Ong, C. N. (1989) · Computers in Biology and Medicine · doi:10.1016/0010-4825(89)90036-x · Cited by 473 · see citing works
On the Comparative Anatomy of Transformations
Tukey, J. W. (1957) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177706875 · Cited by 466 · see citing works
Neuromotor development from 5 to 18 years. Part 1: timed performance
Largo, R. H., Caflisch, J., Hug, F. P., Muggli, K., Molnar, A. A., Molinari, L., Sheehy, A., & Gasser, T. (2001) · Developmental Medicine & Child Neurology · doi:10.1017/s0012162201000810 · Cited by 202 · see citing works
Self-citationApproximate Interval Estimation for a Certain Intraclass Correlation Coefficient
Fleiss, J. L., & Shrout, P. E. (1978) · Psychometrika · doi:10.1007/bf02293867 · Cited by 148 · see citing works
Pearson, K., Lee, A., & Bramley-Moore, L. (1899) · Philosophical Transactions of the Royal Society of London Series A Containing Papers of a Mathematical or Physical Character · doi:10.1098/rsta.1899.0006 · Cited by 134 · see citing works
Neuromotor development from 5 to 18 years. Part 1: timed performance
Largo, R. H., Caflisch, J., Hug, F. P., et al. (2001) · Developmental Medicine & Child Neurology · doi:10.1111/j.1469-8749.2001.tb00739.x · Cited by 54 · see citing works
Reliability Formulas for Independent Decision Data When Reliability Data are Matched
Rajaratnam, N. (1960) · Psychometrika · doi:10.1007/bf02289730 · Cited by 45 · see citing works
Tan, M. T., Qu, Y., Mascha, E. J., & Schubert, A. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990815)18:15 · Cited by 26 · see citing works
Emergency medicine — a house that Jack is building?
Taylor, T. K. F. (1989) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.1989.tb115999.x · Cited by 4 · see citing works
Irwig, L., & SIMPSON, J. (1989) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.1989.tb116001.x · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018) · Frontiers in Public Health · doi:10.3389/fpubh.2018.00149 · Cited by 5,925 · see citing works
Zuo, X.-N., Kelly, C., Adelstein, J. S., Klein, D. F., Castellanos, F. X., & Milham, M. P. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.10.080 · Cited by 744 · see citing works
Belur, J., Tompson, L., Thornton, A., & Simon, M. (2021) · Sociological Methods & Research · doi:10.1177/0049124118799372 · Cited by 753 · see citing works
Swami, V., & Barron, D. (2019) · Body Image · doi:10.1016/j.bodyim.2018.08.014 · Cited by 454 · see citing works
Braun, U., Plichta, M. M., Esslinger, C., Sauer, C., Haddad, L., Grimm, O., Mier, D., Mohnke, S., Heinz, A., Erk, S., et al. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.08.044 · Cited by 424 · see citing works
Berthoz, S., & Hill, E. L. (2005) · European Psychiatry · doi:10.1016/j.eurpsy.2004.06.013 · Cited by 400 · see citing works
Plasma β-amyloid and white matter lesions in AD, MCI, and cerebral amyloid angiopathy
Gurol, M. E., Irizarry, M. C., Smith, E. E., Raju, S., Diaz-Arrastia, R., Bottiglieri, T., Rosand, J., Growdon, J. H., & Greenberg, S. M. (2006) · Neurology · doi:10.1212/01.wnl.0000191403.95453.6a · Cited by 339 · see citing works
Test-retest reliability of evoked BOLD signals from a cognitive-emotive fMRI test battery.
Plichta, M. M., Schwarz, A. J., Grimm, O., Morgen, K., Mier, D., Haddad, L., Gerdes, A. B. M., Sauer, C., Tost, H., Esslinger, C., et al. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.129 · Cited by 287 · see citing works
Staley, J. K., Krishnan-Sarin, S., Cosgrove, K. P., Krantzler, E., Frohlich, E., Perry, E., Dubin, J. A., Estok, K., Brenner, E., Baldwin, R. M., et al. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.0546-06.2006 · Cited by 223 · see citing works
“The interclass coefficient to assess inter-rater reliability (Rousson et al., 2002) ranged between 0.”
BackgroundMethodInfluentialInter-Coder Agreement in Qualitative Coding: Considerations for its Use
Halpin, S. N. (2024) · American Journal of Qualitative Research · doi:10.29333/ajqr/14887 · Cited by 149 · see citing works
KUNTSI, J., ROGERS, H., SWINARD, G., BÖRGER, N., MEERE, J. V. D., RIJSDIJK, F., & ASHERSON, P. (2006) · Psychological Medicine · doi:10.1017/s0033291706008580 · Cited by 126 · see citing works
Soreni, N., Crosbie, J., Ickowicz, A., & Schachar, R. (2009) · Journal of Attention Disorders · doi:10.1177/1087054708326110 · Cited by 121 · see citing works
Test–retest reliability of five frequently used executive tasks in healthy adults
Soveri, A., Lehtonen, M., Karlsson, L. C., Lukasik, K., Antfolk, J., & Laine, M. (2018) · Applied Neuropsychology: Adult · doi:10.1080/23279095.2016.1263795 · Cited by 81 · see citing works
Schuler, M., Musekamp, G., Faller, H., Ehlebracht-König, I., Gutenbrunner, C., Kirchhof, R., Bengel, J., Nolte, S., Osborne, R. H., & Schwarze, M. (2013) · Quality of Life Research · doi:10.1007/s11136-012-0268-6 · Cited by 77 · see citing works
- Authors
- Rousson, V., Gasser, T., & Seifert, B.
- Affiliations
- University of Zurich (CH)
- Corresponding author
- Rousson, Valentin
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2002-10-24
- Volume
- 21
- Issue
- 22
- Pages
- 3431-3446
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2057591576
- Cited by
- 415
- References
- 17
- Citations by year
- 2026: 8 · 2025: 10 · 2024: 21 · 2023: 16 · 2022: 16 · 2021: 41 · 2020: 15 · 2019: 15 · 2018: 28 · 2017: 20 · 2016: 24 · 2015: 26 · 2014: 18 · 2013: 17 · 2012: 20
- FWCI
- 10.12
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Statistics in Medicine (publishedVersion)
- PubMed (publishedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- intra-rater reliability, inter-rater reliability, test-retest reliability, continuous measurement, product moment correlation, intraclass correlation coefficient, systematic errors, learning effects, limits of agreement, test battery, children and adolescents
- MeSH terms
- Child, Female, Humans, Male, Motor Skills, Reproducibility of Results, Models, Statistical, Observer Variation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
