The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Measurement reliability and agreement in psychiatry
Abstract
Psychiatric research has benefited from attention to measurement theories of reliability, and reliability/agreement statistics for psychopathology ratings and diagnoses are regularly reported in empirical reports. Nevertheless, there are still controversies regarding how reliability should be measured, and the amount of resources that should be spent on studying measurement quality in research programs. These issues are discussed in the context of recent theoretical and technical contributions to the statistical analysis of reliability. Special attention is paid to statistical studies published since Kraemer's 1992 review of reliability methods in this journal.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Self-citationDiagnostic and Statistical Manual of Mental Disorders
&NA; (1988) · Alzheimer Disease & Associated Disorders · doi:10.1097/00002093-198802020-00022 · Cited by 12,526
Statistical Theories of Mental Test Scores
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,022
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
Journal of the Royal Statistical Society.
(1923) · The Economic Journal · doi:10.1093/ej/33.130.269 · Cited by 3,219
Fuller, W. A. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316665 · Cited by 2,351
rwg: an assessment of within-group interrater agreement
James, L. R., Demaree, R. G., & Wolf, G. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.2.306 · Cited by 2,221
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Estimation of composite reliability for congeneric measures
Raykov, T. (1997) · Applied Psychological Measurement · doi:10.1177/01466216970212006 · Cited by 1,775
Statistical Methods for Rates and Proportions.
Plewis, I., & Fleiss, J. L. (1982) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2982107 · Cited by 1,731
Cohen, J. (1983) · Applied Psychological Measurement · doi:10.1177/014662168300700301 · Cited by 1,564
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
ON THE METHODS AND THEORY OF RELIABILITY
BARTKO, J. J., & CARPENTER, W. T. (1976) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-197611000-00003 · Cited by 1,310
Generalizability theory: a primer
(1992) · Choice Reviews Online · doi:10.5860/choice.29-4192 · Cited by 1,298
CORRELATION CALCULATED FROM FAULTY DATA
SPEARMAN, C. (1910) · British Journal of Psychology, 1904-1920 · doi:10.1111/j.2044-8295.1910.tb00206.x · Cited by 1,144
(1921) · Mind · doi:10.1093/mind/xxx.119.375 · Cited by 867
Bland, J. M., & Altman, D. G. (1990) · Computers in Biology and Medicine · doi:10.1016/0010-4825(90)90013-f · Cited by 691
Psychiatric diagnosis: Are clinicians still necessary?
Spitzer, R. L. (1983) · Comprehensive Psychiatry · doi:10.1016/0010-440x(83)90032-9 · Cited by 687
SOME EXPERIMENTAL RESULTS IN THE CORRELATION OF MENTAL ABILITIES 1
BROWN, W. (1910) · British Journal of Psychology, 1904-1920 · doi:10.1111/j.2044-8295.1910.tb00207.x · Cited by 559
A disagreement about within-group agreement: disentangling issues of consistency versus consensus
Kozlowski, S. W., & Hattrup, K. (1992) · Journal of Applied Psychology · doi:10.1037/0021-9010.77.2.161 · Cited by 514
A Note On "A Concordance Correlation Coefficient to Evaluate Reproducibility"
Nickerson, C. A. E. (1997) · Biometrics · doi:10.2307/2533516 · Cited by 510
Raykov, T. (1997) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3204_2 · Cited by 481
Errors of Measurement in Statistics
Cochran, W. G. (1968) · Technometrics · doi:10.1080/00401706.1968.10490621 · Cited by 479
A Proposed Solution to the Base Rate Problem in the Kappa Statistic
Spitznagel, E. L. (1985) · Archives of General Psychiatry · doi:10.1001/archpsyc.1985.01790300093012 · Cited by 478
Reliability Studies of Psychiatric Diagnosis: Theory and Practice
Grove, W. M. (1981) · Archives of General Psychiatry · doi:10.1001/archpsyc.1981.01780290042004 · Cited by 449
(2015) · Psychological Reports · doi:10.2466/pr0.117.3.i-xi · Cited by 449
Miller, M. B. (1995) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705519509540013 · Cited by 377
Quantification of Agreement in Psychiatric Diagnosis Revisited
Shrout, P. E. (1987) · Archives of General Psychiatry · doi:10.1001/archpsyc.1987.01800140084013 · Cited by 348
Self-citationProceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability:
F., D. D., & Neyman, J. (1956) · Journal of the American Statistical Association · doi:10.2307/2281496 · Cited by 294
Ramifications of a Population Model for κ as a Coefficient of Reliability
Kraemer, H. C. (1979) · Psychometrika · doi:10.1007/bf02296208 · Cited by 293
2 x 2 Kappa Coefficients: Measures of Agreement or Association
Bloch, D. A., & Kraemer, H. C. (1989) · Biometrics · doi:10.2307/2532052 · Cited by 291
Donner, A., & Eliasziw, M. (1992) · Statistics in Medicine · doi:10.1002/sim.4780111109 · Cited by 266
Sample-size calculations for Cohen's kappa.
Cantor, A. B. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.150 · Cited by 240
"Forming inferences about some intraclass correlations coefficients": Correction.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037//1082-989x.1.4.390 · Cited by 214
Coefficients of Agreement Between Observers and Their Interpretation
Maxwell, A. E. (1977) · British Journal of Psychiatry · doi:10.1192/bjp.130.1.79 · Cited by 194
Measurement precision in test score and item response models.
Mellenbergh, G. J. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.3.293 · Cited by 165
Modeling Agreement among Raters
Tanner, M. A., & Young, M. A. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10477157 · Cited by 161
Design and Analysis of Reliability Studies
Gleser, L. J., & Dunn, G. (1991) · Technometrics · doi:10.2307/1269420 · Cited by 157
Approximate Interval Estimation for a Certain Intraclass Correlation Coefficient
Fleiss, J. L., & Shrout, P. E. (1978) · Psychometrika · doi:10.1007/bf02293867 · Cited by 148
Self-citationMeasures of agreement: A single procedure
Bartko, J. J. (1994) · Statistics in Medicine · doi:10.1002/sim.4780130534 · Cited by 144
Reliability of measurement in psychology: From Spearman-Brown to maximal reliability.
Li, H., Rosenthal, R., & Rubin, D. B. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.98 · Cited by 138
Interrater reliability coefficients cannot be computed when only one stimulus is rated.
Schmidt, F. L., & Hunter, J. E. (1989) · Journal of Applied Psychology · doi:10.1037/0021-9010.74.2.368 · Cited by 126
Measurement of reliability for categorical data in medical research
KRAEMER, H. C. (1992) · Statistical Methods in Medical Research · doi:10.1177/096228029200100204 · Cited by 114
Kappa coefficients in epidemiology: An appraisal of a reappraisal
KRAEMER, H. C., & Bloch, D. A. (1988) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(88)90032-7 · Cited by 109
Reliability and Validity in Binary Ratings
Carey, G. (1978) · Archives of General Psychiatry · doi:10.1001/archpsyc.1978.01770360058007 · Cited by 104
HOW reliable are change‐corrected measures of agreement?
Guggenmoos-Holzmann, I. (1993) · Statistics in Medicine · doi:10.1002/sim.4780122305 · Cited by 87
Measuring Interrater Agreement for Ratings of a Single Target
Lindell, M. K., & Brandt, C. J. (1997) · Applied Psychological Measurement · doi:10.1177/01466216970213006 · Cited by 86
The Intraclass Correlation Coefficient: Distribution-Free Definition and Test
Commenges, D., & Jacqmin, H. (1994) · Biometrics · doi:10.2307/2533395 · Cited by 77
Sample size requirements for the comparison of two or more coefficients of inter-observer agreement
Donner, A. P. (1998) · Statistics in Medicine · Cited by 75
Interval Estimation under Two Study Designs for Kappa with Binary Classifications
Hale, C., & Fleiss, J. L. (1993) · Biometrics · doi:10.2307/2532564 · Cited by 75
Testing the Homogeneity of Kappa Statistics
Donner, A. P., Eliasziw, M., & Klar, N. S. (1996) · Biometrics · doi:10.2307/2533154 · Cited by 74
Donner, A. P., & Eliasziw, M. (1994) · Biometrics · doi:10.2307/2533400 · Cited by 66
Measuring diagnostic agreement.
Langenbucher, J. L., Labouvie, E. W., & Morgenstern, J. (1996) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.64.6.1285 · Cited by 44
Measurement of Interrater Agreement with Adjustment for Covariates
Barlow, W. E. (1996) · Biometrics · doi:10.2307/2532907 · Cited by 40
Statistical Methods for Research Workers
Mather, K. (1945) · Nature · doi:10.1038/155132a0 · Cited by 40
Modelling covariate effects in observer agreement studies: The case of nominal scale agreement
Graham, P. J. (1995) · Statistics in Medicine · doi:10.1002/sim.4780140308 · Cited by 29
Barchard, K. A., & Hakstian, A. R. (1997) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3202_4 · Cited by 26
ANALYSIS OF MULTIVARIATE RELIABILITY STRUCTURES AND THE INDUCED BIAS IN LINEAR MODEL ESTIMATION
Aickin, M. G., & Ritenbaugh, C. (1996) · Statistics in Medicine · Cited by 26
Confidence intervals for the interrater agreement measure kappa
Flack, V. F. (1987) · Communication in Statistics- Theory and Methods · doi:10.1080/03610928708829415 · Cited by 20
Reliability of Ratings for Multiple Judges: Intraclass Correlation and Metric Scales
Fagot, R. F. (1991) · Applied Psychological Measurement · doi:10.1177/014662169101500101 · Cited by 20
Intraclass Correlation for Two-by-Two Tables under Three Sampling Designs
Bodian, C. (1994) · Biometrics · doi:10.2307/2533208 · Cited by 20
Estimating the Reliability of Interview Data
Fleiss, J. L. (1970) · Psychometrika · doi:10.1007/bf02291260 · Cited by 17
An Ordinal Coefficient of Relational Agreement for Multiple Judges
Fagot, R. F. (1994) · Psychometrika · doi:10.1007/bf02295186 · Cited by 11
Rater Reliability: A Maximum Likelihood Confirmatory Factor-Analytic Approach
O’Grady, K. E., & Medoff, D. R. (1991) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr2602_9 · Cited by 6
Lyles, R. H., & Chambless, L. E. (1995) · Statistics in Medicine · doi:10.1002/sim.4780141509 · Cited by 3
Reliability considerations in planning diagnostic validity studies
Fleiss, J. L., & Shrout, P. E. (1989) · Cited by 3
Self-citationWalter (1998) · doi:10.1002/(sici)1097-0258(19980115)17:1<101::aid-sim727>3.0.co;2-e
Research works printed but not linked: 12
- 10.2307/1267450 · doi:10.2307/1267450 · Paper page
- American Psychiatric Association, 1980, Diagnostic and statistical manual of mental disorders
- World Health Organization, 1992, Draft international classification of diseases and related health problems
- Snedecor GW, 1967, Statistical methods
- Brown W, 1910, British Journal of Psychology, 3, 296
- 10.1002/(SICI)1097-0258(19960815)15:15<1647::AID-SIM298>3.0.CO;2-G · doi:10.1002/(sici)1097-0258(19960815)15:15<1647::aid-sim298>3.0.co;2-g
- 10.1037/0022-006X.64.6.1285 · DOI
- 10.1177/014662169301700302 · DOI
- 10.1002/(SICI)1097-0258(19980530)17:10<1157::AID-SIM792>3.0.CO;2-W · doi:10.1002/(sici)1097-0258(19980530)17:10<1157::aid-sim792>3.0.co;2-w
- Konishi S, 1991, Journal of the Royal Statistical Society, 53, 649, 10.1111/j.2517-6161.1991.tb01854.x · doi:10.1111/j.2517-6161.1991.tb01854.x
- 10.1037/1082-989X.1.4.390 · doi:10.1037/1082-989x.1.4.390 · Paper page
- 10.1002/(SICI)1097-0258(19970530)16:10<1097::AID-SIM523>3.0.CO;2-8 · doi:10.1002/(sici)1097-0258(19970530)16:10<1097::aid-sim523>3.0.co;2-8
Cited by
QUANTIFYING TEST-RETEST RELIABILITY USING THE INTRACLASS CORRELATION COEFFICIENT AND THE SEM
WEIR, J. P. (2005) · Journal of Strength and Conditioning Research · doi:10.1519/00124278-200502000-00038 · Cited by 4,842
The kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed
Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M., & Streiner, D. L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.002 · Cited by 2,174
(2013) · Cited by 1,351
Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed
Kottner, J., Audige, L., Brorson, S., Donner, A., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M., & Streiner, D. L. (2011) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.01.016 · Cited by 997
Metadata
- Authors
- Shrout, P. E.
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 1998-06
- Volume
- 7
- Issue
- 3
- Pages
- 301-317
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W2023299366
- Cited by
- 1,216
- References
- 71
- FWCI
- 7.31
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.