The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Measurement reliability and agreement in psychiatry
Abstract
Psychiatric research has benefited from attention to measurement theories of reliability, and reliability/agreement statistics for psychopathology ratings and diagnoses are regularly reported in empirical reports. Nevertheless, there are still controversies regarding how reliability should be measured, and the amount of resources that should be spent on studying measurement quality in research programs. These issues are discussed in the context of recent theoretical and technical contributions to the statistical analysis of reliability. Special attention is paid to statistical studies published since Kraemer's 1992 review of reliability methods in this journal.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Self-citationEstimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
rwg: an assessment of within-group interrater agreement
James, L. R., Demaree, R. G., & Wolf, G. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.2.306 · Cited by 2,221
A disagreement about within-group agreement: disentangling issues of consistency versus consensus
Kozlowski, S. W., & Hattrup, K. (1992) · Journal of Applied Psychology · doi:10.1037/0021-9010.77.2.161 · Cited by 514
Sample-size calculations for Cohen's kappa.
Cantor, A. B. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.150 · Cited by 240
Research works printed but not linked: 69
- 10.1097/00005053-197611000-00003 · DOI · Paper page
- 10.1001/archpsyc.1978.01770360058007 · DOI
- 10.1001/archpsyc.1981.01780290042004 · DOI · Paper page
- 10.1192/bjp.130.1.79 · DOI · Paper page
- 10.1037/0021-9010.74.2.368 · DOI
- 10.1001/archpsyc.1987.01800140084013 · DOI · Paper page
- 10.1016/0010-440X(83)90032-9 · DOI · Paper page
- 10.2307/1267450 · doi:10.2307/1267450 · Paper page
- 10.1177/001316446002000104 · DOI · Paper page
- Cronbach LJ, 1972, The dependability of behavioral measurements: theory of generalizability for scores and profiles
- 10.1007/BF02291260 · DOI
- 10.1007/BF02296208 · DOI · Paper page
- Lord FM, 1968, Statistical theories of mental test scores
- Dunn G, 1989, Design and analysis of reliability studies
- 10.1177/096228029200100204 · DOI
- American Psychiatric Association, 1980, Diagnostic and statistical manual of mental disorders
- American Psychiatric Association, 1994, Diagnostic and statistical manual of mental disorders
- World Health Organization, 1992, Draft international classification of diseases and related health problems
- Snedecor GW, 1967, Statistical methods
- 10.1002/9780470316665 · DOI · Paper page
- 10.2307/2532907 · DOI
- Bartko JJ, 1966, Psychological Reports, 1966, 19
- 10.1037/1082-989X.1.1.30 · DOI · Paper page
- 10.1016/0895-4356(88)90032-7 · DOI
- Spearman C, 1910, British Journal of Psychology, 3, 271
- Brown W, 1910, British Journal of Psychology, 3, 296
- 10.2307/2533400 · DOI
- 10.1016/0010-4825(90)90013-F · DOI · Paper page
- 10.1002/sim.4780122305 · DOI
- 10.1001/archpsyc.1985.01790300093012 · DOI · Paper page
- 10.1177/01466216970213006 · DOI
- 10.1037/1082-989X.1.3.293 · DOI
- 10.1002/(SICI)1097-0258(19960815)15:15<1647::AID-SIM298>3.0.CO;2-G · doi:10.1002/(sici)1097-0258(19960815)15:15<1647::aid-sim298>3.0.co;2-g
- 10.1037/0022-006X.64.6.1285 · DOI
- 10.1002/sim.4780130534 · DOI
- 10.2307/2533516 · DOI · Paper page
- 10.1177/014662169301700302 · doi:10.1177/014662169301700302
- 10.1037/h0028106 · DOI · Paper page
- 10.1002/(SICI)1097-0258(19980115)17:1<101::AID-SIM727>3.0.CO;2-E · doi:10.1002/(sici)1097-0258(19980115)17:1<101::aid-sim727>3.0.co;2-e
- Fisher RA, 1958, Statistical methods for research workers
- 10.1002/(SICI)1097-0258(19980530)17:10<1157::AID-SIM792>3.0.CO;2-W · doi:10.1002/(sici)1097-0258(19980530)17:10<1157::aid-sim792>3.0.co;2-w
- 10.2307/2533154 · DOI
- 10.1177/014662168300700301 · DOI · Paper page
- Fleiss JL, 1981, Statistical methods for rates and proportions
- 10.1080/10705519509540013 · DOI · Paper page
- 10.1207/s15327906mbr2602_9 · DOI
- 10.1177/01466216970212006 · DOI · Paper page
- 10.1207/s15327906mbr3204_2 · DOI · Paper page
- 10.1037/1082-989X.1.1.98 · DOI
- 10.1177/014662169101500101 · DOI
- 10.1007/BF02295186 · DOI
- Konishi S, 1991, Journal of the Royal Statistical Society, 53, 649, 10.1111/j.2517-6161.1991.tb01854.x · doi:10.1111/j.2517-6161.1991.tb01854.x
- Mak TK, 1988, Journal of the Royal Statistical Society, 37, 344
- 10.2307/2533208 · DOI
- 10.1007/BF02293867 · DOI
- 10.1207/s15327906mbr3202_4 · DOI
- 10.1002/sim.4780141509 · DOI
- 10.2307/2532564 · DOI
- Cornfield J, 1956, Proceedings of the third Berkeley symposium on mathematical statistics and probability
- 10.1080/03610928708829415 · DOI
- 10.1002/sim.4780111109 · DOI · Paper page
- 10.2307/2532052 · DOI · Paper page
- 10.1037/1082-989X.1.4.390 · doi:10.1037/1082-989x.1.4.390 · Paper page
- 10.1002/(SICI)1097-0258(19970530)16:10<1097::AID-SIM523>3.0.CO;2-8 · doi:10.1002/(sici)1097-0258(19970530)16:10<1097::aid-sim523>3.0.co;2-8
- 10.2307/2533395 · DOI
- Shavelson RJ, 1991, Generalizability theory: a primer
- 10.1002/sim.4780140308 · DOI
- 10.1080/01621459.1985.10477157 · DOI
- 77 Fleiss JL, Shrout PE. Reliability considerations in planning diagnostic validity studies. New York: Guilford Press, 1989: 279-229.
Cited by
The kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
(2013) · Cited by 1,351
Mintken, P. E., Glynn, P., & Cleland, J. A. (2009) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2008.12.015 · Cited by 600
Swami, V., & Barron, D. (2019) · Body Image · doi:10.1016/j.bodyim.2018.08.014 · Cited by 454
McAlinden, C., Khadka, J., & Pesudovs, K. (2011) · Ophthalmic and Physiological Optics · doi:10.1111/j.1475-1313.2011.00851.x · Cited by 436
Metadata
- Authors
- Shrout, P. E.
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 1998-06
- Volume
- 7
- Issue
- 3
- Pages
- 301-317
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W2023299366
- Cited by
- 1,216
- References
- 7
- FWCI
- 7.31
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Measurement reliability and agreement in psychiatry. https://app.cochat.ai/papers/10.1177/096228029800700306
- The paper
- Shrout, P. E. (1998). Measurement reliability and agreement in psychiatry. Statistical Methods in Medical Research. https://doi.org/10.1177/096228029800700306
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
