The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index.
Review papers : Design and analysis of reliability studies
Abstract
This review covers the design and analysis of essentially two types of reliability study: method comparison studies and generalizability (including inter-rater reliability) experiments. Likelihood-based methods of inference (confirmatory factor analysis and REML estimation of variance components, for example) are advocated, partly because of their ease of use but, primarily as a stimulus to the use of more ambitious designs for the investigation of the quality of measurements. The more sophisticated approaches are not intended to replace the simpler traditional methods, however, but are expected to be used to supplement them.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965
The Hospital Anxiety and Depression Scale
Zigmond, A. S., & Snaith, R. P. (1983) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1983.tb09716.x · Cited by 47,139
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An inventory for measuring depression
BECK, A. T. (1961) · Archives of General Psychiatry · doi:10.1001/archpsyc.1961.01710120031004 · Cited by 38,032
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Statistical Methods for Rates and Proportions.
Everitt, B. S., & Fleiss, J. L. (1981) · Biometrics · doi:10.2307/2530193 · Cited by 8,990
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Development of a Rating Scale for Primary Depressive Illness
HAMILTON, M. (1967) · British Journal of Social and Clinical Psychology · doi:10.1111/j.2044-8260.1967.tb00530.x · Cited by 8,410
XV.—The Correlation between Relatives on the Supposition of Mendelian Inheritance.
Fisher, R. A. (1919) · Transactions of the Royal Society of Edinburgh · doi:10.1017/s0080456800012163 · Cited by 4,387
Recovery of inter-block information when block sizes are unequal
PATTERSON, H. D., & THOMPSON, R. (1971) · Biometrika · doi:10.1093/biomet/58.3.545 · Cited by 4,058
Fleiss, J. L., & Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300309 · Cited by 3,354
The Design and Analysis of Clinical Experiments.
Hills, M., & Fleiss, J. L. (1987) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2982050 · Cited by 3,261
Assessing Reliability and Stability in Panel Models
Wheaton, B., Muthen, B., Alwin, D. F., & Summers, G. F. (1977) · Sociological Methodology · doi:10.2307/270754 · Cited by 3,116
Educational measurement, 3rd ed.
Linn, R. L. (1989) · Cited by 2,883
The detection of psychiatric illness by questionnaire
Levy, R. (1973) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(73)90039-1 · Cited by 2,708
Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems
Harville, D. A. (1977) · Journal of the American Statistical Association · doi:10.1080/01621459.1977.10480998 · Cited by 2,667
The American Journal of Psychiatry
(2011) · American Journal of Psychiatry · doi:10.1176/appi.ajp · Cited by 2,554
Fuller, W. A. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316665 · Cited by 2,351
Statistical analysis of sets of congeneric tests
Jöreskog, K. G. (1971) · Psychometrika · doi:10.1007/bf02291393 · Cited by 1,481
Estimation of Variance and Covariance Components
Henderson, C. R. (1953) · Biometrics · doi:10.2307/3001853 · Cited by 1,311
Generalizability theory: a primer
(1992) · Choice Reviews Online · doi:10.5860/choice.29-4192 · Cited by 1,298
CORRELATION CALCULATED FROM FAULTY DATA
SPEARMAN, C. (1910) · British Journal of Psychology, 1904-1920 · doi:10.1111/j.2044-8295.1910.tb00206.x · Cited by 1,144
Multilevel Models in Educational and Social Research
Tate, R. L., & Goldstein, H. (1988) · Journal of Educational Statistics · doi:10.2307/1164754 · Cited by 993
On various intraclass correlation reliability coefficients.
Bartko, J. J. (1976) · Psychological Bulletin · doi:10.1037//0033-2909.83.5.762 · Cited by 825
Multilevel mixed linear model analysis using iterative generalized least squares
GOLDSTEIN, H. (1986) · Biometrika · doi:10.1093/biomet/73.1.43 · Cited by 802
Journal of The Royal Statistical Society
Blenkinsop, I. H. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1319 · Cited by 800
Estimating the Error Rates of Diagnostic Tests
Hui, S. L., & Walter, S. D. (1980) · Biometrics · doi:10.2307/2530508 · Cited by 780
THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY†
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963) · British Journal of Statistical Psychology · doi:10.1111/j.2044-8317.1963.tb00206.x · Cited by 737
Sample size requirements for reliability studies
Donner, A., & Eliasziw, M. (1987) · Statistics in Medicine · doi:10.1002/sim.4780060404 · Cited by 699
The Reliability of Clinical Methods, Data and Judgments: (First of Two Parts)
Koran, L. M. (1975) · New England Journal of Medicine · doi:10.1056/nejm197509252931307 · Cited by 649
Neurosis and the social environment
Bebbington, P. (1982) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(82)90026-5 · Cited by 638
SOME EXPERIMENTAL RESULTS IN THE CORRELATION OF MENTAL ABILITIES 1
BROWN, W. (1910) · British Journal of Psychology, 1904-1920 · doi:10.1111/j.2044-8295.1910.tb00207.x · Cited by 559
Factor Analysis as A Statistical Method.
Hills, M., Lawley, D. N., & Maxwell, A. E. (1972) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344634 · Cited by 483
Errors of Measurement in Statistics
Cochran, W. G. (1968) · Technometrics · doi:10.1080/00401706.1968.10490621 · Cited by 479
Reliability Studies of Psychiatric Diagnosis: Theory and Practice
Grove, W. M. (1981) · Archives of General Psychiatry · doi:10.1001/archpsyc.1981.01780290042004 · Cited by 449
The Statistical Analysis of Experimental Data.
David, H. T., & Mandel, J. (1966) · Journal of the American Statistical Association · doi:10.2307/2282811 · Cited by 436
Walter, S. (1988) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(88)90110-2 · Cited by 360
Estimation of Variance and Covariance Components in Linear Models
Rao, C. R. (1972) · Journal of the American Statistical Association · doi:10.1080/01621459.1972.10481212 · Cited by 354
Measuring Agreement for Multinomial Data
Davies, M., & Fleiss, J. L. (1982) · Biometrics · doi:10.2307/2529886 · Cited by 353
Quantification of Agreement in Psychiatric Diagnosis Revisited
Shrout, P. E. (1987) · Archives of General Psychiatry · doi:10.1001/archpsyc.1987.01800140084013 · Cited by 348
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Statistical Adjustment of Data
Finkel, A. J. (1944) · Ecology · doi:10.2307/1932029 · Cited by 332
Diagnostic Criteria of American and British Psychiatrists
Kendell, R. E. (1971) · Archives of General Psychiatry · doi:10.1001/archpsyc.1971.01750140027006 · Cited by 317
Adcock, R. J. (1878) · The Analyst · doi:10.2307/2635758 · Cited by 316
Balanced Incomplete Block Designs for Inter-Rater Reliability Studies
Fleiss, J. L. (1981) · Applied Psychological Measurement · doi:10.1177/014662168100500115 · Cited by 313
The value of latent class analysis in medical diagnosis
Rindskopf, D., & Rindskopf, W. (1986) · Statistics in Medicine · doi:10.1002/sim.4780050105 · Cited by 280
Linear Models for Unbalanced Data.
Woodward, P., & Searle, S. R. (1988) · The Statistician · doi:10.2307/2348778 · Cited by 278
Bivariate Agreement Coefficients for Reliability of Data
Krippendorff, K. (1970) · Sociological Methodology · doi:10.2307/270787 · Cited by 245
Monte Carlo Comparison of ANOVA, MIVQUE, REML, and ML Estimators of Variance Components
Swallow, W. H., & Monahan, J. F. (1984) · Technometrics · doi:10.1080/00401706.1984.10487921 · Cited by 227
Statistics for analytical chemists
MacFie, H. (1984) · TrAC Trends in Analytical Chemistry · doi:10.1016/0165-9936(84)85011-6 · Cited by 211
On Estimating Precision of Measuring Instruments and Product Variability
Grubbs, F. E. (1948) · Journal of the American Statistical Association · doi:10.1080/01621459.1948.10483261 · Cited by 210
Estimation and Use of Variance Components
Robinson, D. L. (1987) · The Statistician · doi:10.2307/2988267 · Cited by 202
Linnet, K. (1990) · Statistics in Medicine · doi:10.1002/sim.4780091210 · Cited by 175
YERUSHALMY, J. (1969) · Radiologic Clinics of North America · doi:10.1016/s0033-8389(22)01511-1 · Cited by 173
Design and analysis of reliability studies: the statistical evaluation of measurement errors
(1990) · Choice Reviews Online · doi:10.5860/choice.27-6381 · Cited by 169
Generalizability theory: 1973–1980
Shavelson, R. J., & Webb, N. M. (1981) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1981.tb00625.x · Cited by 165
Clare, A. W., & Cairns, V. E. (1978) · Psychological Medicine · doi:10.1017/s0033291700018808 · Cited by 163
Generalizability of Scores Influenced by Multiple Sources of Variance
Gleser, G. C., Cronbach, L. J., & Rajaratnam, N. (1965) · Psychometrika · doi:10.1007/bf02289531 · Cited by 151
Errors of Measurement, Precision, Accuracy and the Statistical Comparison of Measuring Instruments
Grubbs, F. E. (1973) · Technometrics · doi:10.1080/00401706.1973.10489010 · Cited by 150
Birkelo, C. C., & CHAMBERLAIN, W. E. (1947) · JAMA · doi:10.1001/jama.1947.02880060001001 · Cited by 144
Computerized tomographic scan changes in early schizophrenia – preliminary findings
Turner, S. W., Toone, B., & Brett-Jones, J. R. (1986) · Psychological Medicine · doi:10.1017/s003329170000266x · Cited by 138
Quadratic Estimation of Variance Components
LaMotte, L. R. (1973) · Biometrics · doi:10.2307/2529395 · Cited by 131
Using Latent Class Models to Characterize and Assess Relative Error in Discrete Measurements
Espeland, M. A., & Handelman, S. L. (1989) · Biometrics · doi:10.2307/2531499 · Cited by 129
American review of tuberculosis
(1938) · Tubercle · doi:10.1016/s0041-3879(38)80167-7 · Cited by 125
Reduction of Observation Equations Which Contain More Than One Observed Quantity
Kummell, C. H. (1879) · The Analyst · doi:10.2307/2635646 · Cited by 121
Statistical Methods in Laboratory Medicine
(1992) · Annals of Internal Medicine · doi:10.7326/0003-4819-116-8-704_3 · Cited by 119
Simultaneous Latent Structure Analysis in Several Groups
Clogg, C. C., & Goodman, L. A. (1985) · Sociological Methodology · doi:10.2307/270847 · Cited by 113
Disagreement Between Observers in an Epidemiological Study of Respiratory Disease
Schilling, R. S. F., Hughes, J. P. W., & Dingwall-Fordyce, I. (1955) · BMJ · doi:10.1136/bmj.1.4905.65 · Cited by 104
Variance Components Analysis: A Selective Literature Survey
Khuri, A. I., & Sahai, H. (1985) · International Statistical Review · doi:10.2307/1402893 · Cited by 101
Neuroticism and well-being inventories: measuring traits or states?
Ormel, J. (1983) · Psychological Medicine · doi:10.1017/s0033291700050170 · Cited by 91
A bibliography of publications on observer variability
Feinstein, A. R. (1985) · Journal of Chronic Diseases · doi:10.1016/0021-9681(85)90016-5 · Cited by 82
Is there a “Socratic Effect” in Nonexperimental Panel Studies?
Jagodzinski, W., Kühnel, S., & Schmidt, P. (1987) · Sociological Methods & Research · doi:10.1177/0049124187015003004 · Cited by 67
A Simple 'Synthesis'-Based Method of Variance Component Estimation
Hartley, H. O., Rao, J. N. K., & LaMotte, L. R. (1978) · Biometrics · doi:10.2307/2530013 · Cited by 64
Sampling Errors of Variance Components in Small Sample Multifacet Generalizability Studies
Smith, P. L. (1978) · Journal of Educational Statistics · doi:10.3102/10769986003004319 · Cited by 56
Neuroticism, depression and life events: A structural equation model
Horwood, L. J., & Fergusson, D. M. (1986) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/bf00578744 · Cited by 54
Statistical Comparison of Multiple Analytic Procedures: Application to Clinical Chemistry
Lawton, W. H., Sylvestre, E. A., & Young-Ferraro, B. J. (1979) · Technometrics · doi:10.1080/00401706.1979.10489809 · Cited by 47
A Comparison of Three Different Procedures for Estimating Variance Components
Bush, N., & Anderson, R. L. (1963) · Technometrics · doi:10.1080/00401706.1963.10490121 · Cited by 45
Common, Specific, and Error Variance Components of Factor Models: Estimation with Longitudinal Data
Raffalovich, L. E., & Bohrnstedt, G. W. (1987) · Sociological Methods & Research · doi:10.1177/0049124187015004003 · Cited by 43
Healy, M. J. R. (1989) · Statistics in Medicine · doi:10.1002/sim.4780080802 · Cited by 43
EQS: Structural Equations Program, Version 3.0
Long, J. S., Bentler, P. M., Steiger, J. H., Muthén, B., Jöreskog, K. G., & Sörbom, D. (1990) · Journal of Marketing Research · doi:10.2307/3172597 · Cited by 42
Significance Test for Grubbs's Estimators
Maloney, C. J., & Rastogi, S. C. (1970) · Biometrics · doi:10.2307/2528714 · Cited by 42
The Accuracy of Measuring Leg-length Differences
Nichols, P. J. R., & Bailey, N. T. J. (1955) · BMJ · doi:10.1136/bmj.2.4950.1247 · Cited by 41
Goldsmith, C. H., & Gaylor, D. W. (1970) · Technometrics · doi:10.1080/00401706.1970.10488692 · Cited by 27
Espeland, M. A., Murphy, W. C., & Leverett, D. H. (1988) · Statistics in Medicine · doi:10.1002/sim.4780070306 · Cited by 23
GAINING ACCURACY IN GENERALIZABILITY THEORY: USING MULTIPLE DESIGNS
SMITH, P. L. (1981) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1981.tb00849.x · Cited by 20
Estimating the Reliability of Interview Data
Fleiss, J. L. (1970) · Psychometrika · doi:10.1007/bf02291260 · Cited by 17
Comparison of three devices for measuring blood pressure
Bassein, L., Borghi, C., Costa, F. V., et al. (1985) · Statistics in Medicine · doi:10.1002/sim.4780040316 · Cited by 17
MOSTAFA, M. G. (1967) · Biometrika · doi:10.1093/biomet/54.1-2.127 · Cited by 11
The Calculation of Reliability From a Split-Plot Factorial Design
Brennan, R. L. (1975) · Educational and Psychological Measurement · doi:10.1177/001316447503500404 · Cited by 10
Cited by
Lewis, G., Pelosi, A. J., Araya, R., & Dunn, G. (1992) · Psychological Medicine · doi:10.1017/s0033291700030415 · Cited by 1,487
Self-citationHADDOCK, G., McCARRON, J., TARRIER, N., & FARAGHER, E. B. (1999) · Psychological Medicine · doi:10.1017/s0033291799008661 · Cited by 1,461
Development, validation and utilisation of food-frequency questionnaires – a review
Cade, J., Thompson, R., Burley, V., & Warm, D. (2002) · Public Health Nutrition · doi:10.1079/phn2001318 · Cited by 1,423
Hand, D. J. (2007) · Drug Safety · doi:10.2165/00002018-200730070-00010 · Cited by 1,044
Jones, S. H., Thornicroft, G., Coffey, M., & Dunn, G. (1995) · British Journal of Psychiatry · doi:10.1192/bjp.166.5.654 · Cited by 1,015
Self-citation
Metadata
- Authors
- Dunn, G.
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 1992-08
- Volume
- 1
- Issue
- 2
- Pages
- 123-157
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W2045917085
- Cited by
- 325
- References
- 91
- FWCI
- 17.24
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.