The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Method matters: Understanding diagnostic reliability in DSM-IV and DSM-5.
Open accessAbstract
Diagnostic reliability is essential for the science and practice of psychology, in part because reliability is necessary for validity. Recently, the DSM-5 field trials documented lower diagnostic reliability than past field trials and the general research literature, resulting in substantial criticism of the DSM-5 diagnostic criteria. Rather than indicating specific problems with DSM-5, however, the field trials may have revealed long-standing diagnostic issues that have been hidden due to a reliance on audio/video recordings for estimating reliability. We estimated the reliability of DSM-IV diagnoses using both the standard audio-recording method and the test-retest method used in the DSM-5 field trials, in which different clinicians conduct separate interviews. Psychiatric patients (N = 339) were diagnosed using the SCID-I/P; 218 were diagnosed a second time by an independent interviewer. Diagnostic reliability using the audio-recording method (N = 49) was "good" to "excellent" (M κ = .80) and comparable to the DSM-IV field trials estimates. Reliability using the test-retest method (N = 218) was "poor" to "fair" (M κ = .47) and similar to DSM-5 field-trials' estimates. Despite low test-retest diagnostic reliability, self-reported symptoms were highly stable. Moreover, there was no association between change in self-report and change in diagnostic status. These results demonstrate the influence of method on estimates of diagnostic reliability.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The results were similar to those achieved by diagnostic assessments using more complex and time consuming structured instruments 26,39,40 .”
In the citing paper
“The DSM-5 design, therefore, did not control for information variance and so would almost certainly yield lower reliabilities 26,35 .”
In the citing paper
“First, it bears repeating that the joint-rater (concurrent) method of testing reliability, which constrains the information provided to the two diagnosticians to be identical, usually generates higher kappa values compared to those obtained when separate interviews are conducted 26,51 .”
In the citing paper
“That is, reliability is not solely a property of the classification, but also a product of the method used to estimate it.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Diagnostic and Statistical Manual of Mental Disorders
Williams, J. B. W., & First, M. B. (2013) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199975839.013.104 · Cited by 109,845
Diagnostic and Statistical Manual of Mental Disorders, 3rd ed
Cooper, A. M., & Michels, R. (1981) · American Journal of Psychiatry · doi:10.1176/ajp.138.1.128 · Cited by 7,419
Statistical Methods for Rates and Proportions, 2nd ed
GROVE, W. A. (1981) · American Journal of Psychiatry · doi:10.1176/ajp.138.12.1644-a · Cited by 3,756
The Structured Clinical Interview for DSM-III-R (SCID)
Williams, J. B. W. (1992) · Archives of General Psychiatry · doi:10.1001/archpsyc.1992.01820080038006 · Cited by 1,641 · cited 7×
“Comparing identical diagnoses, reliability in the current study appears slightly lower (kappa = .61 vs. .66) than in Williams et al. (1992) and lower than in Di Nardo et al. (1993) and Brown et al. (2001) (kappa = .45 vs. .60 and .65, respectively).”
BackgroundMethodResultInfluentialRegier, D. A., Narrow, W. E., Clarke, D. E., Kraemer, H. C., Kuramoto, S. J., Kuhl, E. A., & Kupfer, D. J. (2013) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.12070999 · Cited by 1,163 · cited 8×
“Others have suggested that (a) the lack of standardized interviews in the DSM-5 Field Trials (Regier et al., 2013) or (b) sample differences between the DSM-5 Field Trials (which used representative samples) and previous field trials (which did not) contributed to the lower reliabilities (Regier et…”
BackgroundMethodResultInfluentialBrown, T. A., Di Nardo, P. A., Lehman, C. L., & Campbell, L. A. (2001) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.110.1.49 · Cited by 887
The Collaborative Longitudinal Personality Disorders Study: Reliability of Axis I and II Diagnoses
Zanarini, M. C., Skodol, A. E., Bender, D., Dolan, R., Sanislow, C., Schaefer, E., Morey, L. C., Grilo, C. M., Shea, M. T., McGlashan, T. H., et al. (2000) · Journal of Personality Disorders · doi:10.1521/pedi.2000.14.4.291 · Cited by 771 · cited 1×
“We also note that Zanarini et al. (2000) found a similar reduction in diagnostic reliability using a small number of audio-recordings (N = 27) and test-retests (N = 52) in a non-representative patient sample.”
ResultDevelopment and validation of the Inventory of Depression and Anxiety Symptoms (IDAS).
Watson, D., O'Hara, M. W., Simms, L. J., Kotov, R., Chmielewski, M., McDade-Montez, E. A., Gamez, W., & Stuart, S. (2007) · Psychological Assessment · doi:10.1037/1040-3590.19.3.253 · Cited by 740 · cited 2×
“Self-Report Measures—Participants completed the Inventory of Depression and Anxiety Symptoms (IDAS; Watson et al., 2007).”
MethodSelf-citationDiagnosing Personality Disorders
Zimmerman, M. A. (1994) · Archives of General Psychiatry · doi:10.1001/archpsyc.1994.03950030061006 · Cited by 715 · cited 1×
“As such, the audio/videorecording method, which constrains the information provided to the two diagnosticians to be identical, can be expected to generate higher kappa values compared to those obtained when separate interviews are conducted (Kraemer et al., 2012; Zimmerman, 1994).”
ResultMarkon, K. E., Chmielewski, M., & Miller, C. J. (2011) · Psychological Bulletin · doi:10.1037/a0023678 · Cited by 656
Self-citationA Re-analysis of the Reliability of Psychiatric Diagnosis
Spitzer, R. L., & Fleiss, J. L. (1974) · The British Journal of Psychiatry · doi:10.1192/bjp.125.4.341 · Cited by 622 · cited 3×
“Prior to DSM-III (American Psychiatric Association, 1980), diagnostic reliability was poor, due in part to the lack of specific diagnostic criteria (Spitzer & Fleiss, 1974).”
BackgroundDi Nardo, P. A. (1993) · Archives of General Psychiatry · doi:10.1001/archpsyc.1993.01820160009001 · Cited by 498 · cited 2×
“In addition, Brown et al. (2001) and Di Nardo et al. (1993) used several inclusion and exclusion criteria that may have reduced diagnostic “noise.””
MethodDSM-III field trials: I. Initial interrater diagnostic reliability
(1979) · American Journal of Psychiatry · doi:10.1176/ajp.136.6.815 · Cited by 394 · cited 17×
“The mean kappa of .80, as well as those of the majority of diagnoses, would be considered “excellent” by traditional standards (Fleiss, 1981; Spitzer et al., 1979).”
BackgroundMethodResultInfluentialWatson, D., O'Hara, M. W., Chmielewski, M., McDade-Montez, E. A., Koffel, E., Naragon, K., & Stuart, S. (2008) · Psychological Assessment · doi:10.1037/a0012570 · Cited by 215 · cited 2×
“The IDAS scales show strong psychometric properties compared to commonly used depression and anxiety measures (Watson et al., 2007, 2008).”
MethodSelf-citationDSM-5: How Reliable Is Reliable Enough?
KRAEMER, H. C., Kupfer, D. J., Clarke, D. E., Narrow, W. E., & Regier, D. A. (2012) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2011.11010050 · Cited by 180 · cited 6×
“…to which a patient would receive the same diagnosis at different hospitals or clinics, or the extent to which different studies are recruiting similar patients, then the test-retest method provides a more meaningful estimate of diagnostic reliability (Kraemer et al., 2012; Williams et al., 1992).”
MethodResultInfluentialKlerman, G. L., Vaillant, G. E., Spitzer, R. L., & Michels, R. (1984) · American Journal of Psychiatry · doi:10.1176/ajp.141.4.539 · Cited by 153
Metaphorical analysis of psychiatric classification as a psychological test.
Blashfield, R. K., & Livesley, W. J. (1991) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.100.3.262 · Cited by 101 · cited 1×
“As Blashfield & Livesley (1991) noted, high test-retest reliability over short timeframes is essential for diagnostic validity.”
BackgroundMcKenzie, D. P., Mackinnon, A. J., Péladeau, N., Onghena, P., Bruce, P. C., Clarke, D. M., Harrigan, S., & McGorry, P. D. (1996) · Journal of Psychiatric Research · doi:10.1016/s0022-3956(96)00033-7 · Cited by 91
Comparing the temporal stability of self-report and interview assessed personality disorder.
Samuel, D. B., Hopwood, C. J., Ansell, E. B., Morey, L. C., Sanislow, C. A., Markowitz, J. C., Yen, S., Shea, M. T., Skodol, A. E., & Grilo, C. M. (2011) · Journal of Abnormal Psychology · doi:10.1037/a0022647 · Cited by 63
DSM-IV: Empirical guidelines from psychometrics.
Nelson‐Gray, R. O. (1991) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.100.3.308 · Cited by 50 · cited 2×
“It has long been argued that reliability sets an upper limit on validity (Nelson-Gray, 1991; Spitzer & Fleiss, 1974); however, the J Abnorm Psychol.”
BackgroundStandards for DSM-5 Reliability
Spitzer, R. L., Williams, J. B. W., & Endicott, J. (2012) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.12010083 · Cited by 31 · cited 1×
“Others have been far more critical (Frances, 2012; Spitzer, Williams, & Endicott, 2012), arguing that the manual “flunked its reliability tests” (Frances, 2012) and that traditional kappa guidelines should be applied (Frances, 2012; Spitzer et al., 2012).”
Background
Research works printed but not linked: 2
- First M Spitzer RL Gibbon M Williams JBW 2002 Structured clinical interview for DSM-IV-TR axis I disorders, research version, patient edition (SCID-I/P) New York Biometrics Research, New York State Psychiatric Institute
- Fleiss Joseph L 1981 Statistical methods for rates and proportions 2 Hoboken, New Jersey Wiley
Cited by
Sorted by citations
Kotov, R., Krueger, R. F., Watson, D., Achenbach, T. M., Althoff, R. R., Bagby, R. M., Brown, T. A., Carpenter, W. T., Caspi, A., Clark, L. A., et al. (2017) · Journal of Abnormal Psychology · doi:10.1037/abn0000258 · Cited by 3,335
“Second, traditional diagnoses generally show limited reliability, as can be expected when arbitrary categories are forced onto dimensional phenomena (Chmielewski, Clark, Bagby, & Watson, 2015; Markon, 2013).”
Self-citationAn MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027
“…should also report the psychometric properties of the measures in the studied sample and compare them to previous research, when available, as this provides valuable information about the performance of the measure and the quality of data itself (Chmielewski, Clark, Bagby, & Watson, 2015).”
Self-citationIntegrating the Hierarchical Taxonomy of Psychopathology (HiTOP) into clinical practice.
Ruggero, C. J., Kotov, R., Hopwood, C. J., First, M., Clark, L. A., Skodol, A. E., Mullins-Sweatt, S. N., Patrick, C. J., Bach, B., Cicero, D. C., et al. (2019) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000452 · Cited by 333
Self-citationValidity and utility of Hierarchical Taxonomy of Psychopathology (HiTOP): I. Psychosis superspectrum
Kotov, R., Jonas, K. G., Carpenter, W. T., Dretsch, M. N., Eaton, N. R., Forbes, M. K., Forbush, K. T., Hobbs, K. A., Reininghaus, U., Slade, T. N., et al. (2020) · World Psychiatry · doi:10.1002/wps.20730 · Cited by 330
Self-citationWatson, D., Levin‐Aspenson, H. F., Waszczuk, M. A., Conway, C. C., Dalgleish, T., Dretsch, M. N., Eaton, N. R., Forbes, M. K., Forbush, K. T., Hobbs, K. A., et al. (2022) · World Psychiatry · doi:10.1002/wps.20943 · Cited by 311
Self-citation
Metadata
- Authors
- Chmielewski, M., Clark, L. A., Bagby, R. M., & Watson, D.
- Affiliations
- University of Notre Dame (US)
- University of Toronto (CA)
- Venue
- Journal of Abnormal Psychology
- Published
- 2015-08
- Volume
- 124
- Issue
- 3
- Pages
- 764-769
- ISSN
- 0021-843X (ISSN-L)
- 1939-1846
- PMC
- PMC4573819
- OpenAlex
- W2184853928
- Cited by
- 161
- References
- 21
- Citations by year
- 2026: 10 · 2025: 17 · 2024: 16 · 2023: 15 · 2022: 21 · 2021: 21 · 2020: 23 · 2019: 20 · 2018: 10 · 2017: 6 · 2016: 2
- FWCI
- 5.47
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Journal of Abnormal Psychology (publishedVersion, other-oa)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (R01-MH068472)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Arts and Humanities › Philosophy › Mental Health and Psychiatry
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Mental Health Research Topics
- Keywords
- diagnostic reliability, DSM-IV, DSM-5, test-retest reliability, inter-rater reliability, psychiatric diagnosis, self-reported symptoms
- MeSH terms
- Adolescent, Adult, Aged, Aged, 80 and over, Mental Disorders, Diagnosis, Differential, Female, Humans, Male, Middle Aged, Reproducibility of Results, Diagnostic and Statistical Manual of Mental Disorders, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
