The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Method matters: Understanding diagnostic reliability in DSM-IV and DSM-5.
Open accessAbstract
Diagnostic reliability is essential for the science and practice of psychology, in part because reliability is necessary for validity. Recently, the DSM-5 field trials documented lower diagnostic reliability than past field trials and the general research literature, resulting in substantial criticism of the DSM-5 diagnostic criteria. Rather than indicating specific problems with DSM-5, however, the field trials may have revealed long-standing diagnostic issues that have been hidden due to a reliance on audio/video recordings for estimating reliability. We estimated the reliability of DSM-IV diagnoses using both the standard audio-recording method and the test-retest method used in the DSM-5 field trials, in which different clinicians conduct separate interviews. Psychiatric patients (N = 339) were diagnosed using the SCID-I/P; 218 were diagnosed a second time by an independent interviewer. Diagnostic reliability using the audio-recording method (N = 49) was "good" to "excellent" (M κ = .80) and comparable to the DSM-IV field trials estimates. Reliability using the test-retest method (N = 218) was "poor" to "fair" (M κ = .47) and similar to DSM-5 field-trials' estimates. Despite low test-retest diagnostic reliability, self-reported symptoms were highly stable. Moreover, there was no association between change in self-report and change in diagnostic status. These results demonstrate the influence of method on estimates of diagnostic reliability.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The results were similar to those achieved by diagnostic assessments using more complex and time consuming structured instruments 26,39,40 .”
In the citing paper
“The DSM-5 design, therefore, did not control for information variance and so would almost certainly yield lower reliabilities 26,35 .”
In the citing paper
“First, it bears repeating that the joint-rater (concurrent) method of testing reliability, which constrains the information provided to the two diagnosticians to be identical, usually generates higher kappa values compared to those obtained when separate interviews are conducted 26,51 .”
In the citing paper
“That is, reliability is not solely a property of the classification, but also a product of the method used to estimate it.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Diagnostic and Statistical Manual of Mental Disorders
Williams, J. B. W., & First, M. B. (2013) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199975839.013.104 · Cited by 109,845
Diagnostic and Statistical Manual of Mental Disorders, 3rd ed
Cooper, A. M., & Michels, R. (1981) · American Journal of Psychiatry · doi:10.1176/ajp.138.1.128 · Cited by 7,419
Statistical Methods for Rates and Proportions, 2nd ed
GROVE, W. A. (1981) · American Journal of Psychiatry · doi:10.1176/ajp.138.12.1644-a · Cited by 3,756
The Structured Clinical Interview for DSM-III-R (SCID)
Williams, J. B. W. (1992) · Archives of General Psychiatry · doi:10.1001/archpsyc.1992.01820080038006 · Cited by 1,641 · cited 7×
“Comparing identical diagnoses, reliability in the current study appears slightly lower (kappa = .61 vs. .66) than in Williams et al. (1992) and lower than in Di Nardo et al. (1993) and Brown et al. (2001) (kappa = .45 vs. .60 and .65, respectively).”
BackgroundMethodResultInfluentialRegier, D. A., Narrow, W. E., Clarke, D. E., Kraemer, H. C., Kuramoto, S. J., Kuhl, E. A., & Kupfer, D. J. (2013) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.12070999 · Cited by 1,163 · cited 8×
“Others have suggested that (a) the lack of standardized interviews in the DSM-5 Field Trials (Regier et al., 2013) or (b) sample differences between the DSM-5 Field Trials (which used representative samples) and previous field trials (which did not) contributed to the lower reliabilities (Regier et…”
BackgroundMethodResultInfluentialBrown, T. A., Di Nardo, P. A., Lehman, C. L., & Campbell, L. A. (2001) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.110.1.49 · Cited by 887
The Collaborative Longitudinal Personality Disorders Study: Reliability of Axis I and II Diagnoses
Zanarini, M. C., Skodol, A. E., Bender, D., Dolan, R., Sanislow, C., Schaefer, E., Morey, L. C., Grilo, C. M., Shea, M. T., McGlashan, T. H., et al. (2000) · Journal of Personality Disorders · doi:10.1521/pedi.2000.14.4.291 · Cited by 771 · cited 1×
“We also note that Zanarini et al. (2000) found a similar reduction in diagnostic reliability using a small number of audio-recordings (N = 27) and test-retests (N = 52) in a non-representative patient sample.”
ResultDevelopment and validation of the Inventory of Depression and Anxiety Symptoms (IDAS).
Watson, D., O'Hara, M. W., Simms, L. J., Kotov, R., Chmielewski, M., McDade-Montez, E. A., Gamez, W., & Stuart, S. (2007) · Psychological Assessment · doi:10.1037/1040-3590.19.3.253 · Cited by 740 · cited 2×
“Self-Report Measures—Participants completed the Inventory of Depression and Anxiety Symptoms (IDAS; Watson et al., 2007).”
MethodSelf-citationDiagnosing Personality Disorders
Zimmerman, M. A. (1994) · Archives of General Psychiatry · doi:10.1001/archpsyc.1994.03950030061006 · Cited by 715 · cited 1×
“As such, the audio/videorecording method, which constrains the information provided to the two diagnosticians to be identical, can be expected to generate higher kappa values compared to those obtained when separate interviews are conducted (Kraemer et al., 2012; Zimmerman, 1994).”
ResultMarkon, K. E., Chmielewski, M., & Miller, C. J. (2011) · Psychological Bulletin · doi:10.1037/a0023678 · Cited by 656
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- First M Spitzer RL Gibbon M Williams JBW 2002 Structured clinical interview for DSM-IV-TR axis I disorders, research version, patient edition (SCID-I/P) New York Biometrics Research, New York State Psychiatric Institute
- Fleiss Joseph L 1981 Statistical methods for rates and proportions 2 Hoboken, New Jersey Wiley
Cited by
Sorted by citations
Items 101–160 of 160
Assessing Level of Personality Organization With the Psychodiagnostic Chart: A Validity Study
Hinrichs, J., Dauphin, V. B., Munday, C. C., Porcerelli, J. H., Kamoo, R., & Christian-Kliger, P. (2018) · Journal of Personality Assessment · doi:10.1080/00223891.2018.1436062 · Cited by 5
From syndromes to symptoms: Advancing our understanding of mental disorders
Clack, S. (2020) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.17151110.v1 · Cited by 3
Daneshvar, S., Bytamar, J. M., Dehbozorgi, S., Pourmohammad, P., Zeraatpisheh, Z., & Jobson, L. (2025) · BMC Psychology · doi:10.1186/s40359-025-02550-1 · Cited by 3
“Traditional psychiatric diagnoses face challenges like col-linearity, heterogeneity, arbitrary thresholds, and poor interrater reliability (e.g [23–25], which impede research progress.”
4AT screening for delirium in dementia: meta-analysis of diagnostic performance
Keane, A., Utama, K. E., Insyira, D., Angelita, A. J., Yekti, P. R. K., & Malik, K. (2025) · Dementia & Neuropsychologia · doi:10.1590/1980-5764-dn-2025-0378 · Cited by 2
“However, DSM-based diagnosis requires clinical experience, time, and formal training, therefore being less feasible in settings lacking resources or acute — such as the emergency department 6,7 .”
Pendse, S. R., Jain, M., Kumar, N., & Choudhury, M. D. (2026) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2026.100344 · Cited by 2
Classification in Traditional Nosologies
Keeley, J. W., Chung, L., & Kleva, C. S. (2023) · Oxford University Press eBooks · doi:10.1093/med-psych/9780197542521.003.0001 · Cited by 2
Reed, G. M. (2023) · Cambridge University Press eBooks · doi:10.1017/9781009182232 · Cited by 2
The Hierarchical Taxonomy of Psychopathology (HiTOP) Approach to Assessment
Simms, L. J., Wright, A. G. C., Cicero, D. C., Kotov, R., Mullins‐Sweatt, S. N., Sellbom, M., Watson, D., Widiger, T. A., & Zimmermann, J. (2023) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190092689.013.4 · Cited by 2
Self-citationA Bifactor Evaluation of Self-Report and Clinician-Administered Measures of PTSD in Veterans
Raines, A. M., Clauss, K. E., Seidler, D. A., Allan, N. P., Elhai, J. D., Vasterling, J. J., Constans, J. I., Maieritsch, K. P., & Franklin, C. L. (2024) · Assessment · doi:10.1177/10731911241229568 · Cited by 2
“As diagnostic reliability is essential for establishing validity (Chmielewski et al., 2015), future studies should seek to replicate these findings where such information is available.”
Schizophrenia or Other Primary Psychotic Disorders
Gäebel, W., & Salveridou‐Hof, E. (2023) · Cambridge University Press eBooks · doi:10.1017/9781009182232.004 · Cited by 2
Anxiety and Fear-Related Disorders: Generalized Anxiety Disorder
Rosellini, A. J., & Brown, T. A. (2024) · Tasman’s Psychiatry · doi:10.1007/978-3-030-51366-5_74 · Cited by 1
van der Wolf, M., & Robbé, M. D. V. (2022) · Routledge eBooks · doi:10.4324/9781351266482-2 · Cited by 1
Scott, T. M., & Dickson, J. M. (2025) · Journal of Affective Disorders Reports · doi:10.1016/j.jadr.2025.100917 · Cited by 1
The Reliability, Validity and Utility of a Diagnosis in Schizophrenia
Lahiri, A., Yigit, R., & Lawrie, S. M. (2025) · British Journal of Hospital Medicine · doi:10.12968/hmed.2025.0138 · Cited by 1
Facing the fog: barriers and facilitators to communicating uncertainty among psychologists
Booy, L., Quinlan, E., Deans, C., Norris, K., & Matthewson, M. L. (2025) · Psychotherapy Research · doi:10.1080/10503307.2025.2569036 · Cited by 1
“This is further evident in studies demonstrating low reliability of diagnoses across practitioners (Chmielewski et al., 2015; Markon et al., 2011), with different practitioners assigning different diagnoses to the same presentation.”
Reframing EHR Policy in China: Towards Balanced Implementation and Ethical Foundations
Chen, S., Huang, W., Xu, Y., & Yang, J. (2025) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.70273 · Cited by 1
“To ensure intercoder reliability, 20% of the policy clauses were randomly selected for parallel coding [17], and Cohen's kappa ( κ ) coefficient was calculated [18].”
Cessa, R., Ghiani, A., Cenk, E., & Bertamini, M. (2026) · Schizophrenia Research · doi:10.1016/j.schres.2026.01.004 · Cited by 2
Boecking, B., Steinmetzger, K., Rose, M., Brüeggemann, P., & Mazurek, B. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2025.1724030 · Cited by 1
Rijn, E. M. D.-V., Mensinger, M. E., Dougherty, D. D., & Widge, A. S. (2026) · Elsevier eBooks · doi:10.1016/b978-0-323-99742-3.00015-1 · Cited by 2
Rubenzer, S. (2022) · Journal of Forensic Psychology Research and Practice · doi:10.1080/24732850.2021.1991105 · Cited by 1
“However, under more naturalistic conditions (interviews conducted by different examiners, on a different day), kappa coefficients of .45-.47 (typically regarded as fair) have been reported (Chmielewski et al., 2015), with the kappas for about of a quarter of diagnoses in the poor range.”
BackgroundMethodInfluentialDave, B., & O’Connor, C. (2022) · Journal of Affective Disorders Reports · doi:10.1016/j.jadr.2022.100403 · Cited by 3
Mindless Child Psychiatry and Psychosomatics
Morales-Monsalve, M., Maldonado-Durán, J. M., & Chandra, P. (2023) · Springer eBooks · doi:10.1007/978-3-031-18377-5_5 · Cited by 1
Anxiety and Fear-Related Disorders: Generalized Anxiety Disorder
Rosellini, A. J., & Brown, T. A. (2023) · Springer eBooks · doi:10.1007/978-3-030-42825-9_74-1 · Cited by 1
Imprecise Oracles Impose Limits to Predictability in Supervised Learning (Extended Abstract)
Sifar, A., & Srivastava, N. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/661 · Cited by 1
“State-of-the-art predictions attain F-scores of close to 0.9, whereas test-retest k estimates for the most recent DSM-5 instrument are estimated at a much more modest 0 : 47 [Chmielewski et al. , 2015].”
BackgroundInfluentialRosellini, A. J., Tung, E. S., & Brown, T. A. (2018) · Cambridge University Press eBooks · doi:10.1017/9781108140416.012 · Cited by 1
Assessment of Anxiety Disorders and Obsessive-Compulsive Disorder
Peters, L., McLellan, L. F., & Brockveld, K. C. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108235433.024 · Cited by 1
Le, T. P., & Cohen, A. S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108333931.039
Chmielewski, M. S., & Trujillo, M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108333931.064
Self-citationSamuel, D. B., Bucher, M. A., & Suzuki, T. (2020) · Cambridge University Press eBooks · doi:10.1017/9781316995808.007
DSM-5: Επτά χρόνια μετά -εξελίξεις και προκλήσεις στην ταξινόμηση ψυχικών διαταραχών
Χριστοπούλου, Ά., Γκιρλέμη, Ά., Αποστολίδου, Ά., Στεφάνου, Η., Κουτουματσιώτη, Α., Μαρούλη, Ε., Φωτεινού, Μ., Ερμηλίου, Β., Κουτριάς, Α., & Ντάλη, Ε. (2020) · Psychology the Journal of the Hellenic Psychological Society · doi:10.12681/psy_hps.25597
Attachment and pathological personality traits in a psychiatric sample
Radetzki, P. A. (2021) · HARVEST (University of Saskatchewan)
Somma, A., Borroni, S., Carlotta, D., Giarolli, L., Barranca, M., Cerioli, C., Franzoni, C., Masci, E., Manini, R., Busso, S. L., et al. (2019)
Wright, A. G. C., & Hallquist, M. N. (2020) · Cambridge University Press eBooks · doi:10.1017/9781316995808.005
Bucher et al. Personality traits and treatment outcomes meta-analysisPreprint
Bucher, M. A., Suzuki, T., & Samuel, D. B. (2019) · doi:10.31219/osf.io/3ezhc
General Versus Person-specific Models of Psychotic-like Symptoms
Le, T. T. (2021) · Louisiana State University · doi:10.31390/gradschool_dissertations.5572
Extreme Emotional Disturbance: Legal Frameworks and Considerations for Forensic Evaluation
Johnston, E., Gliser, C. P., Haney, J. P., Formon, D. L., Hashimoto, N., & Rossbach, N. (2022) · Behavioral Sciences & the Law · doi:10.1002/bsl.2580
“Although the evidence for its reliability is mixed (e.g., Chmielewski et al., 2015) it has demonstrated incremental validity over prior DSM editions (Stinchfield et al., 2016).”
Resilience and Disruptive, Impulse Control, and Conduct Disorders of Childhood
Goldstein, S., Rider, R., & Velez, A. (2023) · Springer eBooks · doi:10.1007/978-3-031-14728-9_8 · Cited by 1
Soodla, H. L., Lehto, K., Kõiv, K., Vainik, U., Akkermann, K., & Mõttus, R. (2024) · doi:10.31219/osf.io/ak6vx
Schroeder, M., Wilcox, G., & Drefs, M. A. (2024) · Contemporary School Psychology · doi:10.1007/s40688-024-00509-4
“Difficulty tracking error rates is exacerbated by challenges with the reliability and validity of the tools and processes used to diagnose and classify disorders as well as the nature of mental health and developmental disorders (Chmielewski et al., 2015; Sullivan et al., 2019).”
Classification and Diagnostic Systems
Sturmey, P. (2024) · Autism and child psychopathology series · doi:10.1007/978-3-031-66902-6_3
Salvo, M. D. (2024) · Springer eBooks · doi:10.1007/978-3-031-68338-1_3
Levin-Aspenson, H. F., Watson, D., Clark, L. A., & Smith, D. A. (2025) · Clinical Psychological Science · doi:10.1177/21677026241308861
“Categorical diagnoses are associated with (a) attenuated interrater and temporal reliability (Chmielewski et al., 2015) and (b) within-diagnosis heterogeneity (Watson, 2003; Watson & Clark, 2023), both of which likely inhibit the robustness of latent variable models, and (c) artifactual comorbidity…”
Self-citationImproving mental health diagnostic quality through cognitively tractable definitions
Millroth, P., & Collsiöö, A. (2025) · Nature Mental Health · doi:10.1038/s44220-025-00404-8
Under‐Interpretation of Neuroimaging Data in Insanity Assessment: A Hidden Risk
Frangi, C., Schincariol, A., Pietrini, P., Sartori, G., Ferracuti, S., & Scarpazza, C. (2025) · Behavioral Sciences & the Law · doi:10.1002/bsl.70006
“…the consequence that psycho-pathology suffers from low inter‐rater reliability, meaning that different clinicians observing the same patients are likely to reach different diagnostic conclusions (Aboraya et al. 2006; Chmielewski et al. 2015; Miller 2001; Miller et al. 2001; Regier et al. 2013).”
Silent Signals: Using EEG and AI to Detect Hidden Mental Health Patterns
Harbi, M. R. (2025) · International Journal of Scientific Research in Science Engineering and Technology · doi:10.32628/ijsrset2512517
Sabella, K. A., & Lane, I. A. (2025) · International Journal of Behavioral Medicine · doi:10.1007/s12529-025-10390-x
“However, symptoms of a serious mental health condition are often heterogeneous within and across disorders [6], and the temporal and inter-rater reliability of SMHC diagnoses are relatively low [7–9].”
Steijer, S., Steingrímsson, S., Meyerson, N., & Sand, P. K. (2025) · BMC Psychiatry · doi:10.1186/s12888-025-07615-4
“However, this also meant that no formal case conferences or calibration meetings were held during the data collection period, which may have contributed to the comparatively lower ICC estimates observed (17).”
Lee, K., Chun, M., Kwon, J., Choi, J., Lee, S., & Im, C. (2025) · Journal of Affective Disorders · doi:10.1016/j.jad.2025.120985
Sharifibastan, F., Erevik, E. K., Pallesen, S., & Morken, K. T. E. (2023) · Open Science Framework · doi:10.17605/osf.io/g8vxb
Najman, J. M., Bor, W., Williams, G. M., Middeldorp, C. M., Mamun, A. A., Clavarino, A. M., & Scott, J. (2021) · Figshare · doi:10.6084/m9.figshare.15000176
Automatic detection of visual cues associated to depression
Pampouchidou, A. (2018) · theses.fr (ABES) · doi:10.70675/98d0e5cfz8530z4299z8ae3zb6e1b6331af1 · Cited by 4
The Hierarchical Taxonomy of Psychology (HiTOP)
Pham, T., & Forbes, M. K. (2026) · Oxford University Press eBooks · doi:10.1093/9780197769676.003.0006
Amico, F., Shannon, S., & Rondeau, S. (2026) · Journal of Personalized Medicine · doi:10.3390/jpm16050261
“We suggest that this approach could increase confidence in day-to-day psychiatric assessments, offering clinicians the opportunity to rely more on objectively measured patterns of brain activity rather than qualitative assessments [169].”
Test-Retest Reliability of Standardized Diagnostic Interviews for Common Adult Psychiatric Disorders
Xie, W., Nordgaard, J., Sheldrick, R. C., Ahmad, J. F., Gomes, F. A., Frey, B. N., & Duncan, L. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.15039
Mental Health and Serious Drug Crime: A Case–Control Study
Hesse, M., Brummer, J., Magnussen, F. I., Simonsen, E. H., & Søgaard, T. F. (2026) · Criminal Behaviour and Mental Health · doi:10.1002/cbm.70042
“Although such interviews do not guarantee high validity (Chmielewski et al. 2015), they are generally considered superior to diagnoses made without the aid of an interview guide.”
Theoretische Grundlagen und Methoden zur Bestimmung psychischer Störungen
Hengartner, M. P. (2026) · Angewandte Psychologie Kompakt · doi:10.1007/978-3-662-72845-1_2
Kohl, L. T., Oswald, F., Thöne, A., Rausch, J., Görtz‐Dorten, A., Dose, C., Hautmann, C., Treier, A., von Wirth, E., Ketter, J., et al. (2026) · PLoS ONE · doi:10.1371/journal.pone.0357785
“Interestingly, these findings reflect the categorial-dimensional debate [24,67] and the frequently reported lack of agreement between different clinicians [68–70].”
Metadata
- Authors
- Chmielewski, M., Clark, L. A., Bagby, R. M., & Watson, D.
- Affiliations
- University of Notre Dame (US)
- University of Toronto (CA)
- Venue
- Journal of Abnormal Psychology
- Published
- 2015-08
- Volume
- 124
- Issue
- 3
- Pages
- 764-769
- ISSN
- 0021-843X (ISSN-L)
- 1939-1846
- PMC
- PMC4573819
- OpenAlex
- W2184853928
- Cited by
- 161
- References
- 21
- Citations by year
- 2026: 10 · 2025: 17 · 2024: 16 · 2023: 15 · 2022: 21 · 2021: 21 · 2020: 23 · 2019: 20 · 2018: 10 · 2017: 6 · 2016: 2
- FWCI
- 5.47
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Journal of Abnormal Psychology (publishedVersion, other-oa)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (R01-MH068472)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Arts and Humanities › Philosophy › Mental Health and Psychiatry
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Mental Health Research Topics
- Keywords
- diagnostic reliability, DSM-IV, DSM-5, test-retest reliability, inter-rater reliability, psychiatric diagnosis, self-reported symptoms
- MeSH terms
- Adolescent, Adult, Aged, Aged, 80 and over, Mental Disorders, Diagnosis, Differential, Female, Humans, Male, Middle Aged, Reproducibility of Results, Diagnostic and Statistical Manual of Mental Disorders, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
