Method matters: Understanding diagnostic reliability in DSM-IV and DSM-5.
Open accessAbstract
Diagnostic reliability is essential for the science and practice of psychology, in part because reliability is necessary for validity. Recently, the DSM-5 field trials documented lower diagnostic reliability than past field trials and the general research literature, resulting in substantial criticism of the DSM-5 diagnostic criteria. Rather than indicating specific problems with DSM-5, however, the field trials may have revealed long-standing diagnostic issues that have been hidden due to a reliance on audio/video recordings for estimating reliability. We estimated the reliability of DSM-IV diagnoses using both the standard audio-recording method and the test-retest method used in the DSM-5 field trials, in which different clinicians conduct separate interviews. Psychiatric patients (N = 339) were diagnosed using the SCID-I/P; 218 were diagnosed a second time by an independent interviewer. Diagnostic reliability using the audio-recording method (N = 49) was "good" to "excellent" (M κ = .80) and comparable to the DSM-IV field trials estimates. Reliability using the test-retest method (N = 218) was "poor" to "fair" (M κ = .47) and similar to DSM-5 field-trials' estimates. Despite low test-retest diagnostic reliability, self-reported symptoms were highly stable. Moreover, there was no association between change in self-report and change in diagnostic status. These results demonstrate the influence of method on estimates of diagnostic reliability.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The reliability results were similar to those achieved by diagnostic assessments using more complex and time-consuming structured instruments.
“The results were similar to those achieved by diagnostic assessments using more complex and time consuming structured instruments 26,39,40 .”
The DSM-5 design did not control for information variance and would almost certainly yield lower reliabilities.
“The DSM-5 design, therefore, did not control for information variance and so would almost certainly yield lower reliabilities 26,35 .”
The joint-rater concurrent reliability method, which gives both diagnosticians identical information, usually produces higher kappa values than separate interviews.
“First, it bears repeating that the joint-rater (concurrent) method of testing reliability, which constrains the information provided to the two diagnosticians to be identical, usually generates higher kappa values compared to those obtained when separate interviews are conducted 26,51 .”
Reliability is not solely a property of the classification but also a product of the method used to estimate it.
“That is, reliability is not solely a property of the classification, but also a product of the method used to estimate it.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Diagnostic and Statistical Manual of Mental Disorders
Williams, J. B. W., & First, M. B. (2013) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199975839.013.104 · Cited by 109,841 · see citing works
Diagnostic and Statistical Manual of Mental Disorders, 3rd ed
Cooper, A. M., & Michels, R. (1981) · American Journal of Psychiatry · doi:10.1176/ajp.138.1.128 · Cited by 7,417 · see citing works
Statistical Methods for Rates and Proportions, 2nd ed
GROVE, W. A. (1981) · American Journal of Psychiatry · doi:10.1176/ajp.138.12.1644-a · Cited by 3,756 · see citing works
The Structured Clinical Interview for DSM-III-R (SCID)
Williams, J. B. W. (1992) · Archives of General Psychiatry · doi:10.1001/archpsyc.1992.01820080038006 · Cited by 1,627 · cited 7× · see citing works
“Comparing identical diagnoses, reliability in the current study appears slightly lower (kappa = .61 vs. .66) than in Williams et al. (1992) and lower than in Di Nardo et al. (1993) and Brown et al. (2001) (kappa = .45 vs. .60 and .65, respectively).”
BackgroundMethodResultInfluentialRegier, D. A., Narrow, W. E., Clarke, D. E., Kraemer, H. C., Kuramoto, S. J., Kuhl, E. A., & Kupfer, D. J. (2013) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.12070999 · Cited by 1,163 · cited 8× · see citing works
“Others have suggested that (a) the lack of standardized interviews in the DSM-5 Field Trials (Regier et al., 2013) or (b) sample differences between the DSM-5 Field Trials (which used representative samples) and previous field trials (which did not) contributed to the lower reliabilities (Regier et…”
BackgroundMethodResultInfluentialBrown, T. A., Di Nardo, P. A., Lehman, C. L., & Campbell, L. A. (2001) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.110.1.49 · Cited by 888 · see citing works
The Collaborative Longitudinal Personality Disorders Study: Reliability of Axis I and II Diagnoses
Zanarini, M. C., Skodol, A. E., Bender, D., Dolan, R., Sanislow, C., Schaefer, E., Morey, L. C., Grilo, C. M., Shea, M. T., McGlashan, T. H., et al. (2000) · Journal of Personality Disorders · doi:10.1521/pedi.2000.14.4.291 · Cited by 771 · cited 1× · see citing works
“We also note that Zanarini et al. (2000) found a similar reduction in diagnostic reliability using a small number of audio-recordings (N = 27) and test-retests (N = 52) in a non-representative patient sample.”
ResultDevelopment and validation of the Inventory of Depression and Anxiety Symptoms (IDAS).
Watson, D., O'Hara, M. W., Simms, L. J., Kotov, R., Chmielewski, M., McDade-Montez, E. A., Gamez, W., & Stuart, S. (2007) · Psychological Assessment · doi:10.1037/1040-3590.19.3.253 · Cited by 740 · cited 2× · see citing works
“Self-Report Measures—Participants completed the Inventory of Depression and Anxiety Symptoms (IDAS; Watson et al., 2007).”
MethodSelf-citationDiagnosing Personality Disorders
Zimmerman, M. A. (1994) · Archives of General Psychiatry · doi:10.1001/archpsyc.1994.03950030061006 · Cited by 721 · cited 1× · see citing works
“As such, the audio/videorecording method, which constrains the information provided to the two diagnosticians to be identical, can be expected to generate higher kappa values compared to those obtained when separate interviews are conducted (Kraemer et al., 2012; Zimmerman, 1994).”
ResultMarkon, K. E., Chmielewski, M., & Miller, C. J. (2011) · Psychological Bulletin · doi:10.1037/a0023678 · Cited by 656 · see citing works
Self-citationA Re-analysis of the Reliability of Psychiatric Diagnosis
Spitzer, R. L., & Fleiss, J. L. (1974) · The British Journal of Psychiatry · doi:10.1192/bjp.125.4.341 · Cited by 612 · cited 3× · see citing works
“Prior to DSM-III (American Psychiatric Association, 1980), diagnostic reliability was poor, due in part to the lack of specific diagnostic criteria (Spitzer & Fleiss, 1974).”
BackgroundDi Nardo, P. A. (1993) · Archives of General Psychiatry · doi:10.1001/archpsyc.1993.01820160009001 · Cited by 498 · cited 2× · see citing works
“In addition, Brown et al. (2001) and Di Nardo et al. (1993) used several inclusion and exclusion criteria that may have reduced diagnostic “noise.””
MethodDSM-III field trials: I. Initial interrater diagnostic reliability
(1979) · American Journal of Psychiatry · doi:10.1176/ajp.136.6.815 · Cited by 394 · cited 17× · see citing works
“The mean kappa of .80, as well as those of the majority of diagnoses, would be considered “excellent” by traditional standards (Fleiss, 1981; Spitzer et al., 1979).”
BackgroundMethodResultInfluentialWatson, D., O'Hara, M. W., Chmielewski, M., McDade-Montez, E. A., Koffel, E., Naragon, K., & Stuart, S. (2008) · Psychological Assessment · doi:10.1037/a0012570 · Cited by 215 · cited 2× · see citing works
“The IDAS scales show strong psychometric properties compared to commonly used depression and anxiety measures (Watson et al., 2007, 2008).”
MethodSelf-citationDSM-5: How Reliable Is Reliable Enough?
KRAEMER, H. C., Kupfer, D. J., Clarke, D. E., Narrow, W. E., & Regier, D. A. (2012) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2011.11010050 · Cited by 180 · cited 6× · see citing works
“…to which a patient would receive the same diagnosis at different hospitals or clinics, or the extent to which different studies are recruiting similar patients, then the test-retest method provides a more meaningful estimate of diagnostic reliability (Kraemer et al., 2012; Williams et al., 1992).”
MethodResultInfluentialKlerman, G. L., Vaillant, G. E., Spitzer, R. L., & Michels, R. (1984) · American Journal of Psychiatry · doi:10.1176/ajp.141.4.539 · Cited by 153 · see citing works
Metaphorical analysis of psychiatric classification as a psychological test.
Blashfield, R. K., & Livesley, W. J. (1991) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.100.3.262 · Cited by 101 · cited 1× · see citing works
“As Blashfield & Livesley (1991) noted, high test-retest reliability over short timeframes is essential for diagnostic validity.”
BackgroundMcKenzie, D. P., Mackinnon, A. J., Péladeau, N., Onghena, P., Bruce, P. C., Clarke, D. M., Harrigan, S., & McGorry, P. D. (1996) · Journal of Psychiatric Research · doi:10.1016/s0022-3956(96)00033-7 · Cited by 91 · see citing works
Comparing the temporal stability of self-report and interview assessed personality disorder.
Samuel, D. B., Hopwood, C. J., Ansell, E. B., Morey, L. C., Sanislow, C. A., Markowitz, J. C., Yen, S., Shea, M. T., Skodol, A. E., & Grilo, C. M. (2011) · Journal of Abnormal Psychology · doi:10.1037/a0022647 · Cited by 58 · see citing works
DSM-IV: Empirical guidelines from psychometrics.
Nelson‐Gray, R. O. (1991) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.100.3.308 · Cited by 50 · cited 2× · see citing works
“It has long been argued that reliability sets an upper limit on validity (Nelson-Gray, 1991; Spitzer & Fleiss, 1974); however, the J Abnorm Psychol.”
BackgroundStandards for DSM-5 Reliability
Spitzer, R. L., Williams, J. B. W., & Endicott, J. (2012) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.12010083 · Cited by 31 · cited 1× · see citing works
“Others have been far more critical (Frances, 2012; Spitzer, Williams, & Endicott, 2012), arguing that the manual “flunked its reliability tests” (Frances, 2012) and that traditional kappa guidelines should be applied (Frances, 2012; Spitzer et al., 2012).”
Background
Flagged first, then by citations
Research works printed but not linked: 2
- First M Spitzer RL Gibbon M Williams JBW 2002 Structured clinical interview for DSM-IV-TR axis I disorders, research version, patient edition (SCID-I/P) New York Biometrics Research, New York State Psychiatric Institute
- Fleiss Joseph L 1981 Statistical methods for rates and proportions 2 Hoboken, New Jersey Wiley
Cited by
Sorted by citations
Kotov, R., Krueger, R. F., Watson, D., Achenbach, T. M., Althoff, R. R., Bagby, R. M., Brown, T. A., Carpenter, W. T., Caspi, A., Clark, L. A., et al. (2017) · Journal of Abnormal Psychology · doi:10.1037/abn0000258 · Cited by 3,335 · see citing works
“Second, traditional diagnoses generally show limited reliability, as can be expected when arbitrary categories are forced onto dimensional phenomena (Chmielewski, Clark, Bagby, & Watson, 2015; Markon, 2013).”
Self-citationAn MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027 · see citing works
“…should also report the psychometric properties of the measures in the studied sample and compare them to previous research, when available, as this provides valuable information about the performance of the measure and the quality of data itself (Chmielewski, Clark, Bagby, & Watson, 2015).”
Self-citationIntegrating the Hierarchical Taxonomy of Psychopathology (HiTOP) into clinical practice.
Ruggero, C. J., Kotov, R., Hopwood, C. J., First, M., Clark, L. A., Skodol, A. E., Mullins-Sweatt, S. N., Patrick, C. J., Bach, B., Cicero, D. C., et al. (2019) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000452 · Cited by 333 · see citing works
Self-citationValidity and utility of Hierarchical Taxonomy of Psychopathology (HiTOP): I. Psychosis superspectrum
Kotov, R., Jonas, K. G., Carpenter, W. T., Dretsch, M. N., Eaton, N. R., Forbes, M. K., Forbush, K. T., Hobbs, K. A., Reininghaus, U., Slade, T. N., et al. (2020) · World Psychiatry · doi:10.1002/wps.20730 · Cited by 310 · see citing works
Self-citationWatson, D., Levin‐Aspenson, H. F., Waszczuk, M. A., Conway, C. C., Dalgleish, T., Dretsch, M. N., Eaton, N. R., Forbes, M. K., Forbush, K. T., Hobbs, K. A., et al. (2022) · World Psychiatry · doi:10.1002/wps.20943 · Cited by 285 · see citing works
Self-citationKrueger, R. F., Hobbs, K. A., Conway, C., Dick, D. M., Dretsch, M. N., Eaton, N. R., Forbes, M. K., Forbush, K. T., Keyes, K. M., Latzman, R. D., et al. (2021) · World Psychiatry · doi:10.1002/wps.20844 · Cited by 263 · see citing works
Self-citationKhazanov, G. K., & Ruscio, A. M. (2016) · Psychological Bulletin · doi:10.1037/bul0000059 · Cited by 240 · see citing works
“Furthermore, assessments of symptoms are usually more reliable than dichotomous, disorder-based measures (Chmielewski et al., 2015; Naragon-Gainey et al., 2009).”
BackgroundAutomatic Assessment of Depression Based on Visual Cues: A Systematic Review
Pampouchidou, A., Simos, P. G., Kostas Marias, A., Mériaudeau, F., Yang, F., Pediaditis, M., & Tsiknakis, M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2724035 · Cited by 224 · see citing works
“regarding their reliability [15], [16], as diagnosis of MDD is not as consistent as other common medical conditions [17].”
BackgroundPersonality in a Hierarchical Model of Psychopathology
Widiger, T. A., Sellbom, M., Chmielewski, M. S., Clark, L. A., DeYoung, C. G., Kotov, R., Krueger, R. F., Lynam, D. R., Miller, J. D., Mullins‐Sweatt, S. N., et al. (2018) · Clinical Psychological Science · doi:10.1177/2167702618797105 · Cited by 228 · see citing works
Self-citationBucher, M. A., Suzuki, T., & Samuel, D. B. (2019) · Clinical Psychology Review · doi:10.1016/j.cpr.2019.04.002 · Cited by 203 · see citing works
Digital biomarkers of mood disorders and symptom change
Jacobson, N. C., Weingarden, H., & Wilhelm, S. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0078-0 · Cited by 160 · see citing works
“The current approach substantially improved upon earlier results,4 and it classified diagnostic status more accurately than published inter-reliability rates for second raters using the SCID-I/P.10 Results suggest that unobtrusive, passive actigraphy data hold potential to supplement traditional diagnostic assessments…”
ResultReed, G. M., Sharan, P., Rebello, T. J., Keeley, J. W., Elena Medina‐Mora, M., Gureje, O., Luis Ayuso‐Mateos, J., Kanba, S., Khoury, B., Kogan, C. S., et al. (2018) · World Psychiatry · doi:10.1002/wps.20524 · Cited by 126 · see citing works
Waszczuk, M. A., Eaton, N. R., Krueger, R. F., Shackman, A. J., Waldman, I. D., Zald, D. H., Lahey, B. B., Patrick, C. J., Conway, C., Ormel, J., et al. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000486 · Cited by 120 · see citing works
Self-citationHengartner, M. P., & Lehmann, S. N. (2017) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2017.00101 · Cited by 115 · see citing works
Clinical Utility of the Inventory of Depression and Anxiety Symptoms (IDAS)
Stasik-O’Brien, S. M., Brock, R. L., Chmielewski, M., Naragon-Gainey, K., Koffel, E., McDade-Montez, E., O’Hara, M. W., & Watson, D. (2019) · Assessment · doi:10.1177/1073191118790036 · Cited by 106 · see citing works
Self-citationCanalization and plasticity in psychopathology
Carhart-Harris, R. L., Chandaria, S., Erritzøe, D., Gazzaley, A. H., Girn, M., Kettner, H., Mediano, P. A. M., Nutt, D. J., Rosas, F. E., Roseman, L., et al. (2022) · Neuropharmacology · doi:10.1016/j.neuropharm.2022.109398 · Cited by 103 · see citing works
“…given a fuller understanding and appreciation of the adverse contextual conditions out of which the pathology has arisen (Conway et al., 2018; Cleck and Blendy, 2008; Dias-Ferreira et al., 2009; Felitti et al., 2019; Edmiston et al., 2011) – see also (Olthof et al., 2020) and (Sroufe, 1997).”
BackgroundChristensen, T. B., Eikenaes, I., Hummelen, B., Pedersen, G., Nysæter, T., Bender, D., Skodol, A., & Selvik, S. G. (2020) · Personality disorders · doi:10.1037/per0000352 · Cited by 91 · see citing works
Simms, L. J., Wright, A. G. C., Cicero, D. C., Kotov, R., Mullins‐Sweatt, S. N., Sellbom, M., Watson, D., Widiger, T. A., & Zimmermann, J. (2021) · Assessment · doi:10.1177/10731911211015309 · Cited by 99 · see citing works
“…different ways (e.g., Galatzer-Levy & Bryant, 2013; Wright et al., 2013), (c) poor diagnostic reliability for numerous diagnostic categories (e.g., Chmielewski et al., 2015; Regier et al., 2013); (d) arbitrary diagnostic thresholds that erroneously imply that psychopathology varies in kind rather…”
BackgroundSelf-citationHierarchical structure of emotional disorders: From individual symptoms to the spectrum.
Waszczuk, M. A., Kotov, R., Ruggero, C. J., Gámez, W., & Watson, D. (2017) · Journal of Abnormal Psychology · doi:10.1037/abn0000264 · Cited by 97 · see citing works
Self-citationStanton, K. J., McDonnell, C. G., Hayden, E. P., & Watson, D. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000464 · Cited by 91 · see citing works
Self-citationKaralunas, S. L., & Nigg, J. T. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2019.11.002 · Cited by 84 · see citing works
“With regard to comorbidity, the associated disorders are themselves subject to arbitrary cutoffs, potentially adding to heterogeneity and decreasing diagnostic reliability (23, 24).”
BackgroundMuzammel, M., Salam, H., & Othmani, A. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106433 · Cited by 88 · see citing works
Graph Convolutional Networks Reveal Network-Level Functional Dysconnectivity in Schizophrenia
Lei, D., Qin, K., Pinaya, W. H. L., Young, J., van Amelsvoort, T., Marcelis, M. C., Donohoe, G. J., Mothersill, D., Corvin, A. P., Vieira, S., et al. (2022) · Schizophrenia Bulletin · doi:10.1093/schbul/sbac047 · Cited by 80 · see citing works
Classifying Major Depressive Disorder Using fNIRS During Motor Rehabilitation
Zhu, Y., Jayagopal, J. K., Mehta, R. K., Erraguntla, M., Nuamah, J. K., McDonald, A. D., Taylor, H. B., & Chang, S. (2020) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2020.2972270 · Cited by 70 · see citing works
Watson, D., Stanton, K. J., & Clark, L. A. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.09.065 · Cited by 71 · see citing works
Self-citationMohammadkhani, P., Forouzan, A. S., Hooshyari, Z., & Abasi, I. (2020) · Iranian Journal of Psychiatry and Behavioral Sciences · doi:10.5812/ijpbs.100930 · Cited by 67 · see citing works
“Diagnostic reliability and validity have an important role in the progress of science and the practice of clinical psychology and psychiatry, without which the accurate identification of risk factors, interpretation of psychopathology, and efficacy of treatment would be erroneous (21, 22).”
BackgroundKemp, K. C., Bathery, A. J., Barrantes‐Vidal, N., & Kwapil, T. R. (2020) · Assessment · doi:10.1177/1073191119900008 · Cited by 54 · see citing works
Mullins‐Sweatt, S. N., Hopwood, C. J., Chmielewski, M. S., Meyer, N. A., Min, J., Helle, A. C., & Walgren, M. D. (2019) · Personality and Mental Health · doi:10.1002/pmh.1464 · Cited by 53 · see citing works
Self-citationScarpazza, C., Ha, M., Baecker, L., Garcia‐Dias, R., Pinaya, W. H. L., Vieira, S., & Mechelli, A. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-0798-6 · Cited by 51 · see citing works
Hagerty, S. L. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916221079597 · Cited by 50 · see citing works
“Furthermore, diagnoses are less reliable when arbitrary categories are imposed onto dimensional phenomenon (Chmielewski et al., 2015; Markon, 2013).”
BackgroundPrecision Medicine Approaches to Mental Health Care
Scala, J. J., Ganz, A. B., & Snyder, M. P. (2022) · Physiology · doi:10.1152/physiol.00013.2022 · Cited by 47 · see citing works
Christensen, T. B., Paap, M. C. S., Arnesen, M., Koritzinsky, K., Nysæter, T. E., Eikenæs, I. U.-M., Selvik, S. G., Walther, K., Torgersen, S. O., Bender, D. S., et al. (2018) · Journal of Personality Assessment · doi:10.1080/00223891.2018.1483377 · Cited by 46 · see citing works
“For the test–retest IRR study, we report the criteria used to examine the reliability in DSM–5 field trials based on a test–retest design with two different raters (Chmielewski et al., 2015; Clarke et al., 2013; Kraemer, Kupfer, Narrow, Clarke, & Regier, 2010).”
MethodPerkins, E. R., Joyner, K. J., Patrick, C. J., Bartholow, B. D., Latzman, R. D., DeYoung, C. G., Kotov, R., Reininghaus, U., Cooper, S. E., Afzali, M. H., et al. (2020) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2020.22.1/eperkins · Cited by 47 · see citing works
Prioritization of Psychopathological Symptoms and Clinical Characterization in Psychiatric Diagnoses
Leucht, S., van Os, J., Jäger, M., & Davis, J. M. (2024) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2024.2652 · Cited by 48 · see citing works
Philippot, P., Bouvard, M., Baeyens, C., & Dethier, V. (2019) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2340 · Cited by 40 · see citing works
Pampouchidou, A., Pediaditis, M., Kazantzaki, E., Sfakianakis, S. G., Apostolaki, I., Argyraki, K., Manousos, D., Mériaudeau, F., Kostas Marias, A., Yang, F., et al. (2020) · Machine Vision and Applications · doi:10.1007/s00138-020-01080-7 · Cited by 38 · see citing works
Spatial–Temporal EEG Fusion Based on Neural Network for Major Depressive Disorder Detection
Zhang, B., Wei, D., Yan, G., Li, X., Su, Y., & Cai, H. (2023) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-023-00567-x · Cited by 35 · see citing works
“But traditional interview diagnostic methods using scales are time-consuming and laborious [6–8], as well as are affected by the diagnosis level of clinicians, thus traditional diagnostic methods have strong subjectivity [9].”
MethodFractures in the framework: limitations of classification systems inpsychiatry
Kapadia, M., Desai, M., & Parikh, R. M. (2020) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2020.22.1/rparikh · Cited by 34 · see citing works
Conway, C., Mansolf, M. A., & Reise, S. P. (2019) · Psychological Assessment · doi:10.1037/pas0000695 · Cited by 33 · see citing works
Christie, L., Inman, J., Davys, D., & Cook, P. A. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.12.080 · Cited by 32 · see citing works
Psychometrics and Psychological Assessment
Hunsley, J. D., & Allan, T. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108235433.002 · Cited by 33 · see citing works
Bartlett, E. A., Klein, D. N., Li, K., DeLorenzo, C., Kotov, R., & Perlman, G. (2019) · Biological Psychiatry · doi:10.1016/j.biopsych.2019.04.027 · Cited by 29 · see citing works
Establishing the accuracy of self-diagnosis in psychiatry
Fellowes, S. (2025) · Philosophical Psychology · doi:10.1080/09515089.2024.2327823 · Cited by 29 · see citing works
“…the reliability (in the technical sense described above) of psychiatric diagnoses have been going on for many decades (Kendell, 1975, see Aragona, 2015 and; Solomon, 2022 for history) and there is disagreement about how high the reliability of the DSM is (Chmielewski et al., 2015; Vanheule, 2017).”
Borderline personality disorder diagnosis in a new key
Mulay, A. L., Waugh, M. H., Fillauer, J. P., Bender, D. S., Bram, A. D., Cain, N. M., Caligor, E., Forbes, M. K., Goodrich, L. B., Kamphuis, J. H., et al. (2019) · Borderline Personality Disorder and Emotion Dysregulation · doi:10.1186/s40479-019-0116-1 · Cited by 30 · see citing works
Development of the Thought Disorder Measure for the Hierarchical Taxonomy of Psychopathology
Cicero, D. C., Jonas, K. G., Chmielewski, M. S., Martin, E. A., Docherty, A. R., Berzon, J., Haltigan, J. D., Reininghaus, U., Caspi, A., Graziolplene, R. G., et al. (2021) · Assessment · doi:10.1177/10731911211015355 · Cited by 27 · see citing works
“The HiTOP system conceptualizes psychopathology dimensionally and hierarchically, which addresses several common problems with psychiatric taxonomy, including excessive comorbidity, arbitrary cut points between disorder and normality, heterogeneity of disorders, unreliable diagnoses, and the existence of subthreshold…”
BackgroundSelf-citationSemcho, S. A., Southward, M. W., Stumpp, N. E., MacLean, D. L., Hood, C. O., Wolitzky-Taylor, K., & Sauer-Zavala, S. (2023) · Journal of Emotion and Psychopathology · doi:10.55913/joep.v1i1.9 · Cited by 26 · see citing works
A new conception and subsequent taxonomy of clinical psychological problems
Bakker, G. (2019) · BMC Psychology · doi:10.1186/s40359-019-0318-8 · Cited by 27 · see citing works
Somma, A., Borroni, S., Gialdi, G., Carlotta, D., Giarolli, L., Barranca, M., Cerioli, C., Franzoni, C., Masci, E., Manini, R., et al. (2020) · Journal of Personality Disorders · doi:10.1521/pedi_2020_34_511 · Cited by 26 · see citing works
Irak, M., & Albayrak, E. O. (2018) · Psychological Reports · doi:10.1177/0033294118813844 · Cited by 23 · see citing works
Hasratian, A. M., Meuret, A. E., Chmielewski, M. S., & Ritz, T. (2022) · Behavior Therapy · doi:10.1016/j.beth.2022.04.009 · Cited by 16 · see citing works
Self-citationLin, H., Fang, J. S., Zhang, J., Zhang, X., Piao, W., & Liu, Y. (2024) · Sensors · doi:10.3390/s24216815 · Cited by 20 · see citing works
Few-Electrode EEG from the Wearable Devices Using Domain Adaptation for Depression Detection
Wu, W., Ma, L., Lian, B., Cai, W., & Zhao, X. (2022) · Biosensors · doi:10.3390/bios12121087 · Cited by 18 · see citing works
“The current mainstream diagnostic approach is based on the diagnostic and statistical manual of mental disorders (DSM-IV) [8] and other similar approaches to psychiatric rating scales [9].”
MethodAn evolutionary life history approach to understanding mental health
Han, W., & Chen, B. (2020) · General Psychiatry · doi:10.1136/gpsych-2019-100113 · Cited by 18 · see citing works
Weekers, L. C., Hutsebaut, J., Zimmermann, J., & Kamphuis, J. H. (2021) · Personality Disorders Theory Research and Treatment · doi:10.1037/per0000512 · Cited by 18 · see citing works
“Higher convergence is to be expected when the same clinical information is presented in a standardized way, because information variance in how patients express their characteristics is again eliminated (Chmielewski et al., 2015; Kraemer, 2014).”
A multidisciplinary approach to insanity assessment as a way to reduce cognitive biases
Scarpazza, C., Zampieri, I., Miolla, A., Melis, G., Pietrini, P., & Sartori, G. (2020) · Forensic Science International · doi:10.1016/j.forsciint.2020.110652 · Cited by 16 · see citing works
Sun, S., Liu, L., Shao, X., Yan, C., Li, X., & Hu, B. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3181690 · Cited by 16 · see citing works
“The most widely used diagnostic criteria for depression are based on the Diagnostic and Statistical Manual of Mental Disorders-IV (DSM-IV) [6] and International Classification of Diseases-10 (ICD-10) [7].”
InfluentialDevelopment and validation of the Inventory of Depression and Anxiety Symptoms–II—German version.
Wester, R. A., Rubel, J. A., Zimmermann, J., Hall, M., Kaven, L., & Watson, D. (2022) · Psychological Assessment · doi:10.1037/pas0001185 · Cited by 16 · see citing works
Self-citationOkazaki, K., Miura, K., Matsumoto, J., Hasegawa, N., Fujimoto, M., Yamamori, H., Yasuda, Y., Makinodan, M., & Hashimoto, R. (2023) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.13553 · Cited by 17 · see citing works
Kogan, C. S., & Paterniti, S. (2017) · The Canadian Journal of Psychiatry · doi:10.1177/0706743717717253 · Cited by 17 · see citing works
“For example, Chmielewski et al.(43) showed that reliability for DSM-5 categories is significantly better when employing an interrater methodology using semi-structured interviews (e.”
MethodInfluentialSeery, C., Bramham, J., & O’Connor, C. (2021) · Psychosis · doi:10.1080/17522439.2021.1901302 · Cited by 15 · see citing works
“, 2018) and poor reliability between clinicians (Chmielewski et al., 2015).”
BackgroundLee, K., Kwon, J., Chun, M., Choi, J., Lee, S., & Im, C. (2024) · Depression and Anxiety · doi:10.1155/2024/4459867 · Cited by 14 · see citing works
“However, the current conventional MDD diagnosis methods based on the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV or DSM-5) [7, 8] sometimes lead to misdiagnosis because these face-to-face interviews rely solely on the subjective responses of patients for diagnosis [9, 10].”
Schizophrenia: a classic battle ground of nature versus nurture debate
Clair, D. S., & Lang, B. (2021) · 中国科学通报：英文版 · doi:10.1016/j.scib.2021.01.032 · Cited by 15 · see citing works
Clinical validation of brief mental health scales for use in South African occupational healthcare
Wijk, C. H. V., Martin, J. H., & Maree, D. J. F. (2021) · SA Journal of Industrial Psychology · doi:10.4102/sajip.v47i0.1895 · Cited by 13 · see citing works
Heller, A. S., Stamatis, C. A., Puccetti, N. A., & Timpano, K. R. (2021) · Journal of Abnormal Psychology · doi:10.1037/abn0000670 · Cited by 14 · see citing works
Traynor, J. M., Wrege, J. S., Walter, M., & Ruocco, A. C. (2021) · Psychological Medicine · doi:10.1017/s0033291721002865 · Cited by 11 · see citing works
Progress, Possibilities, and Pitfalls in Electroretinography Research in Psychiatry
Silverstein, S. M., & Thompson, J. L. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2019.10.028 · Cited by 10 · see citing works
Scarpazza, C., Miolla, A., Zampieri, I., Melis, G., Sartori, G., Ferracuti, S., & Pietrini, P. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.597918 · Cited by 10 · see citing works
Ma, C., Wang, Z., Li, C., Lu, J., Long, J., Li, R., Wu, Q., Jiang, H., Du, J., Li, R., et al. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.781992 · Cited by 10 · see citing works
“Ko et al. also supported the utility of ICD11 and DSM-5 in identifying individuals who need treatment for problematic gaming behaviors (24).”
MethodInfluentialValidating Biobehavioral Technologies for Use in Clinical Psychiatry
Cohen, A. S., Cox, C. R., Tucker, R. P., Mitchell, K. R., Schwartz, E. K., Le, T. P., Foltz, P. W., Holmlund, T. B., & Elvevåg, B. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.503323 · Cited by 10 · see citing works
Gharraee, B., Shabani, A., Masoumian, S., Zamirinejad, S., Yaghmaeezadeh, H., Khanjani, S., & Ghahremani, S. (2022) · East Asian Archives of Psychiatry · doi:10.12809/eaap2208 · Cited by 10 · see citing works
Tunnell, N. C., Suterwala, M., & Meuret, A. E. (2019) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2019.10.004 · Cited by 9 · see citing works
Gharraee, B., Shabani, A., Masoumian, S., Zamirinejad, S., Yaghmaeezadeh, H., Khanjani, S., & Ghahremani, S. (2021) · doi:10.21203/rs.3.rs-670472/v1 · Cited by 9 · see citing works
Sun, X., So, S. H., Chung, L. K.-H., Chiu, C., Chan, R. C. K., & Leung, P. W. L. (2022) · Schizophrenia Research · doi:10.1016/j.schres.2022.01.005 · Cited by 9 · see citing works
A Systematic Review of Sleep–Wake Disorder Diagnostic Criteria Reliability Studies
Gauld, C., Lopez, R., Philip, P., Taillard, J., Morin, C. M., Geoffroy, P. A., & Micoulaud‐Franchi, J. (2022) · Biomedicines · doi:10.3390/biomedicines10071616 · Cited by 9 · see citing works
“The aim was to identify a typology of methodologies of reliability studies used in sleep medicine, inspired by what has been reported in psychiatry in field trials of DSM reliability studies [6,7,17].”
MethodInfluentialOnofa, L. E. U., Odunleye, M., Kola, L., & Gureje, O. (2019) · Archives of Medical Research · doi:10.1016/j.arcmed.2020.01.004 · Cited by 8 · see citing works
Does the millennial generation of women experience more mental illness than their mothers?
Najman, J. M., Bor, W., Williams, G. M., Middeldorp, C. M., Mamun, A. A., Clavarino, A. M., & Scott, J. G. (2021) · BMC Psychiatry · doi:10.1186/s12888-021-03361-5 · Cited by 8 · see citing works
Diagnostic Systems and Models: DSM-5 Criteria, ICD-11 Guidelines, and Dimensional Horizons
Bach, B., Bo, S., & Keeley, J. W. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-818697-8.00111-4 · Cited by 7 · see citing works
Somma, A., Krueger, R. F., Markon, K. E., Alajmo, V. B. M., Arlotta, E., Beretta, S., Boni, F., Busso, S. L., Manini, R., Nazzaro, G., et al. (2019) · Journal of Personality Disorders · doi:10.1521/pedi_2019_33_430 · Cited by 7 · see citing works
Cervin, M., Pozza, A., Barcaccia, B., & Dèttore, D. (2020) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2020.102300 · Cited by 7 · see citing works
Somma, A., Krueger, R., Markon, K., Gialdi, G., Frau, C., & Fossati, A. (2022) · Personality and Mental Health · doi:10.1002/pmh.1556 · Cited by 6 · see citing works
Coding social interactions in naturalistic settings: The taxonomy of dyadic conversation
Mulwa, K. W., & Kucker, S. C. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-02033-w · Cited by 5 · see citing works
“However, consistent with standards for assessment validity in the field, validity and reliability ought to be confirmed for each unique study (see also Chmielewski et al., 2015).”
Soodla, H. L., & Akkermann, K. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1043394 · Cited by 5 · see citing works
“org hinderances in treatment planning (Chmielewski et al., 2015; Hopwood et al., 2020).”
Pérez, G. R., Stasik-O’Brien, S. M., Laifer, L. M., & Brock, R. L. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19031662 · Cited by 6 · see citing works
“Examining the symptoms of a disorder may be more reliable than depending on diagnostic categories and can yield incremental information [38].”
BackgroundTorres-Rosado, L., Lozano, O. M., Sánchez‐García, M., Fernández‐Calderón, F., & Díaz, C. (2023) · World Journal of Psychiatry · doi:10.5498/wjp.v13.i6.278 · Cited by 6 · see citing works
Romanian Inventory of Depression and Anxiety Symptoms (IDAS-II)
Petre, L. M., Gheorghe, D. A., Watson, D., & Mitrofan, L. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1159380 · Cited by 5 · see citing works
“org problematic reliability and validity of the Diagnostic and Statistical Manual of Mental Disorders (DSM; American Psychiatric Association, 1994) diagnoses (Chmielewski et al., 2015).”
BackgroundSelf-citationFunctional Connectivity Biomarkers in Schizophrenia
Howell, A. M., & Anticevic, A. (2024) · Advances in neurobiology · doi:10.1007/978-3-031-69491-2_10 · Cited by 5 · see citing works
Botha, D. (2020) · Psychological Thought · doi:10.37708/psyct.v13i2.471 · Cited by 5 · see citing works
“This drew considerable adverse comments in regard to the reliability of DSM-5 diagnostic criteria (Chmielewski et al., 2015).”
BackgroundFournier, J. C. J., Roberts, N., & Ford, K. L. (2020) · Journal of Personality · doi:10.1111/jopy.12605 · Cited by 5 · see citing works
Heinzmann, S., Etzler, S., Hartmann, A., Klein, E. M., Herpertz, S., Pape, M., Heinzmann, S., Doering, S., Hofmann, T., Rose, M., et al. (2025) · European Eating Disorders Review · doi:10.1002/erv.3183 · Cited by 4 · see citing works
“In sum, the study underlines the importance of a dimensional assessment of personality impairment, which was found to be associated with greater reliability and diagnostic stability than its categorial counterpart (Chmielewski et al. 2015).”
Safeguarding the Quality of Forensic Assessment in Sentencing
der Wolf, M. V. (2022) · Routledge eBooks · doi:10.4324/9781351266482 · Cited by 4 · see citing works
Mihura, J. L., Boyette, L., Görner, K. J., Kleiger, J. H., Jowers, C. E., & Ales, F. (2024) · Schizophrenia Research · doi:10.1016/j.schres.2024.06.051 · Cited by 4 · see citing works
Keel, P. K., Bodell, L. P., Ali, S. I., Starkey, A. G., Trotta, J., Luxama, J. W., Halfhide, C., Hill, N. G., Appelbaum, J., & Williams, D. L. (2025) · JMIR Research Protocols · doi:10.2196/66554 · Cited by 4 · see citing works
“95 [94].”
Pham, H. T., Berenbaum, S. A., DiLalla, L. F., Corley, R. P., Wadsworth, S. J., Reynolds, C. A., & Beltz, A. M. (2025) · Psychological Medicine · doi:10.1017/s0033291725000820 · Cited by 4 · see citing works
“This also means that our results are not likely to change with any changes in diagnostic criteria (Chmielewski, Clark, Bagby, & Watson, 2015).”
Levy, K. N., & Ellison, W. D. (2021) · Training and Education in Professional Psychology · doi:10.1037/tep0000376 · Cited by 4 · see citing works
Radetzki, P. A., Wrath, A. J., McWilliams, L. A., Olson, T. R., Adams, S., Souza, D. D., Lau, B., & Adams, G. C. (2022) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0000000000001569 · Cited by 5 · see citing works
Integrating a Dimensional, Hierarchical Taxonomy of Psychopathology into Clinical Practice_Preprint
Ruggero, C. J. (2018) · doi:10.31234/osf.io/r2jt6 · Cited by 4 · see citing works
Schizophrenia recognition via eye movement features in video paradigm
Li, S., Liu, Z., Zhang, D. H., Song, Y., Xu, L., Zhang, T., & Wang, J. (2025) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-025-02112-x · Cited by 3 · see citing works
Reed, G. M. (2023) · Cambridge University Press eBooks · doi:10.1017/9781009182232.002 · Cited by 3 · see citing works
Razali, H. Y. H., & Yusof, A. N. M. (2024) · Journal of Medical Ethics · doi:10.1136/jme-2024-110086 · Cited by 3 · see citing works
Dynamics of Decision-Making: The Issue of Reliability in Diagnosis
Vanheule, S. (2017) · Springer eBooks · doi:10.1007/978-3-319-44669-1_2 · Cited by 3 · see citing works
- Authors
- Chmielewski, M., Clark, L. A., Bagby, R. M., & Watson, D.
- Affiliations
- University of Notre Dame (US)
- University of Toronto (CA)
- Venue
- Journal of Abnormal Psychology
- Published
- 2015-08
- Volume
- 124
- Issue
- 3
- Pages
- 764-769
- ISSN
- 0021-843X (ISSN-L)
- 1939-1846
- PMC
- PMC4573819
- OpenAlex
- W2184853928
- Cited by
- 161
- References
- 21
- Citations by year
- 2026: 10 · 2025: 17 · 2024: 16 · 2023: 15 · 2022: 21 · 2021: 21 · 2020: 23 · 2019: 20 · 2018: 10 · 2017: 6 · 2016: 2
- FWCI
- 5.47
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Journal of Abnormal Psychology (publishedVersion, other-oa)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (R01-MH068472)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Arts and Humanities › Philosophy › Mental Health and Psychiatry
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Mental Health Research Topics
- Keywords
- diagnostic reliability, DSM-IV, DSM-5, test-retest reliability, inter-rater reliability, psychiatric diagnosis, self-reported symptoms
- MeSH terms
- Adolescent, Adult, Aged, Aged, 80 and over, Mental Disorders, Diagnosis, Differential, Female, Humans, Male, Middle Aged, Reproducibility of Results, Diagnostic and Statistical Manual of Mental Disorders, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
