CoChat
Sign Up
Cited by 126Reliability and Agreement in MeasurementMental Health and PsychiatryMental Health Treatment and Access

The ICD‐11 developmental field study of reliability of diagnoses of high‐burden mental disorders: results among adult patients in mental health settings of 13 countries

Open access
Reed, Geoffrey M., Sharan, Pratap, Rebello, Tahilia J., Keeley, Jared W., Elena Medina‐Mora, María, Gureje, Oye, Luis Ayuso‐Mateos, José, Kanba, Shigenobu +37 more·2018·World Psychiatry
DOI 10.1002/wps.20524PMC PMC5980511
Open this paper in CoChat — free accountOpen PDF

Abstract

Reliable, clinically useful, and globally applicable diagnostic classification of mental disorders is an essential foundation for global mental health. The World Health Organization (WHO) is nearing completion of the 11th revision of the International Classification of Diseases and Related Health Problems (ICD‐11). The present study assessed inter‐diagnostician reliability of mental disorders accounting for the greatest proportion of global disease burden and the highest levels of service utilization – schizophrenia and other primary psychotic disorders, mood disorders, anxiety and fear‐related disorders, and disorders specifically associated with stress – among adult patients presenting for treatment at 28 participating centers in 13 countries. A concurrent joint‐rater design was used, focusing specifically on whether two clinicians, relying on the same clinical information, agreed on the diagnosis when separately applying the ICD‐11 diagnostic guidelines. A total of 1,806 patients were assessed by 339 clinicians in the local language. Intraclass kappa coefficients for diagnoses weighted by site and study prevalence ranged from 0.45 (dysthymic disorder) to 0.88 (social anxiety disorder) and would be considered moderate to almost perfect for all diagnoses. Overall, the reliability of the ICD‐11 diagnostic guidelines was superior to that previously reported for equivalent ICD‐10 guidelines. These data provide support for the suitability of the ICD‐11 diagnostic guidelines for implementation at a global level. The findings will inform further revision of the ICD‐11 diagnostic guidelines prior to their publication and the development of programs to support professional training and implementation of the ICD‐11 by WHO member states.

Integrity signals

Registry

Crossref · 10.1002/wps.20524

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 41 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    Point estimates of kappa were generally lower for disorders with smaller samples.

    “In general, point estimates of kappa were lower for disorders for which smaller samples were obtained.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The lower bound of the confidence interval for kappa was only in the fair range for acute and transient psychotic disorder (0.27), dysthymic disorder (0.28), and post-traumatic stress disorder (0.33).

    “However, the lower bound estimates were only in the fair range (from 0.2 to 0.4) for acute and transient psychotic disorder (0.27), dysthymic disorder (0.28), and posttraumatic stress disorder (0.33).”
    Quote in the paperSubstantial claim
  • Findingreasoning

    The differences in kappa values between ICD-11 and ICD-10 comparisons tended to be modest.

    “These differences tended to be modest.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The lower bound of the confidence interval for kappa was above 0.4 for 13 of the 16 disorders.

    “The lower bound estimates of the confidence interval for kappa were higher than 0.4 (fair reliability) for 13 of the 16 disorders.”
    Quote in the paperSubstantial claim
  • Findingown data

    In 10 of the 14 possible comparisons between the ICD-11 CDDG EIFS and the ICD-10 CDDG field study, kappa values were higher for ICD-11.

    “in 10 of the 14 possible comparisons between the ICD- 11 CDDG EIFS and the ICD-10 CDDG field study, the kappa values were higher for ICD-11.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 43
  • Classified as probably not a substantial claim. 31

  • May combine two claims.

    Findingown data

    Across the study, intraclass kappa coefficients weighted by site and study prevalence ranged from 0.45 for dysthymic disorder to 0.88 for social anxiety disorder, and all calculated diagnoses fell in the moderate to almost perfect range.

    “Intraclass kappa coefficients for diagnoses weighted by site and study prevalence ranged from 0.45 (dysthymic disorder) to 0.88 (social anxiety disorder) and would be considered moderate to almost perfect for all diagnoses.”
    • Also stated in: § RESULTS · hedged
      “The point estimate of kappa ranged from 0.45 (dysthymic disorder) to 0.88 (social anxiety disorder) and would be considered moderate to almost perfect according to Landis and Koch adjectives for all diagnoses for which it was calculated.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for bipolar I disorder was 0.84, which is almost perfect.

    “The kappa estimates were almost perfect for schizophrenia (0.87) and bipolar I disorder (0.84)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for adjustment disorder was 0.73, which is substantial.

    “substantial for schizoaffective disorder (0.66), delusional disorder (0.69), bipolar II disorder (0.62), single episode depressive disorder (0.64), recurrent depressive disorder (0.74), generalized anxiety disorder (0.62), agoraphobia (0.62), and adjustment disorder (0.73)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for panic disorder was 0.57, which is moderate.

    “moderate for acute and transient psychotic disorder (0.45), dysthymic disorder (0.45), panic disorder (0.57), post-traumatic stress disorder (0.49), and the newly introduced diagnosis of complex post-traumatic stress disorder (0.56)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for recurrent depressive disorder was 0.74, which is substantial.

    “substantial for schizoaffective disorder (0.66), delusional disorder (0.69), bipolar II disorder (0.62), single episode depressive disorder (0.64), recurrent depressive disorder (0.74), generalized anxiety disorder (0.62), agoraphobia (0.62), and adjustment disorder (0.73)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for post-traumatic stress disorder was 0.49, which is moderate.

    “moderate for acute and transient psychotic disorder (0.45), dysthymic disorder (0.45), panic disorder (0.57), post-traumatic stress disorder (0.49), and the newly introduced diagnosis of complex post-traumatic stress disorder (0.56)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for schizoaffective disorder was 0.66, which is substantial.

    “substantial for schizoaffective disorder (0.66), delusional disorder (0.69), bipolar II disorder (0.62), single episode depressive disorder (0.64), recurrent depressive disorder (0.74), generalized anxiety disorder (0.62), agoraphobia (0.62), and adjustment disorder (0.73)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for single episode depressive disorder was 0.64, which is substantial.

    “substantial for schizoaffective disorder (0.66), delusional disorder (0.69), bipolar II disorder (0.62), single episode depressive disorder (0.64), recurrent depressive disorder (0.74), generalized anxiety disorder (0.62), agoraphobia (0.62), and adjustment disorder (0.73)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for acute and transient psychotic disorder was 0.45, which is moderate.

    “moderate for acute and transient psychotic disorder (0.45), dysthymic disorder (0.45), panic disorder (0.57), post-traumatic stress disorder (0.49), and the newly introduced diagnosis of complex post-traumatic stress disorder (0.56)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimates were precise for all calculated diagnoses, with confidence intervals below 0.5 and standard errors below 0.1.

    “The estimates of kappa were precise for all diagnoses for which it was calculated (confidence interval <0.5; standard error <0.1).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for generalized anxiety disorder was 0.62, which is substantial.

    “substantial for schizoaffective disorder (0.66), delusional disorder (0.69), bipolar II disorder (0.62), single episode depressive disorder (0.64), recurrent depressive disorder (0.74), generalized anxiety disorder (0.62), agoraphobia (0.62), and adjustment disorder (0.73)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for complex post-traumatic stress disorder was 0.56, which is moderate.

    “moderate for acute and transient psychotic disorder (0.45), dysthymic disorder (0.45), panic disorder (0.57), post-traumatic stress disorder (0.49), and the newly introduced diagnosis of complex post-traumatic stress disorder (0.56)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for dysthymic disorder was 0.45, which is moderate.

    “moderate for acute and transient psychotic disorder (0.45), dysthymic disorder (0.45), panic disorder (0.57), post-traumatic stress disorder (0.49), and the newly introduced diagnosis of complex post-traumatic stress disorder (0.56)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for delusional disorder was 0.69, which is substantial.

    “substantial for schizoaffective disorder (0.66), delusional disorder (0.69), bipolar II disorder (0.62), single episode depressive disorder (0.64), recurrent depressive disorder (0.74), generalized anxiety disorder (0.62), agoraphobia (0.62), and adjustment disorder (0.73)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All common and high-burden disorders in the adult population covered by the study were diagnosed with at least satisfactory reliability, and in most cases excellent reliability, by clinician raters including advanced trainees and experienced clinicians.

    “The results of the ICD-11 EIFS show that all common and high-burden disorders in the adult population covered in the current study were diagnosed with at least satisfactory -and in most cases excellent -reliability by a sample of clinician raters that included advanced trainees in psychiatry as well as more experienced clinicians.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for social anxiety disorder was 0.88, which is almost perfect.

    “ranged from 0.45 (dysthymic disorder) to 0.88 (social anxiety disorder)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for bipolar II disorder was 0.62, which is substantial.

    “substantial for schizoaffective disorder (0.66), delusional disorder (0.69), bipolar II disorder (0.62), single episode depressive disorder (0.64), recurrent depressive disorder (0.74), generalized anxiety disorder (0.62), agoraphobia (0.62), and adjustment disorder (0.73)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for schizophrenia was 0.87, which is almost perfect.

    “The kappa estimates were almost perfect for schizophrenia (0.87) and bipolar I disorder (0.84)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The kappa estimate for agoraphobia was 0.62, which is substantial.

    “substantial for schizoaffective disorder (0.66), delusional disorder (0.69), bipolar II disorder (0.62), single episode depressive disorder (0.64), recurrent depressive disorder (0.74), generalized anxiety disorder (0.62), agoraphobia (0.62), and adjustment disorder (0.73)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Those differences were small, all below 0.2, and the ICD-11 CDDG reliability estimates for these categories remained in the moderate range.

    “However, these differences are modest in size (in all cases <0.2), and the reliability estimates for the ICD-11 CDDG in these categories are still in the moderate range.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The ICD-11 CDDG diagnoses of acute and transient psychotic disorder, panic disorder, and post-traumatic stress disorder appeared less reliable than their ICD-10 CDDG equivalents, although these differences were not statistically analyzed.

    “The ICD-11 CDDG diagnoses of acute and transient psychotic disorder, panic disorder, and posttraumatic stress disorder seemed to have lower reliability than the equivalent categories in the ICD-10 CDDG, though it was not considered appropriate to analyze these differences statistically.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Because all diagnoses with lower bound confidence interval estimates below 0.4 were made less often, higher reliability for these disorders might be achieved in larger samples.

    “All diagnoses with lower bound confidence interval estimates of kappa (<0.4) were made less often, suggesting that higher reliability for these disorders might accrue in samples of larger sizes.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The range and length of the diagnostic interviews were substantially consistent with usual practice in participating mental health centers.

    “The range and length of the diagnostic interviews were therefore substantially consistent with usual practice in participating mental health centers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Data quality was established through continuous monitoring of data collection by local research staff and through programming functions in Qualtrics such as forced response and content validation.

    “Data quality was established through continuous monitoring of the data collection procedures by local research staff at each site and through use of programming functions within Qualtrics TM , such as forced response and content validation options.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The more consistent presentation of information in the ICD-11 CDDG compared with the ICD-10 CDDG likely helped reliability.

    “In no case were changes proposed solely to improve reliability, though the more consistent presentation of information in the ICD-11 CDDG as compared to the ICD-10 CDDG 13 likely helped in this regard.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] source

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    The published DSM-III field trial provided kappa values for disorder groupings rather than specific disorders, which would tend to maximize reliability results.

    “The published results of DSM-III field trial provided kappa values for disorder groupings rather than for specific disorders 32 , which would tend to maximize reliability results because disagreements within a grouping are substantially more likely than disagreements concerning disorders from different groupings.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Clinician raters who conducted joint-rater interviews and were also referring clinicians were not permitted to interview any patient they had referred.

    “At most sites, clinician raters who conducted joint-rater interviews were also part of the pool of referring clinicians, in which case they were not permitted to interview any patient they had referred.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Clinician pairings were varied as much as possible given availability and scheduling, and participating clinicians alternated as primary interviewer and observer.

    “Clinician pairings were varied as much as possible given constraints of availability and scheduling, and participating clinicians alternated as primary interviewer and observer.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The ICD-11 CDDG PTSD diagnosis is narrower than the ICD-10 diagnosis and requires here-and-now re-experiencing of intrusive symptoms and functional impairment.

    “The ICD-11 CDDG for post-traumatic stress disorder diagnosis are conceptually narrower than the ICD-10 ones, and now require the presence of re-experiencing of intrusive symptoms in the “here and now”, as opposed to only experiencing intrusive memories of the traumatic event, as well as the presence of functional impairment 5 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Research shows that worry is a central feature of generalized anxiety disorder.

    “based in part on studies that show that worry is a central characteristic of the disorder 42 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [42] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The ICD-11 conceptualization of social anxiety disorder expanded the feared negative evaluations to include cultural variants and a wider range of behavioural responses to social stimuli.

    “The ICD-11 conceptualization of social anxiety disorder has broadened the concept of ways in which the person could fear being negatively evaluated by others to include cultural variants of the disorder (i.e., fears of humiliation, embarrassment, rejection, or being offensive) as well as the range of behaviours in response to social stimuli 44,45 .”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [44]

      Not checked in the free run.

    • cites [45] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Changes to the ICD-11 CDDG relative to the ICD-10 CDDG were proposed by expert working groups based on available scientific evidence and with explicit attention to clinical utility and global applicability.

    “Changes to the ICD-11 CDDG relative to the ICD-10 CDDG were proposed by expert working groups based on the available scientific evidence and with explicit attention to additional sources of information related to clinical utility and global applicability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The ICD-11 CDDG reconceptualized agoraphobia to cover a wider range of feared stimuli and related behaviours, partly to better represent situations reported in low- and middle-income countries.

    “Agoraphobia is reconceptualized to include a broader array of feared stimuli (fear of situations, fear of specific negative outcomes) and behaviours manifested in response to these stimuli (avoidance or entering the situations under specific conditions or enduring the situation with intense fear/ anxiety), partly to allow for situations that may be more representative of those reported in low-and”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [43]

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    The joint-rater design was used to minimize information variance and to focus on whether two clinicians relying on the same clinical information agree on diagnoses when separately applying the ICD-11 guidelines.

    “The joint-rater design was used in order to minimize information variance and to focus specifically on the question of whether two clinicians, relying on the same clinical information, agree on the diagnoses to be assigned to the patient when separately applying the ICD-11 diagnostic guidelines.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    An earlier Internet-based study found that clinicians did not consistently apply the proposed ICD-11 requirement for re-experiencing the traumatic event.

    “However, an earlier Internet-based study on disorders specifically associated with stress 9 showed that clinicians did not consistently apply the proposed ICD-11 guidelines regarding the required element of re-experiencing of the traumatic event(s).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The training sessions were not dissimilar to those that clinicians might realistically receive when the ICD-11 is implemented in their countries.

    “Training sessions were therefore not dissimilar to those that clinicians might realistically receive when the ICD-11 is implemented in their countries.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Patients were interviewed by two clinicians who had no prior clinical contact with the patient.

    “Following informed consent, patients were interviewed by two clinicians who had no prior clinical contact with the patient.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    ICD-11 acute and transient psychotic disorder now covers only acute psychoses with polymorphic presentation, making it not strictly comparable with the broader concept tested in the ICD-10 field trial.

    “ICD-11 acute and transient psychotic disorder now exclusively comprises acute psychoses with “polymorphic” presentation 3 , which is not strictly comparable to the broader concept tested in the ICD-10 field trial 16 .”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [3] source

      Not checked in the free run.

    • cites [16] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Advanced residents in psychiatry could function as interviewers but were always paired with a fully qualified individual.

    “Advanced residents in psychiatry (following completion of first two years of residency) could function as interviewers but were always paired with a fully qualified individual.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    The DSM-5 design did not control for information variance and would almost certainly yield lower reliabilities.

    “The DSM-5 design, therefore, did not control for information variance and so would almost certainly yield lower reliabilities 26,35 .”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [26] source

      Not checked in the free run.

    • cites [35] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All clinician raters were qualified to make mental disorder diagnoses independently as part of their scope of practice according to their countries' practice standards.

    “According to the practice standards of their countries, all clinician raters were qualified to make mental disorders diagnoses independently as a part of their scope of practice.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prior studies suggest that the threshold for an ICD-10 PTSD diagnosis is relatively low.

    “Studies have also suggested that the threshold for an ICD-10 diagnosis of the disorder is relatively low 48,49 .”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [48] source

      Not checked in the free run.

    • cites [49] source

      Not checked in the free run.

  • May combine two claims.

    Premisenone given

    Most ICD-10 diagnoses correspond closely to proposed ICD-11 diagnoses, but they are not identical.

    “Further, though most ICD-10 diagnoses correspond closely to proposed ICD-11 diagnoses, they are not identical.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The ICD-11 PTSD model has received increasing empirical support.

    “This model has garnered increasing empirical support 50 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [50] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The ICD-10 Diagnostic Criteria for Research field trial involved a structured diagnostic instrument covering the diagnostic criteria for assessment.

    “The ICD-10 Diagnostic Criteria for Research field trial involved the use of a structured diagnostic instrument that covered the diagnostic criteria for assessment 33 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Only diagnoses that occurred at least 30 times across the study were included, because less frequent diagnoses were not considered to have sufficient stability for the evaluation.

    “Only diagnoses that occurred at least 30 times across the study were included in these analyses, as diagnoses assigned less frequently were not considered to have sufficient stability for the present evaluation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The DSM has historically and currently used a longitudinal approach, and most earlier studies were based on it.

    “This is different from the longitudinal approach historically and currently taken by the DSM, on which most previous studies have been based 46,47 .”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [46] source

      Not checked in the free run.

    • cites [47] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Clinician raters independently arrived at a diagnostic formulation of up to three diagnoses and independently provided data after the interview.

    “Based on the interview, and in some cases additional supplementary material provided to both clinicians (e.g., patient file excluding current or prior psychiatric diagnoses and psychotropic medication prescriptions, interviews with family members), clinician raters independently arrived at a diagnostic formulation consisting of up to three diagnoses.”
    • Also stated in: § Study design and procedures · plain
      “Following the interview, both clinician raters independently provided data based on the interview using a secure webbased data collection system.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    The case conference methodology used in the ICD-10 field study is likely to produce more consensus-based results, with correspondingly higher reliability.

    “The case conference methodology is likely to produce more consensus-based results, in which reliability would be correspondingly higher.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Schizoaffective disorder is not uncommon in clinical populations, and its reliability remains debated.

    “Schizoaffective disorder is not a rare diagnosis in clinical populations, and its reliability is subject to ongoing discussion 46 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [46] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Jager and colleagues reviewed six studies, found kappa values from 0.08 to 0.63, and concluded that only one study showed good agreement.

    “Jager et al 47 reviewed six studies and reported kappa scores between 0.08 and 0.63, concluding that only one study showed good agreement.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Findings from DSM-5 field trials indicate that more broadly defined disorders tend to have higher reliability.

    “Data from DSM-5 field trials suggest that disorders that are more broadly defined have higher reliability 25 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductnone given

    The participating institutions were mostly high-status secondary or tertiary care centers where clinician training in diagnosis and interviewing likely meets the highest national standards.

    “Participating institutions were typically high-status secondary or tertiary care centers, where the training of clinicians in diagnostic classification and interviewing is likely to meet the highest national standards.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The ICD-11 CDDG removed adjustment disorder subtypes because evidence for their validity or clinical utility was lacking.

    “Similarly, the subtypes of adjustment disorder have been eliminated from ICD-11 CDDG because they lacked evidence for validity or clinical utility 5 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The concurrent joint-rater design focused specifically on the diagnostic guidelines themselves as a source of unreliability.

    “The concurrent joint-rater design employed in ICD-11 EIFS focused specifically on the role of the diagnostic guidelines themselves as a source of unreliability.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The four disorder groups targeted by the study account for the greatest proportion of global disease burden among mental disorders and the highest levels of service utilization in mental health settings.

    “These diagnoses account for the greatest proportion of global disease burden among mental disorders 37 and the highest levels of service utilization in mental health settings.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [37] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The ICD-11 CDDG broadened the concept of generalized anxiety disorder by adding worry as an alternative essential feature alongside generalized apprehension and physiological symptoms.

    “The conceptualization of generalized anxiety disorder has been broadened in ICD-11 CDDG to include worry as an alternative essential feature to generalized apprehension and accompanying physiological symptoms 7 , based in part on studies that show that worry is a central characteristic of the disorder 42 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The study's clinician interviewers likely had a particular interest in diagnostic classification and in learning about ICD-11.

    “Clinician interviewers in the study would also have had some specific interest in diagnostic classification and in learning about the ICD-11.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The current study covered only a small share of the many mental disorder diagnoses applicable to adults, focusing on high-burden conditions that account for most services at participating centers.

    “Finally, the current study assessed only a relatively small proportion of the wide range of mental disorder diagnoses that may be applied to adult patients, focusing on those that are responsible for the highest level of disease burden and account for the greatest proportions of mental health services in participating centers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The global applicability of the ICD-11 CDDG conceptualization for commonly diagnosed mental disorders is supported by reliability testing across 28 sites in 13 countries and five languages, using a naturalistic field study design and replicable training.

    “The global applicability of the ICD-11 CDDG conceptualization of commonly diagnosed mental disorders is supported by the assessment of reliability of these guidelines in diverse settings (across 28 sites in 13 countries and in five languages) using a naturalistic field study design and a training approach that can easily be replicated for ICD-11 implementation.”
    Quote in the paperSubstantial claim
60 of 60

  • The paper relies on this without support.

    The 28 study sites in 13 countries are sufficiently representative of global mental health settings for the reliability results to support implementation of the ICD-11 diagnostic guidelines at a global level.

    “The EIFS centers are located in countries that diverge widely in terms of languages, culture, and resource levels.”
    “These data provide support for the suitability of the ICD-11 diagnostic guidelines for implementation at a global level.”

  • The authors state this as an assumption.

    The reliability estimates from this study are higher than those that will be obtained in routine practice in all settings where the ICD-11 CDDG is implemented.

    “It can therefore be assumed that the reliabilities obtained in the study are higher than those that will be obtained in usual practice across all settings where the ICD-11 CDDG will be implemented.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • As internet-based and clinical field trials have demonstrated for psychotic disorders, diagnostic reliabilities for ICD-11 compared to ICD-10 have markedly improved for most diagnostic categories in this section.

    “As internet-based and clinical field trials have demonstrated for psychotic disorders, diagnostic reliabilities for the ICD-11 compared to the ICD-10 have markedly improved for most of the diagnostic categories in this section [31].”

    From Mental, behavioral and neurodevelopmental disorders in the ICD-11: an international perspective on key changes and controversies (2020) · cites this paper as [31]

    The Discussion reports generally higher ICD-11 diagnostic reliability than previously reported for ICD-10, including improved reliability for schizoaffective disorder. But this paper reports a clinical field study, not an internet-based trial; it also says acute and transient psychotic disorder appeared less reliable under ICD-11 and cautions that methodological differences preclude strict comparisons. Thus it does not establish a marked improvement across most psychotic-disorder categories as stated. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The ICD-11 guidelines for depression have substantial inter-clinician reliability and very good clinical utility in field trials.

    “The ICD-11 guidelines for depression have been found to have substantial inter-clinician reliability and very good clinical utility in field trials [31, 52].”

    From Mental, behavioral and neurodevelopmental disorders in the ICD-11: an international perspective on key changes and controversies (2020) · cites this paper as [31]

    The Results (page not available in our copy) classify reliability as substantial for single-episode depressive disorder (κ=0.64) and recurrent depressive disorder (κ=0.74). The paper does not report a clinical-utility rating for the ICD-11 depression guidelines, so it does not establish the claimed “very good” clinical utility. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • The results of ICD-11 field studies in international clinical settings call the assumption that a strict criteria-based approach leads to greater reliability into question.

    “The results of ICD-11 field studies in international clinical settings 94 also call this assumption into question.”

    From An organization‐ and category‐level comparison of diagnostic requirements for mental disorders in <scp>ICD</scp> ‐11 and <scp>DSM</scp> ‐5 (2021) · cites this paper as [94]

  • Transdiagnostic approaches have been suggested to complement current psychiatric nosography, which is intrinsically limited, in particular in young people, by integrating clinical staging models and optimizing preventive efforts.

    “Transdiagnostic approaches have been suggested to complement current psychiatric nosography 208 , which is intrinsically limited, in particular in young people 209,210 , by integrating clinical staging models and optimizing preventive efforts.”

    From Preventive psychiatry: a blueprint for improving the mental health of young people (2021) · cites this paper as [209]

The citation lists haven’t been fetched yet

References 52 · Cited by 126

Research works printed but not linked: 3

  1. Maj M, 2012, The ICD‐11 classification of mood and anxiety disorders: background and options, World Psychiatry, 11
  2. First MB., 2007, Counterpoint – there isn't enough evidence available to speculate on the reliability of diagnoses in clinical settings, Psychiatry, 4, 24
  3. Stein DJ., 2012, Agoraphobia and panic disorder: options for ICD‐11, World Psychiatry, 11, 89

Authors
Reed, G. M., Sharan, P., Rebello, T. J., Keeley, J. W., Elena Medina‐Mora, M., Gureje, O., Luis Ayuso‐Mateos, J., Kanba, S., Khoury, B., Kogan, C. S., Krasnov, V. N., Maj, M., de Jesus Mari, J., Stein, D. J., Zhao, M., Akiyama, T., Andrews, H. F., Asevedo, E., Cheour, M., Domínguez‐Martínez, T., El‐Khoury, J., Fiorillo, A., Grenier, J., Gupta, N., Kola, L., Kulygina, M., Leal‐Leturia, I., Luciano, M., Lusu, B., Nicolas, J., Martínez‐López, I., Matsumoto, C., Umukoro Onofa, L., Paterniti, S., Purnima, S., Robles, R., Sahu, M. K., Sibeko, G., Zhong, N., First, M. B., Gaebel, W., Lovell, A. M., Maruta, T., Roberts, M. C., & Pike, K. M.
Affiliations
  • Columbia University Irving Medical Center (US)
  • All India Institute of Medical Sciences (IN)
  • Virginia Commonwealth University (US)
  • National Institute of Psychiatry Ramón de la Fuente Muñiz Mexico City Mexico
  • University of Ibadan (NG)
  • Department of Psychiatry Universidad Autonoma de Madrid, IIS‐P and Instituto de Salud Carlos III, Centro de Investigación Biomédica en Red de Salud Mental (CIBERSAM) Madrid Spain
  • Kyushu University (JP)
  • American University of Beirut Medical Center (LB)
  • University of Ottawa (CA)
  • Federal Medical Research Centre for Psychiatry and Narcology (RU)
  • University of Campania "Luigi Vanvitelli" (IT)
  • Department of Psychiatry Universidade Federal de São Paulo São Paulo Brazil
  • South African Medical Research Council (ZA)
  • Shanghai Jiao Tong University (CN)
  • NTT Medical Center (JP)
  • New York State Psychiatric Institute (US)
  • Universidade Federal de São Paulo (BR)
  • Tunis El Manar University (TN)
  • Instituto Nacional de Psiquiatría Ramón de la Fuente Muñiz (MX)
  • Government Medical College and Hospital (IN)
  • Instituto de Salud Carlos III (ES)
  • The Japanese Society of Psychiatry and Neurology (JP)
  • Federal Neuropsychiatric Hospital Aro Abeokuta Nigeria
  • Pt. Jawahar Lal Nehru Memorial Medical College (IN)
  • Department of Psychiatry and Psychotherapy, Medical Faculty Heinrich‐Heine University Düsseldorf Germany
  • Inserm (FR)
  • Seitoku University (JP)
  • University of Kansas (US)
ORCID
Venue
World Psychiatry
Publisher
Wiley
Published
2018-05-24
Volume
17
Issue
2
Pages
174-186
ISSN
  • 1723-8617 (ISSN-L)
  • 2051-5545
DOI
10.1002/wps.20524
PMC
PMC5980511
OpenAlex
W2805159053
Semantic Scholar
b9c8eaf463cacd57c267f06c053b5a0138b6d17b
Cited by
126
References
52
Citations by year
2026: 3 · 2025: 9 · 2024: 13 · 2023: 14 · 2022: 17 · 2021: 19 · 2020: 20 · 2019: 23 · 2018: 7 · 2017: 1
FWCI
13.14
Type
journal article
Language
en
Open access
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations
  • World Psychiatry (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • HAL (submittedVersion)
  • Archivio Istituzionale della Ricerca - Università degli Studi della Campania "Luigi Vanvitelli" (submittedVersion)
Funders
  • World Health Organization
  • University of Ottawa
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
  • Social Sciences › Arts and Humanities › Philosophy › Mental Health and Psychiatry
  • Social Sciences › Psychology › Social Psychology › Mental Health Treatment and Access
Keywords
ICD-11, diagnostic reliability, interobserver agreement, mental illness, schizophrenia, psychosis, mood disorders, anxiety disorders, stress-related disorders, global mental health, diagnostic guidelines, clinician training

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review