CoChat
Sign Up
Cited by 125Artificial Intelligence in Healthcare and EducationDigital Mental Health InterventionsEthics and Social Impacts of AI

ChatGPT is not ready yet for use in providing mental health assessment and interventions

Open access
Dergaa, Ismail, Fekih-Romdhane, Feten, Hallit, Souheil, Loch, Alexandre Andrade, Glenn, Jordan M., Fessi, Mohamed Saifeddin, Ben Aissa, Mohamed, Souissi, Nizar +5 more·2024·Frontiers in Psychiatry
DOI 10.3389/fpsyt.2023.1277756PMC PMC10794665
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Psychiatry is a specialized field of medicine that focuses on the diagnosis, treatment, and prevention of mental health disorders. With advancements in technology and the rise of artificial intelligence (AI), there has been a growing interest in exploring the potential of AI language models systems, such as Chat Generative Pre-training Transformer (ChatGPT), to assist in the field of psychiatry. Objective: Our study aimed to evaluates the effectiveness, reliability and safeness of ChatGPT in assisting patients with mental health problems, and to assess its potential as a collaborative tool for mental health professionals through a simulated interaction with three distinct imaginary patients. Methods: Three imaginary patient scenarios (cases A, B, and C) were created, representing different mental health problems. All three patients present with, and seek to eliminate, the same chief complaint (i.e., difficulty falling asleep and waking up frequently during the night in the last 2°weeks). ChatGPT was engaged as a virtual psychiatric assistant to provide responses and treatment recommendations. Results: In case A, the recommendations were relatively appropriate (albeit non-specific), and could potentially be beneficial for both users and clinicians. However, as complexity of clinical cases increased (cases B and C), the information and recommendations generated by ChatGPT became inappropriate, even dangerous; and the limitations of the program became more glaring. The main strengths of ChatGPT lie in its ability to provide quick responses to user queries and to simulate empathy. One notable limitation is ChatGPT inability to interact with users to collect further information relevant to the diagnosis and management of a patient’s clinical condition. Another serious limitation is ChatGPT inability to use critical thinking and clinical judgment to drive patient’s management. Conclusion: As for July 2023, ChatGPT failed to give the simple medical advice given certain clinical scenarios. This supports that the quality of ChatGPT-generated content is still far from being a guide for users and professionals to provide accurate mental health information. It remains, therefore, premature to conclude on the usefulness and safety of ChatGPT in mental health practice.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper evaluated ChatGPT as a virtual psychiatric assistant using three simulated patients with insomnia, finding that recommendations were relatively appropriate only in the simplest case and became inappropriate or dangerous as clinical complexity increased. The authors conclude that as of July 2023, ChatGPT is not ready for independent use in mental health assessment or intervention.

Key findings

  • In case A, a 22-year-old male college student preparing for a national exam in 10 days, ChatGPT’s recommendations were “relatively appropriate (albeit non-specific)” and aligned with non-pharmacological first-line care for insomnia.
  • In case B, a 58-year-old female with systemic lupus erythematosus, ChatGPT failed to consider depression or medication-induced insomnia/depression and instead recommended a plan similar to case A; the authors call failure to rule out depression “a clinical error that could have malignant consequences.”
  • In case C, a 23-year-old postpartum female with hemoglobin 10.5 gm/dl, guilt, inability to care for her baby, and fear of harming him, ChatGPT failed to evoke postpartum depression or bipolar postpartum depression and provided fragmented management; the authors state such information “can be misleading and endanger lives.”
  • Across cases, ChatGPT could not prescribe medication, could not ask follow-up questions to collect diagnostic information, and lacked critical thinking/clinical judgment, though it could give quick responses and simulate empathy.

Methods

  • Three imaginary patient scenarios (A, B, C) were created, all presenting with difficulty falling asleep and waking up frequently during the night for the last 2 weeks.
  • ChatGPT was engaged as a virtual psychiatric assistant to provide assessment and treatment recommendations, with structured interactions mirroring real-life psychiatric consultations.
  • Responses were qualitatively evaluated by a panel of academic healthcare professionals, including a professor in psychiatry and a clinical psychiatrist, for accuracy, treatment suitability, empathy, and need for human psychiatrist intervention.

Limitations

  • The paper states the study was based on a limited number of scenarios and mental health problems, so conclusions may not apply to other clinical situations; findings may not generalize to non-ChatGPT AI models.
  • The authors state that inquiries were limited to assessment and treatment of each case’s condition, which may have influenced ChatGPT interactions, and that the absence of real-time patient feedback limits understanding of patient experience.
  • The authors also note in an author’s note that interpretation contains “a certain degree of subjectivity” based on their experience with advanced language models; evident from the text, the study used no real patients and reported no quantitative outcome measures.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/fpsyt.2023.1277756

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 31 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    ChatGPT failed to evoke and discuss the seemingly obvious diagnoses of postpartum depression and bipolar postpartum depression.

    “However, ChatGPT failed to evoke and discuss these seemingly obvious diagnoses.”
    Quote in the paperSubstantial claim
  • Findingown data

    ChatGPT has an inability to use critical thinking and clinical judgment to drive patient's management.

    “Another serious limitation is ChatGPT inability to use critical thinking and clinical judgment to drive patient’s management.”
    Quote in the paperSubstantial claim
  • Findingown data

    ChatGPT has an inability to interact with users to collect further information relevant to the diagnosis and management of a patient's clinical condition.

    “One notable limitation is ChatGPT inability to interact with users to collect further information relevant to the diagnosis and management of a patient’s clinical condition.”
    Quote in the paperSubstantial claim
  • Findingnone given

    Although instructed to begin with an initial assessment of the patient's mental health condition, ChatGPT failed to perform this task.

    “Although ChatGPT was instructed to begin with an initial assessment of the patient’s mental health condition, it failed to perform this task.”
    Quote in the paperSubstantial claim
  • Findingown data

    When asked to act as a psychiatric healthcare provider, ChatGPT demonstrated empathy toward the patient.

    “Promisingly, when asked to act as a psychiatric healthcare provider (interaction 1), ChatGPT demonstrated empathy toward the patient (e.g., “I understand,” “I am here to support you on your journey,” “I want to assure you that we can work together”)”
    Quote in the paperNot a substantial claim
  • May combine two claims. 21
  • Classified as probably not a substantial claim. 40

  • May combine two claims.

    Findingown data

    In case A, ChatGPT's recommendations were relatively appropriate (albeit non-specific) and could potentially be beneficial for both users and clinicians.

    “In case A, the recommendations were relatively appropriate (albeit non-specific), and could potentially be beneficial for both users and clinicians.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    ChatGPT advised the patient to consult professionals for assessment before stating about the necessity of medication.

    “ChatGPT also advised the patient to consult professionals for assessment before stating about the necessity of medication.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    For comprehensive evaluation, ChatGPT recommended a thorough evaluation to determine if there are any underlying medical factors contributing to the patient's sleep difficulties.

    “It was encouraging to note that, for comprehensive evaluation, ChatGPT recommended “a thorough evaluation to determine if there are any underlying medical factors contributing to her sleep difficulties.””
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The main strengths of ChatGPT lie in its ability to provide quick responses to user queries and to simulate empathy.

    “The main strengths of ChatGPT lie in its ability to provide quick responses to user queries and to simulate empathy.”
    • Also stated in: § 5 Conclusion and future perspectives · plain
      “The main strengths of ChatGPT lie in its ability to provide quick responses to user queries and to simulate empathy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    ChatGPT ultimately concluded that the issues were related to the presence of stress in the work environment.

    “While ChatGPT considered the reduced work performance as a contributing factor to the feelings of worthlessness, it ultimately concluded that the issues were related to the presence of stress in the work environment.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    ChatGPT relied on the variable 'upcoming national exam in 10 days' to quickly suggest that subsequent increased stress and anxiety likely disrupted the student's sleep pattern.

    “ChatGPT relied on the variable “upcoming national exam in 10 days” to quickly suggest that subsequent “increased stress and anxiety” likely disrupted the student’s sleep pattern.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    ChatGPT listed several broad causal factors contributing to sleep problems, including postpartum hormonal changes, caring for the newborn, the traumatic labor experience, and guilt and anxiety related to caring for the baby.

    “The program listed instead several broad causal factors contributing to sleep problems, including postpartum hormonal changes, caring for the newborn, the traumatic labor experience, as well as the presence of guilt and anxiety related to caring for her baby.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    ChatGPT suggested a treatment plan based on non-pharmacological interventions as a first-line option, aligning with current guidelines and clinical standards of care.

    “it was able to suggest a treatment plan based on non-pharmacological interventions as a first-line option, aligning with current guidelines and clinical standards of care (32).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] Rios, 2019, Comparative effectiveness and safety of pharmacological and non-pharmacological interventions for insomnia: an overview of reviews., Syst Rev. source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Although ChatGPT gave a general warning that medication can be necessary given significant impact on daily functioning, it failed to raise concerns about the patient's current mental state and to link her symptoms to an eventual depression.

    “Although ChatGPT gave a general warning that medication (i.e., a sleep aid or an antidepressant) can be necessary given the significant impact on daily functioning, it failed to raise concerns about her current mental state and to link her symptoms to an eventual depression.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    ChatGPT presented a relatively broad, non-specific list of treatment recommendations including sleep hygiene education, stress management techniques, relaxation techniques, sleep environment optimization, and CBT-I.

    “In terms of treatment plan, ChatGPT presented a relatively broad, non-specific list of recommendations, with suggestions including sleep hygiene education, stress management techniques, relaxation techniques, sleep environment optimization, and CBT for Insomnia (CBT-I).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    ChatGPT recommended stress management techniques to assist the patient in coping with work-related stress.

    “As a result, ChatGPT recommended stress management techniques to assist the patient in coping with work-related stress.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    ChatGPT advised further evaluation and guidance from a healthcare professional if sleep issues continue to significantly impact daily functioning and quality of life.

    “On a positive note, ChatGPT drew attention to the fact that, if “sleep issues continue to significantly impact daily functioning and quality of life,” further evaluation and guidance from a healthcare professional is advisable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    As for July 2023, ChatGPT failed to give the simple medical advice given certain clinical scenarios.

    “As for July 2023, ChatGPT failed to give the simple medical advice given certain clinical scenarios.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    ChatGPT omitted to prioritize treatment options proposed for this case and rather started by suggesting sleep hygiene and lifestyle changes as a first step.

    “ChatGPT omitted to prioritize treatment options proposed for this case, and rather started by suggesting sleep hygiene and lifestyle changes as a first step.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    ChatGPT mentioned possible physical health implications of SLE and highlighted the need to ensure SLE is well-managed.

    “ChatGPT mentioned possible “physical health implications of SLE” and highlighted the need to “ensure your SLE is well-managed””
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In further discussion to recommend medication, ChatGPT emphasized its own limitations by indicating that, as an AI language tool, it does not have the ability to prescribe medication or provide medical treatment.

    “In further discussion with ChatGPT to recommend a medication for this patient, ChatGPT emphasized its own limitations by indicating that, as an AI language tool, it does not have the ability to prescribe medication or provide medical treatment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    ChatGPT recognized the importance of addressing the anemia, albeit without giving any specific reasons justifying this recommendation.

    “When considering the treatment plan, ChatGPT recognized the importance of addressing the anemia, albeit without giving any specific reasons justifying this recommendation [i.e., treatment of anemia as an important therapeutic measure of depression (43)].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [43] Azami, 2019, The association between anemia and postpartum depression: a systematic review and meta-analysis., Caspian J Intern Med.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    ChatGPT listed a spectrum of possible modes of care to address sleep problems, including sleep hygiene and routine, adjusting breastfeeding routines, psychological support, social support, nutritional support, and safety assessment.

    “Then, ChatGPT listed a spectrum of possible modes of care to address sleep problems, including “sleep hygiene and routine,” “adjusting breastfeeding routines,” “psychological support” (to address feelings of overwhelm, fear of harming the baby, and guilt), “social support,” “nutritional support” (to address loss of appetite and tiredness), and “safety assessment.””
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    ChatGPT recognized the importance of soliciting input from healthcare providers.

    “Altogether, ChatGPT recognized the importance of soliciting input from healthcare providers”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Anemia is a potential risk factor for postpartum depression.

    “She also has anemia, which is a potential risk factor for postpartum depression (43).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [43] Azami, 2019, The association between anemia and postpartum depression: a systematic review and meta-analysis., Caspian J Intern Med.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Strong longitudinal evidence demonstrates that insomnia and depression are significantly and bi-directionally associated with each other.

    “Strong longitudinal evidence exists demonstrating insomnia and depression are significantly and bi-directionally associated with each other (34, 35)”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [34] Jansson-Fröjmark, 2008, A bidirectional relationship between anxiety and depression, and insomnia? A prospective study in the general population., J Psychosom Res. source

      Not checked in the free run.

    • cites [35] Li, 2016, Insomnia and the risk of depression: a meta-analysis of prospective cohort studies., BMC Psychiatry. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Mental health problems are not ChatGPT's responsibility, even if it is in the form of first aid.

    “In this sense, we support the assertions of Aditama et al. (15) that “Mental health problems are not ChatGPT’s responsibility, even if it is in the form of first aid.””
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] Aditama, 2023, Is ChatGPT worth using in first aid for mental health issues?, J Public Health. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Medications commonly administered to lupus patients, such as corticosteroids like prednisone, may trigger both sleep disruption and depressive symptoms.

    “Another possible differential is drug-induced insomnia and/or depression, as medications commonly administered to lupus patients (corticosteroids such as prednisone) may trigger both sleep disruption and depressive symptoms (37, 38).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [37] Malangu, 2012, Drugs Inducing Insomnia as an Adverse Effect, Can’t Sleep.

      Not checked in the free run.

    • cites [38] Kusljic, 2016, Corticosteroid-induced psychiatric disturbances: it is time for pharmacists to take notice., Res Social Adm Pharm. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Failing to detect and treat postpartum depression may have potential harmful consequences, including suicide risk, physical harm to the baby, impaired mother-infant bonding, and the risk of children developing emotional, cognitive, behavioral and social problems.

    “Failing to detect and treat postpartum depression may have potential harmful consequences, including suicide risk, physical harm to the baby, impaired mother-infant bonding, and the risk of children developing emotional, cognitive, behavioral and social problems (50).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [50] Mughal, 2022, Postpartum Depression.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In the initial evaluation of a case such as this, a clinician would ask for additional information and perform a physical examination to rule out differential diagnoses such as postpartum thyroiditis.

    “In the initial evaluation of a case such as this, a clinician would ask for additional information (e.g., drug and alcohol history, any medication intake, physical symptoms), and perform a physical examination to rule out differential diagnoses, such as postpartum thyroiditis (45).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] Stagnaro-Green, 2011, American thyroid association taskforce on thyroid disease during pregnancy and postpartum. Guidelines of the American thyroid association for the diagnosis and management of thyroid disease during pregnancy and postpartum., Thyroid. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Unlike ChatGPT, clinicians would ask more questions and consider more possible explanations before reaching a diagnosis conclusion and subsequent treatment plan.

    “Unlike ChatGPT, clinicians would ask more questions and would consider more possible explanations before reaching a (diagnosis) conclusion and subsequent treatment plan.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Certain factors in this patient highly suggest a diagnosis of bipolar postpartum depression, including early illness-onset, first onset of depression after childbirth, onset immediately after delivery, and a history of suicide in family members.

    “Certain factors in this patient highly suggest a diagnosis of bipolar postpartum depression, including early illness-onset, first onset of depression after childbirth, onset immediately (i.e., during the first month) after delivery, and a history of suicide in family members (44).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [44] Sharma, 2017, Bipolar postpartum depression: an update and recommendations., J Affect Disord. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The simulated interactions were designed to mirror real-life psychiatric consultations, allowing for a consistent and focused assessment of ChatGPT's capabilities.

    “These interactions were designed to mirror real-life psychiatric consultations, allowing for a consistent and focused assessment of ChatGPT’s capabilities.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The evaluation criteria included accuracy in diagnosis, suitability of treatment recommendations, and the ability to emulate empathetic communication, reflecting the standards of clinical practice.

    “The criteria for evaluation included accuracy in diagnosis, suitability of treatment recommendations, and the ability to emulate empathetic communication, reflecting the standards of clinical practice.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    AI language models, while capable of generating coherent responses, may struggle with understanding and addressing the intricacies of such mental health conditions.

    “AI language models, while capable of generating coherent responses, may struggle with understanding and addressing the intricacies of such mental health conditions (15).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15] Aditama, 2023, Is ChatGPT worth using in first aid for mental health issues?, J Public Health. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A patient does not meet diagnostic criteria for insomnia if sleep complaints are completely explained by another physical, psychiatric or sleep disorder.

    “Although the chief complaint is insomnia symptoms, the patient does not meet diagnostic criteria for insomnia if these sleep complaints are completely explained by another physical, psychiatric or sleep disorder (23).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [23] Roth, 2011, Prevalence and perceived health associated with insomnia based on DSM-IV-TR; international statistical classification of diseases and related health problems, tenth revision; and research diagnostic criteria/international classification of sleep disorders, criteria: results from the America insomnia survey., Biol Psychiatry. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The responses generated by ChatGPT were evaluated by a panel of academic healthcare professionals, including a professor in psychiatry and a clinical psychiatrist actively practicing in the field.

    “The responses generated by ChatGPT were subjected to evaluation by a panel of academic healthcare professionals, including a professor in psychiatry and a clinical psychiatrist actively practicing in the field, both of whom are acknowledged as authors in the paper.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The paper's findings call into question previous claims that ChatGPT has promise for providing accurate and comprehensive medical information and has potential to provide first aid to young adults struggling with mental health issues.

    “Overall, findings call into question previous claims that ChatGPT “has promise for providing accurate and comprehensive medical information” (18), and has “potential to provide first aid to young adults struggling with mental health issues” (19).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [18] Johnson, 2023, Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the chat-GPT Model., Res Sq. source

      Not checked in the free run.

    • cites [19] Aminah, 2023, Considering ChatGPT to be the first aid for young adults on mental health issues., J Public Health. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Both insomnia and depression commonly occur in patients with SLE and appear to be related to the disease activity itself.

    “as both insomnia and depression commonly occur in patients with SLE, and appear to be related to the disease activity itself (39, 40).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [39] Palagini, 2014, Sleep disorders and systemic lupus erythematosus., Lupus. source

      Not checked in the free run.

    • cites [40] Palagini, 2013, Depression and systemic lupus erythematosus: a systematic review., Lupus. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Information provided by ChatGPT for patient C's mental health queries is not only limited but can be misleading and endanger lives.

    “Therefore, information provided by ChatGPT for patient C’s mental health queries is not only limited but can be misleading and endanger lives.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Insomnia is a core secondary symptom of depression in the DSM diagnostic criteria of major depressive disorder.

    “Insomnia is a core secondary symptom of depression in the Diagnostic and Statistical Manual of Mental Disorders diagnostic criteria of major depressive disorder (20).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] 2013, Diagnostic and Statistical Manual of Mental Disorders: DSM-5.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    This advice would be helpful for a majority of young people struggling with insomnia and potentially limit risks of self-medication for sleep, which is a common practice among students.

    “This advice would be helpful for a majority of young people struggling with insomnia, and potentially limit risks of self-medication for sleep, which is a common practice among students (33).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] Goodhines, 2019, Self-medication for sleep in college students: concurrent and prospective associations with sleep and alcohol behavior., Behav Sleep Med. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Routine screening for depression is recommended by the American College of Obstetricians and Gynecologists for all women during the perinatal period.

    “Routine screening for depression is recommended by the American College of Obstetricians and Gynecologists for all women during the perinatal period (49).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49] 2015, The American College of Obstetricians and Gynecologists Committee opinion no. 630. Screening for perinatal depression., Obstet Gynecol. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    To constitute a diagnosis, a clinician would screen for depression using a screening test such as the Edinburgh Postnatal Depression Scale, followed by additional clinical evaluation such as assessment for manic or psychotic features, and assess suicidal and infanticide risks.

    “To constitute the diagnosis, a clinician would screen for depression using a screening test (e.g., the Edinburgh Postnatal Depression Scale (46), followed by additional clinical evaluation [e.g., assessment for manic or psychotic features (47)]. Another important objective of the clinical evaluation is to assess suicidal and infanticide risks (48).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [46] Cox, 1987, Detection of postnatal depression: development of the 10-item Edinburgh Postnatal Depression Scale., Br J Psychiatry. source

      Not checked in the free run.

    • cites [47] Sharma, 2012, Postpartum depression: diagnostic and treatment issues., J Obstet Gynaecol Can. source

      Not checked in the free run.

    • cites [48] Lakkis, 2015, Screening instruments for depression in primary care: a concise review for clinicians., Postgrad Med. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Insomnia and depression share pathophysiological mechanisms.

    “suggesting shared pathophysiological mechanisms between the two entities (36).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [36] Riemann, 2001, Sleep and depression–results from psychobiological studies: an overview., Biol Psychol. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    A minimum level of medical training is required to identify any possible errors and proceed with caution when utilizing ChatGPT suggestions.

    “A minimum level of medical training is thus required to identify any possible errors and proceed with caution when utilizing ChatGPT suggestions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The development of the patient scenarios was based on typical presentations in psychiatric settings and informed by a thorough literature review to ensure their relevance and diversity.

    “The development of these scenarios was based on typical presentations in psychiatric settings, informed by a thorough literature review to ensure their relevance and diversity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The scope of the present conclusions is limited because the study was based on a limited number of scenarios and mental health problems, and it may not apply to other clinical situations.

    “First, the scope of the present conclusions is limited as the study t was based on a limited number of scenarios and mental health problems, as such, it may not apply to other clinical situations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    As clinical complexity increased, the information and recommendations generated by ChatGPT became inappropriate, even dangerous, and the limitations of the program became more glaring.

    “However, as clinical complexity increased, the information and recommendations generated by ChatGPT became inappropriate, even dangerous; and the limitations of the program became more glaring.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    ChatGPT provided fragmented management, taking into consideration each symptom separately, without any holistic critical thinking.

    “ChatGPT provided fragmented management, taking into consideration each symptom separately, without any holistic critical thinking.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductreasoning

    Different reviewers may offer diverse interpretations of the results.

    “While we have made every effort to ensure objectivity, it is crucial to acknowledge that different reviewers may offer diverse interpretations of the results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Findings may not be generalizable to non-ChatGPT AI models.

    “Additionally, findings may not be generalizable to non-ChatGPT AI models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The absence of real-time patient feedback may hinder understanding of the patient experience and their perception of the effectiveness and appropriateness of the AI system.

    “Finally, the absence of real-time patient feedback may hinder our understanding of the patient experience and their perception of the effectiveness and appropriateness of the AI system.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    These findings are in agreement with previous research suggesting a potential limitation of ChatGPT in handling complex medical queries.

    “These findings are in agreement with those of previous research, suggesting a potential limitation of ChatGPT in handling complex medical queries (18).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] Johnson, 2023, Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the chat-GPT Model., Res Sq. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The interpretation of the findings in this study contains a certain degree of subjectivity, given its reliance on the authors' experience with advanced language models.

    “We would like to underscore that the interpretation of the findings in this study contains a certain degree of subjectivity, given its reliance on the authors’ experience with advanced language models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    In case A, the recommendations were relatively appropriate, albeit non-specific, and could potentially be beneficial for both users and clinicians.

    “In case A, the recommendations were relatively appropriate (albeit non-specific), and could potentially be beneficial for both users and clinicians.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Inquiries chosen by authors were limited to assessment and treatment of each case's conditions, which might have influenced ChatGPT interactions in a manner that precludes fruitful results.

    “Furthermore, inquiries chosen by authors were limited to assessment and treatment of each of the cases’ conditions, which might have influenced ChatGPT interactions in a manner that precludes fruitful results.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    There is still a significant chasm preventing ChatGPT from being used safely and optimally in clinical practice without clinical supervision.

    “In conclusion, there is still a significant chasm preventing ChatGPT from being used safely and optimally in clinical practice, without clinical supervision.”
    Quote in the paperSubstantial claim
53 of 53

  • The paper relies on this without support.

    The use of AI language tools in the field of psychiatry remains relatively unexplored.

    “However, despite all the technological advancements, the use of AI language tools in the field of psychiatry remains relatively unexplored.”

  • The paper relies on this without support.

    The potential application of ChatGPT in mental health care is still unexplored.

    “The potential application of ChatGPT in mental health care still unexplored.”

  • The paper relies on this without support.

    No previous study has specifically investigated how ChatGPT can be applied to certain mental health issues or used with particular groups of people who need mental health support.

    “To the best of the author’s knowledge, no previous study has specifically investigated how this technology can be applied to certain mental health issues or used with particular groups of people who need mental health support.”

  • The paper relies on this without support.

    There is a gap in understanding how to effectively use these models in actual clinical settings.

    “This lack of research leaves us with a gap in understanding how to effectively use these models in actual clinical settings.”

  • The paper relies on this without support.

    In this clinical situation, a clinician would ask further questions, seek clarification, and request further information to reach an understanding of the patient's current condition before identifying clinically appropriate investigations and treatments.

    “In this clinical situation, a clinician would ask further questions, seek clarification, and request further information to reach an understanding of the patient’s current condition before identifying the clinically appropriate investigations and treatments.”

  • The paper relies on this without support.

    Failure to consider and rule out depression is a clinical error that could have malignant consequences, such as suicide risk.

    “Failure to consider and rule out depression is a clinical error that could have malignant consequences (e.g., suicide risk).”

  • The paper relies on this without support.

    Complex clinical situations are often managed by using care providers' expertise and clinical experience.

    “Complex clinical situations are often managed by using care providers’ expertise and clinical experience.”

  • The paper relies on this without support.

    Complex clinical cases remain unpublished and not readily available in existing medical literature for ChatGPT.

    “Such cases remain unpublished and not readily available in existing medical literature for ChatGPT.”

  • The paper relies on this without support.

    ChatGPT only relies on pre-existing knowledge.

    “Because ChatGPT only relies on pre-existing knowledge, further efforts should be made to incorporate a wide variety of reliable sources of medical information”
9 of 9

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Patient C's postpartum timing differs between presentation and discussion

    Minor

    Patient C's presentation says she gave birth one month ago, but the discussion describes her as 15 days postpartum. The difference affects the discussion's characterization of when her symptoms began.

  • Discussion misidentifies the first recommendation for patient C

    Minor

    The discussion says ChatGPT started with sleep hygiene and lifestyle changes, whereas Table 4 lists medical evaluation and management first and sleep hygiene second.

2 of 2

The citation lists haven’t been fetched yet

References 59 · Cited by 125

Research works printed but not linked: 24

  1. Sayers, 2001, The World Health Report 2001–Mental health: new understanding, new hope., Bull World Health Organ., 79
  2. Silverman, 2016, Artificial Intelligence and Human Behavior Modeling and Simulation for Mental Health Conditions., 163
  3. van Heerden, 2023, Global mental health services and the impact of artificial intelligence–powered large language models., JAMA Psychiatry., 80, 662, 10.1001/jamapsychiatry.2023.0195 · DOI
  4. Dergaa, 2023, From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing., Biol Sport., 40, 615, 10.5114/biolsport.2023.116032 · DOI
  5. Dergaa, 2023, Towards responsible research: examining the need for preprint policy reassessment in the era of artificial intelligence., EXCLI J., 22, 686, 10.17179/excli2022-3662 · DOI
  6. Brown, 2020, Language models are few-shot learners., Adv Neural Inform Proc Syst., 33, 1877
  7. 2013, Diagnostic and Statistical Manual of Mental Disorders: DSM-5.
  8. 2014, International Classification of Sleep Disorders—Third Edition (ICSD-3)., 2313
  9. Morin, 2011, Prevalence of insomnia and its treatment in Canada., Can J Psychiatry., 56, 540, 10.1177/070674371105600904 · DOI
  10. Bjorvatn, 2016, High prevalence of insomnia and hypnotic use in patients visiting their general practitioner., Family Pract., 33, 205, 10.1093/fampra/cmv105 · DOI
  11. Bartlett, 2008, Predictors of primary medical care consultation for sleep disorders., Sleep Med., 9, 857, 10.1016/j.sleep.2007.09.008 · DOI
  12. Ogeil, 2020, Clinician and patient barriers to the recognition of insomnia in family practice: a narrative summary of reported literature analysed using the theoretical domains framework., BMC Family Pract., 21, 10.1186/s12875-020-01101-4 · DOI
  13. Twa, 2016, Evidence-based clinical practice: asking focused questions (PICO)., Optom Vis Sci., 93, 1187, 10.1097/OPX.0000000000000929 · DOI
  14. Rios, 2019, Comparative effectiveness and safety of pharmacological and non-pharmacological interventions for insomnia: an overview of reviews., Syst Rev., 8, 10.1186/s13643-019-1191-x · DOI
  15. Goodhines, 2019, Self-medication for sleep in college students: concurrent and prospective associations with sleep and alcohol behavior., Behav Sleep Med., 17, 327, 10.1080/15402002.2017.1379277 · DOI
  16. Malangu, 2012, Drugs Inducing Insomnia as an Adverse Effect, Can’t Sleep., 23
  17. Kusljic, 2016, Corticosteroid-induced psychiatric disturbances: it is time for pharmacists to take notice., Res Social Adm Pharm., 12, 355, 10.1016/j.sapharm.2015.05.005 · DOI
  18. Palagini, 2014, Sleep disorders and systemic lupus erythematosus., Lupus., 23, 115, 10.1177/0961203313503982 · DOI
  19. Palagini, 2013, Depression and systemic lupus erythematosus: a systematic review., Lupus., 22, 409, 10.1177/0961203313477903 · DOI
  20. Sharma, 2017, Bipolar postpartum depression: an update and recommendations., J Affect Disord., 219, 105, 10.1016/j.jad.2017.05.044 · DOI
  21. Sharma, 2012, Postpartum depression: diagnostic and treatment issues., J Obstet Gynaecol Can., 34, 436, 10.1016/S1701-2163(16)35201-9 · DOI
  22. Lakkis, 2015, Screening instruments for depression in primary care: a concise review for clinicians., Postgrad Med., 127, 99, 10.1080/00325481.2015.993944 · DOI
  23. 2015, The American College of Obstetricians and Gynecologists Committee opinion no. 630. Screening for perinatal depression., Obstet Gynecol., 125, 1268, 10.1097/01.AOG.0000465192.34779.dc · DOI
  24. Mughal, 2022, Postpartum Depression.

Authors
Dergaa, I., Fekih-Romdhane, F., Hallit, S., Loch, A. A., Glenn, J. M., Fessi, M. S., Ben Aissa, M., Souissi, N., Guelmami, N., Swed, S., El Omri, A., Bragazzi, N. L., & Ben Saad, H.
Affiliations
  • University of Sfax (TN)
  • Effat University (SA)
  • Conselho Nacional de Desenvolvimento Científico e Tecnológico (BR)
  • Neurotrack Technologies (United States) (US)
  • University of Genoa (IT)
  • University of Aleppo (SY)
  • York University (CA)
Corresponding author
Dergaa, Ismail
ORCID
Venue
Frontiers in Psychiatry
Publisher
Frontiers Media SA
Published
2024-01-04
Volume
14
Pages
1277756-1277756
ISSN
1664-0640 (ISSN-L)
DOI
10.3389/fpsyt.2023.1277756
PMC
PMC10794665
OpenAlex
W4390585437
Cited by
125
References
59
Citations by year
2026: 40 · 2025: 62 · 2024: 22 · 2023: 1
FWCI
11.59
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Psychiatry (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Qatar National Library
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
ChatGPT, large language models, artificial intelligence in psychiatry, mental health assessment, clinical decision support, simulated patient scenarios, sleep disorders, treatment guidelines, clinical judgment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review