CoChat
Sign Up
Cited by 133Digital Mental Health InterventionsMental Health Treatment and AccessArtificial Intelligence in Healthcare and Education

Closing the accessibility gap to mental health treatment with a personalized self-referral chatbot

Open access
Habicht, Johanna, Viswanathan, Sruthi, Carrington, Ben, Hauser, Tobias U., Harper, Ross, Rollwage, Max·2024·Nature Medicine
DOI 10.1038/s41591-023-02766-x

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Inequality in treatment access is a pressing issue in most healthcare systems across many medical disciplines. In mental healthcare, reduced treatment access for minorities is ubiquitous but remedies are sparse. Here we demonstrate that digital tools can reduce the accessibility gap by addressing several key barriers. In a multisite observational study of 129,400 patients within England's NHS services, we evaluated the impact of a personalized artificial intelligence-enabled self-referral chatbot on patient referral volume and diversity in ethnicity, gender and sexual orientation. We found that services that used this digital solution identified substantially increased referrals (15% increase versus 6% increase in control services). Critically, this increase was particularly pronounced in minorities, such as nonbinary (179% increase) and ethnic minority individuals (29% increase). Using natural language processing to analyze qualitative feedback from 42,332 individuals, we found that the chatbot's human-free nature and the patients' self-realization of their need for treatment were potential drivers for the observed improvement in the diversity of access. This provides strong evidence that digital tools may help overcome the pervasive inequality in mental healthcare.

AI snapshot

Written from the paper’s full text.

TL;DR

In a multi-site observational study of ∼129,400 patients across 28 NHS Talking Therapies services, an AI-enabled self-referral chatbot was associated with a 15% increase in total referrals versus a 6% increase in matched control services. The tool disproportionately increased referrals among minority groups, and NLP analysis of 42,332 feedback entries suggested human-free interaction and self-realisation as key mechanisms.

Key findings

  • Services using the AI-enabled self-referral tool had a 15% increase in total referrals, compared with 6% in matched services using traditional self-referral methods; the difference was significant (χ²(1)=86.3, p<0.0001).
  • Referral increases were larger for minority groups: non-binary 235%, bisexual 30%, and ethnic minority 31%, versus female 18%, male 16%, heterosexual 14%, and White 15%. Odds ratios were non-binary vs female 2.83 (CI 2.264–3.545), non-binary vs male 2.90 (CI 2.314–3.629), bisexual vs heterosexual 1.14 (CI 1.074–1.219), ethnic minorities vs White 1.14 (CI 1.081–1.202), Asian/Asian British 1.21 (CI 1.116–1.318), and Black/Black British 1.24 (CI 1.108–1.378).
  • The tool did not negatively affect access to assessments: a two-way ANOVA found no significant interaction between time period and tool usage (F(1,13)=0.09, p=0.764).
  • NLP classification of 42,332 feedback entries found 89% positive, 7% neutral, and 4% negative feedback. Gender minority and bisexual individuals mentioned the tool’s human-free nature more than majority groups (χ²(1)=22.7, p<0.0001; χ²(1)=62.4, p<0.0001), while Asian and Black ethnic groups mentioned self-realisation more than White individuals (χ²(1)=46.6, p<0.0001) and convenience less (χ²(1)=24.2, p<0.0001).

Methods

  • Multi-site real-world retrospective observational study using routine NHS Talking Therapies data from 28 services: 14 implemented the AI-enabled self-referral tool (Limbic Access) and 14 matched controls did not; ∼129,400 patients were included, with three-month pre- and post-implementation periods and staggered service launch times.
  • Services were matched using Euclidean distance on two normalized features from the pre-implementation period; total referrals were compared with Chi-squared tests, demographic changes with percentage change and logistic regression odds ratios, and a self-referral-only sensitivity analysis was conducted.
  • Qualitative feedback from September 2021 to March 2023: 157,416 patients used the tool, 29% (N=46,166) gave free-text feedback, and after excluding responses under 10 characters, 42,332 entries remained. Reflexive thematic analysis identified nine themes; SBERT, PCA, and multi-class logistic regression with 100-fold cross-validation achieved a micro-average F1 score of 0.64; Chi-squared tests with Bonferroni correction compared theme proportions.

Limitations

  • The paper states possible cultural differences between NHS Talking Therapies services and that services adopting the tool may be inherently more open to digital health innovation, which could influence referral numbers; it notes implementation was not part of wider digital innovation programs and the increase was tightly linked to launch.
  • Evident from the text: the observational design cannot establish causality, and matching was based only on two normalized pre-period referral features, leaving potential residual confounding.
  • Evident from the text: the NLP analysis covers only the 29% of users who provided free-text feedback and excludes entries under 10 characters, and the classification model had moderate performance (micro-average F1=0.64), so feedback findings may not represent all users.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41591-023-02766-x

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

0 of 31 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    There was a 31% increase in referrals for ethnic minorities.

    “We identified a 31% increase in referrals for ethnic minorities, whereas the increase for the White group was 15% (Figure 3C), suggesting that different ethnic groups were more represented after the implementation of the AI-enabled self-referral tool.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The data used are recorded in routine practice by all NHS Talking Therapies services and reported by NHS Digital.
  • Findingown data

    There was an increase for Asian and Asian British groups compared to White individuals (OR=1.21, CI=[1.116,1.318], p<0.0001).

    “We particularly saw an increase for Asian and Asian British groups (odds-ratio =1.21, CI = [1.116,1.318],p < 0.0001) and Black and Black British groups (odds-ratio = 1.24,CI = [1.108,1.378],p < 0.0001) compared to White individuals (Figure 3D).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The data used are recorded in routine practice by all NHS Talking Therapies services and reported by NHS Digital.
  • Findingown data

    A two-way ANOVA revealed no significant interaction between time period and tool usage on the percentage of people assessed (F(1, 13) = 0.09, p = 0.764).

    “A two-way ANOVA revealed no significant interaction between the time period and tool usage (F(1, 13) = 0.09, p = 0.764).”
    Quote in the paperSubstantial claim
  • Findingown data

    There were no differences in the number of referrals for individuals from other ethnic groups compared to White individuals (OR=1.08, CI=[0.913,1.272], p=0.377).

    “We did not see any differences in the number of referrals for individuals from other ethnic groups (odds-ratio =1.08, CI = [0.913,1.272],p = 0.377) and Mixed ethnic group (odds-ratio =1.01, CI = [0.917,1.107],p = 0.877) compared to White individuals.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The data used are recorded in routine practice by all NHS Talking Therapies services and reported by NHS Digital.
  • Findingown data

    Referrals by people whose sexual orientation was gay/lesbian increased by 19%.

    “We found that referrals by people whose sexual orientation was bi-sexual increased by 30%, referrals by people whose sexual orientation is gay/lesbian increased by 19%, whereas referrals from heterosexuals increased by 14% (Figure 3B).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The data used are recorded in routine practice by all NHS Talking Therapies services and reported by NHS Digital.
  • May combine two claims. 29
  • Classified as probably not a substantial claim. 15

  • May combine two claims.

    Findingown data

    Overall, 89% of the free text feedback was classified as positive, 7% neutral and 4% negative.

    “Overall, 89% of the free text feedback was classified as positive, 7% neutral and 4% negative.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The themes following convenience were hope of having taken the first step to access support, absence of human involvement reducing stigma and judgment, and increased self-realization of need for support.

    “This was followed by the hope of having taken the first step to access support, the absence of human involvement reducing stigma and judgment and increased self-realization that they might suffer from a mental health condition and need support caused by the self-referral process.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Services that used the AI-enabled self-referral tool showed a 15% increase in referrals, whereas the matched services that did not use the tool showed a 6% increase.

    “Specifically, services that used the AI-enabled self-referral tool showed a 15% increase in referrals, whereas the matched services that did not use the tool showed a 6% increase.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    At the time of writing, the self-referral tool has facilitated the referral of approximately 157,000 patients, meaning approximately 14,000 additional patients have found their way into treatment owing to this technical solution.

    “At the time of writing, this self-referral tool has facilitated the referral of ∼157,000 patients, which means that ∼14,000 additional patients have found their way into treatment owing to this technical solution.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A positive percentage change in referrals was observed for people with all sexual orientations, with the highest percentage increase for individuals who are bisexual.

    “A positive percentage change in referrals was observed for people with all sexual orientations, with the highest percentage increase in referrals for individuals who are bi-sexual.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A positive percentage change in referrals was observed for all genders, with the highest percentage increase for individuals who identify as non-binary.

    “A positive percentage change in referrals was observed for all genders, with the highest percentage increase in referrals for individuals who identify as non-binary.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A positive percentage change in referrals was observed for all ethnic groups, with the highest percentage increase for ethnic minorities.

    “A positive percentage change in referrals was observed for all ethnic groups, with the highest percentage increase in referrals for ethnic minorities.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    The AI-enabled self-referral tool automatically opens on the service's website, eliminating the need for patients to search for contact information or referral form.

    “Furthermore, the AI-enabled self-referral tool automatically opens on the service's website, eliminating the need for patients to search for contact information or referral form.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The supply of mental healthcare is not matching the demand as mental health services are widely underfunded and understaffed.

    “Firstly, the supply of mental healthcare is not matching the demand as mental health services are widely underfunded and understaffed [World Health Organization, 2021b, Mahase, 2020].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites World Health Organization, 2021b

      Not checked in the free run.

    • cites Mahase, 2020 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Digital technologies and AI have been proposed as potential solutions to these challenges.

    “Digital technologies and Artificial Intelligence (AI) have been proposed as potential solutions to these challenges [Rudd and Beidas, 2020, Taylor et al., 2020, Kola et al., 2021].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Rudd and Beidas, 2020 source

      Not checked in the free run.

    • cites Taylor et al., 2020 source

      Not checked in the free run.

    • cites Kola et al., 2021 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Existing solutions may not be optimised or digitally streamlined, leading to lower completion rates and limited access to services.

    “However, existing solutions may not be optimised or digitally streamlined, leading to lower completion rates and limited access to services [NHS Digital, 2019].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites NHS Digital, 2019

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Digital technologies can remove some of the barriers to accessing mental health services, and thus increase accessibility.

    “This suggests that digital technologies can remove some of the barriers to accessing mental health services, and thus increase accessibility.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Not every individual experiencing mental health problems seeks help, or they delay seeking it for years.

    “Secondly, not every individual experiencing mental health problems seeks help, or they delay seeking it for years [Wang et al., 2007, Department of Health andSocial Care, 2023], resulting in unmet needs and inadequate support at the right time.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Wang et al., 2007

      Not checked in the free run.

    • cites Department of Health and Social Care, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Digital solutions might be especially well-positioned to help individuals who are seeking support to overcome barriers, for example by offering more flexibility and reducing stigma.

    “Excitingly, digital solutions might be especially well-positioned to help individuals who are seeking support to overcome barriers, for example by offering more flexibility and reducing stigma [Rodríguez-Rivas et al., 2022].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rodríguez-Rivas et al., 2022 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    As many as two-thirds of individuals with depression do not receive the necessary care.

    “Notably, a significant proportion of mental health problems remain undiagnosed and untreated, with estimates suggesting that as many as two-thirds of individuals with depression do not receive the necessary care [Moitra et al., 2022].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Moitra et al., 2022 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Services were matched using Euclidean distance based on two normalized features (total referrals in pre-period).

    “In order to match services, we calculated the Euclidean distance based on these two (normalized) features and took the service with the lowest distance to the NHS Talking Therapies service using the AI-enabled self-referral tool, where a low distance implies a high degree of similarity.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Bonferroni correction was used to account for multiple comparisons.

    “To account for multiple comparisons, we used Bonferroni correction.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    For minority groups, the tool's human-free nature and its ability to increase patients' awareness of their treatment needs were two significant drivers.

    “Additionally, for minority groups, we identified two significant drivers: the tool's human-free nature, which allowed patients to seek help without fear of being judged or discriminated against, and its ability to increase patients' awareness of their treatment needs.”
    • Also stated in: § Abstract · plain
      “We found that the tool's humanfree nature and its ability to improve the perceived need for treatment were the main drivers for improved diversity.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The feedback entries provided by patients are representative of the experiences of all patients who used the tool.

  • May combine two claims.

    Interpretationown data

    Different demographic groups might experience different advantages which might contribute to increased accessibility for these groups.

    “indicating that different groups might experience different advantages which might contribute to the increased accessibility for these groups.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The AI-enabled self-referral tool facilitates a general increase in referrals and improves the diversity of referrals.

    “In conclusion, these results suggest that the AI-enabled self-referral tool facilitates a general increase in referrals, as well as improves the diversity of referrals.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Initial codes were discussed with the larger group of researchers, and consensus was reached on the resulting themes.

    “Initial codes were discussed with the larger group of researchers, and a consensus was reached on the resulting themes (listed in the table below) after two meetings.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The analysis was repeated for only self-referrals to ensure total referrals did not increase due to other referral forms.

    “To ensure that the total number of referrals did not increase due to other forms of referrals other than self-referrals (e.g.GP referrals) increasing in the time period, we repeated the analysis for only self-referrals in the same services at the same time period (Supplementary Figure 1).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The NLP model's accuracy was evaluated through a 100-fold cross-validation approach.

    “We used a multi-class logistic regression model and evaluated its accuracy through a 100-fold cross-validation approach.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The NLP analysis excluded responses containing fewer than 10 characters to ensure only meaningful insights were included.

    “To ensure our analysis only included meaningful insights, we excluded any responses containing fewer than 10 characters, resulting in a final sample of 42,332 feedback entries.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    AI-enabled chatbots have the potential to increase accessibility to mental health services for all, and to alleviate barriers faced by disadvantaged populations.

    “These findings suggest that AI-enabled chatbots have the potential to increase accessibility to mental health services for all, and to alleviate barriers faced by disadvantaged populations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationcitation

    Our results support the notion that digital technologies can support reaching the SDGs and support more people in accessing the services they need.

    “Our results further support the notion that digital technologies can support reaching the SDGs [Lancet, 2021] and support more people in accessing the services they need.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lancet, 2021 source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductnone given

    Implementation of the AI-enabled self-referral tool was not part of wider digital innovation programs for any of the services and the increase in referrals was tightly linked to the launch of the AI-enabled self-referral tool.

    “Nevertheless, implementation of the AI-enabled self-referral tool was not part of wider digital innovation programs for any of the services and the increase in referrals was tightly linked to the launch of the AI-enabled self-referral tool.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Addressing these barriers with an AI-enabled self-referral tool has likely lowered the threshold for seeking help, resulting in higher referral rates.

    “Addressing these barriers with an AI-enabled self-referral tool has likely lowered the threshold for seeking help, resulting in higher referral rates as indicated by our data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Utilizing an AI-enabled self-referral tool could reduce barriers to access, resulting in more referrals and improved treatment efficacy and a decrease in the burden on individuals and society.

    “Utilizing an AI-enabled self-referral tool could reduce barriers to access, resulting in more referrals and improved treatment efficacy [Rollwage et al., 2022] and a decrease in the burden on individuals and society.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Rollwage et al., 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductreasoning

    Data was collected as part of a standard national data reporting process that undergoes quality checks, so there is no concern about missing data or inconsistent reporting.

    “As the data was collected as part of a standard national data reporting process that undergoes quality checks, there is no concern about missing data or that the data is reported in different ways.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    The staggered design of the self-referral tool implementation at different time points accounted for natural changes in referral numbers.

    “Furthermore, the staggered design of the self-referral tool implementation at different time points accounted for any natural changes in referral numbers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Some of these barriers might be especially well addressed by an AI-enabled self-referral tool's human-free aspect and the reduced stigma.

    “Some of these barriers might be especially well addressed by an AI-enabled self-referral tool's human-free aspect and the reduced stigma.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Ethnic minorities mainly mentioned how the referral process helped them to realise that they might be suffering from mental health problems or the need for treatment, which may reflect negative attitudes and lack of perceived need for treatment.

    “Ethnic minorities mainly mentioned how the referral process helped them to realise that they might be suffering from mental health problems or the need for treatment which may reflect the negative attitudes and lack of perceived need for treatment these individuals might face.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The findings demonstrate the potential of AI-enabled conversational tools to facilitate the self-referral process for mental health services and make it more accessible for different groups of people.

    “The findings of this research demonstrate the potential of AI-enabled conversational tools to facilitate the self-referral process for mental health services and make it more accessible for different groups of people.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Such solutions may remove these barriers and may be valued more by these minority groups, leading to a higher number of referrals.

    “Such solutions may remove these barriers and may be valued more by these minority groups, leading to a higher number of referrals.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Data was collected under real-world conditions from multiple services, providing a larger sample size and improving the generalisability of the results.

    “In addition, the data was collected under real-world conditions from multiple services, which provides a larger sample size and thereby improves the generalisability of the results.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Minority groups may face different barriers and value different aspects when seeking mental health treatment than majority groups.

    “In conclusion, our findings suggest that minority groups may face different barriers and value different aspects when seeking mental health treatment than majority groups.”
    • Also stated in: § 4 Discussion · hedged
      “Interestingly, we found that minority groups mention different themes in their feedback, suggesting that the barriers they face and their needs are different to majority groups.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Findings from previous research suggest that digital technologies may facilitate help-seeking for mental health and make it easier for people to disclose personal information.

    “Findings from previous research suggest that digital technologies may facilitate help-seeking for mental health [Johnson et al., 2022] and make it easier for people to disclose personal information [Te Pas et al., 2020, Bhakta et al., 2014].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Johnson et al., 2022 source

      Not checked in the free run.

    • cites Te Pas et al., 2020, Bhakta et al., 2014 source

      Not checked in the free run.

38 of 38

  • The paper relies on this without support.

    NHS Digital publicly available data undergoes quality checks and is comprehensive enough for high-quality analysis.

    “The publicly available data undergoes quality checks and is a comprehensive dataset that facilitates high-quality analysis.”

  • The paper relies on this without support.

    The routine questionnaires and screening questions have not been typically collected at the point of referral in NHS Talking Therapies, but rather during the first assessment on a phone call.

    “These routine questionnaires and screening questions have not been typically collected at the point of referral in NHS Talking Therapies, but rather during the first assessment on a phone call.”

  • The paper relies on this without support.

    The feedback entries provided by patients are representative of the experiences of all patients who used the tool.

    “From September 2021 to March 2023, a total of 157,416 patients used this novel self-referral approach, with 29% of these patients offering free-text feedback on their experience (N = 46,166).”

  • The paper relies on this without support.

    Mental health support is effective for many mental disorders.

    “Mental health support can reduce the burden and has been found to be highly effective for many mental disorders.”

  • The paper relies on this without support.

    Less is known about the marginal impact of AI technologies in supporting individuals of differing demographics seeking help.

    “less is known about the marginal impact of AI technologies in supporting individuals of differing demographics seeking help.”

  • The paper relies on this without support.

    The data used are recorded in routine practice by all NHS Talking Therapies services and reported by NHS Digital.

    “The data used in this study are recorded in routine practice by all NHS Talking Therapies services and are reported to the public monthly, quarterly, and annually by NHS Digital (NHS Digital).”

  • The paper relies on this without support.

    An increase in referrals can potentially lead to longer wait times for clinical assessment and may reduce the number of people who access these assessments.

    “An increase in referrals can potentially lead to longer wait times for clinical assessment and may reduce the number of people who access these assessments.”

  • The paper relies on this without support.

    The baseline increase in referrals in matched services is likely due to the national initiative to boost access across all NHS Talking Therapies services.

    “This baseline increase is likely to reflect the national initiative to boost access across all NHS Talking Therapies services.”

  • The paper relies on this without support.

    Non-binary, bisexual, and ethnic minority groups have been previously underrepresented in mental healthcare treatment.

    “These groups have been previously underrepresented in mental healthcare treatment”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A personalized self-referral chatbot in the UK, geared towards mental health and available to the general population, was able to increase referrals from 6% to 15%, with a particular impact on ethnic minorities (29% increase).

    “A personalized self-referral chatbot in the UK, geared towards mental health and available to the general population, was able to increase referrals from 6% to 15%, with a particular impact on ethnic minorities (29% increase) [131].”

    From Polycystic Ovary Syndrome and the Internet of Things: A Scoping Review (2024) · cites this paper as [131]

    Section 3.1 reports a 15% increase in referrals after the chatbot was introduced, compared with a 6% increase at matched services using other referral methods—not a rise from 6% to 15% within the chatbot services. The same section reports a 31% increase for ethnic minorities, rather than 29%. (LLM, conf=0.91)

    Experimental — accuracy not yet measured

  • Studies have shown that chatbots are able to generate quality and empathetic responses to patients and substantially increase referrals.

    “Studies have shown that chatbots are able to generate quality and empathetic responses to patients [130] and substantially increase referrals [131].”

    From Polycystic Ovary Syndrome and the Internet of Things: A Scoping Review (2024) · cites this paper as [131]

  • The duration of chatbot intervention ranged from 20 minutes to 3 months (mean 3.63, SD 1.74 wk).

    “The duration of chatbot intervention ranged from 20 minutes [59] to 3 months [56] (mean 3.63, SD 1.74 wk).”

    From Generative AI Mental Health Chatbots as Therapeutic Tools: Systematic Review and Meta-Analysis of Their Role in Reducing Mental Health Issues (2025) · cites this paper as [56]

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Not confirmedχ²(1) = 14.6reported p = .0036 · recomputed p < .001

    “χ 2 (1) = 14.6, p = .0036”

  • ConsistentF(1, 13) = 0.09reported p = .764 · recomputed p = .769

    “F(1, 13) = 0.09, p = .764”

  • Consistentχ²(1) = 30.8reported p < .0001 · recomputed p < .001

    “χ 2 (1) = 30.8,p < .0001”

3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Bhakta, R., Savin-Baden, M. & Tombs, G. Sharing secrets with robots? In Proc. EdMedia+ Innovate Learning (eds Viteli, J. & Leikomaa, M.) 2295–2301 (Association for the Advancement of Computing in Education, 2014).

Cited by

Sorted by citations

Related

Authors
Habicht, J., Viswanathan, S., Carrington, B., Hauser, T. U., Harper, R., & Rollwage, M.
Affiliations
University College London (GB)
Corresponding author
Habicht, Johanna
ORCID
Venue
Nature Medicine
Publisher
Springer Science and Business Media LLC
Published
2024-02
Volume
30
Issue
2
Pages
595-602
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
DOI
10.1038/s41591-023-02766-x
OpenAlex
W4391542646
Semantic Scholar
8f3fb98b9e03c9b4f2ad9d48ddb32f1873d61653
Cited by
132
References
49
Citations by year
2026: 63 · 2025: 46 · 2024: 22 · 2023: 1
FWCI
51.95
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion)
  • medRxiv (acceptedVersion)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Mental Health Treatment and Access
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
access to mental health care, artificial intelligence, health disparities, ethnic minorities, nonbinary patients, gender diversity, digital health tools, NHS mental health services, patient feedback
MeSH terms
Artificial Intelligence, Ethnicity, Female, Health Services Accessibility, Humans, Male, Mental Health, Minority Groups, Referral and Consultation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review