Regulating AI in Mental Health: Ethics of Care Perspective
Open accessPublished version of a preprint
Abstract
This article contends that the responsible artificial intelligence (AI) approach—which is the dominant ethics approach ruling most regulatory and ethical guidance—falls short because it overlooks the impact of AI on human relationships. Focusing only on responsible AI principles reinforces a narrow concept of accountability and responsibility of companies developing AI. This article proposes that applying the ethics of care approach to AI regulation can offer a more comprehensive regulatory and ethical framework that addresses AI’s impact on human relationships. This dual approach is essential for the effective regulation of AI in the domain of mental health care. The article delves into the emergence of the new “therapeutic” area facilitated by AI-based bots, which operate without a therapist. The article highlights the difficulties involved, mainly the absence of a defined duty of care toward users, and shows how implementing ethics of care can establish clear responsibilities for developers. It also sheds light on the potential for emotional manipulation and the risks involved. In conclusion, the article proposes a series of considerations grounded in the ethics of care for the developmental process of AI-powered therapeutic tools.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Manipulation by LLM-CBs may occur based on users' vulnerabilities and negative thought patterns that users shared beforehand.
“Manipulation may occur based on users’ vulnerabilities and negative thought patterns that users shared beforehand39,49.”
From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [39]
The “Manipulation and Vulnerability” section says emotional AI may manipulate people by exploiting their vulnerabilities and identifies a concern about AI-chatbot manipulation. The “Affective Computing and Emotional AI” section discusses chatbots interpreting thoughts and feelings, but our copy does not say manipulation is based on negative thought patterns users shared beforehand. No page numbers are available in our copy. (LLM, conf=0.85)
Experimental — accuracy not yet measuredData security concerns stem from security risks relating to sensitive health data provided by users or by clinicians.
“Data security concerns (mentioned by 13 articles) stem from security risks relating to sensitive health data provided by users or by clinicians36–39,44,47,50,51,55.”
From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [39]
The Privacy and Safety and Security sections (pages unavailable) link sensitive medical data to stricter security standards and discuss unauthorized breaches. The confidentiality section also describes sensitive information recorded by mental health apps. These passages support the concern about users’ health data, but do not specifically say that the data at risk may be provided by clinicians. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382 · see citing works
Fjeld, J., Achten, N., Hilligoss, H., Nagy, A., & Srikumar, M. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3518482 · Cited by 998 · cited 2× · see citing works
“Most AI regulatory documents and guidance are based on common principles [13], which are referred to as “responsible AI.””
BackgroundLi, H., Zhang, R., Lee, Y.-C., Kraut, R. E., & Mohr, D. C. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00979-5 · Cited by 598 · cited 1× · see citing works
“Indeed, some applications have already been criticized as lacking clinical validation [5].”
BackgroundOnline manipulation: hidden influences in a digital world
Susser, D., Roessler, B., & Nissenbaum, H. F. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3306006 · Cited by 341 · see citing works
ChatGPT outperforms humans in emotional awareness evaluations
Elyoseph, Z., Hadar-Shoval, D., Asraf, K., & Lvovsky, M. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1199058 · Cited by 324 · see citing works
Vulnerability and Inevitable Inequality
Albertson Fineman, M. (2017) · Oslo Law Review · doi:10.18261/issn.2387-3299-2017-03-02 · Cited by 227 · cited 1× · see citing works
“According to Fineman [49], vulnerability extends beyond specific individuals or groups known as “vulnerable populations.””
BackgroundArtificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril
Series, T. L. H. S., & National Academy of Medicine (2019) · doi:10.17226/27111 · Cited by 227 · see citing works
Trustworthy Augmented Intelligence in Health Care
Crigger, E., Reinbold, K., Hanson, C., Kao, A., Blake, K., & Irons, M. (2022) · Journal of Medical Systems · doi:10.1007/s10916-021-01790-z · Cited by 132 · see citing works
Flagged first, then by citations
Research works printed but not linked: 21
- [1]Koutsouleris N Hauser TU Skvortsova V de Choudhury M From promise to practice: towards the realisation of AI-informed mental health care Lancet Digit Health 2022 4 11 e829 e840 10.1016/S2589-7500(22)00153-4 36229346 S2589-7500(22)00153-4 36229346 · DOI · Paper page
- [2]Lee EE Torous J de Choudhury M Depp CA Graham SA Kim H Paulus MP Krystal JH Jeste DV Artificial intelligence for mental health care: clinical applications, barriers, facilitators, and artificial wisdom Biol Psychiatry Cogn Neurosci Neuroimaging 2021 6 9 856 864 10.1016/j.bpsc.2021.02.001 33571718 S2451-9022(21)00046-X 33571718 PMC8349367 · DOI
- [3]Tal A Elyoseph Z Haber Y Angert T Gur T Simon T Asman O The artificial third: utilizing ChatGPT in mental health Am J Bioeth 2023 23 10 74 77 10.1080/15265161.2023.2250297 37812102 37812102 · DOI
- [6]Wellner G Mykhailov D Caring in an algorithmic world: ethical perspectives for designers and developers in building AI algorithms to fight fake news Sci Eng Ethics 2023 08 09 29 4 30 10.1007/s11948-023-00450-4 37555995 10.1007/s11948-023-00450-4 37555995 · DOI
- [7]Yew GCK Trust in and ethical design of Carebots: the case for ethics of care Int J Soc Robot 2021 13 4 629 645 10.1007/s12369-020-00653-w 32837630 653 32837630 PMC7245509 · DOI
- [8]Cohn J In A different code: artificial intelligence and the ethics of care Int Rev Inf Ethics 2020 28 1 7 10.29173/irie383 · DOI
- [10]Noddings N Caring: A Feminine Approach to Ethics and Moral Education 1984 California University of California Press
- [11]Held V The Ethics of Care: Personal, Political, and Global. 2nd ed 2006 Oxford, New York Oxford University Press
- [12]Kittay EF Love's Labor: Essays on Women, Equality and Dependency 1999 New York Routledge
- [20]Beauchamp TL Childress JF Principles of Biomedical Ethics 2019 New York Oxford University Press
- [34]West RL Jurisprudence and gender U Chi L Rev 1988 55 1 1 10.2307/1599769 · DOI
- [35]Gilligan C In A Different Voice: Psychological Theory and Women's Development 1982 London, England Harvard University Press
- [36]Tronto J Moral Boundaries: A Political Argument for an Ethic of Care 1993 New York Routledge 63 64
- [37]Tronto J Caring Democracy: Markets, Equality, and Justice 2013 New York NYU Press
- [38]Engster D Hamington M Care Ethics and Political Theory 2015 New York Oxford University Press
- [39]Sedlakova J Trachsel M Conversational artificial intelligence in psychotherapy: a new therapeutic tool or agent? Am J Bioeth 2023 23 5 4 13 10.5167/uzh-218039 35362368 35362368 · DOI
- [43]Stark L Hoey J The ethics of emotion in artificial intelligence systems 2021 FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency 2021 March 01 Virtual Event, Canada
- [45]Hadar-Shoval D Elyoseph Z Lvovsky M The plasticity of ChatGPT's mentalizing abilities: personalization for personality structures Front Psychiatry 2023 14 1234397 10.3389/fpsyt.2023.1234397 37720897 37720897 PMC10503434 · DOI
- [48]Cohen T Regulating manipulative artificial intelligence SCRIPTed: J Law Technol Soc 2023 20 1 203 242 10.2966/scrip.200123.203 · DOI
- [50]Bielby P Not ‘us’ and ‘them’: towards a normative legal theory of mental health vulnerability Int J Law Context 2018 15 1 1 17 10.1017/s1744552318000149 · DOI
- [52]Elyoseph Z Gur T Haber Y Simon T Angert T Navon Y Tal A Asman O An ethical perspective on the democratization of mental health with generative artificial intelligence JMIR Mental Health (forthcoming) 2024 10.2196/preprints.58011 PMC11500620 39417792 · DOI
Cited by
Sorted by citations
Does the Digital Therapeutic Alliance Exist? Integrative Review
Malouin-Lachance, A., Capolupo, J., Laplante, C., & Hudon, A. (2025) · JMIR Mental Health · doi:10.2196/69294 · Cited by 103 · see citing works
A scoping review on the mental health harms of LLM-based chatbots
Diel, A., Torous, J., Cuijpers, P., Kleesiek, J., Nensa, F., Weber, N., Faust, F., Lalgi, T. J., Mellis, F. S., Teufel, M., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03054-x · Cited by 4 · see citing works
Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaboration
Xu, J., Wu, X., Zhang, Z., Xu, Y., Yang, Y., Zhu, J., Xiao, L., Wu, W., & He, L. (2026) · arXiv · doi:10.48550/arxiv.2606.18259
“High-risk domains demand clinical triage with human-in-the-loop escalation for suicidal ideation, self-harm or crisis; audit trails recording model version, confidence display and the accountable actor; and vulnerability-aware defaults [9, 94, 95].”
- Authors
- Tavory, T.
- Affiliations
- Bar-Ilan University (IL)
- Corresponding author
- Tavory, Tamar
- ORCID
- Tavory, Tamar
- Venue
- JMIR Mental Health
- Publisher
- JMIR Publications Inc.
- Published
- 2024-09-19
- Volume
- 11
- Pages
- e58493
- ISSN
- 2368-7959 (ISSN-L)
- PMC
- PMC11450345
- OpenAlex
- W4400840784
- Cited by
- 108
- References
- 8
- Citations by year
- 2026: 64 · 2025: 41 · 2024: 3
- FWCI
- 44.76
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Mental Health (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- AI regulation, care ethics, mental health services, responsible AI, accountability, AI chatbots, duty of care, emotional manipulation, developer responsibility, interpersonal relationships
- MeSH terms
- Artificial Intelligence, Humans, Mental Health, Mental Health Services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
