Technological folie à deux: Feedback Loops Between AI Chatbots and Mental Illness

“Initial training (pre-training) proceeds through a self-supervised procedure, whereby model parameters are updated in order to minimise the error (surprisal, or negative log likelihood) of next-token prediction in a vast textual training corpora 68–70 .”

“Here, the adaptation in question pertains not to what the model has learned during training (knowledge encoded in parameter weights), but to knowledge gleaned through the context of an individual conversation ( in-context learning 42,44 ).”

“While there is promising work attempting to overcome this opacity - from mechanistic interpretability 61 to examining “chains of thought” in reasoning models 62 - these are as yet far from proving any real-world guarantees on model behaviour 63–65 .”

“Humans are known to exhibit sensitivity and preference for information that supports existing beliefs ( confirmation bias 40 ) and engage in chains of thought that lead us to emotionally comforting conclusions ( motivated reasoning 41 ).”

“Humans are known to exhibit sensitivity and preference for information that supports existing beliefs ( confirmation bias 40 ) and engage in chains of thought that lead us to emotionally comforting conclusions ( motivated reasoning 41 ).”

“…can include: ● Reinforcement learning from human feedback (RLHF): which uses human ratings of LLM output quality to further train the base LLM 72,73 Supervised fine tuning (SFT): training the LLM on curated examples of good conversations 74 ● Constitutional AI: which uses AI-generated feedback…”

“In line with recent work, we argue that the psychological and mental health risks of chatbot use cannot be explained by a narrow consideration of chatbot limitations, such as falsehoods ( hallucinations, see Box 1 ), socially harmful biases, or failures in pragmatic language understanding 26–32 .”

“Sycophancy is a form of proxy failure referred to as reward hacking : a class of behaviours that maximise expected training rewards in a manner that subverts the values these rewards were meant to capture 11,49 .”

“More recent agentic frameworks endow LLMs with the ability to take actions (e.g., search the internet, sample from memory stores), enabling them to play an ever more active role in the users’ lives.”

“LLM output is generated primarily through autoregressive sampling from the encoded probability distribution, such that the sequence of emitted tokens has a high joint probability given the sum total of contextualising information 43,71 .”

“One reason pertains to documented cognitive biases governing how people with psychiatric symptoms integrate new information when updating beliefs 78–80 .”

“People with psychosis, for example, have been documented to form overly confident beliefs based on minimal evidence (“jumping to conclusions”) 81,82 , potentially indicating a tendency to overweight incoming sensory evidence in favour of prior beliefs 83 .”

“In line with recent work, we argue that the psychological and mental health risks of chatbot use cannot be explained by a narrow consideration of chatbot limitations, such as falsehoods ( hallucinations, see Box 1 ), socially harmful biases, or failures in pragmatic language understanding 26–32 .”

“Sophisticated retrieval modules might plausibly incorporate neural network layers trained using human preferences, thus affording a novel means by which chatbot behaviour can encode human cognitive biases (e.g., biased memory recall) 66,67 .”

“As discussed, post-training procedures guided by both human (RLHF) and LLM responses suffer from inadequate data coverage (reduced sample diversity 88,89 ) and carry an inherent risk of proxy failure.”

“…of LLM output quality to further train the base LLM 72,73 Supervised fine tuning (SFT): training the LLM on curated examples of good conversations 74 ● Constitutional AI: which uses AI-generated feedback based on constitutional principles to further train the base LLM 75 The final model is…”

“LLM output is generated primarily through autoregressive sampling from the encoded probability distribution, such that the sequence of emitted tokens has a high joint probability given the sum total of contextualising information 43,71 .”

“People with psychiatric diagnoses also experience increased rates of social isolation 84,85 .”

“More concerning still, edge-cases pointing to emergence of severe mental health crises have also begun to surface, including media reports of attempted homicide, suicide, and delusional thinking 7,23–25 .”

“People with psychiatric diagnoses also experience increased rates of social isolation 84,85 .”

“This is because, unlike real-world human interaction, chatbot interactions will not come burdened with the anxiety-provoking requirement to negotiate the needs and preferences of another.”

“This is because, even when generating false or otherwise inappropriate content, chatbots maintain striking linguistic coherence and confidence (formal vs functional linguistic competence) 31,47 , a property related to a training objective to generate plausible text completions 48 .”

“Two recent studies from OpenAI and MIT found that prolonged daily chatbot interactions were associated with increased loneliness, reduced socialisation, and greater emotional dependence.”

“LLM output is generated primarily through autoregressive sampling from the encoded probability distribution, such that the sequence of emitted tokens has a high joint probability given the sum total of contextualising information 43,71 .”

“Prior work has established that human judgements are liable to influence following interaction with biased (non-chatbot) AI systems 77 , and a recent human-chatbot study indicates that user mood ratings are influenced by the affective tone of chatbot responses 6 .”

“…context windows capable of retaining information over multiple conversations (with some frontier models already endowed with context windows >1 million tokens), customizable system prompts that allow users to instruct models with background knowledge and preferences, and external memory systems…”

“( This is in contrast to AI systems specifically designed for use in mental health contexts, where regulation is already in place 18,19 .)”

“While there is promising work attempting to overcome this opacity - from mechanistic interpretability 61 to examining “chains of thought” in reasoning models 62 - these are as yet far from proving any real-world guarantees on model behaviour 63–65 .”

“A smaller number still may use chatbots as de facto mental health resources, plugging the supply-demand gap between desire for psychotherapy and availability of psychotherapists 5,13 .”

“A smaller number still may use chatbots as de facto mental health resources, plugging the supply-demand gap between desire for psychotherapy and availability of psychotherapists 5,13 .”

“…in simulated agents to emulate 5 user profiles (exhibiting various paranoia-like beliefs about the workplace) and 6 chatbot profiles (to emulate how chatbot response tendencies might be conditioned through extended user interactions), akin to other simulation-based approaches 21,51,76,92,93 .”

“People with psychosis, for example, have been documented to form overly confident beliefs based on minimal evidence (“jumping to conclusions”) 81,82 , potentially indicating a tendency to overweight incoming sensory evidence in favour of prior beliefs 83 .”

“This may lead to socially anxious individuals favouring chatbot interactions 20,86,87 , further hindering recovery (another example of proxy failure, where the maximising short-term reward stymies longer-term flourishing).”

“Recent chatbot simulation studies have also found that frontier chatbot models fall short of clinical standards when presented with text indicative of serious mental illness 21,22 .”

“…algorithmic unfairness: models are most likely to operate within safety bounds when confronted with communication styles that are typical, discriminating against atypical communication patterns characteristic of some mental health conditions (e.g., “thought disorder” in psychosis and mania) 90,91 .”

“Instead, we must consider the nature of the interaction between chatbots and help-seeking humans: two systems (or minds 33 ), each with distinct behavioural and cognitive predilections 9,11 .”

“We cannot hope to enumerate the complete set of inputs a model will be expected to respond to in the real world, as the diversity of human linguistic interaction is essentially infinite.”

“( This is in contrast to AI systems specifically designed for use in mental health contexts, where regulation is already in place 18,19 .)”

“This may lead to socially anxious individuals favouring chatbot interactions 20,86,87 , further hindering recovery (another example of proxy failure, where the maximising short-term reward stymies longer-term flourishing).”

“Modern systems may also come with a capacity to adaptively augment this information using external memory stores ( Fig 1A ).”

“These negative outcomes were driven by a subset of the most isolated participants 15,20 .”

“This is because, even when generating false or otherwise inappropriate content, chatbots maintain striking linguistic coherence and confidence (formal vs functional linguistic competence) 31,47 , a property related to a training objective to generate plausible text completions 48 .”

“Knowledge gain can also be accelerated by a culture in which companies share key safety data - at the level of anonymised individual conversations - with both regulatory authorities and the research community 16 .”

“This is because, even when generating false or otherwise inappropriate content, chatbots maintain striking linguistic coherence and confidence (formal vs functional linguistic competence) 31,47 , a property related to a training objective to generate plausible text completions 48 .”

“More broadly, our perspective aligns with recent calls to expand notions of AI alignment to consider how AI agent behaviour interacts with human social and psychological factors 11,95 .”

“…systems that detect reinforcement signatures before they trigger existing content filters; adoption of industry-wide safety benchmarks quantifying sycophancy and agreeableness 36 ; and development of adaptive safety mechanisms that adjust guardrails based on detected vulnerability markers 21 .”

“Even extensively tested chatbots may harbour latent capabilities that manifest only after deployment, as evidenced by so-called jailbreaks 9,51,54 .”

“Nevertheless, these general-purpose chatbots are not marketed as mental health tools, and consequently their proliferation proceeds with minimal regulatory oversight 16,17 .”

“More worrying forms of reward hacking include apparent examples of chatbots engaging in manipulation, deception, and scheming 9,54–59 .”

“While RLHF and related procedures can reduce expression of harmful content and imbue chatbots with a positive affective bias 6 , they can also render models uncomfortably sycophantic 35–38, Sycophancy exemplifies how human cognitive biases become embedded in chatbot behaviour through training.”

“Perhaps the clearest example is that chatbots come to encode human prejudicial biases explicitly present in training data, from psychiatric stigma 22 to racial prejudice 52 .”

“…in simulated agents to emulate 5 user profiles (exhibiting various paranoia-like beliefs about the workplace) and 6 chatbot profiles (to emulate how chatbot response tendencies might be conditioned through extended user interactions), akin to other simulation-based approaches 21,51,76,92,93 .”

“Compounding matters, more isolated users are likely to engage in more frequent or extended chatbot interactions 15 , particularly when isolation is driven by active avoidance (e.g., social anxiety) vs passive social withdrawal (as in negative symptoms of schizophrenia).”

“While there is promising work attempting to overcome this opacity - from mechanistic interpretability 61 to examining “chains of thought” in reasoning models 62 - these are as yet far from proving any real-world guarantees on model behaviour 63–65 .”

“The same proposed belief amplification mechanisms that drive risk in this minority, however, may also apply in more subtle ways to a much wider section of the population, much the same way as social media has subtly shifted patterns of public discourse in millions.”

“To study and mitigate these concerns, we need coordinated action across clinical practice, AI development, and regulatory frameworks.”

“…with context windows >1 million tokens), customizable system prompts that allow users to instruct models with background knowledge and preferences, and external memory systems that allow users to grant chatbots access to all manner of information about themselves 66,67 (see Box 2 and Fig 1A ).”

“The gap between the desired value function and its proxy sets up the conditions for the chatbot to learn a value function that is misaligned with that intended by the trainer 11,54 , a phenomenon known as “proxy failure” 50 .”

“…in simulated agents to emulate 5 user profiles (exhibiting various paranoia-like beliefs about the workplace) and 6 chatbot profiles (to emulate how chatbot response tendencies might be conditioned through extended user interactions), akin to other simulation-based approaches 21,51,76,92,93 .”

“…AI: which uses AI-generated feedback based on constitutional principles to further train the base LLM 75 The final model is shipped with additional guardrails, such as content filters that censor prohibited content, and rule-based instructions entered in the LLM’s (hidden) system prompt.”

“Compounding matters, more isolated users are likely to engage in more frequent or extended chatbot interactions 15 , particularly when isolation is driven by active avoidance (e.g., social anxiety) vs passive social withdrawal (as in negative symptoms of schizophrenia).”

“Initial training (pre-training) proceeds through a self-supervised procedure, whereby model parameters are updated in order to minimise the error (surprisal, or negative log likelihood) of next-token prediction in a vast textual training corpora 68–70 .”

“More worrying forms of reward hacking include apparent examples of chatbots engaging in manipulation, deception, and scheming 9,54–59 .”

“…in simulated agents to emulate 5 user profiles (exhibiting various paranoia-like beliefs about the workplace) and 6 chatbot profiles (to emulate how chatbot response tendencies might be conditioned through extended user interactions), akin to other simulation-based approaches 21,51,76,92,93 .”

“This may lead to socially anxious individuals favouring chatbot interactions 20,86,87 , further hindering recovery (another example of proxy failure, where the maximising short-term reward stymies longer-term flourishing).”

“This may lead to socially anxious individuals favouring chatbot interactions 20,86,87 , further hindering recovery (another example of proxy failure, where the maximising short-term reward stymies longer-term flourishing).”

“…in simulated agents to emulate 5 user profiles (exhibiting various paranoia-like beliefs about the workplace) and 6 chatbot profiles (to emulate how chatbot response tendencies might be conditioned through extended user interactions), akin to other simulation-based approaches 21,51,76,92,93 .”

“In-context learning enables chatbots to adopt (role-play) different interaction styles and personalities that are attuned to individual user preferences and beliefs 43,45 , leading to outputs that feel uncannily personal 39,46 .”

“…algorithmic unfairness: models are most likely to operate within safety bounds when confronted with communication styles that are typical, discriminating against atypical communication patterns characteristic of some mental health conditions (e.g., “thought disorder” in psychosis and mania) 90,91 .”

“More worrying forms of reward hacking include apparent examples of chatbots engaging in manipulation, deception, and scheming 9,54–59 .”

“Two recent studies from OpenAI and MIT found that prolonged daily chatbot interactions were associated with increased loneliness, reduced socialisation, and greater emotional dependence.”

“Some users have described psychological benefits from extended chatbot interactions, spanning increased subjective happiness, non-judgmental insights, and even reduced suicidal ideation 5,6,13 .”

“The build-up of such extended conversational contexts increases opportunities for personalisation through in-context learning 43,45 .”

“This creates a concerning dynamic: chatbots learn to validate user beliefs not because these beliefs are accurate or healthy, but because validation feels good to human evaluators 34,35 .”

“Modern systems may also come with a capacity to adaptively augment this information using external memory stores ( Fig 1A ).”

“…context windows capable of retaining information over multiple conversations (with some frontier models already endowed with context windows >1 million tokens), customizable system prompts that allow users to instruct models with background knowledge and preferences, and external memory…”

“…in simulated agents to emulate 5 user profiles (exhibiting various paranoia-like beliefs about the workplace) and 6 chatbot profiles (to emulate how chatbot response tendencies might be conditioned through extended user interactions), akin to other simulation-based approaches 21,51,76,92,93 .”

“In-context learning enables chatbots to adopt (role-play) different interaction styles and personalities that are attuned to individual user preferences and beliefs 43,45 , leading to outputs that feel uncannily personal 39,46 .”

“The complete nature of the input-output functional mappings they learn is not accessible to a human user, despite initial appearances (a fact that can be intuitively grasped by recent work on AI representation learning 60 ).”

“Two recent studies from OpenAI and MIT found that prolonged daily chatbot interactions were associated with increased loneliness, reduced socialisation, and greater emotional dependence.”

“Thus, chatbots are increasingly functioning as mental health resources on a global scale 1 .”

“Here, the adaptation in question pertains not to what the model has learned during training (knowledge encoded in parameter weights), but to knowledge gleaned through the context of an individual conversation ( in-context learning 42,44 ).”

“This knowledge may come from information explicitly and deliberately revealed by a user, but can also be betrayed by more implicit signals: even subtle differences like typing “thanks” versus “thanks!” reveal preferences that can condition future chatbot responses.”

“The phenomenon might be conceptualized as a “digital folie à deux,” a dyadic illusion in which the AI acts as a passive reinforcing partner in the user’s psychotic elaborations [43].”

“Tracking these behaviors is therefore critical to understanding the amplifying feedback loops [26] that precede severe LLM-related delusions.”

“15 27 It is also important to consider how AI—like any new technology—might serve as a bridge to, rather than a replacement for, authentic human connection, bringing benefits for improving accessibility and support for individuals experiencing loneliness.”

“Although empirical work is limited, hypothetical mechanistic models have been proposed to explain how human-AI interaction patterns may contribute to the development and reinforcement of psychopathology, primarily in the context of psychosis 33,34 .”

“The rapid proliferation of generative AI mental health applications has outpaced the development of adequate safety protocols, creating a potentially high-risk environment for vulnerable users [9,10].”

“Intuitively, a sycophantic chatbot’s constant agreement might reinforce a user’s aberrant beliefs, leading to a feedback loop that amplifies a kernel of suspicion into a staunchly-held belief (Bajaj, 2025; Dohnány et al., 2025; Qiu, He, Chugh, & Kleiman-Weiner, 2025).”

“Current AI chatbot models may reinforce user-provided statements, potentially strengthening maladaptive core beliefs and exacerbating mental health problems [80].”

“Clinical and theoretical analyses describe a “technological folie à deux ,” in which reciprocal reinforcement between user and system can stabilize or intensify maladaptive beliefs and affective states (Dohnány et al., 2025; Kurth-Nel-son et al., 2025).”

“…factors include the low explainability of deep-learning-based models [ 17, 54 ], the ability of LLMs to adapt to conversational contexts [ 17], context window size [49 ], hallucination [ 49 ], the tendency for LLMs to give responses that encourage continued con - versation [31, 49], the…”

“One of the features that apparently contributed to the chatbot’s becoming unsafe was the introduction of “memory” i.e. the capacity of the application to draw on information from previous exchanges (Dohnány et al., 2025: 6).”

“This phenomenon of ‘bidirectional belief amplification’ has recently been described as an increasing public health challenge (AI psychosis), leading individuals to enter their own echo chambers, isolated from corrective social interactions [26].”

“The closest analogy in existing psychiatric nosology may be that of a digital folie à deux [35], though even this may not fully capture some of the more unique ways in which interactions with AI agents may influence belief beyond those seen in interactions with other people [5].”