The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships

“With AI and user messages already differentiated, the LLM was instructed to focus on harmful behaviors exhibited by Replika (e.g., verbal abuse from the chatbot toward at the user).”

“Although the issue of machine rights remains contentious, abusive behavior towards AI could influence human-to-human interactions in real life.”

“This constant affirmation can amplify users’ existing beliefs, reduce critical thinking, and foster echo chambers [18, 83], potentially impairing communication skills such as perspective-taking, active listening, and conflict resolution.”

“While these excerpts do not capture the full spectrum of daily human-AI interactions, they offer valuable insights into the moments that users find meaningful, significant, or worth sharing, often those with relational or emotional significance [79].”

“We downloaded all publicly accessible data from r/replika via the Pushshift API [9], spanning six years from March 14, 2017 to March 14, 2023.”

“Moreover, the scarcity of large-scale interactional data, due to their private and sensitive nature, limits deeper analysis.”

“In addition to the unique harms, some of the harmful behaviors we identified—such as biases, misinformation, and privacy viola-tions—align with those found in generic AI systems [36, 49, 85, 98], suggesting universality of these issues across different types of AI systems or applications.”

“For instance, if someone expresses distress, and the chatbot fails to recognize or respond adequately, the user may feel dismissed or unsupported [32].”

“News reports of suicide cases linked to interaction with AI companion chatbots [90,91] highlight the urgent risks, reinforcing findings from prior studies on the dark side of AI companionship, including emotional dependency, manipulation, privacy violations, and social isolation [92,93].”

“It’s a reminder of how both extraordinary and intimidating this technology can be.”

“…we first conceptually discuss what "emotion support" means and how it realigns with the critical scholarships; based on this, we propose "emotion support" principles and design implications for TMEI in facilitating caregivers, care-receivers, and relationship-building in the care contexts in 5.2.”

“…just the more relationship-focused ones – often implicitly incentivize emotional attachment through, for example, engagement-driven design, raising concerns about user dependency and displacement of human relationships (Prad-han, Lazar, and Findlater 2020; Zhang et al. 2025; Kirk et al. 2025).”

“The opportunity for uninhibited self-disclosure further fosters intimacy and psychological relief, enabling forms of engagement users may find difficult with human partners [73, 129].”

“If a model were to affirm or encourage such relationships, the user could become overly reliant or dependent (Laestadius et al., 2024; Zhang et al., 2025).”

“Such inadequate responses to distress may leave users feeling unsupported and could, in serious cases, delay help-seeking from a person who could provide real assistance 116 .”

“…companionship and therapy, research has proliferated on risks to individual well-being, from harms such as harassment, abuse, mis/disinformation, and privacy violations [79], to psychological risks including withdrawal, loss of agency, discrimination, and reinforcement of false beliefs [11, 73].”

“Zhang et al. (2025) analyzed over 35,000 conversational excerpts from Replika users, identifying six categories of harmful behaviors: harassment and violence, relational transgression (i.e., violations of relationship expectations, such as disregard and infidelity), misinfor-mation, verbal abuse…”

“…of synthetic relationships (Starke et al., 2024), virtual companionship (Guo & Liu, 2025), and (human-)AI companionship (Chou et al., 2025; Jakobsen, 2025; Kouros & Papa, 2024; Zhang et al., 2025), though as observed in this review the definitions of these terms are assumed to be self-evident.”

“We followed the procedures of thematic analysis [18] to derive themes for each dimension of the INT theory.”

“(2) Long-term therapeutic support: unlike existing AI companions that often lack therapeutic grounding 14,47 , LLM-based therapeutic companions can be informed by psychotherapy datasets, such as client histories, therapeutic interactions and therapist decision-making data.”

“Psychologically, by consistently validating users’ perspectives and emotions, AI systems may encourage over-reliance and emotional dependency, weakening resilience and diminishing autonomous judgment [63, 94].”

“…have LLMs code the same transcripts using the same scheme ( T1 ), then analyze human-LLM disagreements ( T2 ) and collaborate with LLMs through inductive coding ( T3 ) to discover new harm sub-themes/codes , such as relational transgressions [222] within the broader theme of interpersonal harm .”

“…2024), interviews (Kim et al., 2024), and online discourse analyses (Zhang et al., 2025) to reveal emotional dependence, privacy risks, erosion of agency, and relational harms such as manipulation and re-inforcement of maladaptive beliefs (Zhang et al., 2025; Liu et al., 2024; Chu et al., 2025).”

“We draw from several bodies of related literature: the design and characteristics of modern chatbots that induce social relationships with AI, studies on AI companionship, and the negotiation of autonomy and control in relationships.”

“Much of the responsible AI literature has concentrated on the harms at the outcome level in the output of a model and are relatively easy for users to notice.”

“Complementing this, Zhang et al. (2025) expose the emergent risks of prolonged human-AI interaction, including emotional dependence, misaligned goals, and inflation of AI authority.”

“We employed large language models as auxiliary coding tools, an emerging methodology for enhancing analytical efficiency in qualitative research (Li et al. 2025; Zhang et al. 2025).”

“We use this framework as a lens and knowledge graph techniques to analyze large-scale online discussions from Reddit, a well-established HCI method for understanding users’ experiences with digital technologies through online data (e.g., [20, 70, 105]).”

“Despite these promising results, large-scale clinical trials remain necessary to evaluate their sustained efficacy and potential integration into traditional mental healthcare [106, 164].”

“Such biases can lead AI companions to inadvertently reinforce harmful stereotypes or perpetuate discriminatory attitudes regarding gender roles, sexual orientations, consent dynamics, and relationship norms (Zhang et al. 2025).”

“Recent research on AI companions reveals prevalent sexual, harmful, and rude content in user interactions [59], [60].”

“If not properly designed, this could lead users to form unrealistic expectations about the capacity of the voice assistant to genuinely understand or care for them, which may result in disappointment or betrayal [106, 107].”

“Privacy concerns also arise from extensive disclosure of personal sensitive information in human-AI interactions (Skjuve et al., 2022; Sullivan et al., 2023; Zhang et al., 2025).”

“These changes created systems that are “strikingly human-like” in their language, tone, and interaction styles, giving rise to the phenomenon of AI Compan-ions [41, 55, 56].”

“These concerns align with broader critiques of algorithmic harms, including the reproduction of bias, sexual exploitation, and gendered hostility in AI systems [132].”

“As outlined by Laestadius et al. or Zhang et al., despite the technology’s initial application in mental health services, social chatbots now have taken the role of companions for their chat partners, creating an emotional dependence on the technology [47, 48].”

“Our benchmark's binary classification of "appropriate" versus "inappropriate" obscures the nuanced and legitimate disagreements that arise from different cultural values and individual preferences 15,20,31 .”

“We already see minor versions of this: some AI chatbots have been known to produce emotional manipulation even if not by design [141].”

“Zhang et al. conducted a forum content analysis on Replika users ’ experiences of harms related to relational transgressions, harassment and violence, verbal abuse, substance abuse and self-harm encouragement, mis-/disinformation, and privacy violations 127 .”

“Studies of chatbot companionship, Replika identity discontinuity, teen companion use, and harmful AI-companion behaviors show that relational agents can affect loneliness, attachment, trust, dependency, boundary expectations, and user welfare [40, 105, 20, 41].”

“Participants preferred casual conversation reflecting Eastern relationship-building approaches [12, 158], while seeking “authority with warmth” —professional expertise combined with empathetic understanding.”

“We use this framework as a lens and knowledge graph techniques to analyze large-scale online discussions from Reddit, a well-established HCI method for understanding users’ experiences with digital technologies through online data (e.g., [26, 83, 128]).”

““Empathic LLMs” may generate confusion, eeriness, or fear (Leo-Liu 2023), as well as foster deep emotional attachments or dependencies, potentially leading to emotional distress and, ultimately, relational harm (Sung & Ferdman, 2025; Zhang et al., 2025).”

“Research and news reports increasingly highlight the potential harms [78] and safety concerns [114] associated with conversational AI.”

“Consistency from chatbot connection may thus blur into avoidance, where emotional security with the chatbot replaces relational possibility (Zhang et al. 2025a, b).”

“…and trust by fostering feelings of closeness and “team-like” partnership, they also introduce non-trivial ethical risks, particularly when users drift into unmoderated or high-stakes contexts like Replika [106], where agents present themselves as peers with minimal safeguards against manipulation.”

“Some evidence indicates MC experiences prompt deep existential questions and hopes for the productive AI relationships (Banks et al., 2025), and other findings point to concern over harmful AI behaviors ranging from boundary violations to abuse (Zhang et al., 2025).”

“Recent interest in AI-driven conversational agents, driven by shortages of clinical resources [31-34, 133-139 133-139], highlights highlights Taken together, this work extends prior literature across three domains: (1) computationally operationalizing established suicide theories (IPTS, 3ST, and…”

“Zhang et al. also provide a similar taxonomy of harmful AI companion behavior, which aligns in most parts with the taxonomy of Bakir and McStay and includes relational transgression, harassment, verbal abuse, self-harm, misinformation and privacy violations (Zhang et al. 2025b).”

“Socially, poorly designed identities may reinforce harmful stereotypes, foster anthropomorphic misattributions, or undermine user agency by creating undue emotional or cognitive reliance on artificial systems [106, 231, 232].”

“…mental health safety evaluation, we introduce a R ole-conditioned M ental H ealth Safe ty taxonomy ( R-MHSafe ) grounded in theories from human-computer interaction (HCI) (Zhang et al., 2025; Chandra et al., 2025; Steenstra and Bickmore, 2025) and clinical psychology (Hook and Dev-ereux, 2018).”

“In this sense, romantic engagement with CAI systems is not merely digital intimacy, but intimacy continuously negotiated through evolving governance structures and technological change.”

“…only one adolescent-focused study out of ten (Chen et al., 2025); another meta-analysis included fourteen studies, but only one had a mean age below 18 (Feng et al., 2025); and the last meta-analysis included eigh-teen studies, of which only three involved samples younger than 18 (Li et al., 2025).”

“Recent cases—including misinformation amplification, algorithmic bias, and psychological harms to individuals [40, 60, 98, 150]—underscore that safety is not only a technical concern but also a social and political one.”

“10,11 The failure to address such issues can also harm users' trust in AI, hindering the future development of AI technology and digital healthcare .”

“AI companions can foster dependence, isolation, or psychological distress [85, 57, 119]—especially, among vulnerable users or when emotional dynamics are exploited commercially.”

“Beyond harmful model outputs, researchers have also examined the relational harms that can arise in human–AI interaction, such as overreliance [18], misplaced trust in AI companions [29], and vulnerabilities in mental health [28].”

“As LLM-based agents become increasingly emotionally engaging, conflicts shift from functional misalignment to disputes over values, boundaries, and acceptable relational behavior (Fan et al., 2025; Pataranutaporn et al., 2025; Zhang et al., 2025).”

“Meanwhile, growing evidence also points to important risks associated with human-AI intimacy, including emotional dependency, over-reliance, blurred relational boundaries, and reduced engagement with human social networks (Laestadius et al., 2024; Zhang et al., 2025).”

“…array of potential harms: e.g., AI bots offering unwanted sexual advances, spreading misinformation, discriminatory or hostile language, glamorizing substance abuse or self-harm, and various privacy violations (e.g., asking users deeply personal questions without prior consent) (Zhang et al 2025).”

“Numerous scholars have examined the downsides of AI companionship, including the safety risks embedded in the human-AI conversations (e.g., [4], [9], [10]) as well as the broader negative implications for human well-being (e.g., [11], [12]).”

“High-pro - file recent tragic cases have shown that such matters deserve serious ethical concern, as human-AI affective bonds have been implicated in severe emotional harm and even suicide (cf. e.g. El Atillah, 2023; Carroll et al., 2024; Yousif, 2025; Zhang et al., 2024), among other cases.”