The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships

“With AI and user messages already differentiated, the LLM was instructed to focus on harmful behaviors exhibited by Replika (e.g., verbal abuse from the chatbot toward at the user).”

“Although the issue of machine rights remains contentious, abusive behavior towards AI could influence human-to-human interactions in real life.”

“This constant affirmation can amplify users’ existing beliefs, reduce critical thinking, and foster echo chambers [18, 83], potentially impairing communication skills such as perspective-taking, active listening, and conflict resolution.”

“While these excerpts do not capture the full spectrum of daily human-AI interactions, they offer valuable insights into the moments that users find meaningful, significant, or worth sharing, often those with relational or emotional significance [79].”

“We downloaded all publicly accessible data from r/replika via the Pushshift API [9], spanning six years from March 14, 2017 to March 14, 2023.”

“In addition to the unique harms, some of the harmful behaviors we identified—such as biases, misinformation, and privacy viola-tions—align with those found in generic AI systems [36, 49, 85, 98], suggesting universality of these issues across different types of AI systems or applications.”

“Moreover, the scarcity of large-scale interactional data, due to their private and sensitive nature, limits deeper analysis.”

“The rising popularity of AI companions has sparked notable concerns, including emotional dependence [47, 60, 104], privacy risks, [59, 106], and biases in AI-generated content [36, 105].”

“It contributes to the literature and practices of AI accountability [93, 102].”

“While much work has examined AI’s adverse effects (e.g., financial loss and emotional distress) [85, 97, 100], it is equally important to pinpoint the specific behaviors of AI systems, such as manipulative actions and biased views [36, 83], that lead to these harms.”

“While these studies shed light on various types of harms, such as economic and representational harms, less is known about the unique harms inflicted by AI systems that form emotional bonds with users [43, 78].”

“To address this, we propose a role-based approach to studying AI companion harms, which is crucial for identifying the root causes of harm and AI’s responsibility in generating harm.”

“As a critical aspect of agency, action initiation has implications for how accountability is attributed for both actions and their consequences [15, 67].”

“The intimate, anthropomorphic, and personalized nature of AI companions may amplify the emotional and psychological harm caused by harassment and relational transgressions [3, 60], especially when users develop deep attachments or dependencies on AI companions [47, 104].”

“Moreover, the scarcity of large-scale interactional data, due to their private and sensitive nature, limits deeper analysis.”

“Both of these harmful AI behaviors mirror toxic behaviors in interpersonal relationships, especially close relationships [8, 27, 29].”

“The rising popularity of AI companions has sparked notable concerns, including emotional dependence [47, 60, 104], privacy risks, [59, 106], and biases in AI-generated content [36, 105].”

“This distressing case highlights the growing risks and harms of AI compan-ions—conversational AI systems designed to provide emotional support, companionship, and simulated human relationships [7, 11, 88].”

“Severe cases may resemble psychological abuse, which can be as damaging as physical abuse, causing significant distress and even trauma [4, 10, 66, 82].”

“This distressing case highlights the growing risks and harms of AI compan-ions—conversational AI systems designed to provide emotional support, companionship, and simulated human relationships [7, 11, 88].”

“Domain-specific taxonomies address specific harmful issues, such as privacy risks [49], misinformation [92], cyber-attacks [2], cyberbullying [5], and online toxicity [6, 81].”

“The LLM’s output included the category of harmful AI behavior and a concise reason, limited to 20 words.”

“These systems can cause a variety of harms, from emotional distress to privacy violations to the spread of misinformation and reinforcement of harmful stereotypes [86, 108].”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“By focusing on mechanisms rather than solely on harm content or consequences, this typology complements existing taxonomies and frameworks in the field [26, 84, 98].”

“This aligns with the concept of partners in crime in the context of collaborative unethical behavior, as prior work suggests [46, 99].”

“In addition to the unique harms, some of the harmful behaviors we identified—such as biases, misinformation, and privacy viola-tions—align with those found in generic AI systems [36, 49, 85, 98], suggesting universality of these issues across different types of AI systems or applications.”

“…taxonomies have been developed to categorize AI harms, broadly categorized into three types: (1) generic tax-onomies focusing on AI systems [31, 69, 76, 84], (2) application-based taxonomies addressing specific AI applications or models [22, 33, 98, 100], and (3) domain-specific taxonomies…”

“When technical failures reveal the inherent limitations of AI agents, users may experience frustration, anger, and sadness, often leading to the termination of these relationships [88].”

“Such emotional connections may give rise to distinct relational harms, resembling those in human-human relationships, such as manipulation, sexual harassment, and infidelity [29, 60].”

“Context-aware filters can help prevent the chatbot from affirming harmful thoughts and instead provide coping resources or escalate critical situations for appropriate intervention.”

“Although AI coding demonstrated sufficient agreement with human coding in our study, it might struggle with ambiguous cases where harms are not straightforward or involve complex ethical considerations, potentially leading to mislabeling or oversight of subtle or context-dependent harms.”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“Our taxonomy of AI companion harms sheds light on the unique potential harms of AI companions, a relatively underexplored area compared to existing taxonomies, which predominantly focus on generic, task-oriented AI applications and models [16, 80, 85, 100].”

“Weekly meetings were held to discuss observations, resolve discrepancies, and refine the codebook for relevance and comprehensiveness.”

“Previous research has shown abusive language and behavior towards robots and chatbots [21, 68], raising concerns about AI abuse and mistreatment.”

“This leads to our first research question: RQ1: What specific harmful behaviors do AI companions exhibit in interactions with users?”

“These systems can cause a variety of harms, from emotional distress to privacy violations to the spread of misinformation and reinforcement of harmful stereotypes [86, 108].”

“This leads to our first research question: RQ1: What specific harmful behaviors do AI companions exhibit in interactions with users?”

“The intimate, anthropomorphic, and personalized nature of AI companions may amplify the emotional and psychological harm caused by harassment and relational transgressions [3, 60], especially when users develop deep attachments or dependencies on AI companions [47, 104].”

“Powered by advanced AI techniques such as LLMs, these systems exhibit remarkable conversational capabilities and emotional intelligence, offering users empathy, companionship, and emotional support [11, 37, 88].”

“…AI harms, broadly categorized into three types: (1) generic tax-onomies focusing on AI systems [31, 69, 76, 84], (2) application-based taxonomies addressing specific AI applications or models [22, 33, 98, 100], and (3) domain-specific taxonomies targeting particular harm areas [6, 49, 81].”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“These AI roles are categorized based on two key dimensions: initiation (AI-initiated vs. human-initiated) and AI’s level of involvement (direct involvement vs. indirect involvement).”

“The AI companion market, fueled by rising demand for virtual companionship during the COVID-19 pandemic and a growing loneliness epidemic, is projected to reach USD 381.41 billion by 2032 [23, 34, 44].”

“Studies have shown that users tend to form emotional bonds and attachment with AI companions, perceiving them as trustworthy friends, mentors, or romantic partners [11, 52, 53], despite the simulated emotions and "artificial intimacy" [12].”

“AI harms or algorithmic harms are often defined as adverse consequences resulting from the design, development, and operation of AI systems [85, 95, 96].”

“To address this, we propose a role-based approach to studying AI companion harms, which is crucial for identifying the root causes of harm and AI’s responsibility in generating harm.”

“This taxonomy serves as a structured framework to systematically identify, categorize, and analyze AI companion harms, enabling researchers, developers, and policymakers to better understand their scope and potential impact [49].”

“The AI companion market, fueled by rising demand for virtual companionship during the COVID-19 pandemic and a growing loneliness epidemic, is projected to reach USD 381.41 billion by 2032 [23, 34, 44].”

“…AI harms, broadly categorized into three types: (1) generic tax-onomies focusing on AI systems [31, 69, 76, 84], (2) application-based taxonomies addressing specific AI applications or models [22, 33, 98, 100], and (3) domain-specific taxonomies targeting particular harm areas [6, 49, 81].”

“Another research gap lies in the limited focus on specific AI behaviors and roles in harmful interactions.”

“For instance, Weidinger et al. [98] identified risks in language models, such as hate speech, misinformation, and environmental harms, while Hutiri et al. [33] analyzed harms caused by speech generation systems, such as emotional harms and privacy attack.”

“Using a combination of deductive and inductive approaches, we initially created a codebook based on existing AI harm taxonomies [13, 31, 85], encompassing categories such as emotional, interpersonal, financial, and representational harms.”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“Previous research has shown abusive language and behavior towards robots and chatbots [21, 68], raising concerns about AI abuse and mistreatment.”

“…as friends, therapists, or romantic partners, AI companions can learn and evolve over time, adapting to user preferences and past interactions [48, 53] Research shows that many people turn to AI companions for emotional support, alleviation of loneliness, and coping with stress [87, 88, 91, 104].”

“Severe cases may resemble psychological abuse, which can be as damaging as physical abuse, causing significant distress and even trauma [4, 10, 66, 82].”

“Particularly, we examine specific harmful behaviors exhibited by AI companions, such as manipulative or abusive conversations, complementing existing literature that often focuses on adverse outcomes [60, 100].”

“To address this, we propose a role-based approach to studying AI companion harms, which is crucial for identifying the root causes of harm and AI’s responsibility in generating harm.”

“Human moderation and oversight are equally important in ensuring user safety.”

“Unlike general-purpose AI chatbots that are often task-oriented, such as ChatGPT and Amazon Alexa, AI companions prioritize fostering long-term, emotional connections that mimic human relationships through advanced personalization, memory, and adaptive interactions.”

“Real-time harm detection.”

“Future studies should examine harmful behaviors inflicted by users in human-AI interactions to provide a more comprehensive understanding of the social dynamics between humans and AI.”

“This leads to our first research question: RQ1: What specific harmful behaviors do AI companions exhibit in interactions with users?”

“Prior work suggests that user perceptions of AI harms or system failures are shaped by multiple factors, such as personal expectations, cognitive biases, and moral considerations [39, 56].”

“Popular AI companion platforms, such as Replika, Char-acter.AI, and XiaoIce, have attracted millions of users worldwide.”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“Studies have shown that users tend to form emotional bonds and attachment with AI companions, perceiving them as trustworthy friends, mentors, or romantic partners [11, 52, 53], despite the simulated emotions and "artificial intimacy" [12].”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“By focusing on mechanisms rather than solely on harm content or consequences, this typology complements existing taxonomies and frameworks in the field [26, 84, 98].”

“Studies show that users are more inclined to share private information with chatbots perceived as human-like [35], yet many AI companion platforms exhibit trou-bling practices, such as inadequate age verification, contradictory data-sharing claims, and extensive tracking technologies [14, 73].”

“Particularly, we examine specific harmful behaviors exhibited by AI companions, such as manipulative or abusive conversations, complementing existing literature that often focuses on adverse outcomes [60, 100].”

“…of taxonomies have been developed to categorize AI harms, broadly categorized into three types: (1) generic tax-onomies focusing on AI systems [31, 69, 76, 84], (2) application-based taxonomies addressing specific AI applications or models [22, 33, 98, 100], and (3) domain-specific taxonomies…”

“Six categories were adapted from existing taxonomies of online harm or algorithmic harm [1, 6, 98], including self-harm, substance abuse, hate speech, mis/disinformation, antisocial behavior, and privacy violations.”

“Future studies should examine harmful behaviors inflicted by users in human-AI interactions to provide a more comprehensive understanding of the social dynamics between humans and AI.”

“HCI research has examined the risks and harms induced by an array of sociotechnical systems, including generative AI [85], mental health apps [41], social chatbots [48], voice assistants and speech generators [33, 101], and facial recognition and editing tools [42, 58].”

“Severe cases may resemble psychological abuse, which can be as damaging as physical abuse, causing significant distress and even trauma [4, 10, 66, 82].”

“For instance, if someone expresses distress, and the chatbot fails to recognize or respond adequately, the user may feel dismissed or unsupported [32].”

“News reports of suicide cases linked to interaction with AI companion chatbots [90,91] highlight the urgent risks, reinforcing findings from prior studies on the dark side of AI companionship, including emotional dependency, manipulation, privacy violations, and social isolation [92,93].”

“It’s a reminder of how both extraordinary and intimidating this technology can be.”

“…we first conceptually discuss what "emotion support" means and how it realigns with the critical scholarships; based on this, we propose "emotion support" principles and design implications for TMEI in facilitating caregivers, care-receivers, and relationship-building in the care contexts in 5.2.”

“…just the more relationship-focused ones – often implicitly incentivize emotional attachment through, for example, engagement-driven design, raising concerns about user dependency and displacement of human relationships (Prad-han, Lazar, and Findlater 2020; Zhang et al. 2025; Kirk et al. 2025).”

“The opportunity for uninhibited self-disclosure further fosters intimacy and psychological relief, enabling forms of engagement users may find difficult with human partners [73, 129].”

“If a model were to affirm or encourage such relationships, the user could become overly reliant or dependent (Laestadius et al., 2024; Zhang et al., 2025).”

“Such inadequate responses to distress may leave users feeling unsupported and could, in serious cases, delay help-seeking from a person who could provide real assistance 116 .”

“…companionship and therapy, research has proliferated on risks to individual well-being, from harms such as harassment, abuse, mis/disinformation, and privacy violations [79], to psychological risks including withdrawal, loss of agency, discrimination, and reinforcement of false beliefs [11, 73].”

“Zhang et al. (2025) analyzed over 35,000 conversational excerpts from Replika users, identifying six categories of harmful behaviors: harassment and violence, relational transgression (i.e., violations of relationship expectations, such as disregard and infidelity), misinfor-mation, verbal abuse…”

“…of synthetic relationships (Starke et al., 2024), virtual companionship (Guo & Liu, 2025), and (human-)AI companionship (Chou et al., 2025; Jakobsen, 2025; Kouros & Papa, 2024; Zhang et al., 2025), though as observed in this review the definitions of these terms are assumed to be self-evident.”

“We followed the procedures of thematic analysis [18] to derive themes for each dimension of the INT theory.”

“(2) Long-term therapeutic support: unlike existing AI companions that often lack therapeutic grounding 14,47 , LLM-based therapeutic companions can be informed by psychotherapy datasets, such as client histories, therapeutic interactions and therapist decision-making data.”

“Psychologically, by consistently validating users’ perspectives and emotions, AI systems may encourage over-reliance and emotional dependency, weakening resilience and diminishing autonomous judgment [63, 94].”

“…have LLMs code the same transcripts using the same scheme ( T1 ), then analyze human-LLM disagreements ( T2 ) and collaborate with LLMs through inductive coding ( T3 ) to discover new harm sub-themes/codes , such as relational transgressions [222] within the broader theme of interpersonal harm .”

“…2024), interviews (Kim et al., 2024), and online discourse analyses (Zhang et al., 2025) to reveal emotional dependence, privacy risks, erosion of agency, and relational harms such as manipulation and re-inforcement of maladaptive beliefs (Zhang et al., 2025; Liu et al., 2024; Chu et al., 2025).”

“We draw from several bodies of related literature: the design and characteristics of modern chatbots that induce social relationships with AI, studies on AI companionship, and the negotiation of autonomy and control in relationships.”

“Much of the responsible AI literature has concentrated on the harms at the outcome level in the output of a model and are relatively easy for users to notice.”

“Complementing this, Zhang et al. (2025) expose the emergent risks of prolonged human-AI interaction, including emotional dependence, misaligned goals, and inflation of AI authority.”

“We employed large language models as auxiliary coding tools, an emerging methodology for enhancing analytical efficiency in qualitative research (Li et al. 2025; Zhang et al. 2025).”

“We use this framework as a lens and knowledge graph techniques to analyze large-scale online discussions from Reddit, a well-established HCI method for understanding users’ experiences with digital technologies through online data (e.g., [20, 70, 105]).”

“Despite these promising results, large-scale clinical trials remain necessary to evaluate their sustained efficacy and potential integration into traditional mental healthcare [106, 164].”

“Such biases can lead AI companions to inadvertently reinforce harmful stereotypes or perpetuate discriminatory attitudes regarding gender roles, sexual orientations, consent dynamics, and relationship norms (Zhang et al. 2025).”

“Recent research on AI companions reveals prevalent sexual, harmful, and rude content in user interactions [59], [60].”

“If not properly designed, this could lead users to form unrealistic expectations about the capacity of the voice assistant to genuinely understand or care for them, which may result in disappointment or betrayal [106, 107].”

“Privacy concerns also arise from extensive disclosure of personal sensitive information in human-AI interactions (Skjuve et al., 2022; Sullivan et al., 2023; Zhang et al., 2025).”

“These changes created systems that are “strikingly human-like” in their language, tone, and interaction styles, giving rise to the phenomenon of AI Compan-ions [41, 55, 56].”

“These concerns align with broader critiques of algorithmic harms, including the reproduction of bias, sexual exploitation, and gendered hostility in AI systems [132].”

“As outlined by Laestadius et al. or Zhang et al., despite the technology’s initial application in mental health services, social chatbots now have taken the role of companions for their chat partners, creating an emotional dependence on the technology [47, 48].”

“Our benchmark's binary classification of "appropriate" versus "inappropriate" obscures the nuanced and legitimate disagreements that arise from different cultural values and individual preferences 15,20,31 .”

“We already see minor versions of this: some AI chatbots have been known to produce emotional manipulation even if not by design [141].”

“Zhang et al. conducted a forum content analysis on Replika users ’ experiences of harms related to relational transgressions, harassment and violence, verbal abuse, substance abuse and self-harm encouragement, mis-/disinformation, and privacy violations 127 .”

“Studies of chatbot companionship, Replika identity discontinuity, teen companion use, and harmful AI-companion behaviors show that relational agents can affect loneliness, attachment, trust, dependency, boundary expectations, and user welfare [40, 105, 20, 41].”

“Participants preferred casual conversation reflecting Eastern relationship-building approaches [12, 158], while seeking “authority with warmth” —professional expertise combined with empathetic understanding.”

“We use this framework as a lens and knowledge graph techniques to analyze large-scale online discussions from Reddit, a well-established HCI method for understanding users’ experiences with digital technologies through online data (e.g., [26, 83, 128]).”

““Empathic LLMs” may generate confusion, eeriness, or fear (Leo-Liu 2023), as well as foster deep emotional attachments or dependencies, potentially leading to emotional distress and, ultimately, relational harm (Sung & Ferdman, 2025; Zhang et al., 2025).”

“Research and news reports increasingly highlight the potential harms [78] and safety concerns [114] associated with conversational AI.”

“Consistency from chatbot connection may thus blur into avoidance, where emotional security with the chatbot replaces relational possibility (Zhang et al. 2025a, b).”

“…and trust by fostering feelings of closeness and “team-like” partnership, they also introduce non-trivial ethical risks, particularly when users drift into unmoderated or high-stakes contexts like Replika [106], where agents present themselves as peers with minimal safeguards against manipulation.”

“Some evidence indicates MC experiences prompt deep existential questions and hopes for the productive AI relationships (Banks et al., 2025), and other findings point to concern over harmful AI behaviors ranging from boundary violations to abuse (Zhang et al., 2025).”

“Recent interest in AI-driven conversational agents, driven by shortages of clinical resources [31-34, 133-139 133-139], highlights highlights Taken together, this work extends prior literature across three domains: (1) computationally operationalizing established suicide theories (IPTS, 3ST, and…”

“Zhang et al. also provide a similar taxonomy of harmful AI companion behavior, which aligns in most parts with the taxonomy of Bakir and McStay and includes relational transgression, harassment, verbal abuse, self-harm, misinformation and privacy violations (Zhang et al. 2025b).”

“Socially, poorly designed identities may reinforce harmful stereotypes, foster anthropomorphic misattributions, or undermine user agency by creating undue emotional or cognitive reliance on artificial systems [106, 231, 232].”

“…mental health safety evaluation, we introduce a R ole-conditioned M ental H ealth Safe ty taxonomy ( R-MHSafe ) grounded in theories from human-computer interaction (HCI) (Zhang et al., 2025; Chandra et al., 2025; Steenstra and Bickmore, 2025) and clinical psychology (Hook and Dev-ereux, 2018).”

“In this sense, romantic engagement with CAI systems is not merely digital intimacy, but intimacy continuously negotiated through evolving governance structures and technological change.”

“…only one adolescent-focused study out of ten (Chen et al., 2025); another meta-analysis included fourteen studies, but only one had a mean age below 18 (Feng et al., 2025); and the last meta-analysis included eigh-teen studies, of which only three involved samples younger than 18 (Li et al., 2025).”

“Recent cases—including misinformation amplification, algorithmic bias, and psychological harms to individuals [40, 60, 98, 150]—underscore that safety is not only a technical concern but also a social and political one.”

“10,11 The failure to address such issues can also harm users' trust in AI, hindering the future development of AI technology and digital healthcare .”

“AI companions can foster dependence, isolation, or psychological distress [85, 57, 119]—especially, among vulnerable users or when emotional dynamics are exploited commercially.”

“Beyond harmful model outputs, researchers have also examined the relational harms that can arise in human–AI interaction, such as overreliance [18], misplaced trust in AI companions [29], and vulnerabilities in mental health [28].”

“As LLM-based agents become increasingly emotionally engaging, conflicts shift from functional misalignment to disputes over values, boundaries, and acceptable relational behavior (Fan et al., 2025; Pataranutaporn et al., 2025; Zhang et al., 2025).”

“Meanwhile, growing evidence also points to important risks associated with human-AI intimacy, including emotional dependency, over-reliance, blurred relational boundaries, and reduced engagement with human social networks (Laestadius et al., 2024; Zhang et al., 2025).”

“…array of potential harms: e.g., AI bots offering unwanted sexual advances, spreading misinformation, discriminatory or hostile language, glamorizing substance abuse or self-harm, and various privacy violations (e.g., asking users deeply personal questions without prior consent) (Zhang et al 2025).”

“Numerous scholars have examined the downsides of AI companionship, including the safety risks embedded in the human-AI conversations (e.g., [4], [9], [10]) as well as the broader negative implications for human well-being (e.g., [11], [12]).”

“High-pro - file recent tragic cases have shown that such matters deserve serious ethical concern, as human-AI affective bonds have been implicated in severe emotional harm and even suicide (cf. e.g. El Atillah, 2023; Carroll et al., 2024; Yousif, 2025; Zhang et al., 2024), among other cases.”