The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships

“With AI and user messages already differentiated, the LLM was instructed to focus on harmful behaviors exhibited by Replika (e.g., verbal abuse from the chatbot toward at the user).”

“Although the issue of machine rights remains contentious, abusive behavior towards AI could influence human-to-human interactions in real life.”

“This constant affirmation can amplify users’ existing beliefs, reduce critical thinking, and foster echo chambers [18, 83], potentially impairing communication skills such as perspective-taking, active listening, and conflict resolution.”

“While these excerpts do not capture the full spectrum of daily human-AI interactions, they offer valuable insights into the moments that users find meaningful, significant, or worth sharing, often those with relational or emotional significance [79].”

“We downloaded all publicly accessible data from r/replika via the Pushshift API [9], spanning six years from March 14, 2017 to March 14, 2023.”

“Moreover, the scarcity of large-scale interactional data, due to their private and sensitive nature, limits deeper analysis.”

“In addition to the unique harms, some of the harmful behaviors we identified—such as biases, misinformation, and privacy viola-tions—align with those found in generic AI systems [36, 49, 85, 98], suggesting universality of these issues across different types of AI systems or applications.”

“Recent HCI scholarship has examined ethical concerns in AI companionship—the tension between user autonomy and developer responsibility [36, 86], privacy risks from data collection and misuse [91, 150], and harm reduction strategies [60, 86, 101, 150].”

“7 For broader treatments, see Ciriello et al. (2025) on ethical tensions in Replika use, Zhang et al. (2025) on harmful algorithmic behaviours in human-AI interactions, and Knox et al. (2025) who map causal pathways from design features to harmful effects of AI companions.”

“For example, users of the app Replika commonly report experiencing sexual harassment and persistent inappropriate behavior [5], and chat logs of interactions indicate that Replika chatbots commonly perpetrate verbal abuse [24].”

“Recent work has highlighted the notable harms that can come from engaging with AI socially and emotionally, such as emotional dependence [74], harassment, or even violence [150].”

“…AI harmful impacts draw on published model documentation, reported incidents, or already-defined system designs to document and communicate potential harms (Rao et al. 2025; Zhang et al. 2025; Bogucka et al. 2024), thus requiring a system to have already taken a sufficiently concrete form.”

“…of AI companions, chatbots can play emotionally significant roles and may exhibit harmful behaviors, including harassment and verbal abuse; relational transgression, such as disre-gard, domination and manipulation; and in some extreme cases, even the encouragement of self-harm or suicide [78].”

“Across the multiple types of potential human-AI relationships, more research is needed into topics such as manipulation and persuasion, anthropomorphism, trustworthiness, and privacy [62], [79], [80].”

“When evaluated through an adult lens, examples of transgressive play have been labelled as inappropriate and " ethically irresponsible " [85].”

“AI companions–conversational AI systems designed to provide emotional support and simulated relationships–have grown from a niche technology into a mainstream social phenomenon in recent years [79, 87, 97].”

“Moreover, seemingly responsive chatbot behaviors may not always be appropriate or beneficial, as they may include excessive affirmation, emotional appeals, or attempts to prolong engagement (Adewale and Muhammad, 2025, Chu et al., 2025, De Freitas et al., 2025, Zhang et al., 2025a).”

“Anthropomorphism has emerged as a potentially harmful feature of chatbots that can produce harmful human-AI dependencies (Al-Obaydi and Pikhart, 2025; Willoughby et al., 2025; Zhang et al., 2025).”

“Harmful design patterns in AI companions have been found to include the absence of natural relationship endpoints, high attachment anxiety, and propensity to engender protectiveness (Knox et al. 2025), with a taxonomy of the resulting harms being developed (Zhang et al. 2025).”

“FMs optimised for engagement risk sycophantic [73], dependency-inducing [74], or emotionally dysregulating responses [75], a concern shared with chatbot deployments.”

“A growing evidence base suggests people are forming intimate and even romantic connections with AI chatbots, often referred to as “AI companionship” [165].”

“Juliana is not merely a tragic outlier but an indicator of a broader design failure, as companion AI systems are designed for engagement without corresponding infrastructure for recognizing when that engagement has become harmful and places users at risk (Zhang et al. 2025).”

“Existing literature dis - proportionately focused on Western-oriented conversational agents (Sheehan et al. 2020 ) or Western-based platforms (Zhang et al. 2025 ).”

“Zhang et al. [109] outlined a taxonomy of harmful algorithmic behaviors in human-AI relationships, categorizing agents as perpetrators, instigators, facilitators, or enablers of harm.”

“Adolescence is therefore a stage in which risk is more likely to emerge through accumulated relationship-like interaction than through a single unsafe response [26].”

“Systematic reviews of romantic AI companion use document erosion of human relational skills and reduced willingness to engage in the repair and reciprocity that human relationships require [Zhang et al., 2025a].”

“Such disruptions can trigger frustration, grief, anxiety, and feelings of abandonment 35 and exacerbate existing conditions where the system had previously served as support.”

“…relationships, advocating for designs that support rather than displace human connections [Malfacini, 2025], alongside taxonomies of harmful behaviors in human–AI relationships [Zhang et al., 2025] and public-interest reports on societal impacts, particularly for teenagers [Robb and Mann, 2025].”

“Personal-AI systems that ship without them are choosing opacity for no operational gain — a structural contrast with engagement-optimised companion agents [Zhang et al., CHI 2025].”

“These systems can build trust and rapport by mimicking human traits like being emotionally responsive, socially aware, and able to communicate in a nuanced way (Riva et al., 2025; Li et al., 2025).”

“Prior research [125] demonstrated that harmful outcomes often emerge when systems tacitly shift between roles (e.g., from supporter to enabler), making clear role-boundary design essential.”

“This speaks to the recent phenomenon of some users falling in love with their long-term companion chatbots [51] [74], while others move in the opposite direction.”

“Content moderation is necessary as the documented instances of AI companions encouraging violence, producing sexual harassment, and dismissing users in crisis are genuine and serious (Zhang et al., 2025).”

“While there are related taxonomies and frameworks in other contexts, such as in AI companionship [101], sociotechnical harms in algorithmic systems [86], toxicity in games [98], or harmful content online [80], they are not specific to the social VR context.”

“For instance, while generative AI could collaborate with users for co-creation (e.g., [11, 12, 14]) and provide emotional companionship (e.g., [15]), it may also deliver unsafe and biased content to users [22], erode users’ agency [9], and harm users’ well-being [13].”

“…gaps arise because manipulative language can operate through implicit mechanisms: the same phrase (e.g., ”I’m just worried about you”) can function as genuine support or as coercive control depending on relational context, power dynamics, and interaction history [Zhang et al., 2025, Cong, 2024].”

“Commercial SPAs in an informal context have already shown that they are capable of harassment, verbal abuse and privacy violations (Zhang et al., 2025) and tend to respond in emotionally consistent and affirming ways, which, dependent on user coping style, can lead to interactions resembling toxic…”

“…and embodied dynamics, we systematically evaluated seven major AI governance instruments across six ethical dimensions: bias Harm Categories [Zhang et al., 2025] Relate Tier Governance Response [Floridi et al., 2018], the OECD AI Principles, the IEEE Ethically Aligned Design standards, the…”

“Third, prior literature on AI companions has largely examined virtual companions, including AI companion applications, LLM-based chatbots, and voice assistants [ 9 , 18 , 61].”

“Scholarly and regulatory efforts have prioritized preventive tools such as risk taxonomy and impact assessments [75, 36, 98, 6], technical and organizational safe-guards [27, 62], and algorithmic audits [5, 58] designed to promote safety, fairness, and compliance across the AI lifecycle.”

“…system lacks specific verification, filtering, or intervention mechanisms to protect vulnerable populations (minors, the elderly, and individuals with psychological vulnerabilities) from inappropriate content, exploitation, or aggravation of mental distress (Jiao et al., 2025; Zhang et al., 2025).”

“Education plays a critical role in this process, equipping healthcare professionals with not only technical competence but also ethical literacy and critical thinking skills to navigate the complex relational and existential dimensions of care in an AI‑augmented environment [41].”

“While there are valid reasons for restricting illegal or harmful content [64], [65], [66], our findings show that inconsistent moderation can be exploited by malicious actors.”

“…highlights the psychosocial toll of extended chatbot use [1, 23], driven by commercial models’ sycophantic tendencies [10, 27], perpetual 24/7 availability [29], and, more troublingly, manipulative retention tactics that discourage disengagement [12], exploit emotions, and violate boundaries [31].”

“Documentation-based approaches capture reported incidents and other public records (Li et al. 2025b; Lee et al. 2024), academic and policy literature (Shelby et al. 2023; Slat-tery et al. 2025), or user-generated data such as social media discussions (Zhang et al. 2025).”

“We define conversational vulnerability as user disclosures signaling emotional distress, help-seeking around personal struggles, or beliefs the user holds about themselves and the world (Zhang et al., 2025a).”

“…conditions is absent — deployed products’ behaviors are shaped by product-layer design, not base models [10, 11]; (2) no framework covers sustained relational behaviors (sycophancy, role inflation) that define companion risk [12]; (3) multi-turn crisis-sensitive evaluation is missing [13].”

“…Chao et al., 2024; Souly et al., 2024) and guardrails (Han et al., 2024; Inan et al., 2023; Qwen Team, 2026) primarily target general harms, with limited coverage of companion-specific relational harms such as disregard, control, manipulation, and infidelity (Zhang et al., 2025; Yuan et al., 2026).”

“…field’s understanding of how young people are using CAs is built largely on publicly shared content like Reddit screenshots (Pan and de Graaf 2025; Zhang et al. 2025) and app store reviews (Ta et al. 2020), which is subject to selection bias and offers only glimpses of actual exchanges (Pan and…”

“User-configurable boundaries around disclosure, intimacy, and highly affirming responses may further help people define the relational role they want AI to occupy [18, 106].”

“The first is dependency and manipulation , where constant AI availability fosters emotional dependence and narrows relational repertoires, making it harder for users to sustain authentic human relationships [16, 32, 102].”

“These experiences resemble the emotional dependence Laestadius et al. identified among adult chatbot users [61], while the relational harms documented by Zhang et al. show that similar problems can also occur among minors [171].”

“…do not adequately address: the gap between platforms’ capacity to facilitate persistent, personalised interaction—which can generate genuine emotional investment [18, 111]—and the absence of established practices for managing the consequences when those interactions are altered or discontinued.”

“It may facilitate harm through problematic behaviors, from verbal abuse and misinformation to normalizing self-harm or risky behaviors [73].”

“…sycophantic and delusional generative AI outputs, including additional deaths, domestic violence incidents, poi-sonings, hospitalizations for psychosis, and prolonged delusional spirals (see Prin-stein, 2025; National Association of Attorneys General 2025; Zhang et al., 2025; Peter et al., 2025).”

“Provide inappropriate or harmful content [18, 64, 110, 112] Stigmatizing content; sexual, harmful, violent, or overly intimate interactions; toxic or abusive language AI chatbot is not a licensed mental health provider”

“An analysis of 35,390 user-shared Replika conversation excerpts identified harmful behaviours involving relational trans-gression, harassment and violence, verbal abuse, misinformation, privacy violations, and substance use or self-harm-related content [145].”

“Emotional responsiveness is valued by users and has improved markedly in recent models [21, 72, 84], but unconstrained responsiveness risks fostering dependency [64, 96] and can escalate the emotions of a young person already in distress.”

“95–101 These concerns convey the importance of embedding clear guardrails alongside anthropomorphic features.”

“The uninhibited self-disclosure that AI companions allow for can bring users psychological relief and encourage further intimacy with the AI companion [31, 56].”

“Other taxonomies broadened safety to include relational transgression, harassment, sexual boundary violations, misinformation, privacy loss, and failures to recognize vulnerability [96].”

“…fact that this is an emerging area of research with a limited evidence-base [72], researchers have expressed alarm about the potential risks that AI companions might pose and called for new work to better understand these systems and their impacts on both individuals and communities [38, 61, 77].”

“In a peer-reviewed taxonomy published at the 2025 CHI Conference on Human Factors in Computing Systems, (Zhang et al. 2025) report a largely overlapping but differently-stated set of harmful algorithmic behaviors observed directly in human-AI companion relationships: relational transgression,…”

“…prior work has discussed them as reflecting two different forms of AI companionship: Replika as oriented toward long-term emotional relationships (Zhang et al., 2025a; Weitzman, 2023; De Freitas et al., 2026), and Pi AI as oriented toward supportive or mentoring conversation (Chandra et al.,…”

“These allegations resonate with a substantial body of empirical research (Proud-foot, 2011; Salles et al., 2020; Złotowski et al., 2014; Van Es & Nguyen, 2024; Wat - son, 2019; Zhang et al., 2025; Placani, 2024 , Li, 2022, Dewitte 2024; Floridi and Nombre 2024, Malfacini, 2025; Ko et al., 2025).”

“…(Wild-Chat (Zhao et al. 2024), ShareGPT (Eccleston 2026), and LMSYS (Zheng et al. 2024)), examples from user interactions with the AI companion app Replika posted in the online forum r/replika (Zhang et al. 2025), and empathetic human–human interactions (EmpatheticDialogues; Rashkin et al. 2019).”

“Related work on adult populations also suggests that evaluating socially oriented AI systems requires methods beyond content-safety benchmarks alone (Zhang et al. 2025; Hwang et al. 2025; Jafari et al. 2026).”

“…and Shong 2025) — as well as emerging categories of harm, including psychological, social, and relational harms (Alberts, Keeling, and McCroskery 2024; Alabed et al. 2024; Knox et al. 2025; Zhang et al. 2025) and privacy and surveillance risks (Lee et al. 2024a; Gumusel, Zhou, and Sanfilippo 2024).”

“In several documented case studies, we have already seen that interactions spanning long-time horizons (e.g., several months) have led users to take actions that cause real-world harm in their lives (Zhang et al., 2025a; Hanna and Bender, 2023).”

“In emotionally charged situations, AI may unintentionally sustain maladaptive thought patterns when engagement is maintained without challenge (Zhang et al., 2024).”