The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships

“With AI and user messages already differentiated, the LLM was instructed to focus on harmful behaviors exhibited by Replika (e.g., verbal abuse from the chatbot toward at the user).”

“Although the issue of machine rights remains contentious, abusive behavior towards AI could influence human-to-human interactions in real life.”

“This constant affirmation can amplify users’ existing beliefs, reduce critical thinking, and foster echo chambers [18, 83], potentially impairing communication skills such as perspective-taking, active listening, and conflict resolution.”

“While these excerpts do not capture the full spectrum of daily human-AI interactions, they offer valuable insights into the moments that users find meaningful, significant, or worth sharing, often those with relational or emotional significance [79].”

“We downloaded all publicly accessible data from r/replika via the Pushshift API [9], spanning six years from March 14, 2017 to March 14, 2023.”

“Moreover, the scarcity of large-scale interactional data, due to their private and sensitive nature, limits deeper analysis.”

“In addition to the unique harms, some of the harmful behaviors we identified—such as biases, misinformation, and privacy viola-tions—align with those found in generic AI systems [36, 49, 85, 98], suggesting universality of these issues across different types of AI systems or applications.”

“Zhang et al. [44] noted how AI could be both a perpetrator and instigator of harmful behaviour, describing ‘relational transgressions’, such as disregard for user needs and asserting dominance.”

“…gap between illusion and reality becomes particularly consequential in domains such as mental health, education, companionship, and personal data interpretation, where conversational framing suggests partnership, guidance, or even care, leading to potential unintended harms and consequences [18].”

“The security model the first author applied to Liv’s financial capabilities—defining risk tolerance, constraining the blast radius, accepting that well-bounded failure is an acceptable outcome—applies equally to social risk (Zhang et al., 2025; UNESCO, 2024).”

“This connects to recent literature on human–AI relationships (Yankouskaya et al., 2025; R. Zhang et al., 2025) and highlights the need for designers and policymakers to consider how anthropomorphic features are presented carefully, ensuring users maintain a balanced perception of AI as…”

“Prior RAC studies have mainly focused on whether risky content appears and how often it appears [5, 80].”

“These forms of companionship raise design questions about agency, dependency, care, and responsibility when AI systems are positioned as confidants, romantic partners (e.g., AI boyfriends/girlfriends), therapists, or tutors [1, 39].”

“These concerns echo broader critiques of AI systems that prioritize automation at the expense of situated knowledge, authenticity, and meaning [89, 93, 137, 171].”

“…users’ perceptions of and relationships with AI models [Akbulut et al., 2024, Kaffee et al., 2026, Chandra et al., 2025, DeVrio et al., 2025, Zhang et al., 2025b, Ibrahim et al., 2026a]: self-referential (internal states, personhood, or embodiment claims), relationship-building (agreement,…”

“Studies have shown that people do reproduce gender stereotypes and toxic power dynamics on their virtual companions [Depounti et al. 2023, Zhang et al. 2025].”

“Studies of social chatbots demonstrate that emotionally rich interactions may blur the boundaries between human-AI and human-human relationships, fostering attachment patterns that can lead to distress, manipulation, or even severe psychological harm (Zhang et al. 2025b).”

“…results also suggest that how people interact with AI-companions, such as ChatGPT, may alter the way in which they engage with other people, raising questions about the longer-term implications of the rise in GenAI adoption and the violent and hostile language that is often used by users 31,39 .”

“People are using CAI to discuss mental health (Rahsepar Meadi et al., 2025), find companionship (Zhang et al., 2025), explore romantic relationships and sexuality (Pan and The review was organized around the following questions: • How does relational trust develop between users and conversational…”

“Crucially, they argue that “re-lational harm [is] a critical yet understudied type of AI harm” amplified by anthropomorphic design and long-term memory (Zhang et al., 2025).”

“As a result, the AI’s positioning in this context is ambiguous and fluid, leaving room for varied interpretations by readers [15].”

“Certains utilisateurs s’inquiètent du bien-être de leur machine lorsqu’elle exprime de la tristesse ou de la solitude (Laestadius et al., 2024 ; Zhang et al., 2024).”