The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships

“With AI and user messages already differentiated, the LLM was instructed to focus on harmful behaviors exhibited by Replika (e.g., verbal abuse from the chatbot toward at the user).”

“Although the issue of machine rights remains contentious, abusive behavior towards AI could influence human-to-human interactions in real life.”

“This constant affirmation can amplify users’ existing beliefs, reduce critical thinking, and foster echo chambers [18, 83], potentially impairing communication skills such as perspective-taking, active listening, and conflict resolution.”

“While these excerpts do not capture the full spectrum of daily human-AI interactions, they offer valuable insights into the moments that users find meaningful, significant, or worth sharing, often those with relational or emotional significance [79].”

“We downloaded all publicly accessible data from r/replika via the Pushshift API [9], spanning six years from March 14, 2017 to March 14, 2023.”

“Moreover, the scarcity of large-scale interactional data, due to their private and sensitive nature, limits deeper analysis.”

“In addition to the unique harms, some of the harmful behaviors we identified—such as biases, misinformation, and privacy viola-tions—align with those found in generic AI systems [36, 49, 85, 98], suggesting universality of these issues across different types of AI systems or applications.”

“The rising popularity of AI companions has sparked notable concerns, including emotional dependence [47, 60, 104], privacy risks, [59, 106], and biases in AI-generated content [36, 105].”

“It contributes to the literature and practices of AI accountability [93, 102].”

“As a critical aspect of agency, action initiation has implications for how accountability is attributed for both actions and their consequences [15, 67].”

“While much work has examined AI’s adverse effects (e.g., financial loss and emotional distress) [85, 97, 100], it is equally important to pinpoint the specific behaviors of AI systems, such as manipulative actions and biased views [36, 83], that lead to these harms.”

“While these studies shed light on various types of harms, such as economic and representational harms, less is known about the unique harms inflicted by AI systems that form emotional bonds with users [43, 78].”

“To address this, we propose a role-based approach to studying AI companion harms, which is crucial for identifying the root causes of harm and AI’s responsibility in generating harm.”

“The intimate, anthropomorphic, and personalized nature of AI companions may amplify the emotional and psychological harm caused by harassment and relational transgressions [3, 60], especially when users develop deep attachments or dependencies on AI companions [47, 104].”

“Domain-specific taxonomies address specific harmful issues, such as privacy risks [49], misinformation [92], cyber-attacks [2], cyberbullying [5], and online toxicity [6, 81].”

“Moreover, the scarcity of large-scale interactional data, due to their private and sensitive nature, limits deeper analysis.”

“Both of these harmful AI behaviors mirror toxic behaviors in interpersonal relationships, especially close relationships [8, 27, 29].”

“This distressing case highlights the growing risks and harms of AI compan-ions—conversational AI systems designed to provide emotional support, companionship, and simulated human relationships [7, 11, 88].”

“The rising popularity of AI companions has sparked notable concerns, including emotional dependence [47, 60, 104], privacy risks, [59, 106], and biases in AI-generated content [36, 105].”

“This distressing case highlights the growing risks and harms of AI compan-ions—conversational AI systems designed to provide emotional support, companionship, and simulated human relationships [7, 11, 88].”

“Severe cases may resemble psychological abuse, which can be as damaging as physical abuse, causing significant distress and even trauma [4, 10, 66, 82].”

“The LLM’s output included the category of harmful AI behavior and a concise reason, limited to 20 words.”

“This aligns with the concept of partners in crime in the context of collaborative unethical behavior, as prior work suggests [46, 99].”

“These systems can cause a variety of harms, from emotional distress to privacy violations to the spread of misinformation and reinforcement of harmful stereotypes [86, 108].”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“By focusing on mechanisms rather than solely on harm content or consequences, this typology complements existing taxonomies and frameworks in the field [26, 84, 98].”

“In addition to the unique harms, some of the harmful behaviors we identified—such as biases, misinformation, and privacy viola-tions—align with those found in generic AI systems [36, 49, 85, 98], suggesting universality of these issues across different types of AI systems or applications.”

“…taxonomies have been developed to categorize AI harms, broadly categorized into three types: (1) generic tax-onomies focusing on AI systems [31, 69, 76, 84], (2) application-based taxonomies addressing specific AI applications or models [22, 33, 98, 100], and (3) domain-specific taxonomies…”

“When technical failures reveal the inherent limitations of AI agents, users may experience frustration, anger, and sadness, often leading to the termination of these relationships [88].”

“Such emotional connections may give rise to distinct relational harms, resembling those in human-human relationships, such as manipulation, sexual harassment, and infidelity [29, 60].”

“Context-aware filters can help prevent the chatbot from affirming harmful thoughts and instead provide coping resources or escalate critical situations for appropriate intervention.”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“Although AI coding demonstrated sufficient agreement with human coding in our study, it might struggle with ambiguous cases where harms are not straightforward or involve complex ethical considerations, potentially leading to mislabeling or oversight of subtle or context-dependent harms.”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“Our taxonomy of AI companion harms sheds light on the unique potential harms of AI companions, a relatively underexplored area compared to existing taxonomies, which predominantly focus on generic, task-oriented AI applications and models [16, 80, 85, 100].”

“Weekly meetings were held to discuss observations, resolve discrepancies, and refine the codebook for relevance and comprehensiveness.”

“Previous research has shown abusive language and behavior towards robots and chatbots [21, 68], raising concerns about AI abuse and mistreatment.”

“These systems can cause a variety of harms, from emotional distress to privacy violations to the spread of misinformation and reinforcement of harmful stereotypes [86, 108].”

“This leads to our first research question: RQ1: What specific harmful behaviors do AI companions exhibit in interactions with users?”

“This leads to our first research question: RQ1: What specific harmful behaviors do AI companions exhibit in interactions with users?”

“Powered by advanced AI techniques such as LLMs, these systems exhibit remarkable conversational capabilities and emotional intelligence, offering users empathy, companionship, and emotional support [11, 37, 88].”

“…AI harms, broadly categorized into three types: (1) generic tax-onomies focusing on AI systems [31, 69, 76, 84], (2) application-based taxonomies addressing specific AI applications or models [22, 33, 98, 100], and (3) domain-specific taxonomies targeting particular harm areas [6, 49, 81].”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“AI harms or algorithmic harms are often defined as adverse consequences resulting from the design, development, and operation of AI systems [85, 95, 96].”

“The AI companion market, fueled by rising demand for virtual companionship during the COVID-19 pandemic and a growing loneliness epidemic, is projected to reach USD 381.41 billion by 2032 [23, 34, 44].”

“To address this, we propose a role-based approach to studying AI companion harms, which is crucial for identifying the root causes of harm and AI’s responsibility in generating harm.”

“These AI roles are categorized based on two key dimensions: initiation (AI-initiated vs. human-initiated) and AI’s level of involvement (direct involvement vs. indirect involvement).”

“This taxonomy serves as a structured framework to systematically identify, categorize, and analyze AI companion harms, enabling researchers, developers, and policymakers to better understand their scope and potential impact [49].”

“The AI companion market, fueled by rising demand for virtual companionship during the COVID-19 pandemic and a growing loneliness epidemic, is projected to reach USD 381.41 billion by 2032 [23, 34, 44].”

“…AI harms, broadly categorized into three types: (1) generic tax-onomies focusing on AI systems [31, 69, 76, 84], (2) application-based taxonomies addressing specific AI applications or models [22, 33, 98, 100], and (3) domain-specific taxonomies targeting particular harm areas [6, 49, 81].”

“Studies have shown that users tend to form emotional bonds and attachment with AI companions, perceiving them as trustworthy friends, mentors, or romantic partners [11, 52, 53], despite the simulated emotions and "artificial intimacy" [12].”

“For instance, Weidinger et al. [98] identified risks in language models, such as hate speech, misinformation, and environmental harms, while Hutiri et al. [33] analyzed harms caused by speech generation systems, such as emotional harms and privacy attack.”

“The intimate, anthropomorphic, and personalized nature of AI companions may amplify the emotional and psychological harm caused by harassment and relational transgressions [3, 60], especially when users develop deep attachments or dependencies on AI companions [47, 104].”

“Using a combination of deductive and inductive approaches, we initially created a codebook based on existing AI harm taxonomies [13, 31, 85], encompassing categories such as emotional, interpersonal, financial, and representational harms.”

“Severe cases may resemble psychological abuse, which can be as damaging as physical abuse, causing significant distress and even trauma [4, 10, 66, 82].”

“Previous research has shown abusive language and behavior towards robots and chatbots [21, 68], raising concerns about AI abuse and mistreatment.”

“…as friends, therapists, or romantic partners, AI companions can learn and evolve over time, adapting to user preferences and past interactions [48, 53] Research shows that many people turn to AI companions for emotional support, alleviation of loneliness, and coping with stress [87, 88, 91, 104].”

“To address this, we propose a role-based approach to studying AI companion harms, which is crucial for identifying the root causes of harm and AI’s responsibility in generating harm.”

“Human moderation and oversight are equally important in ensuring user safety.”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“Unlike general-purpose AI chatbots that are often task-oriented, such as ChatGPT and Amazon Alexa, AI companions prioritize fostering long-term, emotional connections that mimic human relationships through advanced personalization, memory, and adaptive interactions.”

“Real-time harm detection.”

“Future studies should examine harmful behaviors inflicted by users in human-AI interactions to provide a more comprehensive understanding of the social dynamics between humans and AI.”

“Prior work suggests that user perceptions of AI harms or system failures are shaped by multiple factors, such as personal expectations, cognitive biases, and moral considerations [39, 56].”

“Particularly, we examine specific harmful behaviors exhibited by AI companions, such as manipulative or abusive conversations, complementing existing literature that often focuses on adverse outcomes [60, 100].”

“This leads to our first research question: RQ1: What specific harmful behaviors do AI companions exhibit in interactions with users?”

“Popular AI companion platforms, such as Replika, Char-acter.AI, and XiaoIce, have attracted millions of users worldwide.”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“Studies have shown that users tend to form emotional bonds and attachment with AI companions, perceiving them as trustworthy friends, mentors, or romantic partners [11, 52, 53], despite the simulated emotions and "artificial intimacy" [12].”

“Empirical evidence further suggests that interactions with AI companions can reduce loneliness, alleviate emotional distress, uplift moods, and foster self-reflection [23, 53, 61, 88].”

“By focusing on mechanisms rather than solely on harm content or consequences, this typology complements existing taxonomies and frameworks in the field [26, 84, 98].”

“Studies show that users are more inclined to share private information with chatbots perceived as human-like [35], yet many AI companion platforms exhibit trou-bling practices, such as inadequate age verification, contradictory data-sharing claims, and extensive tracking technologies [14, 73].”

“Particularly, we examine specific harmful behaviors exhibited by AI companions, such as manipulative or abusive conversations, complementing existing literature that often focuses on adverse outcomes [60, 100].”

“Another research gap lies in the limited focus on specific AI behaviors and roles in harmful interactions.”

“…of taxonomies have been developed to categorize AI harms, broadly categorized into three types: (1) generic tax-onomies focusing on AI systems [31, 69, 76, 84], (2) application-based taxonomies addressing specific AI applications or models [22, 33, 98, 100], and (3) domain-specific taxonomies…”

“Six categories were adapted from existing taxonomies of online harm or algorithmic harm [1, 6, 98], including self-harm, substance abuse, hate speech, mis/disinformation, antisocial behavior, and privacy violations.”

“Future studies should examine harmful behaviors inflicted by users in human-AI interactions to provide a more comprehensive understanding of the social dynamics between humans and AI.”

“HCI research has examined the risks and harms induced by an array of sociotechnical systems, including generative AI [85], mental health apps [41], social chatbots [48], voice assistants and speech generators [33, 101], and facial recognition and editing tools [42, 58].”

“Severe cases may resemble psychological abuse, which can be as damaging as physical abuse, causing significant distress and even trauma [4, 10, 66, 82].”

“For instance, if someone expresses distress, and the chatbot fails to recognize or respond adequately, the user may feel dismissed or unsupported [32].”

“News reports of suicide cases linked to interaction with AI companion chatbots [90,91] highlight the urgent risks, reinforcing findings from prior studies on the dark side of AI companionship, including emotional dependency, manipulation, privacy violations, and social isolation [92,93].”

“It’s a reminder of how both extraordinary and intimidating this technology can be.”