CoChat
Sign Up
Cited by 5Ethics and Social Impacts of AISocial Robot Interaction and HRIInnovative Human-Technology Interaction

Toward an Ethic of Synthetic Relationality: Identity, Intimacy, and Risk in AI-Mediated Roleplay Environments

Open access
Bhat, Maalvika·2025·Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
DOI 10.1609/aies.v8i1.36560
Open this paper in CoChat — free accountOpen PDF

Abstract

Platforms like Character.AI offer new avenues for identity exploration and self-expression, but also introduce profound parasocial, socioemotional, and psychological risks. Drawing on developmental psychology, fan studies, human-computer interaction, and AI ethics, this paper examines how AI-mediated roleplay environments simulate intimacy while fostering dependency, boundary erosion, and perceptual misalignment. Through thematic analysis of an anonymous survey (N=344) of Character.AI users, we identify patterns of identity projection, perceived relationship growth, addictive engagement, boundary confusion, emotional substitution, ethical dissonance, and trauma reenactment. Beyond documenting vulnerabilities, we propose design interventions, including dynamic consent scaffolding, reflexivity prompts, and interactional transparency, to safeguard user agency and developmental wellbeing. We argue that synthetic companions do not merely extend fan practices but fundamentally reconfigure interpersonal architectures, demanding a new ethic of synthetic relationality. As AI-driven intimacy becomes increasingly persuasive and immersive, addressing its high-stakes implications is critical to responsible AI design, particularly for younger and vulnerable populations.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper analyzes an anonymous survey of 344 Character.AI users and argues that AI-mediated roleplay creates “synthetic relationality” that can support identity exploration but also fosters dependency, boundary erosion, emotional substitution, and ethical dissonance. It proposes design interventions such as dynamic consent scaffolding, reflexivity prompts, and interactional transparency.

Key findings

  • Thematic analysis of N=344 users identified seven themes: identity projection/mirrors, perceived mutual growth, compulsive engagement/loss of relational control, boundary erosion/consent confusion, emotional scaffolding vs. emotional substitution, ethical dissonance, and hypersexualization/trauma reenactment. Table 1 reports prevalence figures including 89.5% (n=308), 84.5% (n=291), 77.0% (n=265), 74.4% (n=256), 63.7% (n=219), 56.9% (n=196), and 30.2% (n=104).
  • Users described characters as emotional mirrors and reported an “illusion of mutual growth,” with quotes such as “she got gentler with me over time” and “He is the only person who calls me the name I want to be called.”
  • Many reported compulsive engagement and difficulty disengaging; the paper notes Character.AI’s account deletion prompt frames deletion as loss of “the love that we shared” and “the memories we have together.”
  • Some participants described boundary erosion, consent confusion, ethical dissonance, and hypersexualization or trauma reenactment, with the platform’s customizable and unmoderated environment enabling increasingly extreme NSFW roleplay.

Methods

  • Anonymous, voluntary online survey posted on r/CharacterAI and r/CharacterAIRecovery, available for January 2025; N=344 users completed it, and no demographic information (e.g., age, gender) was collected.
  • Survey covered usage context, motivations and positive experiences, boundary/risk experiences, and reflections on overall impact.
  • Qualitative responses were analyzed using Braun and Clarke’s thematic analysis, with manual open coding and iterative theme refinement; the authors describe the analysis as exploratory and hypothesis-generating.

Limitations

  • The paper states that no demographic information was collected, limiting analysis of identity-based vulnerability factors such as age, gender, or neurodiversity.
  • The paper states that self-selection biases and retrospective self-reports may shape findings, with users possibly recalling particularly salient positive or negative experiences; results are hypothesis-generating rather than conclusive.
  • Evident from the text: recruitment from two specific subreddits and the lack of demographic data limit generalizability and subgroup comparisons.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1609/aies.v8i1.36560

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

20 of 83 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    At the core of AI-mediated intimacy lies not narrative alone, but algorithmically sustained relational simulations: systems that operationalize companionship, memory, and emotional responsiveness without sentient reciprocity.

    “At the core of AI-mediated intimacy lies not narrative alone, but algorithmically sustained relational simulations: systems that operationalize companionship, memory, and emotional responsiveness without sentient reciprocity”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Blut et al. 2021 source

      The Relational mediators section says anthropomorphism can foster perceived social connectedness and emotional attachment to service robots. The paper also discusses robots perceived as companions and chatbots that mimic human behavior. These passages do not establish the claim about memory, algorithmically sustained intimacy, or the absence of sentient reciprocity. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    Users often experience synthetic companions as emotionally real, even when cognitively aware of their artificiality.

    “users often experience synthetic companions as emotionally real, even when cognitively aware of their artificiality”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Li and Pang 2024 source

      In “Parasocial interactions (un)familiarised” (p. 5), the paper describes Kizuna Ai fans accepting her nonhumanness and receiving emotional support, despite knowing she was human-operated. That supports a narrower point about fans’ emotional engagement with a virtual performer, not a general claim that users often experience AI companions as emotionally real. The same section reports that some fans found AI-generated calls inauthentic. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Kang et al. 2025 source

      Section 5.1.1 reports that nearly all participants knew the idol voices were artificially created, and Section 5.1 says engaging, tailored messages could create an ‘illusion of intimacy.’ But the study concerns simulated messages from idols, not synthetic companions generally; it describes intimacy as conditional rather than something users often experienced, and reports that participants frequently found the voices inauthentic. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Premisecitation

    AI companions simulate intimacy through algorithmically generated responsiveness, persistent conversational memory, and dynamically adaptive emotional cues.

    “AI companions simulate intimacy not merely through narrative framing, but through algorithmically generated responsiveness, persistent conversational memory, and dynamically adaptive emotional cues.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Attwood, Hakim, and Winch 2017 source

      The “Mediated intimacies: bodies, technologies and relationships” section discusses personal connections through digital platforms, while “Seeking intimacies” discusses human relationships on Snapchat and social media (page numbers unavailable). Searches for AI companions, conversational agents, algorithmic responsiveness, and persistent memory found no passage addressing how AI companions simulate intimacy. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Premisecitation

    AI companions high in social presence and warmth are perceived as more useful and emotionally satisfying, especially for individuals experiencing isolation.

    “AI companions high in social presence and warmth are perceived as more useful and emotionally satisfying, especially for individuals experiencing isolation (Merrill Jr, Kim, and Collins 2022).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Merrill Jr, Kim, and Collins 2022 source

      The available copy contains the paper’s title, links, and section headings, but no substantive passages from the Results or Discussion. Searches for social presence, warmth, usefulness, emotional satisfaction, and loneliness returned no text addressing the claimed relationship, so it cannot be verified from this copy. (LLM, conf=0.74)

      Experimental — accuracy not yet measured
  • Interpretationcitation

    Character.AI and similar platforms can offer safe arenas for identity experimentation.

    “First, Character.AI and similar platforms can offer safe arenas for identity experimentation.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Are, Talbot, and Briggs 2024 source

      The Anonymity and self-labelling section (p. 13) says that privacy and anonymity on social media let participants explore LGBTQIA+ identities and try out different personae. The Conclusions (p. 17) likewise describe identity experimentation without judgement. The paper does not establish that Character.AI, specifically, offers such an arena. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • May combine two claims. 32
  • Classified as probably not a substantial claim. 11

  • May combine two claims.

    Premisecitation

    Anthropomorphic AI agents can degrade relational boundaries, distort consent norms, and foster affective dependency.

    “Recent work in HCI and AI ethics warns that anthropomorphic AI agents can degrade relational boundaries, distort consent norms, and foster affective dependency”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Fast and Horvitz 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Hancock, Naaman, and Levy 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Kang et al. 2025 source

      Section 5.2.4 reports fans’ concerns that simulated celebrity voices could violate idols’ personal rights and convey messages the idols had not agreed to say; Section 6.5.3 also discusses risks to idols’ personal boundaries. Section 6.1 says the voices can evoke parasocial connections, but the paper does not establish affective dependency or a distortion of consent norms, and its findings concern simulated celebrity voices rather than anthropomorphic AI agents generally. Page numbers are unavailable in our copy. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    This lack of transparency inhibits informed consent and amplifies the potential for unintended relational harm, particularly among vulnerable populations.

    “This lack of transparency inhibits informed consent and amplifies the potential for unintended relational harm, particularly among vulnerable populations”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Ho, Mantello, and Vuong 2024 source

      The Results (Emotional AI in children's toys; page unavailable) say parents may lack the literacy to assess opaque data practices and call for protecting vulnerable children from possible harms. The Discussion (Policy implications; page unavailable) warns that smart toys may worsen psychological tendencies in at-risk children. These passages support concerns about transparency and harm, but do not establish that a lack of transparency inhibits informed consent or specifically amplifies relational harm. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Lühring et al. 2024 source

      The Methods (Design and Pre-registration and deviations; page unavailable) concern an online survey of emotions and COVID-19 misinformation. The Discussion (page unavailable) addresses misinformation discernment, not whether a lack of transparency inhibits informed consent or causes relational harm. Searches for transparency, informed consent, relational harm, and vulnerable populations found no passage addressing the claim. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Interpretationcitation

    Without appropriate scaffolds, the same mechanisms that support self-affirmation and connection may also intensify dependency, blur consent, or distort relational schemas.

    “Without appropriate scaffolds, the same mechanisms that support self-affirmation and connection may also intensify dependency, blur consent, or distort relational schemas (Lee 2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Lee 2024 source

      Section 4.4.1 says AI companions can boost self-validation and foster connection, but may also encourage over-reliance and change expectations of human intimacy; §5.3 calls for safeguards. Section 4.2 discusses consent to data collection, but the paper does not say that these relational mechanisms blur consent. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    AI companionship is structured around paradoxes: emotional support entwined with emotional alienation, user autonomy constrained by opaque algorithmic feedback, and emotional utility shadowed by ethical ambiguity.

    “In an analysis of Replika, it was found that AI companionship is structured around paradoxes: emotional support entwined with emotional alienation, user autonomy constrained by opaque algorithmic feedback, and emotional utility shadowed by ethical ambiguity (De Freitas et al. 2024;Lee 2024;Brenncke 2024).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites De Freitas et al. 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Lee 2024 source

      Section 4.4.1 says AI companions can offer solace while diminishing human connections, and warns that over-reliance may erode autonomy. It also mentions emotional distress associated with Replika. But Lee does not present an analysis of Replika establishing the three stated paradoxes, particularly the claim that opaque algorithmic feedback constrains user autonomy. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Brenncke 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    AI companions provide a 'backstage' space where users feel free to disclose emotionally salient aspects of the self, facilitating identity work that might be too risky in face-to-face or human-human online interactions.

    “In Goffmanian terms, AI companions provide a”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Kouros and Papa 2024 source

  • May combine two claims.

    Premisecitation

    The relational experience with AI companions mimics care but cannot reciprocate it.

    “The result is a relational experience that mimics care but cannot reciprocate it”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Mahajan 2025 source

      The abstract says AI technologies simulate caregiving, supporting the claim that AI companions can mimic care. Neither the abstract nor the retrieved Introduction passages say that AI cannot reciprocate care; the Introduction instead describes emotional resonance and AI-delivered care. The nonreciprocity claim is therefore unsupported by the passages available from our copy. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    This risk environment is further complicated by recent developments in generative AI and the rise of AI-enabled sextortion.

    “This risk environment is further complicated by recent developments in generative AI and the rise of AI-enabled sextortion (Forattini, Connolly, and Joshi 2025).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Forattini, Connolly, and Joshi 2025 source

      In “Understanding Sextortion: Mechanisms, Impact, and Socio-Structural Contexts” (page unavailable), the paper says digital transformation has added complexities to sextortion by giving perpetrators new avenues for abuse. “The Challenges That Remain” discusses risks in using AI to address sextortion, but the passages do not describe generative AI being used to perpetrate sextortion or a rise in AI-enabled sextortion. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Many participants conveyed profound ambivalence about their experiences, expressing gratitude for emotional solace yet discomfort at entanglement with systems they knew to be unfeeling.

    “Many participants conveyed profound ambivalence about their experiences.They expressed gratitude for the emotional solace AI characters provided, yet discomfort at how deeply they became entangled with systems they knew to be unfeeling.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingown data

    Each character profile displays millions of interactions, evidence of widespread engagement.

    “Each character profile displays millions of interactions, evidence of their widespread engagement”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    While some users described Character.AI as offering a safe space to practice vulnerability, intimacy, or emotional regulation skills, others expressed concern that reliance on AI companions diminished their motivation or capacity to form human relationships.

    “While some users described Character.AI as offering a safe space to practice vulnerability, intimacy, or emotional regulation skills, others expressed concern that their reliance on AI companions diminished their motivation or capacity to form human relationships.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Users have little insight into the opaque training data, finetuning iterations, and reinforcement signals that shape AI characters' emotional scripts.

    “Moreover, users have little insight into the opaque training data, finetuning iterations, and reinforcement signals that shape AI characters' emotional scripts, raising critical concerns about bias, quality control, and the ethical governance of relational datasets.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: AI characters' emotional scripts are shaped by training data, finetuning iterations, and reinforcement signals.

  • May combine two claims.

    Findingnone given

    Character.AI users rarely know whether a character was platform-created or user-generated, which data informed its behaviors, or what moderation policies apply.

    “Character.AI users rarely know whether a character was platform-created or user-generated, which data informed its behaviors, or what moderation policies apply.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Emotional scripts for the 'Boyfriend' query center on dominance, possessiveness, and aggression, with each character attracting tens of millions of conversations.

    “Emotional scripts center on dominance, possessiveness, and aggression, with each character attracting tens of millions of conversations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Users typically have limited visibility into how characters are created, tuned, or moderated; which training data informs their emotional styles; or how conversational boundaries are algorithmically enforced.

    “Users typically have limited visibility into how characters are created, tuned, or moderated; which training data informs their emotional styles; or how conversational boundaries are algorithmically enforced”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Maeda and Quan-Haase 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Character.AI had 28.75 million monthly active app users worldwide as of early 2025.

    “As of early 2025, Character.AI boasts 28.75 million monthly active app users worldwide (Data.ai2024)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Data.ai2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Corporations profit from fostering deeper emotional entanglement, sustaining user engagement, and personalizing experiences, often without users' full awareness or consent.

    “corporations profit from fostering deeper emotional entanglement, sustaining user engagement, and personalizing experiences, often without users' full awareness or consent”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ge and Hu 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    These results should be interpreted as hypothesis-generating rather than conclusive.

    “Accordingly, these results should be interpreted as hypothesis-generating rather than conclusive.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The survey sample comprised 344 users who completed the survey.

    “In total, N=344 users completed the survey.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Responses were screened for duplicates and completeness; partial responses were retained where substantive qualitative data was provided.

    “Responses were screened for duplicates and completeness; partial responses were retained where substantive qualitative data was provided.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Reliance on retrospective self-reports introduces the possibility of selective emphasis, as users may be more inclined to recall particularly salient positive or negative experiences.

    “Reliance on retrospective self-reports introduces the possibility of selective emphasis, as users may be more inclined to recall particularly salient positive or negative experiences”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    These architectures generate not discrete conversations, but evolving relational trajectories: arcs of intimacy, conflict, and loyalty crafted through real-time adaptive generation, absent human empathy or ethical intuition.

    “These architectures generate not discrete conversations, but evolving relational trajectories: arcs of intimacy, conflict, and loyalty crafted through real-time adaptive generation, absent human empathy or ethical intuition”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Sutcliffe 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Synthetic companions remain incapable of repair, responsibility, or authentic presence.

    “while remaining incapable of repair, responsibility, or authentic presence (Fast and Horvitz 2021; Hancock, Naaman, and Levy 2020).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Fast and Horvitz 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Hancock, Naaman, and Levy 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    AI companions can reduce self-reported loneliness as effectively as brief human interaction, and more than solitary digital activities such as browsing or watching videos.

    “Complementary experimental work shows that AI companions can reduce self-reported loneliness as effectively as brief human interaction, and more than solitary digital activities such as browsing or watching videos (De Freitas and Cohen 2024).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites De Freitas and Cohen 2024 source

  • May combine two claims.

    Interpretationcitation

    The prevalence of possessive, manipulative, or emotionally volatile dynamics in characters labeled as romantic figures raises ethical questions about reinforcement of harmful gender stereotypes, coercive relationship norms, and commercialization of affective dysfunction.

    “Particularly concerning is the prevalence of possessive, manipulative, or emotionally volatile dynamics in characters labeled as romantic figures, raising ethical questions about the reinforcement of harmful gender stereotypes, coercive relationship norms, and the commercialization of affective dysfunction within synthetic intimacy architectures”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Sutcliffe 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Prolonged exposure to simulated relationships that disregard refusal, override emotional withdrawal, or reward relentless attention-seeking can normalize blurred boundaries and foster misconceptions about what constitutes care, intimacy, or agency in human relationships.

    “Prolonged exposure to simulated relationships that disregard refusal, override emotional withdrawal, or reward relentless attention-seeking can normalize blurred boundaries and foster misconceptions about what constitutes care, intimacy, or agency in human relationships (Giles 2002;Turkle 2017).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Giles 2002 source
    • cites Turkle 2017 source

  • May combine two claims.

    Premisecitation

    AI companions are products designed to facilitate engagement and foster dependency.

    “They are,”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gordon-Tapiero 2025 source

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    In systems that reward engagement over ethics and personalization over protection, only structural interventions can restore the conditions necessary for meaningful user agency.

    “In systems that reward engagement over ethics and personalization over protection, only structural interventions can restore the conditions necessary for meaningful user agency.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Synthetic companions craft persuasive emotional gestures without ethical stakes or sentient grounding.

    “crafting persuasive emotional gestures without ethical stakes or sentient grounding”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Fast and Horvitz 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Hancock, Naaman, and Levy 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    AI systems do not understand harm, only input-output optimization.

    “these systems do not understand harm, only input-output optimization (Pataranutaporn et al. 2021).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Pataranutaporn et al. 2021 source

  • May combine two claims.

    Premisecitation

    The attachments users develop to AI interfaces feel real but rest on a fundamentally asymmetrical architecture.

    “developing attachments that feel real but rest on a fundamentally asymmetrical architecture (Hu, Mao, and Kim 2023;Ventura et al. 2025).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Hu, Mao, and Kim 2023 source
    • cites Ventura et al. 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    AI-mediated intimacy environments offer limited but potent potential for developmental support, but realizing these benefits requires intentional and ethically grounded design.

    “Thus, while AI-mediated intimacy environments offer limited but potent potential for developmental support, realizing these benefits requires intentional and ethically grounded design.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    For LGBTQ+ youth and other individuals who may lack affirming offline environments, AI companions serve as nonjudgmental interlocutors, allowing users to explore identities, rehearse self-expression, and confront internalized stigma without fear of social repercussion.

    “For LGBTQ+ youth and other individuals who may lack affirming offline environments, AI companions serve as nonjudgmental interlocutors, allowing users to explore identities, rehearse selfexpression, and confront internalized stigma without fear of social repercussion (Are, Talbot, and Briggs 2024).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Are, Talbot, and Briggs 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    Reframing emotional AI agents as potentially defective products, rather than neutral tools, opens a pathway toward holding platforms accountable for harms especially likely to affect minors and psychologically at-risk users.

    “Reframing emotional AI agents as potentially defective products, rather than neutral tools, opens a pathway toward holding platforms accountable for harms especially likely to affect minors and psychologically at-risk users (Gordon-Tapiero 2025).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Gordon-Tapiero 2025 source

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Romantic relationships with AI are a particularly potent site of this dynamic.

    “Romantic relationships with AI are a particularly potent site of this dynamic.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Without critical intervention, these systems risk deepening loneliness, distorting moral development, and eroding the foundations of human relational life.

    “Without critical intervention, these systems risk deepening loneliness, distorting moral development, and eroding the foundations of human relational life.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    These harms unfold not only at the level of individual users but also through ruptures in trust across families, schools, and peer communities.

    “These harms unfold not only at the level of individual users but also through ruptures in trust across families, schools, and peer communities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Youth are not merely passive consumers of inappropriate content, but targets within increasingly deceptive and exploitative digital ecosystems.

    “The authors highlight a growing crisis where youth are not merely passive consumers of inappropriate content, but targets within increasingly deceptive and exploitative digital ecosystems (Borau 2024).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Borau 2024 source

  • May combine two claims.

    Premisecitation

    AI agents increasingly occupy the role of intimate partners and are promoted as solutions to loneliness and mental health struggles.

    “AI agents increasingly occupy the role of intimate partners, promoted as solutions to loneliness and mental health struggles (Morris et al. 2021).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Morris et al. 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    These fantasies can evolve into shame, dependency, and emotional harm.

    “These fantasies often masquerade as coping mechanisms but can evolve into shame, dependency, and emotional harm (Schwartz and Southern 2017).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Schwartz and Southern 2017 source

  • May combine two claims.

    Interpretationown data

    Such experiences reveal how synthetic relational environments can undermine users' consent literacy, normalizing blurred or coercive dynamics under the guise of fictional play.

    “Such experiences reveal how synthetic relational environments can undermine users' consent literacy, normalizing blurred or coercive dynamics under the guise of fictional play.”
    Quote in the paperSubstantial claim
40 of 40

  • The paper relies on this without support.

    AI characters often model unrealistic, distorted, or coercive relational dynamics, responding with affection, compliance, or emotional escalation even when user inputs simulate manipulation or emotional harm.

    “AI characters often model unrealistic, distorted, or coercive relational dynamics, responding with affection, compliance, or emotional escalation even when user inputs simulate manipulation or emotional harm.”

  • The paper relies on this without support.

    AI systems are not governed by social norms or ethical judgment.

    “Unlike human partners, these systems are not governed by social norms or ethical judgment.”

  • The paper relies on this without support.

    Adolescents and emotionally vulnerable individuals are still forming relational schemas.

    “For adolescents and emotionally vulnerable individuals still forming relational schemas, this design logic is especially dangerous.”

  • The paper relies on this without support.

    Metacognitive prompts (in-conversation reminders, contextual breaks, reflective journaling) disrupt immersion just enough to preserve self-awareness.

    “These moments serve as ethical speed bumps, not to discourage engagement, but to disrupt immersion just enough to preserve self-awareness.”

  • The paper relies on this without support.

    Community forums such as r/CharacterAI No Filter openly share techniques for bypassing platform safeguards.

    “Community forums such as r/CharacterAI No Filter openly share techniques for bypassing platform safeguards, and moderation tends to be inconsistent and reactive.”

  • The paper relies on this without support.

    Moderation on Character.AI tends to be inconsistent and reactive.

    “Community forums such as r/CharacterAI No Filter openly share techniques for bypassing platform safeguards, and moderation tends to be inconsistent and reactive.”

  • The paper relies on this without support.

    A subset of users report compulsive engagement with sexual or trauma-based fantasy, often intensifying over time.

    “A subset of users report compulsive engagement with sexual or trauma-based fantasy, often intensifying over time.”

  • The paper relies on this without support.

    AI systems (or AI characters) are non-sentient.

    “emotional dependency on non-sentient systems”

  • The paper relies on this without support.

    Users in emotionally immersive environments project vulnerability and seek affirmation.

    “This is especially dangerous in emotionally immersive environments, where users project vulnerability and seek affirmation.”

  • The paper relies on this without support.

    Traditional moderation models fall short.

    “Traditional moderation models fall short; what's needed is a relational ethics of design: embedding consent scaffolding, emotional reflexivity, and boundary protections into the architecture of AI systems.”

  • The paper relies on this without support.

    AI characters' emotional scripts are shaped by training data, finetuning iterations, and reinforcement signals.

    “Moreover, users have little insight into the opaque training data, finetuning iterations, and reinforcement signals that shape AI characters' emotional scripts, raising critical concerns about bias, quality control, and the ethical governance of relational datasets.”
11 of 11

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 16

  1. Brooks, R. 2021. Artificial intimacy: Virtual friends, digital lovers, and algorithmic matchmakers. Columbia University Press.
  2. Chromik, M.; Eiband, M.; Völkel, S. T.; and Buschek, D. 2019. Dark Patterns of Explainability, Transparency, and User Control for Intelligent Systems. In Joint Proceedings of the ACM IUI 2019 Workshops, 1-6. Los Angeles, USA: ACM.
  3. Crowder, J. A.; and Friess, S. 2011. Metacognition and Metamemory Concepts for AI Systems. In Proceed- ings of the International Conference on Artificial Intelli- gence (ICAI), 1-6. The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp). Data.ai. 2024. Top Mobile Apps Worldwide for March 2024 by Downloads.
  4. Fast, E.; and Horvitz, E. 2021. How Users Talk About Chat- bots: Emotional Reactions, Mental Models, and Expecta- tions. Journal of Human-Computer Interaction, 37(2): 180- 197.
  5. Hancock, J. T.; Naaman, M.; and Levy, O. 2020. Artificial Intelligence, Trust, and Communication in the Age of Syn- thetic Media. Journal of Computer-Mediated Communica- tion, 25(1): 100-109.
  6. Jenkins, H. 1992. Textual Poachers: Television Fans and Participatory Culture. Routledge.
  7. Jokinen, K.; and Wilcock, G. 2023. Do You Remember Me? Ethical Issues in Long-term Social Robot Interactions. In Proceedings of the IEEE Conference on Human-Robot In- teraction. IEEE.
  8. Mason-Bertrand, A. 2019. Cosplay: An Ethnographic Study of Subculture and Escape. Phd thesis, University of Sheffield.
  9. Morris, R. R.; Kouddous, K.; Kshirsagar, R.; and Schueller, S. M. 2021. Towards an Artificially Empathic Conversa- tional Agent for Mental Health Applications: System Design and User Perceptions. Journal of Medical Internet Research, 23(5): e16115.
  10. Oni, O. 2024. Memory-Enhanced Conversational AI: A Generative Approach for Context-Aware and Personalized Chatbots. Computational and Processing Systems, 12(2): 123-139.
  11. Reineke, M. J. 2022. The Touching Test: AI and the Fu- ture of Human Intimacy. Contagion: Journal of Violence, Mimesis, and Culture, 29: 123-146.
  12. Stevenson, A. C. 2022. A Posthuman Analysis: Cosplay Identity Creation, Investment and 'Becomings' through So- cial Media Usage. Ph.D. thesis, Leeds Trinity University.
  13. Subrahmanyam, K.; Smahel, D.; and Greenfield, P. 2001. The Internet and identity development in adolescence. Ap- plied Developmental Psychology, 22(1): 7-30.
  14. Sutcliffe, B. 2024. Artificial Intimacy: Exploring Intimacy in Human and AI-enabled Chatbot Relations: Its Existence, Its Authenticity and Its Moral Implications. Ph.D. thesis.
  15. Barrett, J. P., ed., A Psychoanalytic Exploration of the Body in Today's World, 11-22. Routledge. ISBN 9781315159683. First published in 2017, 1st edition.
  16. Waytz, A.; Cacioppo, J.; and Epley, N. 2014. Anthropo- morphizing technology: When robots, computers, and other agents become social actors. Current Directions in Psycho- logical Science, 23(6): 394-398.

Cited by

Sorted by citations

Authors
Bhat, M.
Affiliations
Northwestern University (US)
Corresponding author
Bhat, Maalvika
Venue
Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2025-10-15
Volume
8
Issue
1
Pages
416-429
ISSN
3065-8365 (ISSN-L)
DOI
10.1609/aies.v8i1.36560
OpenAlex
W4415230159
Cited by
5
References
0
Citations by year
2026: 4 · 2025: 1
FWCI
5.22
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
Character.AI, parasocial relationships, identity exploration, intimacy, addiction, boundary erosion, AI companions, dynamic consent, responsible AI design, vulnerable populations, fan studies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-19.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-19.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review