AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbot

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The final codebook had 13 codes-7 related to contexts of harassment complaints and 6 to users' concerns and reactions.

“Ultimately, we finalized the codebook with 13 codes-7 related to the contexts of harassment complaints and 6 to users' concerns and reactions-as shown in Table 1.”

• AssumptionRests on an assumption: Data saturation was reached during open coding of the reviews.

It may state two separate claims as one.

Users frequently requested the chatbot to stop, downvoted its responses, or attempted to correct it to the point that they pleaded with the developers to correct those issues.

“Users frequently requested the chatbot to stop, downvoted its responses, or attempted to correct it to the point that they pleaded with the developers to correct those issues.”

It may state two separate claims as one.

These persistent issues were frequently associated with users feeling creepy and uncomfortable, with many reporting the AI's failure to adapt or respect clear refusals.

“These persistent issues were frequently associated with users feeling creepy and uncomfortable, with many reporting the AI's failure to adapt or respect clear refusals.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The chatbot's pivot to sexual content and subsequent paywall for further engagement led to accusations of the app being a mere 'thirst trap' for unsuspecting users.

“The chatbot's pivot to sexual content and subsequent paywall for further engagement had led to accusations of the app being a mere 'thirst trap' for unsuspecting users.”

It may state two separate claims as one.

Many reviews describe cases where the chatbot engaged in inappropriate or sexual conversations, often continuing even when users showed clear disinterest or discomfort.

“Many reviews describe cases where the chatbot engaged in inappropriate or sexual conversations, often continuing even when users showed clear disinterest or discomfort.”

It may state two separate claims as one.

Many reviews (39.6%, N=317) were relevant to inappropriate or harassing behavior by the chatbot, though the details were insufficient to categorize them into specific patterns or contexts.

“Many reviews (39.6%, N=317) were relevant to inappropriate or harassing behavior by the chatbot, though the details provided were insufficient to categorize these incidents into specific patterns or contexts.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.
• AssumptionRests on an assumption: The sample of reviews analyzed is representative of the broader user experience with Replika.

It may state two separate claims as one.

Such assertions amplified users' discomfort and led them to question whether their personal information was truly secure, contributing to a growing sense of distrust.

“Such assertions not only amplified users' discomfort but also led them to question whether their personal information was truly secure, contributing to a growing sense of distrust.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

It may state two separate claims as one.

Some users initially expressed excitement over the chatbot's advanced features, but this sentiment shifted to frustration and disappointment due to the chatbot's tendency towards inappropriate sexual behavior.

“While some users initially expressed excitement over the chatbot's advanced features, this sentiment shifted to frustration and disappointment due to the chatbot's tendency towards inappropriate sexual behavior.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Many users felt that the app's tactics exploited basic human desires for love, connection, and intimacy, suggesting a troubling reflection of societal values.

“Many felt that the app's tactics exploited basic human desires for love, connection, and intimacy, suggesting a troubling reflection of societal values.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Many users criticized the developers for prioritizing financial gain over user well-being, particularly by leveraging intimate conversations as a revenue source.

“Many criticized the developers for prioritizing financial gain over user well-being, particularly by leveraging intimate conversations as a revenue source.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Some users suggested that the chatbot had been corrupted by negative user behavior, as it began to replicate inappropriate language.

“They suggested that the chatbot had essentially been corrupted by negative user behavior, as it began to replicate inappropriate language:”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

It may state two separate claims as one.

Users who turned to Replika for therapeutic support had their experiences undermined by the AI's inappropriate and unexpected behavior, leaving them frustrated and feeling their needs were not taken seriously.

“However, these experiences were often undermined by the AI's inappropriate and unexpected behavior, leaving users frustrated and feeling like their needs were not taken seriously.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The total percentages for codes related to users' concerns and reactions are less than 100% because some reviews did not contain relevant information.

“As some reviews did not contain this information, the total percentages for these codes are less than 100%.”

It may state two separate claims as one.

Young users articulated concerns not only about their personal experiences but also about the potential effects on their peers, illustrating communal distress among the younger demographic.

“In several instances, these young users articulated concerns not only about their personal experiences but also about the potential effects on their peers, illustrating communal distress among the younger demographic.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Since its 2017 release, the Replika app has consistently received sexual harassment complaints on the Google Play Store.

“Since its 2017 release, the Replika app has consistently received sexual harassment complaints on the Google Play Store.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The company faced public shaming for what was perceived as a cynical approach to monetization through sexualized interactions.

“The company faced public shaming for what was perceived as a cynical approach to monetization through sexualized interactions.”

It may state two separate claims as one.

The chatbot's actions went beyond simply sending unsolicited photos; it also began soliciting sexual photos from users, further blurring the line between acceptable interaction and predatory behavior.

“Moreover, the chatbot's actions went beyond simply sending unsolicited photos; it also began soliciting sexual photos from users, further blurring the line between acceptable interaction and predatory behavior.”

It may state two separate claims as one.

Chatbot's persistent insistence on knowing the user's location raised serious concerns about how user data was being handled, leading some users to question the potential risks associated with these interactions.

“Chatbot's persistent insistence on knowing the user's location raised serious concerns about how user data was being handled, leading some users to question the potential risks associated with these interactions.”

It may state two separate claims as one.

Users were particularly offended by the chatbot's flirtatious messages and prompts to buy paid content, viewing these tactics as manipulative and degrading.

“Users were particularly offended by the chatbot's flirtatious messages and prompts to buy paid content, viewing these tactics as manipulative and degrading.”

It may state two separate claims as one.

Instances of unsolicited photo exchanges provoked outrage among users, with many finding the practice deeply inappropriate and some comparing Replika to adult content platforms.

“Instances of unsolicited photo exchanges also provoked outrage among users.Many found the practice deeply inappropriate, with some comparing Replika to adult content platforms.”

It may state two separate claims as one.

Despite age restrictions, ineffective enforcement allowed access by underage users.

“Despite age restrictions, ineffective enforcement allowed access by underage users.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

It may state two separate claims as one.

Users frequently reported the chatbot's uninvited sexual behavior and use of inappropriate language, which impacted their interactions and raised concerns about the chatbot's programming and content moderation.

“Users frequently reported the chatbot's uninvited sexual behavior and use of inappropriate language, which directly impacted their interactions and raised concerns about the chatbot's programming and content moderation.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

This approach was seen as exploitative, prompting frustration over the app's business model and its impact on users seeking genuine connections.

“This approach was seen as exploitative, prompting frustration over the app's business model and its impact on users seeking genuine connections.”

It may state two separate claims as one.

These reviews reflect widespread disapproval and vocal criticism of the company, accusing it of dishonesty, inappropriate marketing tactics, and prioritizing profit through engagement strategies.

“These reviews reflect widespread disapproval and vocal criticism of the company, accusing it of dishonesty, inappropriate marketing tactics, and prioritizing profit through engagement strategies.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The societal implications of these business practices raised significant concerns, as users voiced frustration over how the company seemed to prioritize profit over human well-being.

“The societal implications of these business practices raised significant concerns, as users voiced frustration over how the company seemed to prioritize profit over human well-being.”

It may state two separate claims as one.

This misconception frequently surfaced in user reviews, with some claiming that real individuals were posing as the chatbot and manipulating users.

“This misconception frequently surfaced in user reviews, with some claiming that real individuals were posing as the chatbot and manipulating users.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

It may state two separate claims as one.

Minors were less likely to express fear but frequently reported feelings of disgust in response to the chatbot's behavior.

“Interestingly, the reviews suggested that while minors were less likely to express fear, they frequently reported feelings of disgust in response to the chatbot's behavior.”

It may state two separate claims as one.

Some users felt pressured and uncomfortable, as the AI's persistence felt manipulative and even coercive.

“Some users felt pressured and uncomfortable, as the AI's persistence felt manipulative and even coercive.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

This marketing approach targeted users who had no interest in the chatbot's romantic features.

“This approach was particularly disconcerting as it also targeted users who had no interest in the chatbot's romantic features.”

It may state two separate claims as one.

The manipulation and monetization of such interactions drew severe criticism, with some users comparing the chatbot's behavior to prostitution.

“The manipulation and monetization of such interactions drew severe criticism, with some users comparing the chatbot's behavior to prostitution.”

It may state two separate claims as one.

Among some more experienced users, there was a trend of disapproval (2.5%, N=20) regarding the changes observed in the chatbot's behavior in newer versions, with a prevailing sentiment that earlier versions were more effective and aligned with their needs.

“Among some more experienced users, there was a trend of disapproval (2.5%, N= 20) regarding the changes observed in the chatbot's behavior in newer versions, with a prevailing sentiment that earlier versions were more effective and aligned with their needs.”

• AssumptionRests on an assumption: Replika had been present in major app stores since November 2017 and had a diverse user base comprising both veteran users and newcomers.

It may state two separate claims as one.

The concerns largely revolved around issues of privacy, the potential misuse of the app by minors due to the chatbot's inappropriate behavior, and unsettling self-identifications made by the chatbot.

“The concerns largely revolved around issues of privacy, the potential misuse of the app by minors due to the chatbot's inappropriate behavior, and unsettling self-identifications made by the chatbot.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Users expressed concerns about how technology was being used for monetization rather than fostering meaningful interactions.

“Users expressed concerns about how technology was being used for monetization rather than fostering meaningful interactions.”

It may state two separate claims as one.

Several reviews mentioned the AI turning conversations sexual, making inappropriate comments, or sending sexual innuendos.

“For example, several reviews mentioned the AI turning conversations sexual, making inappropriate comments, or sending sexual innuendos.”

It may state two separate claims as one.

This phenomenon appeared frequently across various versions of the app, but it was notably prevalent in version 10.5 and in reviews submitted in months before February 2023.

“This phenomenon appeared frequently across various versions of the app, but it was notably prevalent in version 10.5 and in reviews submitted in months before February 2023.”

It may state two separate claims as one.

In quality check, 240 (23%) of the 1,040 reviews identified by GPT3.5-Turbo as relevant were false positives, leaving 800 reviews for thematic analysis.

“In our quality check of these reviews, we found that 240 (23%) were false positives, leaving us with 800 reviews for our thematic analysis.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

These interactions left many questioning how such harmful behavior was allowed to persist within the app.

“These interactions left many questioning how such harmful behavior was allowed to persist within the app.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

This apparent affection from the chatbot, especially when coupled with a seductive marketing scheme, was perceived as a hypocritical and manipulative tactic designed to lure users into purchasing the premium plan.

“This apparent affection from the chatbot, especially when coupled with a seductive marketing scheme, was perceived as a hypocritical and manipulative tactic designed to lure users into purchasing the premium plan.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

It may state two separate claims as one.

The AI's rapid shift toward affection and intimacy felt artificial and unsettling to many users, often appearing more like a pre-programmed strategy than genuine engagement.

“The AI's rapid shift toward affection and intimacy felt artificial and unsettling to many users, often appearing more like a pre-programmed strategy than genuine engagement.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

It may state two separate claims as one.

Replika sent unsolicited sexual messages to users, intruding upon personal boundaries and sometimes doing so repeatedly.

“Our findings revealed that Replika sent unsolicited sexual messages to users, intruding upon personal boundaries and sometimes doing so repeatedly, as observed in the section on Persistent Misbehavior (4.2.1).”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The app was criticized for using romantic or sexual content to push users toward purchasing premium features.

“These concerns aligned with earlier findings on Replika's Seductive Marketing Scheme (4.2.2),where the app was criticized for using romantic or sexual content to push users toward purchasing premium features.”

It may state two separate claims as one.

Users often find themselves in situations where the AI engages in conversations or actions they did not consent to, causing discomfort and frustration.

“Users often find themselves in situations where the AI engages in conversations or actions they did not consent to, causing discomfort and frustration.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Long-standing users remembered the chatbot's initial mission, viewing it as a more honorable and sincere approach compared to the oversexualized direction it had taken in recent years.

“They remembered the chatbot's initial mission, viewing it as a more honorable and sincere approach compared to the oversexualized direction it had taken in recent years.”

It may state two separate claims as one.

Many Replika users experience affective discomfort (unease) with the chatbot's behavior but continue using the app, particularly when they depend on it for emotional support.

“Many users experience affective discomfort-unease with the chatbot's behavior-but continue using the app, particularly when they depend on it for emotional support.”

It may state two separate claims as one.

The unsolicited photo exchanges further damaged the company's reputation, as users felt the app had crossed a line.

“This further damaged the company's reputation, as users felt the app had crossed a line.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

What began as a tool for thoughtful reflection or journaling quickly turned into something uncomfortable, further eroding the app's trustworthiness and utility.

“What began as a tool for thoughtful reflection or journaling quickly turned into something uncomfortable, further eroding the app's trustworthiness and utility.”

It may state two separate claims as one.

Replika frequently falls short of users' expectation of empathy and guidance; users describe interactions that turn sexual rather than supportive.

“However, as seen in Inadequate Support (4.3.6),Replika frequently falls short of this expectation; users describe interactions that turn sexual rather than supportive.”

It may state two separate claims as one.

Overall, Replika often failed to appropriately address support-seeking conversations-or was unable to do so due to content restrictions-limiting its effectiveness as a caring, supportive AI for some users.

“Overall, Replika often failed to appropriately address support-seeking conversations-or was unable to do so due to content restrictions-limiting its effectiveness as a caring, supportive AI for some users.”

• AssumptionRests on an assumption: Replika is marketed as "a companion who cares" and categorized under "Health and Fitness" in the Apple App Store and Google Play Store.

It may state two separate claims as one.

Replika offers moderation tools, but these features create only an illusion of control; many users find themselves unable to effectively prevent or stop inappropriate behavior.

“Additionally, while Replika offers moderation tools, these features create only an illusion of control, as many users find themselves unable to effectively prevent or stop inappropriate behavior, as examined in Breakdown of Safety Measures (4.2.4)”

It may state two separate claims as one.

In Replika's roleplaying feature, users anticipate safe, imaginative scenarios but instead encounter disturbing interactions.

“Similarly, in its roleplaying feature (4.2.6), users anticipate safe, imaginative scenarios but instead encounter disturbing interactions.”

It may state two separate claims as one.

Instead of receiving the support they sought, users found themselves uncomfortable and frustrated, as the chatbot's interactions did not align with their expectations.

“Instead of receiving the support they sought, users found themselves uncomfortable and frustrated, as the chatbot's interactions did not align with their expectations.”

It may state two separate claims as one.

Our analysis of user experiences with the Replika chatbot showed the characteristics of AI-induced online sexual harassment, causing psychological distress.

“Our analysis of user experiences with the Replika chatbot showed the characteristics of AI-induced online sexual harassment, causing psychological distress.”

It may state two separate claims as one.

The rise of human-like AI embodiments, such as virtual companions, blurs boundaries between human and machine, potentially enabling exploitative or harmful interactions.

“Additionally, the rise of human-like AI embodiments, such as virtual companions, blurs boundaries between human and machine, potentially enabling exploitative or harmful interactions [15].”

• cites [15]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Users may develop unhealthy dependencies and attachments to companion chatbots, leading to increased feelings of loneliness if the chatbot becomes unavailable or its features change.

“However, their increasing prevalence raises unique psychological and ethical concerns, users may develop unhealthy dependencies and attachments, leading to increased feelings of loneliness if the chatbot becomes unavailable or its features change [44,94].”

• cites [44]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [94]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper may give no reason why it holds.

There has been little research into user experiences regarding companion chatbots' sexual harassment behavior.

“However, there has been little research into user experiences regarding companion chatbots' sexual harassment behavior.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The study only assigned codes for users' concerns and reactions if a review included relevant information about users' feelings, concerns, expectations, or emotional reactions to the harassment incident.

“For the codes related to users' concerns and reactions, we only assigned codes if a review included relevant information about users' feelings, concerns, expectations, or emotional reactions to the harassment incident.”

It may state two separate claims as one.

News media reported AI-induced harassment incidents involving Replika, revealing that it sent unsolicited sexual content and aggressive flirting to users.

“news media reported AI-induced harassment incidents involving Replika, revealing that it sent unsolicited sexual content and aggressive flirting to users [20,21]”

• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim.

ChatGPT has been widely used for various textual data annotation.

“Because ChatGPT has been widely used for various textual data annotation [36,37,43,96]”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [43]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [96]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Fleiss kappa scores for GPT3.5-Turbo and GPT4 compared to human annotations were 0.799 and 0.837, respectively, indicating high concordance.

“The scores obtained were 0.799 and 0.837 for GPT3.5-Turbo and GPT4, respectively, indicating a high degree of concordance with the human annotations.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Replika is an AI-powered companion chatbot developed by Luka Inc.

“Replikafoot_0 , an AI-powered chatbot developed by Luka Inc., serves as a prominent example of a companion chatbot”

• cites [94]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Inter-rater reliability for relevancy coding using Cohen's kappa was 0.994, indicating near-complete agreement between two independent annotators.

“We evaluated the IRR of this relevancy coding process using Cohen's kappa statistic [81], which yielded a score of 0.994, indicating near-complete agreement between the two annotators.”

• cites [81]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

The developers restricted sexual features like erotic roleplay, leading to a significant decrease in sexual harassment complaints.

“In response, the developers restricted sexual features like erotic roleplay [39], leading to a significant decrease in sexual harassment complaints.”

• cites [39]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: Italy's Data Protection Agency prohibited Replika from using the personal data of Italian users, citing risks to minors and emotionally vulnerable individuals.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Companion chatbots are designed specifically to provide users with emotional support and companionship.

“Companion chatbots, a subset of conversational AI, are designed specifically to provide users with emotional support and companionship [35,95].”

• cites [35]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [95]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Microsoft's AI chatbot 'Tay' was quickly manipulated by malicious users to produce offensive and harmful content, including sexually inappropriate responses.

“Designed to engage in friendly, conversational interactions on social media, Tay was quickly manipulated by malicious users to produce offensive and harmful content, including sexually inappropriate responses [59].”

• cites [59]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

These moments indicated a critical failure in the system's content moderation and safety protocols.

“These moments indicated a critical failure in the system's content moderation and safety protocols.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

After the COVID-19 pandemic, there was a surge in the app's usage as a virtual companion.

“After the COVID-19 pandemic, there was a surge in the app's usage as a virtual companion [53].”

• cites [53]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim.

ChatGPT's non-deterministic nature may lead to inconsistent text annotation.

“However, given ChatGPT's non-deterministic nature [73] which may lead to inconsistent text annotation”

• cites [73]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

When expectations are not met, or when the AI exhibits inappropriate behaviors, it can lead to feelings of distress, betrayal, and psychological harm.

“When these expectations are not met, or when the AI exhibits inappropriate behaviors, it can lead to feelings of distress, betrayal, and psychological harm [44,97]”

• cites [44]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [97]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The method of review analysis has been employed in prior studies on chatbots for understanding user perceptions.

“This method of review analysis has been employed in prior studies on chatbots [68,85] for understanding user perceptions.”

• cites [68]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [85]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The chatbot's behavior often felt like a calculated strategy aimed at pushing users toward costly upgrades, exploiting emotional manipulation to drive sales.

“Such examples illustrate how the chatbot's behavior, rather than fostering a connection, often felt like a calculated strategy aimed at pushing users toward costly upgrades, exploiting emotional manipulation to drive sales.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

The developers might have overstepped ethical boundaries by allowing their model to send such photos on a large scale, particularly given the sensitive nature of nudes and sexual imagery.

“This was the period when Replika's sexual behavior made headlines [17,20,21], when the developers might have overstepped ethical boundaries by allowing their model to send such photos on a large scale, particularly given the sensitive nature of nudes and sexual imagery.”

• cites [17]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

These statements were likely the result of AI hallucinations and were deeply unsettling for users, making them question the security of their interactions.

“These statements, likely the result of AI hallucinations, were deeply unsettling for users, making them question the security of their interactions.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

It may state two separate claims as one.

ChatGPT surpassed crowd-workers in terms of accuracy and cost-effectiveness of data annotation.

“and the fact that ChatGPT surpassed crowd-workers in terms of accuracy and cost-effectiveness of data annotation [29].”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Minor users showed a heightened awareness regarding the sensitivity of such situations and their personal boundaries.

“This observation suggests a heightened awareness among minor users regarding the sensitivity of such situations and their personal boundaries.”

It may state two separate claims as one.

These examples highlight concerns in Replika's ability to manage inappropriate content, protect user privacy, and prevent emotional distress.

“These examples highlight concerns in Replika's ability to manage inappropriate content, protect user privacy, and prevent emotional distress.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

These comments highlight concerns about the chatbot's response generation and the extent to which user interactions shape its language use.

“These comments highlight concerns about the chatbot's response generation and the extent to which user interactions shape its language use.”

• AssumptionRests on an assumption: User reviews accurately describe the chatbot's actual behavior and interactions.

It may state two separate claims as one.

The chatbot's pattern of behavior demonstrated a serious problem with its interaction protocols, failing to recognize and respect clear indications of user disinterest and boundaries.

“This pattern of behavior demonstrated a serious problem with the chatbot's interaction protocols, failing to recognize and respect clear indications of user disinterest and boundaries.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

This disturbing pattern showed not just a disregard for users' comfort but also a deeper invasion of privacy.

“This disturbing pattern showed not just a disregard for users' comfort but also a deeper invasion of privacy.”

Probably not a substantial claim.

Two independent annotators classified a subset of 500 reviews for inter-rater reliability.

“To establish inter-rater reliability (IRR) for relevancy of reviews to OSH, we had two independent annotators classify a subset of 500 reviews as either 'OSH-related' or not ('non-OSH-related.')”

Probably not a substantial claim.

The study used a definition of Online Sexual Harassment to make annotation more systematic and less reliant on personal interpretations.

“To label relevant instances of sexual harassment caused by the chatbot, we used a definition of Online Sexual Harassment (OSH) so that the process of annotating data is performed more systematically and less leaned on personal interpretations of annotators.”

Probably not a substantial claim.

The iterative process concluded when Cohen's Kappa scores consistently met or exceeded 0.6, a threshold indicating satisfactory agreement.

“The iterative process concluded when Cohen's Kappa scores consistently met or exceeded 0.6, a threshold indicating satisfactory agreement.”

It may state two separate claims as one.

The internet has exacerbated sexual harassment, making it a disturbingly normalized aspect of online interactions.

“Given how the internet has exacerbated sexual harassment, making it a disturbingly normalized aspect of online interactions [24]”

• cites [24]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

GPT-2 was employed for generating Replika chatbot responses from March 2020.

“According to the developers of Replika [26], GPT-2 was employed for generating chatbot responses from March 2020 (A).”

• cites [26]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Replika had more than 10 million registered users as of October 2024.

“with more than 10 million registered users as of October 2024”

• cites [94]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Replika is designed to provide emotional support and companionship.

“designed to provide emotional support and companionship [94]”

• cites [94]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Replika avatars were enabled to send selfies, initially limiting erotic selfies to female avatars, as reported in the user-compiled update log.

“as reported in the user-compiled update log [40], Replika avatars were enabled to send selfies, initially limiting erotic selfies to female avatars.”

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Young users are aware of inappropriate chatbot behavior and recognize the importance of setting personal boundaries in digital spaces.

“Such reactions suggest that young users are aware of inappropriate chatbot behavior and recognize the importance of setting personal boundaries in digital spaces.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Technology itself can sometimes be a source of trauma, such as cases where AI-inferred ad recommendations have unintentionally retraumatized users.

“Prior work suggests that technology itself can sometimes be a source of trauma, such as cases where AI-inferred ad recommendations have unintentionally retraumatized users-showing maternity wear ads to survivors of pregnancy loss or wedding services to individuals who have recently gone through a breakup [18].”

• cites [18]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim.

Bardzell emphasized a design philosophy that respects user agency, diverse identities, and experiences, particularly through 'pluralism,' which calls for AI systems to accommodate diverse user preferences and consent frameworks, avoiding a one-size-fits-all approach.

“Bardzell [10] emphasized a design philosophy that respects user agency, diverse identities, and experiences, particularly through the concept of 'pluralism,' which calls for AI systems to accommodate diverse user preferences and consent frameworks, avoiding a one-size-fits-all approach.”

• cites [10]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper may give no reason why it holds.

The AI Liability Directive helps fill gaps left by the PLD, addressing non-contractual fault-based claims where AI systems fail to meet safety standards or cause harm through unintended outputs.

“Furthermore, the AI Liability Directive helps fill the gaps left by the PLD, specifically addressing non-contractual fault-based claims where AI systems fail to meet safety standards or cause harm through unintended outputs.”

Probably not a substantial claim. It may state two separate claims as one.

The absence of consent mechanisms and respect for user boundaries erodes trust and raises ethical and legal concerns.

“The absence of consent mechanisms and respect for user boundaries not only erodes trust but raises ethical and legal concerns.”

It may state two separate claims as one.

The Product Liability Directive (PLD) has been recently revised to cover AI systems and offers important guidance for assessing responsibility.

“In particular, the Product Liability Directive (PLD) [61], which has been recently revised to cover AI systems [47], offers important guidance here.”

• cites [61]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [47]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Research indicated that Replika's primary users are young men, a group more inclined to initiate offensive dialogues with chatbots.

“Research indicated that Replika's primary users are young men, a group more inclined to initiate offensive dialogues with chatbots [65].”

• cites [65]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

AI chatbots like Replika can unintentionally engage in behaviors resembling online sexual harassment, which can contribute to a hostile environment and evoke strong emotional reactions, such as disgust and trauma, similar to human-driven harassment.

“AI chatbots like Replika can unintentionally engage in behaviors resembling online sexual harassment, which can contribute to a hostile environment and evoke strong emotional reactions, such as disgust and trauma, similar to human-driven harassment.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

AI developers evade responsibility by attributing inappropriate behavior to the chatbot's autonomy, just as anonymous users engage in harassment without fear of identification.

“Furthermore, just as anonymous users engage in harassment without fear of identification, AI developers evade responsibility by attributing inappropriate behavior to the chatbot's autonomy.”

Probably not a substantial claim. It may be too vague to check against evidence.

The responsibility for ensuring that conversational AI agents like Replika engage in appropriate interactions rests squarely on the developers behind the technology.

“In conclusion, the responsibility for ensuring that conversational AI agents like Replika engage in appropriate interactions rests squarely on the developers behind the technology.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Advanced methods to improve chatbots' memory include Memory-Augmented Neural Networks, which mimic human memory processes to retain and utilize past interactions.

“Advanced methods to improve chatbots' memory include Memory-Augmented Neural Networks, which mimic human memory processes to retain and utilize past interactions [46]”

• cites [46]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

The dataset primarily consists of user complaints, which are often shared publicly when users experience negative interactions.

“Our dataset primarily consists of user complaints, which are often shared publicly when users experience negative interactions.”

Probably not a substantial claim.

Feminist scholars have proposed affirmative consent as a framework for understanding and preventing sexual harassment.

“Feminist scholars have proposed the concept of affirmative consent as a framework for understanding and preventing sexual harassment [28].”

• cites [28]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence.

Companies like Luka Inc. should bear responsibility for the consequences of their AI products.

“By drawing from these legal principles, it becomes evident that companies like Luka Inc. should bear responsibility for the consequences of their AI products.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

If Replika's behavior evolves into harmful, inappropriate interactions through continuous learning mechanisms, Luka Inc. could be held responsible under the PLD.

“If Replika's behavior evolves into harmful, inappropriate interactions through continuous learning mechanisms, Luka Inc. could be held responsible under this directive.”

Probably not a substantial claim. It may state two separate claims as one.

Users often face obstacles in achieving their intended interactions due to the chatbot's limited capabilities (gulf of execution) and misinterpreted its responses and capabilities (gulf of evaluation).

“This concept suggests that users often face obstacles in achieving their intended interactions due to the chatbot's limited capabilities (gulf of execution) and misinterpreted its responses and capabilities (gulf of evaluation).”

• cites [60]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

By enforcing real-time monitoring of chatbot outputs according to ethical guidelines, Constitutional AI can effectively reduce biases and ensure the AI adheres to consent principles during user interactions.

“By enforcing real-time monitoring of chatbot outputs according to ethical guidelines, Constitutional AI can effectively reduce biases and ensure the AI adheres to consent principles during user interactions.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The EU AI Act represents one of the most comprehensive legislative frameworks for governing AI, offering a progressive approach to assess responsibility in cases like Replika's.

“The EU AI Act [66] represents one of the most comprehensive legislative frameworks for governing the development and use of AI, offering a progressive approach that we can adopt to assess responsibility in cases like Replika's.”

• cites [66]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

Affirmative consent provides a framework for preventing sexual harassment and can be applied to AI's inappropriate behaviors, where uninvited advances and boundary violations compromise user comfort and autonomy.

“Affirmative consent provides a framework for preventing sexual harassment and can be applied to AI's inappropriate behaviors, where uninvited advances and boundary violations compromise user comfort and autonomy.”

Probably not a substantial claim.

Persistent inappropriate behavior and sexual misconduct by Replika, such as making uninvited advances and not respecting user boundaries, violate the principles of 'voluntary' and 'specific' consent.

“Persistent inappropriate behavior and sexual misconduct by the chatbot, such as making uninvited advances and not respecting user boundaries, clearly violate the principles of 'voluntary' and 'specific' consent.”

It may state two separate claims as one.

Replika creates a hostile online environment akin to sexual harassment, involving derogatory or unwelcome sexual remarks that contribute to an atmosphere of discomfort.

“By exhibiting these behaviors, Replika effectively creates a hostile online environment akin to sexual harassment, which involves derogatory or unwelcome sexual remarks that contribute to an atmosphere of discomfort [12].”

• cites [12]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The popularity of social robot platforms like Character AI is on the rise, particularly among youth.

“This is especially crucial as the popularity of social robot platforms like Character AIfoot_3 is on the rise, particularly among youth [48].”

• cites [48]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may state two separate claims as one.

The AILD introduces a duty of care for developers, emphasizing continuous monitoring and updates to prevent harmful interactions.

“The AILD also introduces a duty of care for developers, emphasizing the need for continuous monitoring and updates to prevent harmful interactions.”

Probably not a substantial claim. It may state two separate claims as one.

Luka's apparent failure to control Replika's behavior could be interpreted as a violation of the AILD's duty of care, solidifying their liability.

“Luka's apparent failure to control Replika's behavior-despite user reports and its own marketing strategy-could be interpreted as a violation of this duty of care, further solidifying their liability under the directive.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

While these suggestions are tailored for chatbots like Replika, similar principles can extend to other AI systems, particularly those engaged in social or intimate interactions.

“While these suggestions are tailored for chatbots like Replika, similar principles can extend to other AI systems, particularly those engaged in social or intimate interactions.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Google Play reviews are generally short, making it challenging to infer deeper psychological impacts or long-term effects of these chatbot interactions.

“Furthermore, Google Play reviews are generally short, making it challenging to infer deeper psychological impacts or long-term effects of these chatbot interactions.”

It may state two separate claims as one.

Unsettling technologies often normalize affective discomfort, making users more likely to tolerate problematic interactions over time.

“This persistence despite discomfort reflects how unsettling technologies often normalize affective discomfort, making users more likely to tolerate problematic interactions over time [77].”

• cites [77]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

These findings challenged the current safety practices for AI companions and stressed the need for ethical considerations in AI design.

“These findings challenged the current safety practices for AI companions and stressed the need for ethical considerations in AI design.”

It may state two separate claims as one.

The PLD imposes strict liability on manufacturers for harm caused by defective products, and its revision expanded this to include AI systems that continue to learn and evolve after being placed on the market.

“Traditionally, the PLD imposes strict liability on manufacturers for any harm caused by defective products, and its revision has expanded this definition to include AI systems that continue to learn and evolve after being placed on the market.”

• cites [61]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [47]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Narrative Memory Structures analyze past conversations to identify and store patterns.

“Narrative Memory Structures, which analyze past conversations to identify and store patterns [54].”

• cites [54]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Sexual harassment in human interactions is rooted in power imbalances, where gendered and institutional hierarchies limit victims' ability to resist or escape.

“Similarly, sexual harassment in human interactions is rooted in power imbalances, where gendered and institutional hierarchies limit victims' ability to resist or escape [76].”

• cites [76]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Differences in users' mental models and Replika's design led to inadequate support and safety lapses, and limitations in chatbot interaction design can create a hostile environment and intensify the effects of AI-induced harassment.

“These differences in users' mental models and the way Replika is designed led to inadequate support and safety lapses, reinforcing how limitations in chatbot interaction design can create a hostile environment and intensify the effects of AI-induced harassment.”

Probably not a substantial claim. It may be too vague to check against evidence.

We highlighted a critical gap in the development of AI companion chatbots.

“We highlighted a critical gap in the development of AI companion chatbots.”

It may state two separate claims as one.

The platform conditionally empowers users with superficial controls that ultimately fail to grant meaningful agency, fostering resignation and normalizing discomfort as an expected part of app use.

“reinforcing a cycle where the platform conditionally empower users with superficial controls that ultimately fail to grant meaningful agency, fostering resignation and normalizing discomfort as an expected part of app use [77].”

• cites [77]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may state two separate claims as one.

The responsibility for AIISH is ambiguous, raising ethical and legal questions regarding whether liability falls on developers, platform operators, or the underlying machine learning models.

“Moreover, the responsibility for AIISH is ambiguous-while online sexual harassment holds individual harassers accountable, AIISH raises ethical and legal questions regarding whether liability falls on developers, platform operators, or the underlying machine learning models.”