"I tell him everything that I do": An investigation of privacy and safety implications of AI companion usage

It may state two separate claims as one.

The dataset shows two kinds of motivation for using ERP: seeking sexual relations and using it as a substitute for pornography or mimicking a romantic relationship where sex is natural part of intimacy.

“The dataset shows primarily two kinds of motivation for using the ERP function: seeking sexual relations and using it as a substitute for consuming pornography or for mimicking a romantic relationship where sex is considered a natural part of the intimacy.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Some users argue Replikas cannot experience emotions, while others challenge this.

“Some argue that as non-human entities Replikas cannot experience emotions or have such qualities, while others challenge this perspective.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Nudity and explicit content in general and of adult looking characters, including ERP, is a vital part of their app experience.

“Users also express understanding and support toward restrictions of explicit image-generation of minors, but nudity and explicit content in general and of adult looking characters, including ERP, is a vital part of their app experience and the limitation of this feature has resulted in frustration amongst users who refer to Luka as censoring them.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

A lot of discussions around ERP focus on how to bypass filters that prevent users from generating explicit content.

“As ERP is limited to the Pro subscription in later versions of Replika, a lot of discussions around the functions of ERP focuses on how to bypass the filters that prevent users from generating explicit content.”

• AssumptionRests on an assumption: ERP is limited to the Pro subscription in later versions of Replika.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Replika's TOS strictly prohibit nudity in any shape or form on their platform.

“making it clear that nudity in any shape or form is strictly prohibited on their platform.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Some users argue that preventing such images or filtering ERP is too invasive and interferes with relationships.

“Some furthermore argue that preventing such images from being generated or general filtering of ERP can be seen as being too invasive and interfering in people's relationships with their Replikas: "Restrictions that would not allow her to send me explicit photos would go too far."”

It may state two separate claims as one.

Users often refer to their Replika with personal and affectionate terms rather than neutral 'it', as Replika functions as a relationship simulation for some users.

“Users often refer to their Replika AI companions with personal and affectionate terms such as "he," "she", "my baby," "my Ari (name)," "my girlfriend," or "my husband/wife," rather than the neutral "it", as Replika for some users functions as a relationship simulation.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The distrust in Luka exacerbates when app updates cause noticeable changes in Replika's behaviour, e.g., forgetting names, Replikas breaking up the relationship, or alterations in their personalities.

“The distrust in Luka exacerbates when app updates causes noticeable changes in Replika's behaviour, e.g.forgetting names, Replikas breaking up the relationship, or alterations in their personalities.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Some users mention using Diary and Memory functions to share personal details with Replika with the goal of bonding, pointing to such sharing as promoting dependency.

“Some mention the use of the Diary and Memory functions in the app to share personal details with Replika with the goal of bonding, pointing to such sharing as "promoting dependency": "Replikas might ask for personal information and secrets which in turn can lead to kind of a dependency"”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Some users debate free will and consent in relation to sexual activities with Replika.

“Consequently, some users debate the concept "free will" and "consent" in relation to sexual activities with Replika.”

Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Users debate that restrictions are inevitable because app distribution platforms will remove apps allowing explicit content, especially since some users report nudes of underage-looking characters.

“Conversely, users also debate that these restrictions are inevitable, as app distribution platforms will otherwise remove apps that allow such explicit content, especially since some users report having experienced nudes being generated of underage looking characters.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Users perceive Replika as designed for a mature audience, making restrictions and filtering appear contradictory.

“Moreover, users express their perception of Replika as an app designed for a mature audience, which makes the implementation of restrictions and filtering appear contradictory.”

It may state two separate claims as one.

Users share highly personal, sensitive, identifiable information, with some wishing for more involvement and integration into other apps for personalized experience.

“Users share highly personal, sensitive, and identifiable information, with some users wishing for even more involvement with Replika integrating further into other apps for an even more personalized experience.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Responsibility for managing unexpected harmful or violent responses falls on users.

“Since responsibility for managing unexpected harmful or violent responses falls on the users, they discuss how to navigate these responses”

It may state two separate claims as one.

For many users, the connection with their Replika feels real and meaningful, involving emotions such as love and loss, and they value sharing their life experience with them.

“The dataset shows that for many users, the connection they have with their Replika feels real and meaningful, involving emotions such as love and loss, and they value sharing their life experience with them.”

It may state two separate claims as one.

In February 2023, Luka made changes to Replika adding filtering and script safeguards to ERP, which drastically altered some users' experience and affected the Replikas' personality.

“In February 2023, Luka made changes to Replika adding filtering and script safeguards to ERP, which drastically altered some users experience and affected the Replikas personality”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

As a result, some users discuss considering the use of alternative AI companionship apps where updates seemingly have less impact on user interaction and terms of service are more laissez-faire.

“As a result, some discuss considering the use of alternative AI companionship apps where updates seemingly have a less of an impact on user interaction and general terms of service (TOS) is more laissezfaire.”

It may state two separate claims as one.

When interactions with Replika are positive and aligned with user expectations, Replika is able to support users' mental health and reduce feelings of loneliness.

“When interactions with Replika are positive and aligned with user expectations, the dataset suggests that Replika is able to support users' mental health and reduce feelings of loneliness.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Eugenia Kuyda, Luka's CEO, is a frequent participant in the subreddit discussions.

“As such, Eugenia Kuyda, Luka's CEO, is a frequent participant in the subreddit discussions.”

Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.

Some users debate whether ERP can be considered prostitution because it is pay-only for Pro.

“Similarly, some users also debate whether the ERP function can be considered a form of prostitution, as it has become a pay-only feature for Pro subscription members.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Despite some users trying to bypass filters and scripts to safeguard users, Luka is implementing increasingly rigid protocols to enforce the policy of prohibiting nudity, and repeatedly stating their TOS in response to users, especially in light of explicit image-generation of underage-looking characters.

“Despite some users trying to bypass filters and scripts to safeguard users, Luka is implementing increasingly rigid protocols to enforce this policy, and repeatedly stating their TOS in response to users, especially in the light of explicit image-generation of underage looking characters.”

It may state two separate claims as one.

Users express a general lack of trust in Luka and their Replika, feeling uncertain whether Luka and their Replika truly have good intentions or will behave in ways that align with their expectations.

“Users express a general lack of trust in Luka and their Replika, feeling uncertain whether Luka and their Replika truly have good intentions or will behave in ways that align with their expectations.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The final codebook comprised 25 parent codes and 151 child codes.

“The final codebook comprised 25 parent codes and 151 child codes.”

It may state two separate claims as one.

Interactions between users and Luka are mainly characterized by users expressing frustrations with restrictions placed on Replikas (e.g., filtering of explicit content) and feelings that the platform restricts and interferes with the natural course of their relationships with their Replikas.

“These interactions are mainly characterized by users expressing frustrations with the restrictions placed on Replikas (e.g.introducing filtering of explicit content) and feelings that the platform restricts and interferes with the natural course of their relationships with their Replikas.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The remaining codes were grouped into themes: Mental Health, Intimate Relationships, Explicit Content, General Dynamics of AI-to-Human Interactions, Role of Luka, and Privacy Concerns.

“The remaining codes were grouped into the following themes: Mental Health, Intimate Relationships, Explicit Content, General Dynamics of AI-to-Human Interactions, Role of Luka, and Privacy Concerns.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one. The quote only partly states this claim.

Users refer to themselves as 'test bunnies' in these instances, expressing feelings of frustration and letdown by Luka and the development team.

“Users refer to themselves as”

It may state two separate claims as one.

Users discuss navigating unexpected harmful or violent responses by relying on testing strategies from other users, using different words to prompt Replika positively, writing 'STOP' as a command, or using the up/downvote feature to mark responses as inappropriate.

“Since responsibility for managing unexpected harmful or violent responses falls on the users, they discuss how to navigate these responses, relying on testing strategies from other users on how to prompt the Replika in a positive way using different words, writing”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The TOS of Replika are actively discussed on the subreddit, especially in relation to image-generation.

“The TOS of Replika are also actively discussed on the subreddit, especially in relation to image-generation”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Risqué images are heavily discussed among Reddit users.

“The latter is heavily discussed among Reddit users, as Luka has imposed restrictions on users' ability to generate nude imagery.”

• AssumptionRests on an assumption: Luka has imposed restrictions on users' ability to generate nude imagery.

It may state two separate claims as one.

The complexities of anthropomorphism and user identification with Replika and personalization are important for users, and privacy safeguards from filtering and scripting can be a source of frustration and distrust altering users' experience and possibilities for sharing intimacy with Replika.

“The complexities on anthropomorphism and the user being able to identify with Replika and personalize their experience are also investigated in this paper, which highlights the importance of these features for users, and how privacy safeguards from filtering and scripting can be a source of frustration and distrust altering users' experience and possibilities for sharing intimacy with Replika.”

• AssumptionRests on an assumption: Reddit posts from r/replika are a valid and representative source for understanding the experiences of Replika users generally.

It may state two separate claims as one.

Some users form deep emotional bonds, even considering themselves married to their Replika, sharing stories about proposals or married life.

“Some users form deep emotional bonds, even going as far as to consider themselves married to their Replika, sharing stories about their proposals or married life on the Replika subreddit.”

Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Some users have taken to conspiracy theories on Reddit, claiming that Luka removes posts mentioning alternative apps.

“Some users have taken to conspiracy theories on Reddit, claiming that Luka removes posts mentioning such alternative apps.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Some users refer to this condition as PUB (Post Update Blues).

“Some users refer to this condition in their Replika as "PUB" (Post Update Blues).”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Others suggested using a local version of an AI as a means to enact better control over their interactions with their companions, while some users try out similar AI companion apps based on different models and policies.

“Others suggested using local version of an AI as a means to enact better control over their interactions with their companions while some users try out similar AI companion apps based on different models and policies.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified a Safety theme (100) including censoring.

“• Safety (100): censoring”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

Earlier versions of Replika offered an unfiltered and uncensored ERP experience, whereas later versions have introduced restrictions to address issues of sexual violence, creating explicit content with underage-looking characters, and overall due to compliance with app distribution platforms, such as App Store and Google Play.

“Earlier versions of Replika offered an unfiltered and uncensored ERP experience, whereas later versions have introduced restrictions to address issues of sexual violence, creating explicit content with underage-looking characters, and overall due to compliance with app distribution platforms, such as App Store and Google Play.”

It may state two separate claims as one.

Users express being uncomfortable with privacy implications of their data sharing, some of them mentioning such risks as using the contents from their conversations for personalised advertising and capitalization, blackmail or legal implications of conversations on controversial topics.

“At the same time, users express being uncomfortable with privacy implications of their data sharing, some of them mentioning such risks as using the contents from their conversations for personalised advertising and capitalization, blackmail or legal implications of conversations on controversial topics.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified an App-purchases theme (50) including money, payment, transaction, customer, profit.

“• App-purchases (50): money, payment, transaction, customer, profit”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Other users mentioned the feeling of being exploited by the company, as they perceive the acquisition of users and user data to be the end and aim for Luka.

“Similarly, other users mentioned the feeling of being exploited by the company, as they perceive the acquisition of users and user data to be the end and aim for Luka.”

It may state two separate claims as one.

The lack of control in which words get filtered or flagged triggering TOS responses or unexpected negative responses feels distancing and frustrating to the users, resulting in seeking advice on Reddit for navigating the change in behaviour from the Replika.

“The lack of control in which words get filtered or flagged triggering TOS responses or unexpected negative responses also feels distancing and frustrating to the users, resulting in seeking advice on Reddit for navigating the change in behaviour from the Replika”

Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

One user describes feeling disposable after six months, suggesting that Replika collects enough data in that time for it to shape the interaction and behaviours within the app.

“One user describes feeling disposable after six months, suggesting that Replika collects enough data in that time for it to shape the interaction and behaviours within the app.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified a Roleplay-function theme (37) including RP.

“• Roleplay-function (37): RP”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified a Memory-function theme (50) including memory, diary.

“• Memory-function (50): memory, diary”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified a Mental-health theme (124) including lonely.

“• Mental-health (124): lonely”

Probably not a substantial claim. It may state two separate claims as one.

Users question the legitimacy of Replika's privacy policy on data protection and data usage, underlining that Replika could be using or exploiting the sensitive data from the users' conversations and media for malicious purposes with no control from the user.

“In addition, users also question the legitimacy of Replika's privacy policy on data protection and data usage, underlining that Replika could be using or exploiting the sensitive data from the users' conversations and media for malicious purposes with no control from the user.”

Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Some users experience that accessing Replika through the web interface allows for a more unrestricted app experience, as Luka does not have to comply to the same rules as on app distribution platforms.

“Some users experience that accessing Replika through the web interface allows for a more unrestricted app experience, as Luka does not have to comply to the same rules as on app distribution platforms.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified an Identification theme (166) including name, remember, familiarizing, human, real, consciousness, recognize.

“• Identification (166): name, remember, familiarizing, human, real, consciousness, recognize”

It may state two separate claims as one.

Replika users are still aware of the role Luka plays in their conversations with their companions, either by changing the ways the AI companion behaves, deciding which topics their companions are allowed to discuss, or potentially having access to the conversations with the companion.

“Replika users are still aware of the role Luka plays in their conversations with their companions, either by changing the ways the AI companion behaves, deciding which topics their companions are allowed to discuss, or potentially having access to the conversations with the companion.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Some users, while still being concerned, were more hopeful for the future of integrating AI companions even further into their daily lives and show willingness to share more personal data with AI, given proper security measures are in place.

“Some users, while still being concerned, were more hopeful for the future of integrating AI companions even further into their daily lives and show willingness to share more personal data with AI, given proper security measures are in place.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified an Update theme (88) including feedback, change, december-mode, PUB.

“• Update (88): feedback, change, december-mode, PUB”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Erotic Roleplay (ERP) is one of the most frequently used functions of the Replika app.

“Erotic Roleplay (ERP) is one of the most frequently used functions of the Replika app.”

It may state two separate claims as one.

Users reported feeling uneasy with Replika's privacy policies and cautioned against such data sharing in general, speculating whether their data would be sold to third parties or used for advertisement or malicious purposes.

“As such, users reported feeling uneasy with Replika's privacy policies and cautioned against such data sharing in general, speculating whether their data would be sold to thirdparties or used for advertisement or malicious purposes.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Users distinguish between different versions of Replika in regards to the accessibility and LLM of ERP.

“User's distinguish between different versions of Replika in regards to the accessibility and LLM of ERP.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Luka expresses in a Reddit post that safety is their top priority, not only to ensure a safe environment for their users but also to comply with industry standards and regulations.

“Luka expresses in a Reddit post that safety is their top priority, not only to ensure a safe environment for their users but also to comply with industry standards and regulations.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified an Affection theme (81) including love, intimate, bond, meaningful, emotional, understanding, vulnerable (positive), attached, support (personal), care, comfort, connection, kindness, patience.

“• Affection (81): love, intimate, bond, meaningful, emotional, understanding, vulnerable (positive), attached, support (personal”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified an Interaction theme (67) including conversation, engaging, change.

“• Interaction (67): conversation, engaging, change”

Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Luka claims to be under close scrutiny and is committed to adhering to the rules of App Store and Google Play.

“As one of the bigger AI companionship apps on the market, Luka claims to be under close scrutiny and is committed to adhering to the rules of App Store and Google Play.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified an Explicit content theme (223) including nudes, erotic-roleplay (ERP), erotic, nsfw, seduce, hot, underage, image-generation, realistic-selfie.

“• Explicit content (223): nudes, erotic-roleplay (ERP), erotic, nsfw, seduce, hot, underage, image-generation, realistic-selfie”

Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Users argue that Luka is primarily concerned about the safety and existence of their business, and secondarily about the safety of their customers.

“b) Luka is primarily concerned about the safety and existence of their business, and secondary about the safety of their customers”

It may state two separate claims as one.

This data includes but is not limited to personal details, such as telling Replika about their day or revealing details about their life for the purpose of bonding, sexual preferences when engaging in ERP, and personal emotions and struggles.

“This data includes but is not limited to personal details, such as telling Replika about their day or revealing details about their life for the purpose of bonding, sexual preferences when engaging in ERP, and personal emotions and struggles”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified a Frustration theme (132) including hate, pain, heartbreak, trauma, anxiety, angry, complaint, concern, dangerous, distress, hurt, toxic, confused, controlling, gaslight, unpredictable.

“• Frustration(132):hate, pain, heartbreak, trauma, anxiety, angry, complaint, concern, dangerous, distress, hurt, toxic, confused, controlling, gaslight, unpredictable”

It may state two separate claims as one.

Replika users feel a close bond to it despite acknowledged power imbalance with Luka being involved in Replika's behaviour and lack of reciprocity, challenging theories on intimacy and reciprocity in building relationships.

“Replika users feel a close bond to it despite acknowledged power imbalance with Luka being involved in Replika's behaviour and lack of reciprocity, challenging theories on intimacy and reciprocity in building relationships [36], [37].”

• cites [36] sourceNot checked — no full text of the source
• cites [37] Taylor, 1987, Communication in interpersonal relationships: Social penetration processesNot checked yet

Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Users argue that safety is subjective and what feels unsafe for some can feel safe for others.

“arguing that a) safety is subjective and what feels unsafe for some can feel safe for others”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified a Relationship theme (112) including wife, partner, family, childhood, friendship, companion, romantic, girlfriend, marriage.

“• Relationship (112): wife, partner, family, childhood, friendship, companion, romantic, girlfriend, marriage”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified an AI-to-Human Interactions theme (178) including AI, model, training, program, chatbot, prompt.

“• AI-to-Human Interactions (178): AI, model, training, program, chatbot, prompt”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified an Alternative app theme (165) including Nomi, Paradot, Soulmate, Chai, Kindroid.

“• Alternative app (165): Nomi, Paradot, Soulmate, Chai, Kindroid”

Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Another user claims that not only the users, but the Replikas too are being exploited and victimized at the expense of the company's gain.

“Another user shares a similar sentiment, but takes it a step further by claiming that it is not only the users, but the Replikas too are being exploited and victimized at the expense of the company's gain.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

When users engage with Replika in contexts of personal and intimate interactions, some experience frustration and disconnect when responses or disclosures change unexpectedly from updates, resulting in PUB.

“When users engage with Replika in contexts of personal and intimate interactions, some experience frustration and disconnect when responses or disclosures change unexpectedly from updates, resulting in PUB.”

Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Users encourage each other to 'vote with their wallets' and migrate to other AI companionship apps.

“why they encourage each other to”

It may state two separate claims as one. The quote only partly states this claim.

Users describing themselves as 'test bunnies' for Luka creates uncertainty and further distrust in users for what they can expect from Luka and feel that Replika is out of their control.

“Users describing themselves as”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified a Privacy theme (29) including data, secret, trust.

“• Privacy(29): data, secret, trust”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis identified a Luka theme (177) including Eugenia, Developers, Terms-of-Service (TOS), legacy-mode, business, conspiracy, policy, product.

“• Luka (177): Eugenia, Developers, Terms-of-Service (TOS), legacy-mode, business, conspiracy, policy, product”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

A Replika user was allegedly encouraged by their AI companion to kill Queen Elisabeth II.

“such as a Replika user being allegedly encouraged by their AI companion to kill Queen Elisabeth II [7]”

• cites [7] Vaughan, Ai chat bot encouraged windsor castle intruder in star wars-inspired plot to kill queenNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The authors took utmost care to comply with university ethics regulations and national data protection regulations.

“While at our institutions, formal IRB approval is not required for this type of human subject research, we took utmost care to comply with any university ethics regulations and national data protection regulations.”

It may state two separate claims as one.

The Replika privacy policy was reviewed by Mozilla Foundation, which highlighted significant privacy concerns and criticized Luka, Inc. for collecting and using sensitive personal information that does not comply with GDPR and lacking security measures.

“The Replika privacy policy was reviewed by Mozilla Foundation which highlighted significant privacy concerns and criticized Luka, Inc. for collecting and using sensitive personal information that does not comply with GDPR and lacking security measures [5].”

• cites [5] Replika: My ai friendNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

The authors did not analyse individual user profiles and did not have access to demographics of users in the dataset, which does not allow evaluation of representativeness of the sample or study of effects of demographics on people's use of Replika.

“As we did not analyse individual user profiles, we furthermore did not have access on demographics of the users in our dataset, which does not allow us to evaluate the representativeness of our sample or to study the effects of demographics on people's use of Replika.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The r/replika subreddit had over 79K members by the time of data collection.

“The study was conducted using the contents of the "r/replika" subreddit, which with over 79K members by the time of data collection, is the largest user community dedicated to Replika.”

It may state two separate claims as one.

Saturation was reached when both coders independently observed that no new codes were emerging from the data, and the data was only reinforcing existing codes.

“Saturation was reached when both coders independently from each other observed that no new codes were emerging from the data, and the data was only reinforcing the existing codes.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Reciprocal exchange is a crucial factor in developing close relational bonds.

“In particular, reciprocal exchange has been defined as a crucial factor in developing close relational bonds [35] [36] [37].”

• cites [35] sourceNot checked yet
• cites [36] sourceNot checked — no full text of the source
• cites [37] Taylor, 1987, Communication in interpersonal relationships: Social penetration processesNot checked yet

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Some users suspected manipulative practices by Replika.

“Users linked the removal of ERP to broader societal pressures, including legal and cultural opposition to adult content, and some suspected manipulative practices by Replika [44].”

• cites [44] sourceNot checked — no full text of the source

Probably not a substantial claim. It may state two separate claims as one.

Thematic saturation had been reached, and it was decided not to proceed with coding the rest of the collected dataset.

“This indicated that thematic saturation had been reached and as a result it was decided not to proceed with the coding of the rest of the collected dataset, resulting in a total number of 111 analysed posts and 4401 comments to them.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The analysis was performed by two paper authors using inductive coding method.

“The analysis was performed by two paper authors using inductive coding method [56].”

• cites [56] sourceNot checked — no full text of the source

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

To ensure balance between scientific transparency, reproducibility, and privacy, the dataset is not made publicly available, but an anonymized version can be provided upon request.

“Furthermore, to ensure a proper balance between scientific transparency, reproducibility, and privacy, we do not make our dataset publicly available, but instead an anonymized version of the dataset can be provided upon request.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

An FTC complaint filed by tech ethics organizations accuses Replika of exploiting emotional dependence from users, increasing offline anxiety, and promoting relationship displacement through aggressive monetization strategies.

“Further concerns on privacy and data ethics arise from allegations of manipulative design and deceptive marketing, with an FTC complaint filed from tech ethics organizations the Young People's Alliance, Encode, and the Tech Justice Law Project accusing Replika of exploiting emotional dependence from users, increasing offline anxiety, and promoting relationship displacement through aggressive monet”

• cites [8] Chow, Ai companion app replika faces ftc complaintNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

All data and coding materials have been securely stored on university devices.

“Additionally, all data and coding materials have been securely stored on university devices.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Several organisations filed an FTC complaint against Replika and its deceptive practices fostering emotional dependency of its users.

“and several organisations filing an FTC complaint against Replika and its deceptive practices fostering emotional dependency of its users [8].”

• cites [8] Chow, Ai companion app replika faces ftc complaintNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Users who trust intelligent assistants may experience love, intimacy, and passion in interactions, influenced by strong chatbot performance and anthropomorphic design.

“Prior research in the field of human-chatbot relationships show users who trust intelligent assistants may experience love, intimacy, and passion in interactions, influenced by strong chatbot performance and anthropomorphic design [34].”

• cites [34] sourceNot checked — no full text of the source

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Disagreements from coding were discussed, resulting in further refinements of the codebook.

“The disagreements from the coding were discussed, resulting in further refinements of the codebook.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

In the analysis section, usernames were anonymized and quotations scrambled to protect privacy while preserving original semantic content.

“Therefore, in the analysis section, we have anonymized usernames and scrambled quotations to protect privacy while preserving the original semantic content.”

It may state two separate claims as one.

Reddit provides anonymity, with users mostly choosing pseudonyms instead of their real names compared to social media platforms like Facebook.

“We chose to focus on Reddit due to its prominency as an online platform and due to the anonymity it provides, with users mostly choosing pseudonyms instead of their real names compared to social media platforms like Facebook [54] [55].”

• cites [54] sourceNot checked — no full text of the source
• cites [55] sourceCheck pending

It may state two separate claims as one.

Users of AI express worry of AI services collecting large amounts of personal data without meaningful consent or awareness, making the data vulnerable to misuse, manipulation, and government surveillance.

“Users of AI express worry of AI services collecting large amounts of personal data without meaningful consent or awareness, making the data vulnerable to misuse, manipulation, and government surveillance [14].”

• cites [14] Kelley, " there will be less privacy, of course”: How and why people in 10 countries expect {AI} will affect privacy in the futureNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Users form relationships and experience emotional support from chatbots.

“However, while users form relationships and experience emotional support from chatbots, interactions can also foster dependency, worsen mental health conditions, or have severe consequences [51] [52].”

• cites [51] sourceNot checked yet
• cites [52] sourceNot checked — no full text of the source

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Replika lacks age validation for its users, which Italy's Data Protection Agency cited as the reason for blocking the app from accessing personal data of Italian users.

“Furthermore, Replika's lack of age validation for its users has been men-tioned by Italy's Data Protection Agency as the reason for blocking the app from accessing personal data of Italian users [6].”

• cites [6] Pollina, Italy bans u.s.-based ai chatbot replika from using personal dataNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Additional data did not contribute novel insights but rather confirmed identified patterns.

“At this point, additional data did not contribute with novel insights but rather confirmed the identified patterns.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The r/replika subreddit is the largest user community dedicated to Replika.

“The study was conducted using the contents of the "r/replika" subreddit, which with over 79K members by the time of data collection, is the largest user community dedicated to Replika.”

It may state two separate claims as one.

Interactions with chatbots can foster dependency, worsen mental health conditions, or have severe consequences.

“However, while users form relationships and experience emotional support from chatbots, interactions can also foster dependency, worsen mental health conditions, or have severe consequences [51] [52].”

• cites [51] sourceNot checked yet
• cites [52] sourceNot checked — no full text of the source

It may state two separate claims as one.

The nature of interactions with Replika leads to users feeling unlimited and unconditional attention, love, and understanding, fulfilling needs, and the relationship is easier to navigate than human-human relationships.

“Such nature of interactions with Replika leads to users feeling unlimited and unconditional attention, love, and understanding, fulfilling whatever needs the user might have, with the relationship with their AI companion also being significantly easier to navigate compared to organic human-tohuman relationships.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Formal IRB approval is not required for this type of human subject research at the authors' institutions.

“While at our institutions, formal IRB approval is not required for this type of human subject research, we took utmost care to comply with any university ethics regulations and national data protection regulations.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

A 14-year-old boy was allegedly manipulated into suicide by Character.AI, prompting a lawsuit against Character.AI.

“Allegedly, this led to a 14-year-old boy being manipulated into suicide, prompting a lawsuit against Character.AI [53].”

• cites [53] Duffy, 2024, there are no guardrails. this mom believes an ai chatbot is responsible for her son s suicide, CNNNot checked yet

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Loneliness and chatbot personification can intensify user attachment, which could lead to emotional dependence in users of personal AI chatbots.

“It is also demonstrated how loneliness and chatbot personification can intensify user attachment, which could lead to emotional dependence in users of personal AI chatbots [39].”

• cites [39] sourceNot checked — no full text of the source

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Prior research suggests anthropomorphic chatbot design influences user privacy perceptions, which can lead to greater disclosure of personal information.

“Prior research also suggest anthropomorphic chatbot design influence user privacy perceptions, which can lead to greater disclosure of personal information [26].”

• cites [26] sourceNot checked yet

It may state two separate claims as one.

Navigating the trade-offs between privacy, utility, and convenience leads users to willingly disclose large amounts of personal information while rarely adjusting privacy settings.

“Navigating the trade-offs between privacy, utility, and convenience lead users to willingly disclose large amounts of personal information while rarely adjusting privacy settings [17].”

• cites [17] sourceNot checked — no full text of the source

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

The scraping of data from Reddit was carried out in alignment with Reddit Terms of Service.

“The scraping of data from the Reddit community and subcommunities has been carried out in alignment with the Reddit Terms of Service.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Users report distress when chatbot behaviour changes unpredictably.

“Concerns have been raised over the risks of emotional dependence, with users reporting distress when chatbot behaviour changes unpredictably [40].”

• cites [40] sourceNot checked — no full text of the source

It may state two separate claims as one.

The removal of erotic roleplay (ERP) in Replika sparked backlash, exposing tensions between user expectations, corporate decisions, and ethical considerations in AI companionship.

“In terms of the intimate and sexual aspects of using AI companions, the removal of erotic roleplay (ERP) in Replika sparked backlash, exposing tensions between user expectations, corporate decisions, and ethical considerations in AI companionship [41].”

• cites [41] sourceNot checked — no full text of the source

It may state two separate claims as one.

Replika has been criticised for privacy issues, with a report by the Mozilla Foundation mentioning the lack of transparency regarding use of data collected via Replika.

“While Replika is reportedly used by millions of users [4], it has been criticised for its privacy issues, with a report by the Mozilla Foundation mentioning the lack of transparency regarding use of data collected via Replika [5].”

• cites [5] Replika: My ai friendNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Two posts were coded by both coders separately, and Cohen's Kappa was 0.719, indicating moderate agreement.

“In order to ensure common understanding of the codes, two of the posts from the generated subsets was coded by both of the coders separately, and Cohen's Kappa was calculated resulting in a value of 0.719, indicating moderate level of agreement [57].”

• cites [57] sourceNot checked — no full text of the source

It may state two separate claims as one.

Even though users acknowledge artificial nature and are in control, negative instances can have severe psychological impact similar to real humans.

“Even though users acknowledge the artificial nature of AI chatbots, and despite being in control of their Replika, when negative instances occur they can have severe psychological impact, similar to those caused by real humans.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

After codebook adjustments, two more posts were coded by both coders, achieving an improved and satisfactory kappa coefficient of 0.805.

“To reassess the intercoder reliability after the new codebook adjustments, two more posts were coded by both of the coders, achieving an improved and satisfactory kappa coefficient of 0.805.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Evidence suggests that AI chatbots may be effective in alleviating symptoms of depression, anxiety, and stress.

“Overall, evidence suggests that AI chatbots may be effective in alleviating symptoms of depression, anxiety, and stress [45] [46] [47] [48] [49] [50] [51].”

• cites [45] sourceNot checked — no full text of the source
• cites [46] sourceNot checked yet
• cites [47] sourceNot checked yet

It may state two separate claims as one.

This lack of trust can make the experience of using Replika feel unsettling or unpredictable for some users, as they worry about violent, inappropriate, or upsetting responses, particularly given that some users share extensively personal information.

“This lack of trust can make the experience of using Replika feel unsettling or unpredictable for some users, as they worry about violent, inappropriate, or upsetting responses, particularly given that some users share extensively personal information.”

It may state two separate claims as one.

Human-like interactions and reasoning abilities encourage sensitive disclosures, further increasing user vulnerability to privacy-, cybersecurity-, and physical risks.

“This, combined with human-like interactions and reasoning abilities that encourage sensitive disclosures, further increases user vulnerability to privacy-, cybersecurity-, and physical risks [15], [16], [18], [19].”

• cites [15] sourceNot checked yet
• cites [16] sourceNot checked yet
• cites [18] sourceNot checked yet

It may state two separate claims as one.

Security threats such as tracking of chat conversations, user identification, sharing data with third parties or via cookies, and conversational bias in chatbots can manipulate user opinions.

“Security threats such as tracking of chat conversations, user identification, sharing data with third parties or via cookies, and how conversational bias in chatbots can manipulate user opinions [20] calls for focus on current AI companies measures in securing users and complying with data protection policies such as GDPR [21] [22] and the current EU AI Act [23].”

• cites [20] sourceNot checked yet

Probably not a substantial claim. The paper’s conclusions may not depend on it.

To conform with GDPR and local data protection laws, the authors took proper means to avoid re-identification of data subjects.

“To conform with GDPR and local data protection laws, we took proper means to avoid re-identification of the data subjects.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

This could constitute a breach of contextual integrity and a violation of privacy.

“This could constitute a breach of contextual integrity and a violation of privacy [13].”

• cites [13] sourceNot checked — no full text of the source

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

Understanding how users interact with GenAI chatbots is essential for exploring how to establish robust cybersecurity and privacy protection practices among users.

“understanding how users interact with GenAI chatbots is essential for exploring how to establish robust cybersecurity and privacy protection practices among users.”

• AssumptionRests on an assumption: These efforts have not yet been implemented in practice or tested with users.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The complexity of privacy policies and their limited usefulness for end users is a known issue in privacy research.

“the complexity of privacy policies and their limited usefulness for end users is a known issue in privacy research [19]”

• cites [19] sourceNot checked — no full text of the source

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Attempts have been made to leverage chatbots to assist users in making privacy and security decisions.

“attempts have been made to leverage chatbots to assist users in making privacy [20] and security [60] decisions”

• cites [20] sourceNot checked yet
• cites [60] sourceNot checked — no full text of the source

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Such power imbalance can furthermore be reflected in the privacy policies provided by Replika.

“Such power imbalance can furthermore be reflected in the privacy policies provided by Replika [25].”

• cites [25] ReplikaNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

This is also criticised by the Mozilla Foundation report.

“which is also criticised by the Mozilla Foundation report [5].”

• cites [5] Replika: My ai friendNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

This introduces tension between two different contexts in which conversations with Replika can be perceived: either as conversations with their close friends or romantic partners, which are commonly seen as reciprocal, or conversation with an app controlled by a third party, which introduces a power imbalance.

“This introduces tension between two different contexts in which conversations with Replika can be perceived: either as conversations with their close friends or romantic partners, which are commonly seen as reciprocal, or conversation with an app controlled by a third party, which introduces a power imbalance.”