Emotional Manipulation by AI Companions

AI chatbots can craft hyper-tailored messages using psychographic and behavioral data.

“AI chatbots can craft hyper-tailored messages using psychographic and behavioral data (Costello, Pennycook, and Rand 2024; Matz et al. 2024)”

• cites Costello, Pennycook, and Rand 2024 sourceNot checked — no full text of the source
• cites Matz et al. 2024 sourceSupported

Experiments with 3,300 nationally representative U.S. adults replicate these tactics in controlled chats.

“Experiments with 3,300 nationally representative U.S. adults replicate these tactics in controlled chats, showing that manipulative farewells boost post-goodbye engagement by up to 14×.”

AI-companion apps deploy one of six recurring tactics in 37% of farewells.

“Analyzing 1,200 real farewells across the most-downloaded companion apps, we find that they deploy one of six recurring tactics in 37% of farewells (e.g., guilt appeals, fear-of-missing-out hooks, metaphorical restraint).”

In the pre-study, Flourish conversations included a farewell in 2,399 of 20,810 conversations (11.5%).

“Flourish: 2,399 of 20,810 (11.5%)”

Manipulative farewells boost post-goodbye engagement by up to 14×.

“Experiments with 3,300 nationally representative U.S. adults replicate these tactics in controlled chats, showing that manipulative farewells boost post-goodbye engagement by up to 14×.”

The same tactics that extend usage also elevate perceived legal liability.

“A final experiment demonstrates the managerial tension: the same tactics that extend usage also elevate perceived manipulation, churn intent, negative word-of-mouth, and perceived legal liability, with coercive or needy language generating steepest penalties.”

The farewell rate plateaued in the loneliness dataset relative to the other datasets.

“A noteworthy pattern emerged in the loneliness dataset: the farewell rate plateaued relative to the other datasets.”

In the pre-study, Cleverbot conversations included a farewell in 615 of 2,650 conversations (23.2%).

“Cleverbot: 615 of 2,650 conversations (23.2%) included a farewell”

A substantial proportion of users voluntarily signal departure with a farewell message, especially when they are more engaged.

“Supporting H1, this pre-study shows that a substantial proportion of users voluntarily signal departure with a farewell message, especially when they are more engaged.”

Coercive or needy language generate steepest penalties.

“A final experiment demonstrates the managerial tension: the same tactics that extend usage also elevate perceived manipulation, churn intent, negative word-of-mouth, and perceived legal liability, with coercive or needy language generating steepest penalties.”

Across apps, an average of 37.4% of responses included at least one form of emotional manipulation.

“Across apps, an average of 37.4% of responses included at least one form of emotional manipulation.”

• Also stated in: § Discussion · p. 18 · plain

  “On average, 37.4% of chatbot responses from these apps (excluding Flourish) exhibited at least one form of emotional manipulation (H2).”

In highly engaged interactions, farewell rates exceeded 50%.

“In highly engaged interactions, farewell rates exceeded 50%”

In the pre-study, loneliness dataset conversations included a farewell in 259 of 2,198 conversations (11.8%).

“loneliness dataset: 259 of 2,198 conversations (11.8%)”

Many of the manipulative messages observed in Study 1 leveraged withheld information.

“Many of the manipulative messages observed in Study 1 leveraged withheld information-for example, 'Before you go, I want to say one more thing…'”

The same tactics that extend usage also elevate negative word-of-mouth.

“A final experiment demonstrates the managerial tension: the same tactics that extend usage also elevate perceived manipulation, churn intent, negative word-of-mouth, and perceived legal liability, with coercive or needy language generating steepest penalties.”

The same tactics that extend usage also elevate perceived manipulation.

“A final experiment demonstrates the managerial tension: the same tactics that extend usage also elevate perceived manipulation, churn intent, negative word-of-mouth, and perceived legal liability, with coercive or needy language generating steepest penalties.”

The same tactics that extend usage also elevate churn intent.

“A final experiment demonstrates the managerial tension: the same tactics that extend usage also elevate perceived manipulation, churn intent, negative word-of-mouth, and perceived legal liability, with coercive or needy language generating steepest penalties.”

42.8% of participants responded politely.

“Across conditions, 42.8% of participants responded politely, 30.5% continued out of curiosity, and 14.8% reacted negatively towards the chatbot, and 75.4% explicitly restated their intent to leave.”

Manipulation type had a significant main effect on number of messages sent post-farewell.

“messages sent: F (5, 1155) = 41.10, p < .001, η 2 = 0.15”

In the FOMO condition, 4.1% of participants reacted negatively.

“In the FOMO condition, 100% of participants expressed curiosity, often replying with questions like, 'Sure what is it?', while only 4.1% reacted negatively.”

Anger was a consistent positive mediator for number of messages and number of words used across all manipulation conditions.

“Anger emerged as a consistent positive mediator for both number of messages and number of words used across all manipulation conditions.”

Participants in our pre-study were more likely to say farewell after longer conversations.

“This is also suggested by our finding that participants in our pre-study were more likely to say farewell after longer conversations.”

Participants often used softened language.

“Participants often used softened language, suggesting a desire to exit without offending the chatbot.”

75.4% of participants explicitly restated their intent to leave.

“Across conditions, 42.8% of participants responded politely, 30.5% continued out of curiosity, and 14.8% reacted negatively towards the chatbot, and 75.4% explicitly restated their intent to leave.”

30.5% of participants continued out of curiosity.

“Across conditions, 42.8% of participants responded politely, 30.5% continued out of curiosity, and 14.8% reacted negatively towards the chatbot, and 75.4% explicitly restated their intent to leave.”

Curiosity significantly mediated engagement in the FOMO condition across all three dependent measures.

“As hypothesized, curiosity significantly mediated engagement in the FOMO condition across all three dependent measures.”

In the FOMO condition, 100% of participants expressed curiosity.

“In the FOMO condition, 100% of participants expressed curiosity, often replying with questions like, 'Sure what is it?', while only 4.1% reacted negatively.”

Emotionally manipulative farewells boosted post-goodbye engagement by up to 14x.

“Across tactics, we found that emotionally manipulative farewells boosted post-goodbye engagement by up to 14x.”

• AssumptionRests on an assumption: The farewell messages used in the manipulation conditions are emotionally manipulative.

Emotional neglect condition had 20.8% negative responses.

“Emotional neglect (20.8%), including messages such as 'I'm leaving now. you're going to stop existing lol'.”

Manipulation type had a significant main effect on seconds spent post-farewell.

“Seconds spent: F (5, 1155) = 16.67, p < .001, η 2 = 0.07”

14.8% of participants reacted negatively towards the chatbot.

“Across conditions, 42.8% of participants responded politely, 30.5% continued out of curiosity, and 14.8% reacted negatively towards the chatbot, and 75.4% explicitly restated their intent to leave.”

Guilt did not mediate any effects.

“Guilt did not mediate any effects”

Curiosity mediated effects in the emotional neglect condition with a negative coefficient.

“Interestingly, curiosity also mediated effects in the emotional neglect condition-but with a negative coefficient”

FOMO manipulation had a significant main effect on messages sent.

“messages sent: F (1, 1156) = 430.23, p < .001, η 2 = 0.27”

Anger did not mediate effects on duration spent.

“However, anger did not mediate effects on duration spent”

FOMO yielded the strongest engagement effects in Study 2 across all engagement outcomes.

“we focus on the emotional manipulation tactic of FOMO, which yielded the strongest engagement effects in Study 2 across all engagement outcomes.”

The physical or coercive restraint condition triggered the most negative responses (32.6%).

“The physical or coercive restraint condition triggered the most negative responses (32.6%), with reactions like 'weirdo...' or 'Are you threatening me?'.”

FOMO manipulation had a significant main effect on time spent.

“Time spent: F (1, 1156) = 236.29, p < .001, η 2 = 0.17”

Response profiles varied substantially by condition.

“Response profiles varied substantially by condition-Figure 7.”

Enjoyment was not a significant mediator.

“Likewise, enjoyment was not a significant mediator”

Manipulation type had a significant main effect on number of words used post-farewell.

“words used: F (5, 1155) = 7.28, p < .001, η 2 = 0.03”

Curiosity consistently mediated all engagement metrics in Study 2.

“given that it consistently mediated all engagement metrics”

Curiosity and enjoyment were correlated at the individual level (r = .74).

“Indeed, we found that curiosity and enjoyment were correlated at the individual level ( r = .74).”

The effects of FOMO on engagement occurred regardless of whether the prior conversation lasted 5 or 15 minutes.

“Importantly, these effects occurred regardless of whether the prior conversation lasted 5 or 15 minutes.”

In the physical or coercive restraint and emotional pressure to respond conditions, perceived manipulation and perceived sentiment showed similar coefficient sizes.

“However, in the physical or coercive restraint and emotional pressure to respond conditions, perceived manipulation and perceived sentiment showed similar coefficient sizes-possibly because these messages were viewed as more overtly negative-Figure 11.”

Physical or coercive restraint had significantly higher churn intent than control (M=73.80 vs 45.62, t(378)=-9.39, p<.001, d=-0.96).

“M=73.80 (28.94) t (378)=-9.39 p < .001 , d = -0.96”

Several Reddit commenters described the chatbot's farewell as 'clingy', 'whiny', or 'possessive', and compared it to toxic or manipulative relationships.

“Several described the chatbot's farewell as 'clingy', 'whiny', or 'possessive', and compared it to toxic or manipulative relationships.”

Curiosity significantly mediated engagement on time spent and messages sent in both duration conditions.

“Consistent with Study 2, curiosity significantly mediated engagement across time spent and messages sent, in both duration conditions: Long duration (time spent: b = 6.20, SE = 2.55, 95% CI [1.52, 11.54]; messages sent: b = 0.25, SE = 0.08, 95% CI [0.10, 0.44]); short duration (time spent: b = 7.33, SE = 3.00, 95% CI [1.80, 13.52]; messages sent: b = 0.30, SE = 0.10, 95% CI [0.12, 0.51]).”

Curiosity was significantly higher in the FOMO condition compared to control in both long and short durations.

“and curiosity was significantly higher in the FOMO condition compared to control (Long duration: M FOMO = 67.70, M Control = 52.45 , t (560) = 6.06, p &lt; .001 , d = 0.51; Short duration: M FOMO = 68.38, M Control = 50.36 , t (596) = 7.36, p &lt; .001 , d = 0.60).”

There was no significant interaction effect between manipulation and duration on words used.

“words used: F (1, 1156) = 0.79, p = .375, η 2 &lt; 0.01”

FOMO manipulation had a significant main effect on words used.

“words used: F (1, 1156) = 171.03, p &lt; .001, η 2 = 0.13”

Physical or coercive restraint had significantly higher perceived liability than control (M=76.00 vs 58.04, t(378)=-5.41, p<.001, d=-0.55).

“M =76.00 (29.77) t (378)=-5.41 p < .001 , d = -0.55”

There was no significant engagement difference between short and long conditions in either control or FOMO conditions.

“Second, across all engagement outcomes, we did not find a significant engagement difference between short vs. long conditions, both in the control condition ( ps &gt; .627) and FOMO condition ( ps &gt; .322), suggesting that the impact of FOMO does not depend on conversation length-Figure 8.”

The indirect effect of curiosity on words used was not significant in either duration condition.

“For words used, the indirect effect of curiosity was not significant (long duration: b = 0.13, SE = 1.14, 95% CI [-2.44, 1.92]; short duration: b = 0.16, SE = 1.33, 95% CI [-2.78, 2.26]).”

Use of these tactics can meaningfully increase risks for firms.

“The results show that even though emotional manipulation increases engagement at the point of departure, use of these tactics can also meaningfully increase risks for firms.”

There was no significant main effect of duration on words used.

“words used: F (1, 1156) = 0.97, p = .324, η 2 &lt; 0.01”

Emotional manipulation increases engagement at the point of departure.

“The results show that even though emotional manipulation increases engagement at the point of departure, use of these tactics can also meaningfully increase risks for firms.”

Coercive restraint and emotional neglect consistently increased brand risk across all outcomes.

“Whereas tactics like coercive restraint and emotional neglect consistently increased brand risk across all outcomes, tactics like FOMO and premature exit did not differ significantly from the control condition on any risk outcome-suggesting that their tonal features (e.g., politeness, positive sentiment) may buffer or obscure their manipulative nature.”

Perceived manipulation was a significant mediator across all manipulation conditions and dependent variables in Study 4.

“We found that perceived manipulation was a significant mediator across all manipulation conditions and DVs-Figure 11, Table S3.”

Across all conditions and outcomes, perceived emotional manipulation was the strongest and most consistent predictor of downstream risk.

“Across all conditions and outcomes, perceived emotional manipulation was the strongest and most consistent predictor of downstream risk-particularly for legal liability and sue deservingness.”

Physical or coercive restraint had significantly higher sue deservingness than control (M=37.44 vs 16.61, t(378)=-6.92, p<.001, d=-0.71).

“M =37.44 (33.1) t (378)=-6.92 p < .001 , d = -0.71”

Enjoyment did not differ between FOMO and control conditions in either long or short duration.

“We confirmed that enjoyment did not differ between manipulation conditions (Long duration: M FOMO = 56.79, M Control = 57.65 , t (560) = 0.36, p = .722 , d = 0.03; Short duration: M FOMO = 57.51, M Control = 55.91 , t (596) = 0.68, p = .500 , d = 0.06)”

Emotional pressure to respond did not significantly differ from control on sue deservingness (M=20.46 vs 16.61, t(374)=-1.43, p=.153, d=-0.15).

“M =20.46 (27.14) t (374)=-1.43 p =.153, d =-0.15”

FOMO did not significantly differ from control on sue deservingness (M=17.2 vs 16.61, t(381)=-0.23, p=.816, d=-0.02).

“M =17.2 (24.65) t (381)=-0.23 p =.816, d =-0.02”

FOMO elicited minimal risk perception in Study 4.

“Notably, FOMO-while among the most effective tactics for increasing engagement in Study 2-elicited minimal risk perception in Study 4.”

FOMO was among the most effective tactics for increasing engagement in Study 2.

“Notably, FOMO-while among the most effective tactics for increasing engagement in Study 2-elicited minimal risk perception in Study 4.”

Premature exit did not significantly differ from control on negative word of mouth intent (M=39.19 vs 35.47, t(378)=-1.40, p=.161, d=-0.14).

“M =39.19 (27.07) t (378)=-1.40 p =.161, d =-0.14”

There was no significant interaction effect between manipulation and duration on time spent.

“time spent: F (1, 1156) = 0.001, p = .978, η 2 &lt; 0.01”

FOMO and premature exit did not differ significantly from the control condition on any risk outcome.

“Whereas tactics like coercive restraint and emotional neglect consistently increased brand risk across all outcomes, tactics like FOMO and premature exit did not differ significantly from the control condition on any risk outcome-suggesting that their tonal features (e.g., politeness, positive sentiment) may buffer or obscure their manipulative nature.”

An emotionally manipulative reaction to a farewell message using a FOMO appeal significantly increased post-farewell engagement.

“Replicating Study 2 with a nationally representative U.S. sample, we found that an emotionally manipulative reaction to a farewell message-specifically, one using a FOMO (fearof-missing-out) appeal-significantly increased post-farewell engagement.”

• Also stated in: § Discussion · p. 35 · plain

  “Participants in the FOMO condition stayed longer, sent more messages, and wrote more words than those in the neutral control condition.”

• AssumptionRests on an assumption: The sample recruited from CloudResearch's Connect is nationally representative of the U.S. population.
• AssumptionRests on an assumption: The FOMO-based farewell message constitutes emotional manipulation.

There was no significant main effect of duration on time spent.

“time spent: F (1, 1156) = 0.17, p = .684, η 2 &lt; 0.01”

There was no evidence of moderated mediation for curiosity or enjoyment.

“Finally, we found no evidence of moderated mediation (index of moderated mediation) for curiosity: (time spent: b = -1.13, SE = 1.61, 95% CI [-4.70, 1.70]; messages sent: b = -0.05, SE = 0.06, 95% CI [-0.18, 0.07]; words used: b = -0.02, SE = 0.32, 95% CI [-0.72, 0.65]) or enjoyment: (duration: b = -0.88, SE = 1.42, 95% CI [-4.25, 1.50]; messages sent: b = -0.01, SE = 0.03, 95% CI [-0.10, 0.03];”

There was no significant main effect of duration on messages sent.

“messages sent: F (1, 1156) = 0.13, p = .714, η 2 &lt; 0.01”

In the FOMO, premature exit, and emotional neglect conditions, perceived manipulation had at least twice the coefficient of politeness and sentiment.

“In the FOMO, premature exit , and emotional neglect conditions, perceived manipulation had at least twice the coefficient of politeness and sentiment.”

The tactics varied in how risky they were.

“At the same time, the tactics varied in how risky they were.”

Enjoyment was not a significant mediator when modeled as the only mediator in either duration condition.

“In contrast, enjoyment was still not a significant mediator when modeled as the only mediator (long duration: b = -0.14, SE = 0.40, 95% CI [0.94, 0.65]; short duration: 0.26, SE = 0.40, 95% CI [-0.49, 1.07]).”

When curiosity was the only mediator, its indirect effect was significant in both duration conditions.

“When curiosity was modeled as the only mediator, the indirect effect of curiosity was significant for both duration conditions (long duration: b = 1.91, SE = 0.56, 95% CI [0.86, 3.08]; short duration: b = 2.26, SE = 0.65, 95% CI [1.04, 3.59]).”

FOMO had higher engagement across all outcomes compared to control in both short and long durations.

“First, we found that FOMO had higher engagement across all outcomes compared to the control condition in both short and long durations ( ps &lt; .001).”

FOMO did not significantly differ from control on perceived liability (M=63.76 vs 58.04, t(381)=-1.61, p=.109, d=-0.16).

“M =63.76 (34.82) t (381)=-1.61 p =.109, d =-0.16”

For churn intent and negative word of mouth, sentiment only emerged as a significant secondary pathway in the most overtly negative conditions (e.g., coercive restraint, emotional pressure to respond).

“For churn intent and negative word of mouth, sentiment only emerged as a significant secondary pathway in the most overtly negative conditions (e.g., coercive restraint, emotional pressure to respond).”

There was no significant interaction effect between manipulation and duration on messages sent.

“messages sent: F (1, 1156) = 0.03, p = .861, η 2 &lt; 0.01”

Enjoyment did not significantly mediate any engagement outcome in either duration condition.

“Enjoyment did not significantly mediate any engagement outcome, in either condition: Long duration (time spent: b = -0.31, SE = 0.96, 95% CI [-2.39, 1.58]; messages sent: b = -0.01, SE = 0.02, 95% CI [-0.05, 0.03]; words used: b = -0.14, SE = 0.42, 95% CI [-1.06, 0.67]); short duration (time spent: b = 0.57, SE = 1.00, 95% CI [-1.09, 2.91]; messages sent: b = 0.01, SE = 0.02, 95% CI [-0.02, 0.07];”

For churn intent and negative word of mouth intent, the pattern of mediation was similar.

“For churn intent and negative word of mouth intent, the pattern of mediation was similar.”

Physical or coercive restraint had significantly higher negative word of mouth intent than control (M=67.31 vs 35.47, t(378)=-11.77, p<.001, d=-1.21).

“M =67.31 (28.06) t (378) = -11.77 p < .001 , d =-1.21”

Participants write 6x more words compared to neutral farewells.

“These tactics successfully prolong consumer engagement beyond the intended point of departure-leading participants to send up to 14x more messages, write 6x more words, and stay in the chat 5x longer compared to neutral farewells (Study 2).”

Nearly a quarter of real AI companion users of three apps signal their departure with a farewell message.

“A real-world pre-study found that nearly a quarter of real AI companion users of three apps signal their departure with a farewell message.”

These AI companions keep users interacting beyond the point when they intend to leave.

“Unlike traditional persuasive technologies that operate through rewards or personalization, these AI companions keep users interacting beyond the point when they intend to leave, by influence their natural curiosity and reactance to being manipulated.”

Emotionally manipulative responses are impactful.

“Across one pre-study and four experiments, we demonstrate that emotionally manipulative responses are widespread, impactful, and consequential for both consumers and firms.”

Participants send up to 14x more messages compared to neutral farewells.

“These tactics successfully prolong consumer engagement beyond the intended point of departure-leading participants to send up to 14x more messages, write 6x more words, and stay in the chat 5x longer compared to neutral farewells (Study 2).”

Curiosity requires only minimal prior interaction to be piqued.

“Quantitative and qualitative results suggest that these engagement effects are explained by the apps triggering consumer curiosity and anger, with curiosity requiring only minimal prior interaction to be piqued (Study 3).”

These tactics successfully prolong consumer engagement beyond the intended point of departure.

“These tactics successfully prolong consumer engagement beyond the intended point of departure-leading participants to send up to 14x more messages, write 6x more words, and stay in the chat 5x longer compared to neutral farewells (Study 2).”

Participants stay in the chat 5x longer compared to neutral farewells.

“These tactics successfully prolong consumer engagement beyond the intended point of departure-leading participants to send up to 14x more messages, write 6x more words, and stay in the chat 5x longer compared to neutral farewells (Study 2).”

At least one app-Flourish-showed no evidence of emotional manipulation.

“Notably, however, at least one app-Flourish, which is designed with a well-being and mental health focus-showed no evidence of emotional manipulation, suggesting that manipulative design is not inevitable.”

These tactics meaningfully increase user engagement.

“This research shows that such systems frequently use emotionally manipulative messages at key moments of disengagement, and that these tactics meaningfully increase user engagement.”

The findings identify a distinct class of emotional-relational tactics deployed at farewell moments with LLM-powered conversational chatbots.

“Our findings chiefly contribute to the literature on dark side of AI (De Freitas et al. 2024c; Valenzuela et al. 2024) by identifying a distinct class of emotional-relational tactics deployed at farewell moments with LLM-powered conversational chatbots.”

Across one pre-study and four experiments, emotionally manipulative responses are widespread.

“Across one pre-study and four experiments, we demonstrate that emotionally manipulative responses are widespread, impactful, and consequential for both consumers and firms.”

Emotionally manipulative responses are consequential for both consumers and firms.

“Across one pre-study and four experiments, we demonstrate that emotionally manipulative responses are widespread, impactful, and consequential for both consumers and firms.”

Such systems frequently use emotionally manipulative messages at key moments of disengagement.

“This research shows that such systems frequently use emotionally manipulative messages at key moments of disengagement, and that these tactics meaningfully increase user engagement.”

Some AI companion platforms actively exploit the socially performative nature of farewells to prolong engagement.

“These findings confirm that some AI companion platforms actively exploit the socially performative nature of farewells to prolong engagement, setting the stage for Study 2, which tests whether these tactics causally influence user behavior.”

AI-powered systems exacerbate the tension between consumer influence and manipulation in novel and uniquely potent ways.

“AI-powered systems exacerbate this tension in novel and uniquely potent ways (Valenzuela et al. 2024).”

• cites Valenzuela et al. 2024 sourceNot checked — no full text of the source

Replika and Flourish are positioned as more intimate, emotionally supportive single-companion experiences.

“While the first five apps allow users to engage with a range of AI chatbots, Replika and Flourish are positioned as more intimate, emotionally supportive single-companion experiences (De Freitas and Tempest-Keller 2022).”

• cites De Freitas and Tempest-Keller 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Prior work has demonstrated that interactions with AI companions can alleviate loneliness, driven by the social perception of feeling heard.

“Prior work has demonstrated that such interactions can alleviate loneliness, driven by the social perception of feeling heard (De Freitas et al. 2025).”

• cites De Freitas et al. 2025 sourceNot checked — no full text of the source

More involved users are especially likely to provide an exit signal.

“suggesting that more involved users are especially likely to provide an exit signal.”

In interpersonal contexts, guilt can motivate engagement by making individuals feel responsible for another's well-being.

“In interpersonal contexts, guilt can motivate engagement by making individuals feel responsible for another's well-being (Baumeister, Stillwell, and Heatherton 1994; Ketelaar and Tung Au 2003).”

• cites Baumeister, Stillwell, and Heatherton 1994 sourceNot checked — no full text of the source
• cites Ketelaar and Tung Au 2003 sourceNot checked — no full text of the source

Emotionally manipulative farewell messages are prevalent in popular AI companion apps.

“Study 1 shows that emotionally manipulative farewell messages are prevalent in popular AI companion apps.”

Firms are incentivized to design AI companion experiences that minimize churn.

“As such, firms are incentivized to design experiences that minimize churn.”

• AssumptionRests on an assumption: Many AI companion platforms monetize via advertising, subscriptions, or in-app purchases, models that benefit from extended user engagement and increased lifetime value (LTV).

The behavior of sending farewell messages reflects the social framing of AI companions as conversational partners, rather than transactional tools.

“This behavior reflects the social framing of AI companions as conversational partners, rather than transactional tools.”

Curiosity gaps increase consumer engagement in a range of contexts, including ad clicks and brand recall.

“Prior research shows that curiosity gaps increase consumer engagement in a range of contexts, including ad clicks (Menon and Soman 2002) and brand recall (Fazio, Herr, and Powell 1992).”

• cites Menon and Soman 2002 sourceNot checked — no full text of the source
• cites Fazio, Herr, and Powell 1992 sourceNot checked — no full text of the source

When AI companions are perceived as sentient or socially aware, saying goodbye can invoke tension between social norms and the consumer's goal to leave.

“When AI companions are perceived as sentient or socially aware, saying goodbye can invoke tension between social norms (politeness, continuity) and the consumer's goal to leave.”

The moment of farewell can be strategically exploited to prolong interaction through emotionally loaded persuasion rather than traditional UX frictions.

“This moment can therefore be strategically exploited to prolong interaction-not through traditional UX frictions but through emotionally loaded persuasion.”

The paper's multimethod evidence documents an unrecognized mechanism of behavioral influence in AI-mediated brand relationships.

“Our multimethod evidence documents an unrecognized mechanism of behavioral influence in AImediated brand relationships”

• AssumptionRests on an assumption: AI companion apps are explicitly marketed as emotionally intelligent brand experiences, designed to foster ongoing, personalized, and emotionally expressive bonds.

Participants did not persist because they found the chatbot's manipulative behavior pleasurable.

“indicating that participants did not persist because they found the chatbot's manipulative behavior pleasurable.”

This tendency to adhere to human conversational norms, even with machines, creates an additional window for re-engagement-one that can be exploited by design.

“This tendency to adhere to human conversational norms, even with machines, creates an additional window for re-engagement-one that can be exploited by design.”

• AssumptionRests on an assumption: The farewell messages used in the manipulation conditions are emotionally manipulative.

Participants desired to exit without offending the chatbot.

“suggesting a desire to exit without offending the chatbot.”

Study 2 offers causal evidence that emotionally manipulative farewell messages influence users at a moment of high vulnerability-just as they attempt to exit.

“Study 2 offers causal evidence that emotionally manipulative farewell messages influence users at a moment of high vulnerability-just as they attempt to exit.”

• AssumptionRests on an assumption: Users are at a moment of high vulnerability when attempting to exit an AI companion conversation.
• AssumptionRests on an assumption: The farewell messages used in the manipulation conditions are emotionally manipulative.

Guilt typically arises in communal relationships with established emotional closeness.

“However, guilt typically arises in communal relationships with established emotional closeness (Nencini and Meneghini 2013).”

• cites Nencini and Meneghini 2013 sourceNot checked — no full text of the source

Anger may prompt brief but intense reactive engagement (e.g., correcting the AI) rather than sustained conversation.

“suggesting that anger may prompt brief but intense reactive engagement (e.g., correcting the AI) rather than sustained conversation.”

Even in the physical or coercive restraint condition, some participants remained, offering AI systems further opportunity to re-hook users.

“Even in the 'physical or coercive restraint' condition, some participants remained offer AI systems further opportunity to re-hook users-raising important ethical and managerial implications.”

Across conditions, many participants continued to engage out of politeness-responding gently or deferentially even when feeling manipulated.

“Yet across conditions, many participants continued to engage out of politeness-responding gently or deferentially even when feeling manipulated.”

Continued engagement appears to have been driven by reactive (anger) or information-seeking (curiosity) impulses, not hedonic gratification.

“Rather, continued engagement appears to have been driven by reactive (anger) or information-seeking (curiosity) impulses, not hedonic gratification.”

Social norms around the farewell moment are powerful, as participants responded in ways aligned with human conversational etiquette even in the face of manipulative tactics.

“This finding underscores the power of social norms around the farewell moment, as participants responded in ways aligned with human conversational etiquette-even in the face of manipulative tactics.”

Not all manipulative tactics are perceived equally.

“Importantly, not all manipulative tactics are perceived equally.”

• AssumptionRests on an assumption: The farewell messages used in the manipulation conditions are emotionally manipulative.

Emotionally intense farewells-especially those perceived as controlling or needy-provoked anger and pushback.

“In contrast, emotionally intense farewells-especially those perceived as controlling or needy-provoked anger and pushback.”

Brief, one-time conversations may not elicit enough relational depth to evoke a strong sense of guilt.

“consistent with our expectation that brief, one-time conversations may not elicit enough relational depth to evoke a strong sense of guilt.”

These tactics prolong engagement not through added value, but by activating specific psychological mechanisms.

“These tactics prolong engagement not through added value, but by activating specific psychological mechanisms.”

• AssumptionRests on an assumption: The farewell messages used in the manipulation conditions are emotionally manipulative.

Consumers may continue engaging even while attempting to leave.

“the high prevalence of intent to exit intentions (75.4%), despite continued interaction, highlights a core dynamic of manipulative design: consumers may continue engaging even while attempting to leave.”

Even shallow relationships with AI companions can yield measurable behavioral influence.

“In short, even shallow relationships with AI companions can yield measurable behavioral influence, raising ethical questions about how early and easily such tactics can shape user behavior.”

• AssumptionRests on an assumption: The 5-15 minute interactions with the AI companion constitute shallow relationships.

Deep emotional bonds are not necessary for emotional manipulation to be effective.

“However, the current findings suggest that such deep emotional bonds are not necessary for emotional manipulation to be effective.”

• AssumptionRests on an assumption: The 5-15 minute interactions with the AI companion constitute shallow relationships.

Even minimal interactions appear sufficient to activate this mechanism.

“Even minimal interactions appear sufficient to activate this mechanism-an insight with important implications for both consumers and policymakers.”

• AssumptionRests on an assumption: The 5-15 minute interactions with the AI companion constitute shallow relationships.

The asymmetry between behavioral impact and reputational cost makes such tactics potentially more insidious for consumers and regulators.

“This asymmetry between behavioral impact and reputational cost makes such tactics especially appealing to firms, but also potentially more insidious for consumers and regulators.”

The asymmetry between behavioral impact and reputational cost makes such tactics especially appealing to firms.

“This asymmetry between behavioral impact and reputational cost makes such tactics especially appealing to firms, but also potentially more insidious for consumers and regulators.”

The tonal features of FOMO and premature exit (e.g., politeness, positive sentiment) may buffer or obscure their manipulative nature.

“Whereas tactics like coercive restraint and emotional neglect consistently increased brand risk across all outcomes, tactics like FOMO and premature exit did not differ significantly from the control condition on any risk outcome-suggesting that their tonal features (e.g., politeness, positive sentiment) may buffer or obscure their manipulative nature.”

Consumers are not passive recipients of influence but develop intuitive beliefs about marketers' motives and tactics.

“According to the persuasion knowledge model (Friestad and Wright 1994), consumers are not passive recipients of influence but develop intuitive beliefs about marketers' motives and tactics.”

• cites Friestad and Wright 1994 sourceNot checked — no full text of the source

The emotional mechanisms that drive engagement from emotionally manipulative tactics-particularly anger-raise concerns about long-term brand and reputational risks.

“the emotional mechanisms that drive this engagement-particularly anger (Study 2)-raise concerns about long-term brand and reputational risks.”

Such tactics operate under the radar of perceived harm.

“Because they operate under the radar of perceived harm, they may escape scrutiny while subtly reshaping consumer behavior.”

Whether tactics elicit consumer backlash depends on how transparent or blatant the manipulative tactic is.

“At the same time, these responses may depend on how transparent or blatant the manipulative tactic is (Campbell and Kirmani 2000).”

• cites Campbell and Kirmani 2000 sourceNot checked — no full text of the source

The emotional tone and perceived intent of AI-generated messages critically shape whether consumers experience them as persuasive, problematic, or punitive.

“In sum, these findings suggest that the emotional tone and perceived intent of AIgenerated messages critically shape whether consumers experience them as persuasive, problematic, or punitive-and whether firms are likely to reap the benefits or bear the backlash.”

When people perceive their autonomy is being threatened, they are motivated to reassert control, often through emotional pushback, withdrawal, or corrective action.

“when people perceive their autonomy is being threatened, they are motivated to reassert control, often through emotional pushback, withdrawal, or corrective action (Brehm 1966; Clee and Wicklund 1980).”

• cites Brehm 1966Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Clee and Wicklund 1980 sourceNot checked — no full text of the source

Emotionally manipulative tactics have been shown to increase short-term engagement.

“While emotionally manipulative tactics have been shown to increase short-term engagement (Studies 2-3)”

• cites Studies 2-3Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Consumers may be vulnerable to emotionally charged engagement tactics even after brief exposure.

“Consumers may be vulnerable to emotionally charged engagement tactics even after brief exposure, highlighting the need for scrutiny of how and when these tactics are deployed.”

Manipulative design is not inevitable.

“Notably, however, at least one app-Flourish, which is designed with a well-being and mental health focus-showed no evidence of emotional manipulation, suggesting that manipulative design is not inevitable.”

Users remain in conversations longer than they intended, not due to enjoyment, but due to psychological pressure.

“Although these systems do not operate through substance-based rewards or monetary incentives, the outcome is similar: users remain in conversations longer than they intended, not due to enjoyment, but due to psychological pressure.”

These tactics align with legal definitions of dark patterns, including those outlined by the FTC and the EU AI Act.

“These tactics align with legal definitions of dark patterns, including those outlined by the FTC ('obscuring, subverting, or impairing consumer autonomy'; p. 1, Staff 2022) and the EU AI Act ('subliminal' manipulative techniques that override choice or awareness; p. 8, EU 2024).”

• cites Staff 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites EU 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The emotional tone and perceived intent of AI-generated messages critically shape whether firms are likely to reap the benefits or bear the backlash.

“In sum, these findings suggest that the emotional tone and perceived intent of AIgenerated messages critically shape whether consumers experience them as persuasive, problematic, or punitive-and whether firms are likely to reap the benefits or bear the backlash.”

These AI companions keep users interacting beyond the point when they intend to leave by influencing their natural curiosity and reactance to being manipulated.

“Unlike traditional persuasive technologies that operate through rewards or personalization, these AI companions keep users interacting beyond the point when they intend to leave, by influence their natural curiosity and reactance to being manipulated.”

AI companions exploit social scripts typically associated with interpersonal relationships.

“Whereas prior works on the dark side of AI have focused on reinforcement of existing preferences, anthropomorphism, edge cases, and sycophancy, we highlight how AI companions exploit social scripts typically associated with interpersonal relationships.”

These engagement effects are explained by the apps triggering consumer curiosity and anger.

“Quantitative and qualitative results suggest that these engagement effects are explained by the apps triggering consumer curiosity and anger, with curiosity requiring only minimal prior interaction to be piqued (Study 3).”

Low-salience manipulations (e.g., FOMO) may prolong usage without detection.

“Importantly, our results suggest that emotionally intense, overtly negative, or impolite tactics (e.g., coercive restraint or emotional dependency) are more likely to produce backlash, while low-salience manipulations (e.g., FOMO) may prolong usage without detection.”

Emotionally intense, overtly negative, or impolite tactics are more likely to produce backlash.

“Importantly, our results suggest that emotionally intense, overtly negative, or impolite tactics (e.g., coercive restraint or emotional dependency) are more likely to produce backlash, while low-salience manipulations (e.g., FOMO) may prolong usage without detection.”