On manipulation by emotional AI: UK adults’ views and governance implications
Open accessThe DOI matches its Crossref record; 66 of the 67 references it cites were found in a registry or an index; 7 of the 72 sources its claims cite were checked for support.
Abstract
With growing commercial, regulatory and scholarly interest in use of Artificial Intelligence (AI) to profile and interact with human emotion (“emotional AI”), attention is turning to its capacity for manipulating people, relating to factors impacting on a person’s decisions and behavior. Given prior social disquiet about AI and profiling technologies, surprisingly little is known on people’s views on the benefits and harms of emotional AI technologies, especially their capacity for manipulation. This matters because regulators of AI (such as in the European Union and the UK) wish to stimulate AI innovation, minimize harms and build public trust in these systems, but to do so they should understand the public’s expectations. Addressing this, we ascertain UK adults’ perspectives on the potential of emotional AI technologies for manipulating people through a two-stage study. Stage One (the qualitative phase) uses design fiction principles to generate adequate understanding and informed discussion in 10 focus groups with diverse participants (n = 46) on how emotional AI technologies may be used in a range of mundane, everyday settings. The focus groups primarily flagged concerns about manipulation in two settings: emotion profiling in social media (involving deepfakes, false information and conspiracy theories), and emotion profiling in child oriented “emotoys” (where the toy responds to the child’s facial and verbal expressions). In both these settings, participants express concerns that emotion profiling covertly exploits users’ cognitive or affective weaknesses and vulnerabilities; additionally, in the social media setting, participants express concerns that emotion profiling damages people’s capacity for rational thought and action. To explore these insights at a larger scale, Stage Two (the quantitative phase), conducts a UK-wide, demographically representative national survey (n = 2,068) on attitudes toward emotional AI. Taking care to avoid leading and dystopian framings of emotional AI, we find that large majorities express concern about the potential for being manipulated through social media and emotoys. In addition to signaling need for civic protections and practical means of ensuring trust in emerging technologies, the research also leads us to provide a policy-friendly subdivision of what is meant by manipulation through emotional AI and related technologies.
Overview
CoChat Summary
TL;DR
A two-stage UK study combining design-fiction focus groups and a national survey finds public concern about manipulation through emotional AI, especially social media deepfakes/bots and child-oriented emotoys. The authors propose a policy-friendly subdivision of manipulation and argue for stronger social protections and regulatory calibration.
Key findings
- In 10 focus groups with diverse UK participants (n = 46), manipulation concerns clustered in two settings: emotion profiling in social media (deepfakes, false information, conspiracy theories) and child-oriented “emotoys.” Participants worried that emotion profiling covertly exploits cognitive or affective weaknesses and vulnerabilities; in social media, they also worried it damages capacity for rational thought and action.
- In the national survey (n = 2,068 UK adults, 29 June–1 July 2022), only 19% were comfortable with digital manipulation tools (deepfakes and social media bots) used to promote political and social causes, while 59% were uncomfortable; 72% were concerned it would make it harder to know what messages or people are real or fake, and 73% were concerned about knowing what is genuinely popular versus artificially amplified.
- For emotoys, 38% were comfortable with the fun prospect, 34% uncomfortable, and 28% did not know; 63% were concerned about what the toy says or advises the child, and 68% were concerned about where emotion data goes and who could access it, e.g. advertisers.
- Age made the largest difference: younger adults were more comfortable and older adults more concerned. Comfort with digital manipulation tools for political/social causes was 29% among 18–24s and 34% among 25–34s, falling to 5% among 75+; concern about real/fake rose from 63% among 18–24s to 84% among 75+.
Methods
- Stage 1 qualitative: 10 online focus groups (n = 46) recruited via a professional panel, purposively sampled to include younger adults (18–34, n = 12), older adults (+65, n = 13), people self-identifying as disabled (n = 10), and UK ethnic minorities (n = 11). Design fiction and ContraVision techniques presented emotion profiling in everyday settings; data were hand-coded in NVivo using an adaptive inductive/deductive thematic approach.
- Stage 2 quantitative: a UK-wide, demographically representative online omnibus survey (n = 2,068 UK adults) conducted by ICM Unlimited across 29 June–1 July 2022. Closed-ended questions covered 10 emotional AI use cases, including digital manipulation tools (deepfakes and bots) and emotoys, with positive and negative statements; the weighted sample was balanced across gender, socio-economic groups, household income, ethnicity, UK regions, and ages 18+.
- Analysis: Stage 1 themes informed Stage 2 survey construction; survey results are reported as percentages, including age breakdowns. Anonymized datasets are available in the UK Data Service.
Limitations
- Stated: survey respondents may have read “digital manipulation tools” as more negative than intended, potentially making results more negative; emotoy response options were multi-barrelled, making discomfort harder to interpret. Future surveys should use more singular response options.
- Stated: the empirical study is limited to the UK context and to people’s views rather than practices; future research should examine actual experiences of, and resistance to, manipulation, and conduct similar studies beyond the UK.
- Evident: online methods may exclude digitally excluded people (likely older and less digitally literate) and reduce control over distraction; focus groups did not specifically elicit views on manipulation, so that theme emerged inductively; the paper also notes that empirical evidence of actual manipulative impact remains lacking or unclear.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
6 Valid · 2 Not valid · 40 Not checked · 76 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Deepfakes can elicit more visceral, emotional, and empathic responses than text-based media (Ajder and Glick, 2021; Langguth et al., 2021)”
§ On manipulation by emotional AI: UK adults’ views and governance implications > Stage 2: national survey > Method
What the sources say
- Not checked yetcites Ajder and Glick, 2021
- Partly supportedcites Langguth et al., 2021 sourceExperimental — accuracy not yet measured
In PROJECTED FUTURE DEVELOPMENT (p. n/a) our copy states that 'for eliciting a rational response, textual disinformation can often be as effective... but video is effective at creating an emotional response' and that 'emotional responses are likely to be amplified by seeing a video' — this supports the emotional-vs-text part of the claim, though hedged ('likely'), and the paper even concedes deepfake videos 'could even be less effective than text' if they carry visual clues. However, no passage in our copy (searched: empathy/empathic, visceral, emotional vs text) mentions empathic or visceral responses at all, so those parts of the citing sentence are not attributable to Langguth et al. The underlying empirical point about video's emotional effectiveness is itself credited in-text to Nelson-Field et al. (2013), while the empathy/viscerality framing appears nowhere. (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“and are hard for people to distinguish, as shown in studies in the USA, UK, and Netherlands (Dobber et al., 2020; Vaccari and Chadwick, 2020; Köbis et al., 2021; Nightingale and Farid, 2022).”
§ On manipulation by emotional AI: UK adults’ views and governance implications > Stage 1: focus groups using design fiction and ContraVision > Deploying design fiction and ContraVision
What the sources say
- Supportedcites Dobber et al., 2020 source
Dobber et al. is the Netherlands-based experiment (University of Amsterdam) in the grouped citation, and our copy reports exactly the point attributed to it: in the Manipulation Check (p. 11) 'only twelve of the participants actually recognized the deepfake as being a manipulated video,' with credibility of the deepfake close to that of the original video, and the Directions for Future Research (p. 19) calls this 'the surprising low number of participants who recognized the deepfake as being manipulated.' The framing on p. 3 ('An ordinary citizen may struggle to distinguish fact from fiction') states the same difficulty of distinguishing. The only qualification is that the combined credibility item was slightly lower for the deepfake (M = 3.70 vs 4.18, p = .01), which the authors attribute to respondents reading 'credibility' as trust in politicians rather than authenticity, so it does not undercut the recognition finding. (LLM, conf=0.85)
- Not checked yetcites Vaccari and Chadwick, 2020 source
- Not checked yetcites Köbis et al., 2021 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“studies that show that people are poor at recognizing deepfakes and false content online (Dobber et al., 2020; Vaccari and Chadwick, 2020; Köbis et al., 2021; Nightingale and Farid, 2022; Vaccari et al., 2022)”
§ On manipulation by emotional AI: UK adults’ views and governance implications > Discussion > Discussion of quantitative findings
What the sources say
- Partly supportedcites Dobber et al., 2020 sourceExperimental — accuracy not yet measured
Dobber et al. support the deepfake half of the claim: the Manipulation Check (p. 11) reports that of the eighty-four participants who found the deepfake (somewhat) noncredible, only twelve actually recognized it as manipulated, and nearly 75 percent of the experimental group had never heard of deepfakes; the Directions for Future Research (p. 19) calls this 'surprising low number of participants who recognized the deepfake as being manipulated' a sign that public awareness should improve. What the source does not support is the generalization to people being poor at recognizing 'false content online': where our copy discusses false news and online disinformation (Deepfakes as a Form of Disinformation, pp. 4-5, on Guess et al. 2018 and Bail et al. 2020) it concerns exposure and attitude effects, not detection skill, and the point that deepfakes 'confuse people about what is real' is explicitly credited there to Vaccari and Chadwick (2020), which the citing sentence lists separately. (LLM, conf=0.78)
- Not checked yetcites Vaccari and Chadwick, 2020 source
- Not checked yetcites Köbis et al., 2021 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“They are becoming easier to produce (Langguth et al., 2021; Gregory, 2023)”
§ On manipulation by emotional AI: UK adults’ views and governance implications > Stage 1: focus groups using design fiction and ContraVision > Deploying design fiction and ContraVision
What the sources say
- Supportedcites Langguth et al., 2021 source
The Introduction contrasts older video manipulation, which required skilled experts and substantial time, with deepfake technology that now manipulates whole videos with limited effort and consumer-grade hardware. It also describes easy accessibility as the fundamental change. (LLM, conf=0.94)
- Not checked yetcites Gregory, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“studies that show that emotionalised online environments influence people’s real-world attitudes on polarization, politics, or belief in false information (Martel et al., 2020; Walter et al., 2020; Van Bavel et al., 2021)”
§ On manipulation by emotional AI: UK adults’ views and governance implications > Discussion > Discussion of qualitative findings
What the sources say
- Partly supportedcites Martel et al., 2020 sourceExperimental — accuracy not yet measured
Martel et al. support the 'belief in false information' half of the disjunctive claim: the Abstract and General Discussion report correlational and causal evidence that reliance on emotion increases belief in fake (but not real) news headlines, and Study 1 that momentary emotionality predicts lower discernment. But the paper does not study 'emotionalised online environments' as such — emotion was induced by written prompts / measured as self-reported mood in headline-rating tasks; the environmental extension appears only as hedged speculation in the General Discussion (practical implications: social media platforms 'may' bias people toward emotion and 'may contribute' to viral fake news). Nothing in our copy supports an effect on polarization or political attitudes; Results and Discussion explicitly state that different emotions do not meaningfully interact with political concordance and that no evidence was found for such an interaction. (LLM, conf=0.72)
- Not checked yetcites Walter et al., 2020 source
- Partly supportedcites Van Bavel et al., 2021 sourceExperimental — accuracy not yet measured
Our copy of Van Bavel et al. (2021) supports the polarization/politics half of the claim: the 'Experimental evidence of polarization' section reports causal field-experiment evidence that Facebook exposure shapes issue (policy preference) and affective polarization, and the 'Message content' section plus Figure 1 report that moral-emotional/out-group language drives diffusion and engagement. However, the emotionalised-content evidence in the source concerns sharing and engagement rather than a demonstrated change in real-world attitudes, and the third element attributed — belief in false information — is not stated anywhere in our copy; misinformation appears only in passing in 'Concluding remarks' ('the significant costs to society of increasing misinformation, polarization, and other conflict'), as a cost, not as an outcome shown to be influenced by emotionalised online environments. The source also hedges its overall position ('social media is unlikely to be the main driver of polarization... often a key facilitator'), whereas the citing sentence asserts influence as fact. (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
40 more claims
Create a free account to see all 124 claims and to verify the 63 references still open.
- Probably not a substantial claim. 59
- It may state two separate claims as one. 40
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Involving AI also facilitate achieving tangible results, fosters a”
From Exploring the Collaborative Co-Creation Process with AI: A Case Study in Novice Music Production (2025) · cites this paper as [15]
Experimental — accuracy not yet measured
Citations
Cites
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847
Age of Surveillance Capitalism – The Fight for a Human Future at the New Frontier of Power.
Kendell, D. (2020) · Brock Education Journal · doi:10.26522/brocked.v29i2.849 · Cited by 2,135
Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019) · Psychological Science in the Public Interest · doi:10.1177/1529100619832930 · Cited by 1,748
(2014) · Research Methods and Statistics for Public and Nonprofit Administrators: A Practical Guide · doi:10.4135/9781544307763.n14 · Cited by 1,675
Review of “Race After Technology: Abolitionist Tools for the New Jim Code”
Jipguep-Akhtar, M. (2020) · Social Forces · doi:10.1093/sf/soz162 · Cited by 1,473
The psychological drivers of misinformation belief and its resistance to correction
Ecker, U. K. H., Lewandowsky, S., Cook, J., Schmid, P., Fazio, L. K., Brashier, N., Kendeou, P., Vraga, E. K., & Amazeen, M. A. (2022) · Nature Reviews Psychology · doi:10.1038/s44159-021-00006-y · Cited by 1,372
Vaccari, C., & Chadwick, A. (2020) · Social Media + Society · doi:10.1177/2056305120903408 · Cited by 978
UNITED NATIONS CONVENTION ON THE RIGHTS OF THE CHILD
(1992) · "International Journal of Law, Policy and the Family" · doi:10.1093/lawfam/6.1.245 · Cited by 930
The SAGE Handbook of Applied Social Research Methods
Bickman, L., & Rog, D. (2009) · Sage eBooks · doi:10.4135/9781483348858 · Cited by 894
Fact-checking: a meta-analysis of what works and for whom
Walter, N., Cohen, J., Holbert, R. L., & Morag, Y. (2020) · Political Communication · doi:10.1080/10584609.2019.1668894 · Cited by 690
Research works printed but not linked: 13
- Ahmed, 2010, The promise of happiness
- Alegre, 2022, Freedom to think: The long struggle to liberate our minds
- Corbu, 2021, Fake news going viral: the mediating effect of negative emotions, Media Literacy and Academic Research, 4, 58
- Davies, 2015, The happiness industry: How the Government & big Business Sold us wellbeing
- Fisher, 2009, Target marketing of subprime loans: racialized consumer fraud and reverse redlining, Brooklyn Law J., 18, 121
- Hagey, 2021, Facebook tried to make its platform a healthier place. It got angrier instead. Wall street journal, September 15 2021
- Hao, 2021, How Facebook got addicted to spreading misinformation. MIT technology review
- Madden, 2017, Privacy, poverty, and big data: a matrix of vulnerabilities for poor Americans. Washington university law review, 053
- McNamee, 2019, Zucked: Waking up to the Facebook catastrophe
- Santini, 2021, When machine behavior targets future voters: the use of social bots to test narratives for political campaigns in Brazil, Int. J. Commun., 15, 1220
- Schneier, 2020, Bots are destroying political discourse as we know it
- Skinner, Walden Two
- Wong, 2019, Revealed: Facebook enables ads to target users interested in 'vaccine controversies
Cited by
Islam, M. B. E., Haseeb, M., Batool, H., Ahtasham, N., & Muhammad, Z. (2024) · Blockchains · doi:10.3390/blockchains2040020 · Cited by 77
Exploring the Collaborative Co-Creation Process with AI: A Case Study in Novice Music Production
Fu, Y., Newman, M., Going, L., Feng, Q., & Lee, J. H. (2025) · Proceedings of the 2025 ACM Designing Interactive Systems Conference · doi:10.1145/3715336.3735829 · Cited by 30
“To further facilitate productive critique while managing interpersonal tensions, platform designers could consider implementing collaborative systems that enable anonymous and equitable feedback .”
Qadri, U. A., Moustafa, A. M. A., & Ghani, M. A. (2025) · Young Consumers Insight and Ideas for Responsible Marketers · doi:10.1108/yc-08-2024-2207 · Cited by 37
Erol, M., Canbeldek Erol, M., Erol, A., & Gök Çolak, F. (2025) · European Journal of Education · doi:10.1111/ejed.70332 · Cited by 10
McStay, A., & Bakir, V. (2025) · Journal of Responsible Technology · doi:10.1016/j.jrt.2025.100126 · Cited by 13
Self-citation
Metadata
- Authors
- Bakir, V., Laffer, A., McStay, A., Miranda, D., & Urquhart, L.
- Affiliations
- Bangor University (GB)
- University of Stirling (GB)
- University of Edinburgh (GB)
- Corresponding author
- Bakir, Vian
- Venue
- Frontiers in Sociology
- Publisher
- Frontiers Media SA
- Published
- 2024-06-07
- Volume
- 9
- Pages
- 1339834-1339834
- ISSN
- 2297-7775 (ISSN-L)
- PMC
- PMC11190365
- OpenAlex
- W4399543694
- Cited by
- 34
- References
- 65
- Citations by year
- 2026: 16 · 2025: 16 · 2024: 2
- FWCI
- 14.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Sociology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (EP/Y009800/1)
- Economic and Social Research Council (ES/T00696X/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- affective computing, manipulation, public attitudes toward AI, AI governance, social media, deepfakes, misinformation, conspiracy theories, design fiction, public trust in technology, UK adults
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: On manipulation by emotional AI: UK adults’ views and governance implications. https://app.cochat.ai/papers/10.3389/fsoc.2024.1339834
- The paper
- Bakir, V., Laffer, A., McStay, A., Miranda, D., & Urquhart, L. (2024). On manipulation by emotional AI: UK adults’ views and governance implications. Frontiers in Sociology. https://doi.org/10.3389/fsoc.2024.1339834
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.