CoChat

On manipulation by emotional AI: UK adults’ views and governance implications

Open access
Bakir, Vian, Laffer, Alexander, McStay, Andrew, Miranda, Diana, Urquhart, Lachlan · 2024 · Frontiers in Sociology

The DOI matches its Crossref record; 66 of the 67 references it cites were found in a registry or an index; 7 of the 72 sources its claims cite were checked for support.

Abstract

With growing commercial, regulatory and scholarly interest in use of Artificial Intelligence (AI) to profile and interact with human emotion (“emotional AI”), attention is turning to its capacity for manipulating people, relating to factors impacting on a person’s decisions and behavior. Given prior social disquiet about AI and profiling technologies, surprisingly little is known on people’s views on the benefits and harms of emotional AI technologies, especially their capacity for manipulation. This matters because regulators of AI (such as in the European Union and the UK) wish to stimulate AI innovation, minimize harms and build public trust in these systems, but to do so they should understand the public’s expectations. Addressing this, we ascertain UK adults’ perspectives on the potential of emotional AI technologies for manipulating people through a two-stage study. Stage One (the qualitative phase) uses design fiction principles to generate adequate understanding and informed discussion in 10 focus groups with diverse participants (n = 46) on how emotional AI technologies may be used in a range of mundane, everyday settings. The focus groups primarily flagged concerns about manipulation in two settings: emotion profiling in social media (involving deepfakes, false information and conspiracy theories), and emotion profiling in child oriented “emotoys” (where the toy responds to the child’s facial and verbal expressions). In both these settings, participants express concerns that emotion profiling covertly exploits users’ cognitive or affective weaknesses and vulnerabilities; additionally, in the social media setting, participants express concerns that emotion profiling damages people’s capacity for rational thought and action. To explore these insights at a larger scale, Stage Two (the quantitative phase), conducts a UK-wide, demographically representative national survey (n = 2,068) on attitudes toward emotional AI. Taking care to avoid leading and dystopian framings of emotional AI, we find that large majorities express concern about the potential for being manipulated through social media and emotoys. In addition to signaling need for civic protections and practical means of ensuring trust in emerging technologies, the research also leads us to provide a policy-friendly subdivision of what is meant by manipulation through emotional AI and related technologies.

Overview

CoChat Summary

TL;DR

A two-stage UK study combining design-fiction focus groups and a national survey finds public concern about manipulation through emotional AI, especially social media deepfakes/bots and child-oriented emotoys. The authors propose a policy-friendly subdivision of manipulation and argue for stronger social protections and regulatory calibration.

Key findings

  • In 10 focus groups with diverse UK participants (n = 46), manipulation concerns clustered in two settings: emotion profiling in social media (deepfakes, false information, conspiracy theories) and child-oriented “emotoys.” Participants worried that emotion profiling covertly exploits cognitive or affective weaknesses and vulnerabilities; in social media, they also worried it damages capacity for rational thought and action.
  • In the national survey (n = 2,068 UK adults, 29 June–1 July 2022), only 19% were comfortable with digital manipulation tools (deepfakes and social media bots) used to promote political and social causes, while 59% were uncomfortable; 72% were concerned it would make it harder to know what messages or people are real or fake, and 73% were concerned about knowing what is genuinely popular versus artificially amplified.
  • For emotoys, 38% were comfortable with the fun prospect, 34% uncomfortable, and 28% did not know; 63% were concerned about what the toy says or advises the child, and 68% were concerned about where emotion data goes and who could access it, e.g. advertisers.
  • Age made the largest difference: younger adults were more comfortable and older adults more concerned. Comfort with digital manipulation tools for political/social causes was 29% among 18–24s and 34% among 25–34s, falling to 5% among 75+; concern about real/fake rose from 63% among 18–24s to 84% among 75+.

Methods

  • Stage 1 qualitative: 10 online focus groups (n = 46) recruited via a professional panel, purposively sampled to include younger adults (18–34, n = 12), older adults (+65, n = 13), people self-identifying as disabled (n = 10), and UK ethnic minorities (n = 11). Design fiction and ContraVision techniques presented emotion profiling in everyday settings; data were hand-coded in NVivo using an adaptive inductive/deductive thematic approach.
  • Stage 2 quantitative: a UK-wide, demographically representative online omnibus survey (n = 2,068 UK adults) conducted by ICM Unlimited across 29 June–1 July 2022. Closed-ended questions covered 10 emotional AI use cases, including digital manipulation tools (deepfakes and bots) and emotoys, with positive and negative statements; the weighted sample was balanced across gender, socio-economic groups, household income, ethnicity, UK regions, and ages 18+.
  • Analysis: Stage 1 themes informed Stage 2 survey construction; survey results are reported as percentages, including age breakdowns. Anonymized datasets are available in the UK Data Service.

Limitations

  • Stated: survey respondents may have read “digital manipulation tools” as more negative than intended, potentially making results more negative; emotoy response options were multi-barrelled, making discomfort harder to interpret. Future surveys should use more singular response options.
  • Stated: the empirical study is limited to the UK context and to people’s views rather than practices; future research should examine actual experiences of, and resistance to, manipulation, and conduct similar studies beyond the UK.
  • Evident: online methods may exclude digitally excluded people (likely older and less digitally literate) and reduce control over distraction; focus groups did not specifically elicit views on manipulation, so that theme emerged inductively; the paper also notes that empirical evidence of actual manipulative impact remains lacking or unclear.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/fsoc.2024.1339834

Cites

66 of 67 references verified1 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Social media use case: binary choice.
Figure 1 Social media use case: binary choice.CC BY · PubMed Central
Figure 2 Social media use case: positive ContraVision.
Figure 2 Social media use case: positive ContraVision.CC BY · PubMed Central
Figure 3 Social media use case: negative ContraVision.
Figure 3 Social media use case: negative ContraVision.CC BY · PubMed Central
Figure 4 Emotoy use case: binary choice.
Figure 4 Emotoy use case: binary choice.CC BY · PubMed Central
Figure 5 Emotoy use case: positive and negative ContraVisions.
Figure 5 Emotoy use case: positive and negative ContraVisions.CC BY · PubMed Central

Verification

Claims the paper makes

6 Valid · 2 Not valid · 40 Not checked · 76 Own results and interpretations

40 more claims

Create a free account to see all 124 claims and to verify the 63 references still open.

See all claims — free account
  • Probably not a substantial claim. 59
  • It may state two separate claims as one. 40

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 67 references

Research works printed but not linked: 13

  1. Ahmed, 2010, The promise of happiness
  2. Alegre, 2022, Freedom to think: The long struggle to liberate our minds
  3. Corbu, 2021, Fake news going viral: the mediating effect of negative emotions, Media Literacy and Academic Research, 4, 58
  4. Davies, 2015, The happiness industry: How the Government & big Business Sold us wellbeing
  5. Fisher, 2009, Target marketing of subprime loans: racialized consumer fraud and reverse redlining, Brooklyn Law J., 18, 121
  6. Hagey, 2021, Facebook tried to make its platform a healthier place. It got angrier instead. Wall street journal, September 15 2021
  7. Hao, 2021, How Facebook got addicted to spreading misinformation. MIT technology review
  8. Madden, 2017, Privacy, poverty, and big data: a matrix of vulnerabilities for poor Americans. Washington university law review, 053
  9. McNamee, 2019, Zucked: Waking up to the Facebook catastrophe
  10. Santini, 2021, When machine behavior targets future voters: the use of social bots to test narratives for political campaigns in Brazil, Int. J. Commun., 15, 1220
  11. Schneier, 2020, Bots are destroying political discourse as we know it
  12. Skinner, Walden Two
  13. Wong, 2019, Revealed: Facebook enables ads to target users interested in 'vaccine controversies

Cited by

See all 35

Metadata

Authors
Bakir, V., Laffer, A., McStay, A., Miranda, D., & Urquhart, L.
Affiliations
  • Bangor University (GB)
  • University of Stirling (GB)
  • University of Edinburgh (GB)
Corresponding author
Bakir, Vian
Venue
Frontiers in Sociology
Published
2024-06-07
Volume
9
Pages
1339834-1339834
ISSN
2297-7775 (ISSN-L)
OpenAlex
W4399543694
Cited by
34
References
65
Citations by year
2026: 16 · 2025: 16 · 2024: 2
FWCI
14.09
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Sociology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
Funders
  • Engineering and Physical Sciences Research Council (EP/Y009800/1)
  • Economic and Social Research Council (ES/T00696X/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
affective computing, manipulation, public attitudes toward AI, AI governance, social media, deepfakes, misinformation, conspiracy theories, design fiction, public trust in technology, UK adults

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: On manipulation by emotional AI: UK adults’ views and governance implications. https://app.cochat.ai/papers/10.3389/fsoc.2024.1339834
The paper
Bakir, V., Laffer, A., McStay, A., Miranda, D., & Urquhart, L. (2024). On manipulation by emotional AI: UK adults’ views and governance implications. Frontiers in Sociology. https://doi.org/10.3389/fsoc.2024.1339834

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.