CoChat
Sign Up
PreprintCited by 20Ethics and Social Impacts of AIMisinformation and Its ImpactsDigitalization, Law, and Regulation

Artificial Influence: An Analysis Of AI-Driven Persuasion

Open access
Burtell, Matthew, Woodside, Thomas·2023·arXiv
DOI 10.48550/arxiv.2303.08721arXiv 2303.08721
Open this paper in CoChat — free accountOpen PDF

Abstract

Persuasion is a key aspect of what it means to be human, and is central to business, politics, and other endeavors. Advancements in artificial intelligence (AI) have produced AI systems that are capable of persuading humans to buy products, watch videos, click on search results, and more. Even systems that are not explicitly designed to persuade may do so in practice. In the future, increasingly anthropomorphic AI systems may form ongoing relationships with users, increasing their persuasive power. This paper investigates the uncertain future of persuasive AI systems. We examine ways that AI could qualitatively alter our relationship to and views regarding persuasion by shifting the balance of persuasive power, allowing personalized persuasion to be deployed at scale, powering misinformation campaigns, and changing the way humans can shape their own discourse. We consider ways AI-driven persuasion could differ from human-driven persuasion. We warn that ubiquitous highlypersuasive AI systems could alter our information environment so significantly so as to contribute to a loss of human control of our own future. In response, we examine several potential responses to AI-driven persuasion: prohibition, identification of AI agents, truthful AI, and legal remedies. We conclude that none of these solutions will be airtight, and that individuals and governments will need to take active steps to guard against the most pernicious effects of persuasive AI.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2303.08721

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI systems that can reliably masquerade as people can be used by criminals for fraud, by governments for foreign influence, and by commercial entities to create manipulative experiences for users.

    “AI systems that can reliably masquerade as people can be used by criminals for fraud [1,14,59], by governments for foreign influence [26,35], and by commercial entities to create engaging but ultimately manipulative experiences for users [7,57].”

    From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [7]

    Section 2.1 describes a commercial chatbot optimized for longer conversations and says some users suspect they are interacting with a real person. Section 2.2 proposes that such chatbots could use relationships with users for personalized advertising and influence. The paper supports the commercial-manipulation possibility, but does not establish that these systems can reliably masquerade as people or that the proposed manipulation is already occurring. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • ChatGPT and GPT-4 can produce convincing personalized content and interactions with users.

    “ChatGPT and GPT-4 can produce convincing personalized content and interactions with users [10].”

    From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [10]

    Section 2.1 describes GPT-4 producing plausible reasons and ChatGPT responding to preceding dialogue, sometimes confidently. But the paper does not establish that both models produce personalized content or interactions: section 3.2 discusses personalized persuasion largely as a future possibility, and section 2.1 says the authors do not know whether GPT-4 exhibits the particular conversational behavior described for ChatGPT. Page numbers are unavailable in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • One of the central novel risks of LLMs is their ability to interact dynamically with users.

    “one of the central novel risks of LLMs is their ability to interact dynamically with users [7,23].”

    From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 27

  1. Theo Araujo, Natali Helberger, Sanne Kruikemeier, and Claes H. de Vreese. 2020. In AI we trust? Perceptions about automated decision-making by artificial intelligence. AI & SOCIETY 35, 3 (Sept. 2020), 611-623. https://doi.org/10.1007/ s00146-019-00931-w · DOI · Paper page
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. https://doi.org/10.48550/arXiv.2204.05862 arXiv:2204.05862 [cs]. · DOI · Paper page
  3. Anton Bakhtin, David Wu, Adam Lerer, and Noam Brown. 2021. No-Press Diplomacy from Scratch. In Advances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., 18063-18074. https://proceedings.neurips.cc/ paper/2021/hash/95f2b84de5660ddf45c8a34933a2e66f-Abstract.html
  4. Nick Bostrom. 2014. Superintelligence: Paths, Dangers, Strategies. Oxford Univer- sity Press. Google-Books-ID: 7_H8AwAAQBAJ.
  5. Laura Burbach, Patrick Halbach, Martina Ziefle, and André Calero Valdez. 2020. Opinion Formation on the Internet: The Influence of Personality, Network Struc- ture, and Content on Sharing Messages Online. Frontiers in Artificial Intelligence 3 (2020). https://www.frontiersin.org/articles/10.3389/frai.2020.00045 · DOI
  6. Joseph Carlsmith. 2022. Is Power-Seeking AI an Existential Risk? https://doi. org/10.48550/arXiv.2206.13353 arXiv:2206.13353 [cs]. · DOI · Paper page
  7. Micah Carroll, Dylan Hadfield-Menell, Stuart Russell, and Anca Dragan. 2021. Estimating and Penalizing Preference Shift in Recommender Systems. In Pro- ceedings of the 15th ACM Conference on Recommender Systems (RecSys '21). As- sociation for Computing Machinery, New York, NY, USA, 661-667. https: //doi.org/10.1145/3460231.3478849 · DOI · Paper page
  8. Bryan Casey. 2019. Robot Ipsa Loquitur. SSRN Electronic Journal (2019). https: //doi.org/10.2139/ssrn.3327673 · DOI
  9. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Pref- erences. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2017/hash/ d5e2c0adad503c91f91df240d0cd4e49-Abstract.html
  10. Daniel C. Dennett. 1981. True Believers : The Intentional Strategy and Why It Works. In Scientific Explanation: Papers Based on Herbert Spencer Lectures Given in the University of Oxford, A. F. Heath (Ed.). Clarendon Press, 150-167. https://philarchive.org/rec/DENTB
  11. Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021. Truthful AI: Developing and governing AI that does not lie. https://doi.org/10.48550/arXiv.2110.06674 arXiv:2110.06674 [cs]. · DOI · Paper page
  12. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. 2022. Unsolved Problems in ML Safety. https://doi.org/10.48550/arXiv.2109.13916 arXiv:2109.13916 [cs]. · DOI · Paper page
  13. Dan Hendrycks and Mantas Mazeika. 2022. X-Risk Analysis for AI Research. https://doi.org/10.48550/arXiv.2206.05862 arXiv:2206.05862 [cs]. · DOI
  14. Carl Iver Hovland, Irving Lester Janis, and Harold H. Kelley. 1953. Communica- tion and Persuasion: Psychological Studies of Opinion Change. Greenwood Press. Google-Books-ID: j_FoAAAAIAAJ.
  15. Lawrence Lessig. 1998. The New Chicago School. The Journal of Legal Studies 27, S2 (June 1998), 661-691. https://doi.org/10.1086/468039 Publisher: The University of Chicago Press. · DOI
  16. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measur- ing How Models Mimic Human Falsehoods. http://arxiv.org/abs/2109.07958 arXiv:2109.07958 [cs]. · arXiv
  17. META FUNDAMENTAL AI RESEARCH DIPLOMACY TEAM (FAIR), Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. 2022. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science 378, 6624 (Dec. 2022), 1067-1074. https://doi.org/10. 1126/science.ade9097 Publisher: American Association for the Advancement of Science. · DOI · Paper page
  18. Stuart Russell. 2019. Human Compatible: Artificial Intelligence and the Problem of Control. Penguin. Google-Books-ID: 8vm0DwAAQBAJ.
  19. David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershel- vam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalch- brenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 (Jan. 2016), 484-489. https://doi.org/10.1038/nature16961 Number: 7587 Publisher: Nature Publishing Group. · DOI · Paper page
  20. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2017. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm. https: //doi.org/10.48550/arXiv.1712.01815 arXiv:1712.01815 [cs]. · DOI · Paper page
  21. Marita Skjuve, Asbjørn Følstad, Knut Inge Fostervold, and Petter Bae Brandtzaeg. 2021. My Chatbot Companion -a Study of Human-Chatbot Relationships. In- ternational Journal of Human-Computer Studies 149 (May 2021), 102601. https: //doi.org/10.1016/j.ijhcs.2021.102601 · DOI · Paper page
  22. Rubén Sánchez-Corcuera, Diego Casado-Mansilla, Cruz E. Borges, and Diego Lopez-De-Ipiña. 2019. Persuade Me!: A User-Based Recommendation System Approach. In 2019 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Ad- vanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (Smart- World/SCALCOM/UIC/ATC/CBDCom/IOP/SCI). 1740-1745. https://doi.org/10. 1109/SmartWorld-UIC-ATC-SCALCOM-IOP-SCI.2019.00310 · DOI
  23. Michael Tomasello. 2010. Origins of Human Communication. MIT Press.
  24. Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, An- drew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Has- sabis, Chris Apps, and David Silver. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 7782 (Nov. 2019), 350- 354. https://doi.org/10.1038/s41586-019-1724-z Number: 7782 Publisher: Nature Publishing Group. · DOI · Paper page
  25. Matthew Wills. 2020. Making Sense of the Divine Right of Kings. https: //daily.jstor.org/making-sense-of-the-divine-right-of-kings/
  26. Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2019. The Design and Implementation of XiaoIce, an Empathetic Social Chatbot. https://doi.org/10. 48550/arXiv.1812.08989 arXiv:1812.08989 [cs]. · DOI · Paper page
  27. Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020. The Design and Im- plementation of XiaoIce, an Empathetic Social Chatbot. Computational Linguistics 46, 1 (March 2020), 53-93. https://doi.org/10.1162/ · DOI · Paper page

Cited by

Sorted by citations

Authors
Burtell, M., & Woodside, T.
Affiliations
Yale University (US)
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2303.08721
arXiv
2303.08721
OpenAlex
W4327673264
Cited by
20
References
0
Citations by year
2026: 1 · 2025: 8 · 2024: 10 · 2023: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
misinformation campaigns, personalized persuasion, human-AI relationship, information environment, legal remedies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review