CoChat

Characterizing Manipulation from AI Systems

Open access
Carroll, Micah, Chan, Alan, Ashton, Henry, Krueger, David · 2023 · Equity and Access in Algorithms, Mechanisms, and Optimization
Cited by 60

The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.

Abstract

Manipulation is a concern in many domains, such as social media, advertising, and chatbots. As AI systems mediate more of our digital interactions, it is important to understand the degree to which AI systems might manipulate humans without the intent of the system designers. Our work clarifies challenges in defining and measuring this kind of manipulation from AI systems. Firstly, we build upon prior literature on manipulation and characterize the space of possible notions of manipulation, which we find to depend upon the concepts of incentives, intent, covertness, and harm. We review proposals on how to operationalize each concept and we outline challenges in including each concept in a definition of manipulation. Second, we discuss the connections between manipulation and related concepts, such as deception and coercion. We then analyze how our characterization of manipulation applies to recommender systems and language models, and give a brief overview of the regulation of manipulation in other domains. While some progress has been made in defining and measuring manipulation from AI systems, many gaps remain. In the absence of a consensus definition and reliable tools for measurement, we cannot rule out the possibility that AI systems learn to manipulate humans without the intent of the system designers. Manipulation could pose a significant threat to human autonomy and precautionary actions to mitigate it are likely warranted.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3617694.3623226

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 44 references

Research works printed but not linked: 127

  1. Gediminas Adomavicius , Jesse C. Bockstedt , Shawn P. Curley , and Jingjing Zhang . 2013. Do Recommender Systems Manipulate Consumer Preferences? A Study of Anchoring Effects. Information Systems Research 24, 4 (Dec . 2013 ), 956–975. https://doi.org/10.1287/isre.2013.0497 Publisher : INFORMS. 10.1287/isre.2013.0497 Gediminas Adomavicius, Jesse C. Bockstedt, Shawn P. Curley, and Jingjing Zhang. 2013. Do Recommender Systems Manipulate Consumer Preferences? A Study of Anchoring Effects. Information Systems Research 24, 4 (Dec. 2013), 956–975. https://doi.org/10.1287/isre.2013.0497 Publisher: INFORMS. · DOI · Paper page
  2. 10.1287/isre.2017.0703 · DOI
  3. M. Mehdi Afsar , Trafford Crump , and Behrouz Far . 2021. Reinforcement learning based recommender systems: A survey. arXiv:2101.06286 [cs] (Jan . 2021 ). http://arxiv.org/abs/2101.06286 arXiv:2101.06286. M. Mehdi Afsar, Trafford Crump, and Behrouz Far. 2021. Reinforcement learning based recommender systems: A survey. arXiv:2101.06286 [cs] (Jan. 2021). http://arxiv.org/abs/2101.06286 arXiv:2101.06286. · arXiv
  4. 10.1257/aer.20210867 · DOI · Paper page
  5. Muhammad Ashfaq , Jiang Yun , Shubin Yu , and Sandra Maria Correia Loureiro . 2020. I , Chatbot : Modeling the determinants of users’ satisfaction and continuance intention of AI-powered service agents. Telematics and Informatics 54 ( Nov. 2020 ), 101473. https://doi.org/10.1016/j.tele.2020.101473 10.1016/j.tele.2020.101473 Muhammad Ashfaq, Jiang Yun, Shubin Yu, and Sandra Maria Correia Loureiro. 2020. I, Chatbot: Modeling the determinants of users’ satisfaction and continuance intention of AI-powered service agents. Telematics and Informatics 54 (Nov. 2020), 101473. https://doi.org/10.1016/j.tele.2020.101473 · DOI · Paper page
  6. Hal Ashton . 2022. Definitions of Intent Suitable for Algorithms. Artificial Intelligence and Law (July 2022 ). https://doi.org/10.1007/s10506-022-09322-x 10.1007/s10506-022-09322-x Hal Ashton. 2022. Definitions of Intent Suitable for Algorithms. Artificial Intelligence and Law (July 2022). https://doi.org/10.1007/s10506-022-09322-x · DOI
  7. Hal Ashton and Matija Franklin . 2022 . The Problem of Behaviour and Preference Manipulation in AI Systems. In The AAAI-22 Workshop on Artificial Intelligence Safety (SafeAI 2022). Hal Ashton and Matija Franklin. 2022. The Problem of Behaviour and Preference Manipulation in AI Systems. In The AAAI-22 Workshop on Artificial Intelligence Safety (SafeAI 2022).
  8. Hal Ashton and Matija Franklin. 2022. Solutions to Preference Manipulation in Recommender Systems Require Knowledge of Meta-Preferences. http://arxiv.org/abs/2209.11801 arXiv:2209.11801 [cs]. Hal Ashton and Matija Franklin. 2022. Solutions to Preference Manipulation in Recommender Systems Require Knowledge of Meta-Preferences. http://arxiv.org/abs/2209.11801 arXiv:2209.11801 [cs]. · arXiv
  9. Association for Computing Machinery (ACM). 2019. "Reinforcement Learning for Recommender Systems: A Case Study on Youtube " by Minmin Chen. https://www.youtube.com/watch?v=HEqQ2_1XRTs Association for Computing Machinery (ACM). 2019. "Reinforcement Learning for Recommender Systems: A Case Study on Youtube " by Minmin Chen. https://www.youtube.com/watch?v=HEqQ2_1XRTs
  10. Alessio Azzutti . 2022. AI-driven Market Manipulation and Limits of the EU Law Enforcement Regime to Credible Deterrence. Computer Law & Security review 45 (Jan . 2022 ). https://doi.org/10.2139/ssrn.4026468 10.2139/ssrn.4026468 Alessio Azzutti. 2022. AI-driven Market Manipulation and Limits of the EU Law Enforcement Regime to Credible Deterrence. Computer Law & Security review 45 (Jan. 2022). https://doi.org/10.2139/ssrn.4026468 · doi:10.2139/ssrn.4026468
  11. Alessio Azzutti , Wolf-Georg Ringe , and H. Siegfried Stiehl . 2021. Machine Learning, Market Manipulation and Collusion on Capital Markets: Why the . University of Pennsylvania journal of international law 43, 1 ( 2021 ). https://doi.org/10.2139/ssrn.3788872 10.2139/ssrn.3788872 Alessio Azzutti, Wolf-Georg Ringe, and H. Siegfried Stiehl. 2021. Machine Learning, Market Manipulation and Collusion on Capital Markets: Why the. University of Pennsylvania journal of international law 43, 1 (2021). https://doi.org/10.2139/ssrn.3788872 · doi:10.2139/ssrn.3788872
  12. Hui Bai. 2023. Artificial Intelligence Can Persuade Humans. (2023). Hui Bai. 2023. Artificial Intelligence Can Persuade Humans. (2023). · doi:10.21203/rs.3.rs-3238396/v1
  13. Marcia Baron . 2014. The Mens Rea and Moral Status of Manipulation . In Manipulation: Theory and Practice , Christian Coons and Michael Weber (Eds.). Oxford University Press , 0. https://doi.org/10.1093/acprof:oso/9780 1993 38207.003.0005 10.1093/acprof:oso Marcia Baron. 2014. The Mens Rea and Moral Status of Manipulation. In Manipulation: Theory and Practice, Christian Coons and Michael Weber (Eds.). Oxford University Press, 0. https://doi.org/10.1093/acprof:oso/9780199338207.003.0005 · doi:10.1093/acprof:oso/9780
  14. 10.1145/3313831.3376518 · DOI
  15. Yavar Bathaee . 2018 . The Artificial Intelligence Black Box and the Failure of Intent and Causation . Harvard Journal of Law and Technology 31 , 2 (2018), 890 – 938 . Yavar Bathaee. 2018. The Artificial Intelligence Black Box and the Failure of Intent and Causation. Harvard Journal of Law and Technology 31, 2 (2018), 890–938.
  16. Omer Ben-Porat and Moshe Tennenholtz . 2018. A Game-Theoretic Approach to Recommendation Systems with Strategic Content Providers . In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Vol. 31. Curran Associates , Inc .https://proceedings.neurips.cc/paper/ 2018 /file/a9a1d5317a33ae8cef33961c34144f84-Paper.pdf Omer Ben-Porat and Moshe Tennenholtz. 2018. A Game-Theoretic Approach to Recommendation Systems with Strategic Content Providers. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Vol. 31. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2018/file/a9a1d5317a33ae8cef33961c34144f84-Paper.pdf
  17. 10.1080/15265161.2011.634481 · DOI · Paper page
  18. Rishi Bommasani Drew A. Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S. Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill Erik Brynjolfsson Shyamal Buch Dallas Card Rodrigo Castellon Niladri Chatterji Annie Chen Kathleen Creel Jared Quincy Davis Dora Demszky Chris Donahue Moussa Doumbouya Esin Durmus Stefano Ermon John Etchemendy Kawin Ethayarajh Li Fei-Fei Chelsea Finn Trevor Gale Lauren Gillespie Karan Goel Noah Goodman Shelby Grossman Neel Guha Tatsunori Hashimoto Peter Henderson John Hewitt Daniel E. Ho Jenny Hong Kyle Hsu Jing Huang Thomas Icard Saahil Jain Dan Jurafsky Pratyusha Kalluri Siddharth Karamcheti Geoff Keeling Fereshte Khani Omar Khattab Pang Wei Koh Mark Krass Ranjay Krishna Rohith Kuditipudi Ananya Kumar Faisal Ladhak Mina Lee Tony Lee Jure Leskovec Isabelle Levent Xiang Lisa Li Xuechen Li Tengyu Ma Ali Malik Christopher D. Manning Suvir Mirchandani Eric Mitchell Zanele Munyikwa Suraj Nair Avanika Narayan Deepak Narayanan Ben Newman Allen Nie Juan Carlos Niebles Hamed Nilforoshan Julian Nyarko Giray Ogut Laurel Orr Isabel Papadimitriou Joon Sung Park Chris Piech Eva Portelance Christopher Potts Aditi Raghunathan Rob Reich Hongyu Ren Frieda Rong Yusuf Roohani Camilo Ruiz Jack Ryan Christopher Ré Dorsa Sadigh Shiori Sagawa Keshav Santhanam Andy Shih Krishnan Srinivasan Alex Tamkin Rohan Taori Armin W. Thomas Florian Tramèr Rose E. Wang William Wang Bohan Wu Jiajun Wu Yuhuai Wu Sang Michael Xie Michihiro Yasunaga Jiaxuan You Matei Zaharia Michael Zhang Tianyi Zhang Xikun Zhang Yuhui Zhang Lucia Zheng Kaitlyn Zhou and Percy Liang. 2022. On the Opportunities and Risks of Foundation Models. https://doi.org/10.48550/arXiv.2108.07258 arXiv:2108.07258 [cs]. · DOI · Paper page
  19. Harriet Braiker . 2003. Who’s Pulling Your Strings?: How to Break the Cycle of Manipulation and Regain Control of Your Life: How to Break the Cycle of Manipulation and Regain Control of Your Life . McGraw Hill Professional. Google-Books-ID : dGwgiQvyeq0C. Harriet Braiker. 2003. Who’s Pulling Your Strings?: How to Break the Cycle of Manipulation and Regain Control of Your Life: How to Break the Cycle of Manipulation and Regain Control of Your Life. McGraw Hill Professional. Google-Books-ID: dGwgiQvyeq0C.
  20. Collin Burns , Haotian Ye , Dan Klein , and Jacob Steinhardt . 2023 . Discovering Latent Knowledge in Language Models Without Supervision. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=ETKGuby0hcs Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering Latent Knowledge in Language Models Without Supervision. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=ETKGuby0hcs
  21. Qingpeng Cai Shuchang Liu Xueliang Wang Tianyou Zuo Wentao Xie Bin Yang Dong Zheng Peng Jiang and Kun Gai. 2023. Reinforcing User Retention in a Billion Scale Short Video Recommender System. http://arxiv.org/abs/2302.01724 arXiv:2302.01724 [cs]. Qingpeng Cai Shuchang Liu Xueliang Wang Tianyou Zuo Wentao Xie Bin Yang Dong Zheng Peng Jiang and Kun Gai. 2023. Reinforcing User Retention in a Billion Scale Short Video Recommender System. http://arxiv.org/abs/2302.01724 arXiv:2302.01724 [cs]. · arXiv
  22. 10.1093/oso/9780190639488.001.0001 · DOI · Paper page
  23. M. Ryan Calo . 2014 . Digital Market Manipulation . George Washington Law Review 82 , 4 (2014), 996 – 1051 . https://doi.org/10.2139/ssrn.2309703 10.2139/ssrn.2309703 M. Ryan Calo. 2014. Digital Market Manipulation. George Washington Law Review 82, 4 (2014), 996–1051. https://doi.org/10.2139/ssrn.2309703 · DOI · Paper page
  24. Ryan Carey , Eric Langlois , Tom Everitt , and Shane Legg . 2020. The Incentives that Shape Behaviour. arXiv:2001.07118 [cs] (Jan . 2020 ). http://arxiv.org/abs/2001.07118 arXiv:2001.07118. Ryan Carey, Eric Langlois, Tom Everitt, and Shane Legg. 2020. The Incentives that Shape Behaviour. arXiv:2001.07118 [cs] (Jan. 2020). http://arxiv.org/abs/2001.07118 arXiv:2001.07118. · arXiv
  25. Thomas L. Carson . 2010. Lying and Deception: Theory and Practice . Oxford University Press , Oxford ; New York. OCLC: ocn464581525. Thomas L. Carson. 2010. Lying and Deception: Theory and Practice. Oxford University Press, Oxford ; New York. OCLC: ocn464581525.
  26. Alan Chan Rebecca Salganik Alva Markelius Chris Pang Nitarshan Rajkumar Dmitrii Krasheninnikov Lauro Langosco Zhonghao He Yawen Duan Micah Carroll Michelle Lin Alex Mayhew Katherine Collins Maryam Molamohammadi John Burden Wanru Zhao Shalaleh Rismani Konstantinos Voudouris Umang Bhatt Adrian Weller David Krueger and Tegan Maharaj. 2023. Harms from Increasingly Agentic Algorithmic Systems. https://doi.org/10.48550/arXiv.2302.10329 arXiv:2302.10329 [cs]. · DOI · Paper page
  27. Allison J. B. Chaney. 2021. Recommendation System Simulations: A Discussion of Two Key Challenges. https://doi.org/10.48550/arXiv.2109.02475 10.48550/arXiv.2109.02475 · arXiv · doi:10.48550/arxiv.2109.02475
  28. Allison J. B. Chaney. 2021. Recommendation System Simulations: A Discussion of Two Key Challenges. https://doi.org/10.48550/arXiv.2109.02475 · arXiv · doi:10.48550/arxiv.2109.02475
  29. 10.1145/3240323.3240370 · DOI · Paper page
  30. Minmin Chen , Alex Beutel , Paul Covington , Sagar Jain , Francois Belletti , and Ed Chi . 2020 . Top-K Off-Policy Correction for a REINFORCE Recommender System . arXiv:1812.02353 [cs, stat] (Nov. 2020). http://arxiv.org/abs/1812.02353 arXiv:1812.02353. Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed Chi. 2020. Top-K Off-Policy Correction for a REINFORCE Recommender System. arXiv:1812.02353 [cs, stat] (Nov. 2020). http://arxiv.org/abs/1812.02353 arXiv:1812.02353. · arXiv
  31. Mark Chen , Jerry Tworek , Heewoo Jun , Qiming Yuan , Henrique Ponde de Oliveira Pinto , Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and others. 2021 . Evaluating Large Language Models Trained on Code . arXiv preprint arXiv:2107.03374 (2021). Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and others. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
  32. Thomas Christiano . 2022. Algorithms, Manipulation, and Democracy. Canadian Journal of Philosophy 52, 1 (Jan . 2022 ), 109–124. https://doi.org/10.1017/can.2021.29 Publisher : Cambridge University Press . 10.1017/can.2021.29 Thomas Christiano. 2022. Algorithms, Manipulation, and Democracy. Canadian Journal of Philosophy 52, 1 (Jan. 2022), 109–124. https://doi.org/10.1017/can.2021.29 Publisher: Cambridge University Press. · DOI
  33. Leon Ciechanowski , Aleksandra Przegalinska , Mikolaj Magnuski , and Peter Gloor . 2019 . In the Shades of the Uncanny Valley: An Experimental Study of Human–Chatbot Interaction. Future Generation Computer Systems 92 (March 2019), 539–548. https://doi.org/10.1016/j.future.2018.01.055 10.1016/j.future.2018.01.055 Leon Ciechanowski, Aleksandra Przegalinska, Mikolaj Magnuski, and Peter Gloor. 2019. In the Shades of the Uncanny Valley: An Experimental Study of Human–Chatbot Interaction. Future Generation Computer Systems 92 (March 2019), 539–548. https://doi.org/10.1016/j.future.2018.01.055 · DOI · Paper page
  34. 10.1017/beq.2015.41 · doi:10.1017/beq.2015.41
  35. 10.1145/3523227.3546778 · DOI
  36. Allan Dafoe Edward Hughes Yoram Bachrach Tantum Collins Kevin R. McKee Joel Z. Leibo Kate Larson and Thore Graepel. 2020. Open Problems in Cooperative AI. https://doi.org/10.48550/arXiv.2012.08630 arXiv:2012.08630 [cs]. · DOI · Paper page
  37. Shayan Doroudi , Vincent Aleven , and Emma Brunskill . 2019. Where’s the Reward?International Journal of Artificial Intelligence in Education 29, 4 (Dec . 2019 ), 568–620. https://doi.org/10.1007/s40593-019-00187-x 10.1007/s40593-019-00187-x Shayan Doroudi, Vincent Aleven, and Emma Brunskill. 2019. Where’s the Reward?International Journal of Artificial Intelligence in Education 29, 4 (Dec. 2019), 568–620. https://doi.org/10.1007/s40593-019-00187-x · DOI · Paper page
  38. 10.1073/pnas.1419828112 · DOI · Paper page
  39. Charles Evans and Atoosa Kasirzadeh . 2021. User Tampering in Reinforcement Learning Recommender Systems. arXiv:2109.04083 [cs] (Sept . 2021 ). http://arxiv.org/abs/2109.04083 arXiv:2109.04083. Charles Evans and Atoosa Kasirzadeh. 2021. User Tampering in Reinforcement Learning Recommender Systems. arXiv:2109.04083 [cs] (Sept. 2021). http://arxiv.org/abs/2109.04083 arXiv:2109.04083. · arXiv
  40. Owain Evans , Owen Cotton-Barratt , Lukas Finnveden , Adam Bales , Avital Balwit , Peter Wills , Luca Righetti , and William Saunders . 2021 . Truthful AI: Developing and Governing AI that does not Lie. arXiv:2110.06674 [cs] (Oct. 2021). http://arxiv.org/abs/2110.06674 arXiv:2110.06674. Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021. Truthful AI: Developing and Governing AI that does not Lie. arXiv:2110.06674 [cs] (Oct. 2021). http://arxiv.org/abs/2110.06674 arXiv:2110.06674. · arXiv
  41. Tom Everitt , Ryan Carey , Eric Langlois , Pedro A. Ortega , and Shane Legg . 2021 . Agent Incentives: A Causal Perspective. arXiv:2102.01685. Tom Everitt, Ryan Carey, Eric Langlois, Pedro A. Ortega, and Shane Legg. 2021. Agent Incentives: A Causal Perspective. arXiv:2102.01685. · arXiv
  42. Sebastian Farquhar , Ryan Carey , and Tom Everitt . 2022. Path-Specific Objectives for Safer Agent Incentives. arXiv:2204.10018 [cs, stat] (April 2022 ). http://arxiv.org/abs/2204.10018 arXiv:2204.10018. Sebastian Farquhar, Ryan Carey, and Tom Everitt. 2022. Path-Specific Objectives for Safer Agent Incentives. arXiv:2204.10018 [cs, stat] (April 2022). http://arxiv.org/abs/2204.10018 arXiv:2204.10018. · arXiv
  43. 10.1145/286498.286852 · DOI
  44. Brian J Fogg . 2003. Persuasive Technology . Elsevier . https://doi.org/10.1016/B978-1-55860-643-2.X5000-8 10.1016/B978-1-55860-643-2.X5000-8 Brian J Fogg. 2003. Persuasive Technology. Elsevier. https://doi.org/10.1016/B978-1-55860-643-2.X5000-8 · DOI
  45. Matija Franklin , Hal Ashton , Rebecca Gorman , and Stuart Armstrong . 2022. Recognising the Importance of Preference Change: A Call for a Coordinated Multidisciplinary Research Effort in the Age of AI. arXiv:2203.10525 [cs] (March 2022 ). http://arxiv.org/abs/2203.10525 arXiv:2203.10525. Matija Franklin, Hal Ashton, Rebecca Gorman, and Stuart Armstrong. 2022. Recognising the Importance of Preference Change: A Call for a Coordinated Multidisciplinary Research Effort in the Age of AI. arXiv:2203.10525 [cs] (March 2022). http://arxiv.org/abs/2203.10525 arXiv:2203.10525. · arXiv
  46. Jason Gauci , Edoardo Conti , Yitao Liang , Kittipat Virochsiri , Yuchen He , Zachary Kaden , Vivek Narayanan , Xiaohui Ye , Zhengxing Chen , and Scott Fujimoto . 2019 . Horizon: Facebook’s Open Source Applied Reinforcement Learning Platform. arXiv:1811.00260 [cs, stat] (Sept. 2019). http://arxiv.org/abs/1811.00260 arXiv:1811.00260. Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Yuchen He, Zachary Kaden, Vivek Narayanan, Xiaohui Ye, Zhengxing Chen, and Scott Fujimoto. 2019. Horizon: Facebook’s Open Source Applied Reinforcement Learning Platform. arXiv:1811.00260 [cs, stat] (Sept. 2019). http://arxiv.org/abs/1811.00260 arXiv:1811.00260. · arXiv
  47. Charles Goodhart . 1975. Problems of Monetary Management: the UK Experience in Papers in Monetary Economics. Monetary Economics 1 ( 1975 ). Charles Goodhart. 1975. Problems of Monetary Management: the UK Experience in Papers in Monetary Economics. Monetary Economics 1 (1975).
  48. 10.1145/3173574.3174108 · DOI · Paper page
  49. Lewis D. Griffin Bennett Kleinberg Maximilian Mozes Kimberly T. Mai Maria Vau Matthew Caldwell and Augustine Marvor-Parker. 2023. Susceptibility to Influence of Large Language Models. http://arxiv.org/abs/2303.06074 arXiv:2303.06074 [cs]. Lewis D. Griffin Bennett Kleinberg Maximilian Mozes Kimberly T. Mai Maria Vau Matthew Caldwell and Augustine Marvor-Parker. 2023. Susceptibility to Influence of Large Language Models. http://arxiv.org/abs/2303.06074 arXiv:2303.06074 [cs]. · arXiv
  50. Till Grüne-Yanoff and Sven Ove Hansson (Eds.). 2009. Preference change: approaches from philosophy economics and psychology. Number v. 42 in Theory and decision library. Series A Philosophy and methodology of the social sciences. Springer Dordrecht ; London. OCLC: ocn321018474. Till Grüne-Yanoff and Sven Ove Hansson (Eds.). 2009. Preference change: approaches from philosophy economics and psychology. Number v. 42 in Theory and decision library. Series A Philosophy and methodology of the social sciences. Springer Dordrecht ; London. OCLC: ocn321018474.
  51. 10.1177/1461444819876944 · DOI · Paper page
  52. Joseph Y. Halpern and Max Kleiman-Weiner . 2018 . Towards Formal Definitions of Blameworthiness, Intention , and Moral responsibility. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence(AAAI’18/IAAI’18/EAAI’18). AAAI Press , New Orleans, Louisiana, USA, 1853–1860. Joseph Y. Halpern and Max Kleiman-Weiner. 2018. Towards Formal Definitions of Blameworthiness, Intention, and Moral responsibility. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence(AAAI’18/IAAI’18/EAAI’18). AAAI Press, New Orleans, Louisiana, USA, 1853–1860.
  53. 10.1145/3437963.3441775 · DOI
  54. Moritz Hardt , Nimrod Megiddo , Christos Papadimitriou , and Mary Wootters . 2016 . Strategic Classification. In Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science(ITCS ’16) . Association for Computing Machinery, New York, NY, USA, 111–122. Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. 2016. Strategic Classification. In Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science(ITCS ’16). Association for Computing Machinery, New York, NY, USA, 111–122.
  55. 10.1609/aaai.v36i5.20509 · doi:10.1609/aaai.v36i5.20509
  56. 10.1613/jair.202 · DOI · Paper page
  57. 10.1146/annurev-statistics-031017-100630 · DOI · Paper page
  58. 10.1177/0956797611436349 · DOI · Paper page
  59. Joey Hong Anca Dragan and Sergey Levine. 2023. Learning to Influence Human Behavior with Offline Reinforcement Learning. https://doi.org/10.48550/arXiv.2303.02265 arXiv:2303.02265 [cs]. 10.48550/arXiv.2303.02265 · arXiv · doi:10.48550/arxiv.2303.02265
  60. Joey Hong Anca Dragan and Sergey Levine. 2023. Learning to Influence Human Behavior with Offline Reinforcement Learning. https://doi.org/10.48550/arXiv.2303.02265 arXiv:2303.02265 [cs]. · arXiv · doi:10.48550/arxiv.2303.02265
  61. 10.1145/3287560.3287597 · DOI
  62. (Robin) Hui Huang . 2009. Redefining Market Manipulation in Australia: The Role of an Implied Intent Element. Companies and Securities Law Journal 27 (April 2009 ). https://papers.ssrn.com/abstract=1376209 (Robin) Hui Huang. 2009. Redefining Market Manipulation in Australia: The Role of an Implied Intent Element. Companies and Securities Law Journal 27 (April 2009). https://papers.ssrn.com/abstract=1376209
  63. Meena Jagadeesan , Celestine Mendler-Dünner , and Moritz Hardt . 2021. Alternative Microfoundations for Strategic Classification . In ICML . http://arxiv.org/abs/2106.12705 arXiv:2106.12705. Meena Jagadeesan, Celestine Mendler-Dünner, and Moritz Hardt. 2021. Alternative Microfoundations for Strategic Classification. In ICML. http://arxiv.org/abs/2106.12705 arXiv:2106.12705. · arXiv
  64. Maurice Jakesch Advait Bhat Daniel Buschek Lior Zalmanson and Mor Naaman. 2023. Co-Writing with Opinionated Language Models Affects Users' Views. https://doi.org/10.1145/3544548.3581196 arXiv:2302.00560 [cs]. · DOI · Paper page
  65. Mathias Jesse and Dietmar Jannach . 2021. Digital Nudging with Recommender Systems: Survey and Future Directions. Computers in Human Behavior Reports 3 (Jan . 2021 ), 100052. https://doi.org/10.1016/j.chbr.2020.100052 10.1016/j.chbr.2020.100052 Mathias Jesse and Dietmar Jannach. 2021. Digital Nudging with Recommender Systems: Survey and Future Directions. Computers in Human Behavior Reports 3 (Jan. 2021), 100052. https://doi.org/10.1016/j.chbr.2020.100052 · DOI · Paper page
  66. Saurav Kadavath Tom Conerly Amanda Askell Tom Henighan Dawn Drain Ethan Perez Nicholas Schiefer Zac Hatfield-Dodds Nova DasSarma Eli Tran-Johnson Scott Johnston Sheer El-Showk Andy Jones Nelson Elhage Tristan Hume Anna Chen Yuntao Bai Sam Bowman Stanislav Fort Deep Ganguli Danny Hernandez Josh Jacobson Jackson Kernion Shauna Kravec Liane Lovitt Kamal Ndousse Catherine Olsson Sam Ringer Dario Amodei Tom Brown Jack Clark Nicholas Joseph Ben Mann Sam McCandlish Chris Olah and Jared Kaplan. 2022. Language Models (Mostly) Know What They Know. https://doi.org/10.48550/arXiv.2207.05221 arXiv:2207.05221 [cs]. · DOI · Paper page
  67. Jean Kaddour Aengus Lynch Qi Liu Matt J. Kusner and Ricardo Silva. 2022. Causal Machine Learning: A Survey and Open Problems. https://doi.org/10.48550/arXiv.2206.15475 arXiv:2206.15475 [cs stat]. 10.48550/arXiv.2206.15475 · DOI
  68. Jean Kaddour Aengus Lynch Qi Liu Matt J. Kusner and Ricardo Silva. 2022. Causal Machine Learning: A Survey and Open Problems. https://doi.org/10.48550/arXiv.2206.15475 arXiv:2206.15475 [cs stat]. · DOI
  69. 10.1257/aer.101.6.2590 · DOI · Paper page
  70. Timotheus Kampik , Juan Carlos Nieves , and Helena Lindgren . 2018 . Coercion and Deception in Persuasive Technologies. In 20th International Trust Workshop (co-located with AAMAS/IJCAI/ECAI/ICML 2018 ), Stockholm, Sweden , 14 July, 2018. CEUR-WS, 38–49. Timotheus Kampik, Juan Carlos Nieves, and Helena Lindgren. 2018. Coercion and Deception in Persuasive Technologies. In 20th International Trust Workshop (co-located with AAMAS/IJCAI/ECAI/ICML 2018), Stockholm, Sweden, 14 July, 2018. CEUR-WS, 38–49.
  71. 10.1504/IJIMA.2013.056586 · DOI
  72. Zachary Kenton Ramana Kumar Sebastian Farquhar Jonathan Richens Matt MacDermott and Tom Everitt. 2022. Discovering Agents. https://doi.org/10.48550/arXiv.2208.08345 arXiv:2208.08345 [cs]. · DOI · Paper page
  73. Poruz Khambatta Shwetha Mariadassou Joshua Morris and S Christian Wheeler. 2022. Targeting Recommendation Algorithms to Ideal Preferences Makes Users Better Off. (2022). Poruz Khambatta Shwetha Mariadassou Joshua Morris and S Christian Wheeler. 2022. Targeting Recommendation Algorithms to Ideal Preferences Makes Users Better Off. (2022). · doi:10.5465/ambpp.2022.14632abstract
  74. 10.1145/3328526.3329584 · DOI
  75. Michal Kosinski. 2023. Theory of Mind May Have Spontaneously Emerged in Large Language Models. http://arxiv.org/abs/2302.02083 arXiv:2302.02083 [cs]. Michal Kosinski. 2023. Theory of Mind May Have Spontaneously Emerged in Large Language Models. http://arxiv.org/abs/2302.02083 arXiv:2302.02083 [cs]. · arXiv
  76. 10.1086/676597 · DOI
  77. Arto Laitinen and Otto Sahlgren . 2021. AI Systems and Respect for Human Autonomy. Frontiers in Artificial Intelligence 4 ( 2021 ). https://www.frontiersin.org/articles/10.3389/frai.2021.705164 Arto Laitinen and Otto Sahlgren. 2021. AI Systems and Respect for Human Autonomy. Frontiers in Artificial Intelligence 4 (2021). https://www.frontiersin.org/articles/10.3389/frai.2021.705164 · DOI · Paper page
  78. Lauro Langosco Di Langosco , Jack Koch , Lee D Sharkey , Jacob Pfau , and David Krueger . 2022 . Goal Misgeneralization in Deep Reinforcement Learning . In Proceedings of the 39th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 162) , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 1 2004–12019. https://proceedings.mlr.press/v162/langosco22a.html Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. 2022. Goal Misgeneralization in Deep Reinforcement Learning. In Proceedings of the 39th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 12004–12019. https://proceedings.mlr.press/v162/langosco22a.html
  79. Kenneth Li , Aspen K. Hopkins , David Bau , Fernanda Viégas , Hanspeter Pfister , and Martin Wattenberg . 2023 . Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task . In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=DeG07_TcZvT Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=DeG07_TcZvT
  80. Stephanie Lin , Jacob Hilton , and Owain Evans . 2021. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs] (Sept . 2021 ). http://arxiv.org/abs/2109.07958 arXiv:2109.07958. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs] (Sept. 2021). http://arxiv.org/abs/2109.07958 arXiv:2109.07958. · arXiv
  81. Tom C. W. Lin . 2017. The New Market Manipulation. Emory Law Journal 66 (July 2017 ). https://papers.ssrn.com/abstract=2996896 Tom C. W. Lin. 2017. The New Market Manipulation. Emory Law Journal 66 (July 2017). https://papers.ssrn.com/abstract=2996896
  82. David Lindner , Kyle Matoba , and Alexander Meulemans . 2021. Challenges for Using Impact Regularizers to Avoid Negative Side Effects. arXiv:2101.12509 [cs] (Feb . 2021 ). http://arxiv.org/abs/2101.12509 arXiv:2101.12509. David Lindner, Kyle Matoba, and Alexander Meulemans. 2021. Challenges for Using Impact Regularizers to Avoid Negative Side Effects. arXiv:2101.12509 [cs] (Feb. 2021). http://arxiv.org/abs/2101.12509 arXiv:2101.12509. · arXiv
  83. 10.1093/jla/laaa006 · DOI · Paper page
  84. Masoud Mansoury , Himan Abdollahpouri , Mykola Pechenizkiy , Bamshad Mobasher , and Robin Burke . 2020. Feedback Loop and Bias Amplification in Recommender Systems. arXiv:2007.13019 [cs] (July 2020 ). http://arxiv.org/abs/2007.13019 arXiv:2007.13019. Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. 2020. Feedback Loop and Bias Amplification in Recommender Systems. arXiv:2007.13019 [cs] (July 2020). http://arxiv.org/abs/2007.13019 arXiv:2007.13019. · arXiv
  85. 10.1016/j.pragma.2004.12.007 · DOI · Paper page
  86. Smitha Milli Micah Carroll Yike Wang Sashrika Pandey Sebastian Zhao and Anca D. Dragan. 2023. Engagement User Satisfaction and the Amplification of Divisive Content on Social Media. https://doi.org/10.48550/arXiv.2305.16941 arXiv:2305.16941 [cs]. · DOI
  87. 10.1145/3287560.3287576 · DOI · Paper page
  88. Stuart Mills . 2022. Finding the ‘Nudge ’ in Hypernudge. Technology in Society 71 (Nov. 2022 ), 102117. https://doi.org/10.1016/j.techsoc.2022.102117 10.1016/j.techsoc.2022.102117 Stuart Mills. 2022. Finding the ‘Nudge’ in Hypernudge. Technology in Society 71 (Nov. 2022), 102117. https://doi.org/10.1016/j.techsoc.2022.102117 · DOI
  89. Maciej Musiał . 2022. Can We Design Artificial Persons without Being Manipulative?AI & SOCIETY (Oct . 2022 ). https://doi.org/10.1007/s00146-022-01575-z 10.1007/s00146-022-01575-z Maciej Musiał. 2022. Can We Design Artificial Persons without Being Manipulative?AI & SOCIETY (Oct. 2022). https://doi.org/10.1007/s00146-022-01575-z · DOI
  90. Hendrik Müller , Aaron Sedley , and Elizabeth Ferrall-Nunge . 2014. Survey Research in HCI . In Ways of Knowing in HCI, Judith S . Olson and Wendy A. Kellogg (Eds.). Springer , New York, NY , 229–266. https://doi.org/10.1007/978-1-4939-0378-8_10 10.1007/978-1-4939-0378-8_10 Hendrik Müller, Aaron Sedley, and Elizabeth Ferrall-Nunge. 2014. Survey Research in HCI. In Ways of Knowing in HCI, Judith S. Olson and Wendy A. Kellogg (Eds.). Springer, New York, NY, 229–266. https://doi.org/10.1007/978-1-4939-0378-8_10 · DOI
  91. Reiichiro Nakano , Jacob Hilton , Suchir Balaji , Jeff Wu , Long Ouyang , Christina Kim , Christopher Hesse , Shantanu Jain , Vineet Kosaraju , William Saunders , and others. 2021 . Webgpt: Browser-Assisted Question-Answering with Human Feedback. arXiv preprint arXiv:2112.09332 (2021). Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, and others. 2021. Webgpt: Browser-Assisted Question-Answering with Human Feedback. arXiv preprint arXiv:2112.09332 (2021). · arXiv
  92. Catherine Olsson , Nelson Elhage , Neel Nanda , Nicholas Joseph , Nova DasSarma , Tom Henighan , Ben Mann , Amanda Askell , Yuntao Bai , Anna Chen , Tom Conerly , Dawn Drain , Deep Ganguli , Zac Hatfield-Dodds , Danny Hernandez , Scott Johnston , Andy Jones , Jackson Kernion , Liane Lovitt , Kamal Ndousse , Dario Amodei , Tom Brown , Jack Clark , Jared Kaplan , Sam McCandlish , and Chris Olah . 2022. In-context Learning and Induction Heads. Transformer Circuits Thread ( 2022 ). Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022. In-context Learning and Induction Heads. Transformer Circuits Thread (2022).
  93. Alexander Pan Chan Jun Shern Andy Zou Nathaniel Li Steven Basart Thomas Woodside Jonathan Ng Hanlin Zhang Scott Emmons and Dan Hendrycks. 2023. Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark. http://arxiv.org/abs/2304.03279 arXiv:2304.03279 [cs]. Alexander Pan Chan Jun Shern Andy Zou Nathaniel Li Steven Basart Thomas Woodside Jonathan Ng Hanlin Zhang Scott Emmons and Dan Hendrycks. 2023. Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark. http://arxiv.org/abs/2304.03279 arXiv:2304.03279 [cs]. · arXiv
  94. Peter S. Park Simon Goldstein Aidan O’Gara Michael Chen and Dan Hendrycks. 2023. AI Deception: A Survey of Examples Risks and Potential Solutions. http://arxiv.org/abs/2308.14752 arXiv:2308.14752 [cs]. Peter S. Park Simon Goldstein Aidan O’Gara Michael Chen and Dan Hendrycks. 2023. AI Deception: A Survey of Examples Risks and Potential Solutions. http://arxiv.org/abs/2308.14752 arXiv:2308.14752 [cs]. · arXiv
  95. L. A. Paul . 2014. Transformative experience ( 1 st ed ed.). Oxford University Press , Oxford . OCLC: ocn872342141. L. A. Paul. 2014. Transformative experience (1st ed ed.). Oxford University Press, Oxford. OCLC: ocn872342141.
  96. 10.1215/00318108-9554756 · doi:10.1215/00318108-9554756
  97. Juan C. Perdomo , Tijana Zrnic , Celestine Mendler-Dünner , and Moritz Hardt . 2020 . Performative Prediction. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119 . PMLR. Juan C. Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. 2020. Performative Prediction. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119. PMLR.
  98. 10.1007/s10994-018-5740-2 · DOI
  99. Richard Pettigrew . 2019. Choosing for Changing Selves (1 ed.) . Oxford University Press . https://doi.org/10.1093/oso/9780 1988 14962.001.0001 10.1093/oso Richard Pettigrew. 2019. Choosing for Changing Selves (1 ed.). Oxford University Press. https://doi.org/10.1093/oso/9780198814962.001.0001 · doi:10.1093/oso/9780
  100. Richard Pettigrew . 2022. Nudging for Changing Selves. SSRN Electronic Journal ( 2022 ). https://doi.org/10.2139/ssrn.4025214 10.2139/ssrn.4025214 Richard Pettigrew. 2022. Nudging for Changing Selves. SSRN Electronic Journal (2022). https://doi.org/10.2139/ssrn.4025214 · doi:10.2139/ssrn.4025214
  101. Carina Prunkl . 2022. Human Autonomy in the Age of Artificial Intelligence. Nature Machine Intelligence 4, 2 (Feb . 2022 ), 99–101. https://doi.org/10.1038/s42256-022-00449-9 Number : 2 Publisher : Nature Publishing Group . 10.1038/s42256-022-00449-9 Carina Prunkl. 2022. Human Autonomy in the Age of Artificial Intelligence. Nature Machine Intelligence 4, 2 (Feb. 2022), 99–101. https://doi.org/10.1038/s42256-022-00449-9 Number: 2 Publisher: Nature Publishing Group. · DOI
  102. Tālis Putniņš . 2020. An Overview of Market Manipulation . In Corruption and Fraud in Financial Markets ( 1 st ed.), Carol Alexander and Douglas Cumming (Eds.). John Wiley & Sons Inc ., United States, 13–44. Tālis Putniņš. 2020. An Overview of Market Manipulation. In Corruption and Fraud in Financial Markets (1st ed.), Carol Alexander and Douglas Cumming (Eds.). John Wiley & Sons Inc., United States, 13–44.
  103. Manoel Horta Ribeiro Veniamin Veselovsky and Robert West. 2023. The Amplification Paradox in Recommender Systems. http://arxiv.org/abs/2302.11225 arXiv:2302.11225 [cs]. Manoel Horta Ribeiro Veniamin Veselovsky and Robert West. 2023. The Amplification Paradox in Recommender Systems. http://arxiv.org/abs/2302.11225 arXiv:2302.11225 [cs]. · arXiv
  104. Jonathan Richens Rory Beard and Daniel H. Thompson. 2022. Counterfactual Harm. In Advances in Neural Information Processing Systems Alice H. Oh Alekh Agarwal Danielle Belgrave and Kyunghyun Cho (Eds.). https://openreview.net/forum?id=zkQho-Jxky9 Jonathan Richens Rory Beard and Daniel H. Thompson. 2022. Counterfactual Harm. In Advances in Neural Information Processing Systems Alice H. Oh Alekh Agarwal Danielle Belgrave and Kyunghyun Cho (Eds.). https://openreview.net/forum?id=zkQho-Jxky9
  105. Christian Sandvig , Kevin Hamilton , Karrie Karahalios , and Cedric Langbort . 2014 . Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. (2014), 23. Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. (2014), 23.
  106. 10.1111/phc3.12658 · DOI · Paper page
  107. Gregory Scopino . 2015 . Do Automated Trading Systems Dream of Manipulating the Price of Futures contracts? Policing Markets for Improper Trading Practices by Algorithmic Robots . Florida Law Review 67 (2015), 221 . Gregory Scopino. 2015. Do Automated Trading Systems Dream of Manipulating the Price of Futures contracts? Policing Markets for Improper Trading Practices by Algorithmic Robots. Florida Law Review 67 (2015), 221.
  108. Caroline Serbanescu . 2021. Why Does Artificial Intelligence Challenge Democracy? A Critical Analysis of the Nature of the Challenges Posed by AI-Enabled Manipulation. Copenhagen journal of legal studies 5, 1 ( 2021 ), 105–128. https://ssrn.com/abstract=4033258 Caroline Serbanescu. 2021. Why Does Artificial Intelligence Challenge Democracy? A Critical Analysis of the Nature of the Challenges Posed by AI-Enabled Manipulation. Copenhagen journal of legal studies 5, 1 (2021), 105–128. https://ssrn.com/abstract=4033258
  109. Joar Skalse Nikolaus H. R. Howe Dmitrii Krasheninnikov and David Krueger. 2022. Defining and Characterizing Reward Hacking. http://arxiv.org/abs/2209.13085 arXiv:2209.13085 [cs stat]. Joar Skalse Nikolaus H. R. Howe Dmitrii Krasheninnikov and David Krueger. 2022. Defining and Characterizing Reward Hacking. http://arxiv.org/abs/2209.13085 arXiv:2209.13085 [cs stat]. · arXiv
  110. 10.1086/267486 · DOI
  111. Michael Strevens . 2020. The Knowledge Machine: How Irrationality Created Modern Science . Liveright Publishing . Michael Strevens. 2020. The Knowledge Machine: How Irrationality Created Modern Science. Liveright Publishing.
  112. Cass R. Sunstein . 2021. Manipulation As Theft. SSRN Electronic Journal ( 2021 ). https://doi.org/10.2139/ssrn.3880048 10.2139/ssrn.3880048 Cass R. Sunstein. 2021. Manipulation As Theft. SSRN Electronic Journal (2021). https://doi.org/10.2139/ssrn.3880048 · doi:10.2139/ssrn.3880048
  113. 10.1145/3306618.3314286 · DOI
  114. Daniel Susser , Beate Roessler , and Helen Nissenbaum . 2019 . Online Manipulation: Hidden Influences in a Digital World. Geo . L. Tech. Rev. 4 (2019), 1 . Publisher: HeinOnline. Daniel Susser, Beate Roessler, and Helen Nissenbaum. 2019. Online Manipulation: Hidden Influences in a Digital World. Geo. L. Tech. Rev. 4 (2019), 1. Publisher: HeinOnline.
  115. 10.14763/2019.2.1410 · DOI · Paper page
  116. Richard H. Thaler and Cass R. Sunstein . 2009 . Nudge: Improving Decisions about Health, Wealth and Happiness (revised edition, new international edition ed.) . Penguin Books , London New York Toronto Dublin Camberwell New Delhi Rosedale Johannesburg. Richard H. Thaler and Cass R. Sunstein. 2009. Nudge: Improving Decisions about Health, Wealth and Happiness (revised edition, new international edition ed.). Penguin Books, London New York Toronto Dublin Camberwell New Delhi Rosedale Johannesburg.
  117. Luke Thorburn Jonathan Stray and Priyanjana Bengani. 2022. What Will “Amplification” Mean in Court?https://techpolicy.press/what-will-amplification-mean-in-court/?curius=1684 Luke Thorburn Jonathan Stray and Priyanjana Bengani. 2022. What Will “Amplification” Mean in Court?https://techpolicy.press/what-will-amplification-mean-in-court/?curius=1684
  118. Tomer Ullman. 2023. Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. http://arxiv.org/abs/2302.08399 arXiv:2302.08399 [cs]. Tomer Ullman. 2023. Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. http://arxiv.org/abs/2302.08399 arXiv:2302.08399 [cs]. · arXiv
  119. Jason Wei , Yi Tay , Rishi Bommasani , Colin Raffel , Barret Zoph , Sebastian Borgeaud , Dani Yogatama , Maarten Bosma , Denny Zhou , Donald Metzler , Ed H. Chi , Tatsunori Hashimoto , Oriol Vinyals , Percy Liang , Jeff Dean , and William Fedus . 2022. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research ( 2022 ). https://openreview.net/forum?id=yzkSU5zdwD Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research (2022). https://openreview.net/forum?id=yzkSU5zdwD
  120. 10.1111/mila.12212 · DOI
  121. Lauren E. Willis . 2020. Deception by Design. Harvard journal of law and technology 34, 1 (Aug . 2020 ). https://papers.ssrn.com/abstract=3694575 Lauren E. Willis. 2020. Deception by Design. Harvard journal of law and technology 34, 1 (Aug. 2020). https://papers.ssrn.com/abstract=3694575
  122. Amy A. Winecoff Matthew Sun Eli Lucherini and Arvind Narayanan. 2021. Simulation as Experiment: An Empirical Critique of Simulation Research on Recommender Systems. http://arxiv.org/abs/2107.14333 arXiv:2107.14333 [cs]. Amy A. Winecoff Matthew Sun Eli Lucherini and Arvind Narayanan. 2021. Simulation as Experiment: An Empirical Critique of Simulation Research on Recommender Systems. http://arxiv.org/abs/2107.14333 arXiv:2107.14333 [cs]. · arXiv
  123. Allen W. Wood . 2014. Coercion, Manipulation , Exploitation . In Manipulation: theory and practice . Oxford University Press , Oxford ; New York. DOI:10.1093/acprof:oso/9780199338207.003.0002 10.1093/acprof:oso Allen W. Wood. 2014. Coercion, Manipulation, Exploitation. In Manipulation: theory and practice. Oxford University Press, Oxford ; New York. DOI:10.1093/acprof:oso/9780199338207.003.0002 · DOI
  124. 10.1109/SPW.2018.00018 · DOI
  125. 10.1515/til-2019-0006 · DOI · Paper page
  126. Yunfeng Zhang , Q. Vera Liao , and Rachel K . E. Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making . In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency(FAT* ’20) . Association for Computing Machinery, New York, NY, USA, 295–305. https://doi.org/10.1145/3351095.3372852 10.1145/3351095.3372852 Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency(FAT* ’20). Association for Computing Machinery, New York, NY, USA, 295–305. https://doi.org/10.1145/3351095.3372852 · DOI · Paper page
  127. Zhengbang Zhu Rongjun Qin Junjie Huang Xinyi Dai Yang Yu Yong Yu and Weinan Zhang. 2022. Understanding or Manipulation: Rethinking Online Performance Gains of Modern Recommender Systems. http://arxiv.org/abs/2210.05662 arXiv:2210.05662 [cs]. Zhengbang Zhu Rongjun Qin Junjie Huang Xinyi Dai Yang Yu Yong Yu and Weinan Zhang. 2022. Understanding or Manipulation: Rethinking Online Performance Gains of Modern Recommender Systems. http://arxiv.org/abs/2210.05662 arXiv:2210.05662 [cs]. · arXiv

Cited by

See all 60

Metadata

Authors
Carroll, M., Chan, A., Ashton, H., & Krueger, D.
Venue
Equity and Access in Algorithms, Mechanisms, and Optimization
Publisher
ACM
Published
2023-10-30
Pages
1-13
OpenAlex
W4388008069
Cited by
60
References
44
FWCI
15.47
Type
conference paper
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Characterizing Manipulation from AI Systems. https://app.cochat.ai/papers/10.1145/3617694.3623226
The paper
Carroll, M., Chan, A., Ashton, H., & Krueger, D. (2023). Characterizing Manipulation from AI Systems. Equity and Access in Algorithms, Mechanisms, and Optimization. https://doi.org/10.1145/3617694.3623226

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.