Characterizing Manipulation from AI Systems
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
Manipulation is a concern in many domains, such as social media, advertising, and chatbots. As AI systems mediate more of our digital interactions, it is important to understand the degree to which AI systems might manipulate humans without the intent of the system designers. Our work clarifies challenges in defining and measuring this kind of manipulation from AI systems. Firstly, we build upon prior literature on manipulation and characterize the space of possible notions of manipulation, which we find to depend upon the concepts of incentives, intent, covertness, and harm. We review proposals on how to operationalize each concept and we outline challenges in including each concept in a definition of manipulation. Second, we discuss the connections between manipulation and related concepts, such as deception and coercion. We then analyze how our characterization of manipulation applies to recommender systems and language models, and give a brief overview of the regulation of manipulation in other domains. While some progress has been made in defining and measuring manipulation from AI systems, many gaps remain. In the absence of a consensus definition and reliable tools for measurement, we cannot rule out the possibility that AI systems learn to manipulate humans without the intent of the system designers. Manipulation could pose a significant threat to human autonomy and precautionary actions to mitigate it are likely warranted.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Langosco et al. [102] and Shah et al. [146] show that both language models and general RL agents can pursue different goals in out-of-distribution environments even when trained to perfect accuracy on in-distribution environments.”
BackgroundSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“AI systems have documented problems with truthfulness [55, 86, 104].”
BackgroundWhat Makes Online Content Viral?
Berger, J., & Milkman, K. L. (2012) · Journal of Marketing Research · doi:10.1509/jmr.10.0353 · Cited by 3,178 · cited 1×
“Furthermore, it seems plausible that the spread of angry content [24] or clickbait [179] on social media is in part due to one-timestep manipulative incentives for the recommender: while such issues are likely at least in part due to network or supply-and-demand dynamics [117], the behavior is also…”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Knee-deep in the big muddy: a study of escalating commitment to a chosen course of action
Staw, B. M. (1976) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(76)90005-2 · Cited by 2,061 · cited 1×
“series that they will feel compelled to finish because of cognitive biases such as sunk cost fallacy [152], rather than because of actual value derived.”
BackgroundConcrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 1×
“Another possible source of manipulation is reinforcement learning from human feedback (RLHF) [44].”
BackgroundChatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape
Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., & Torous, J. B. (2019) · The Canadian Journal of Psychiatry · doi:10.1177/0706743719828977 · Cited by 1,048 · cited 1×
“Some recent work has focused on studying the harms and behaviour changes due to LMs. Bender et al. [23], Weidinger et al. [171] outline risks that LMs pose, including informational harms like disinformation.”
Background‘Hypernudge’: big Data as a mode of regulation by design
Yeung, K. (2017) · Information, Communication & Society · doi:10.1080/1369118x.2016.1186713 · Cited by 868 · cited 2×
“Moreover, intentional algorithmically-aided manipulation has been extensively studied in prior research [45, 157, 178, 180].”
BackgroundMethod
Research works printed but not linked: 127
- Gediminas Adomavicius , Jesse C. Bockstedt , Shawn P. Curley , and Jingjing Zhang . 2013. Do Recommender Systems Manipulate Consumer Preferences? A Study of Anchoring Effects. Information Systems Research 24, 4 (Dec . 2013 ), 956–975. https://doi.org/10.1287/isre.2013.0497 Publisher : INFORMS. 10.1287/isre.2013.0497 Gediminas Adomavicius, Jesse C. Bockstedt, Shawn P. Curley, and Jingjing Zhang. 2013. Do Recommender Systems Manipulate Consumer Preferences? A Study of Anchoring Effects. Information Systems Research 24, 4 (Dec. 2013), 956–975. https://doi.org/10.1287/isre.2013.0497 Publisher: INFORMS. · DOI · Paper page
- 10.1287/isre.2017.0703 · DOI
- M. Mehdi Afsar , Trafford Crump , and Behrouz Far . 2021. Reinforcement learning based recommender systems: A survey. arXiv:2101.06286 [cs] (Jan . 2021 ). http://arxiv.org/abs/2101.06286 arXiv:2101.06286. M. Mehdi Afsar, Trafford Crump, and Behrouz Far. 2021. Reinforcement learning based recommender systems: A survey. arXiv:2101.06286 [cs] (Jan. 2021). http://arxiv.org/abs/2101.06286 arXiv:2101.06286. · arXiv
- 10.1257/aer.20210867 · DOI · Paper page
- Muhammad Ashfaq , Jiang Yun , Shubin Yu , and Sandra Maria Correia Loureiro . 2020. I , Chatbot : Modeling the determinants of users’ satisfaction and continuance intention of AI-powered service agents. Telematics and Informatics 54 ( Nov. 2020 ), 101473. https://doi.org/10.1016/j.tele.2020.101473 10.1016/j.tele.2020.101473 Muhammad Ashfaq, Jiang Yun, Shubin Yu, and Sandra Maria Correia Loureiro. 2020. I, Chatbot: Modeling the determinants of users’ satisfaction and continuance intention of AI-powered service agents. Telematics and Informatics 54 (Nov. 2020), 101473. https://doi.org/10.1016/j.tele.2020.101473 · DOI · Paper page
- Hal Ashton . 2022. Definitions of Intent Suitable for Algorithms. Artificial Intelligence and Law (July 2022 ). https://doi.org/10.1007/s10506-022-09322-x 10.1007/s10506-022-09322-x Hal Ashton. 2022. Definitions of Intent Suitable for Algorithms. Artificial Intelligence and Law (July 2022). https://doi.org/10.1007/s10506-022-09322-x · DOI
- Hal Ashton and Matija Franklin . 2022 . The Problem of Behaviour and Preference Manipulation in AI Systems. In The AAAI-22 Workshop on Artificial Intelligence Safety (SafeAI 2022). Hal Ashton and Matija Franklin. 2022. The Problem of Behaviour and Preference Manipulation in AI Systems. In The AAAI-22 Workshop on Artificial Intelligence Safety (SafeAI 2022).
- Hal Ashton and Matija Franklin. 2022. Solutions to Preference Manipulation in Recommender Systems Require Knowledge of Meta-Preferences. http://arxiv.org/abs/2209.11801 arXiv:2209.11801 [cs]. Hal Ashton and Matija Franklin. 2022. Solutions to Preference Manipulation in Recommender Systems Require Knowledge of Meta-Preferences. http://arxiv.org/abs/2209.11801 arXiv:2209.11801 [cs]. · arXiv
- Association for Computing Machinery (ACM). 2019. "Reinforcement Learning for Recommender Systems: A Case Study on Youtube " by Minmin Chen. https://www.youtube.com/watch?v=HEqQ2_1XRTs Association for Computing Machinery (ACM). 2019. "Reinforcement Learning for Recommender Systems: A Case Study on Youtube " by Minmin Chen. https://www.youtube.com/watch?v=HEqQ2_1XRTs
- Alessio Azzutti . 2022. AI-driven Market Manipulation and Limits of the EU Law Enforcement Regime to Credible Deterrence. Computer Law & Security review 45 (Jan . 2022 ). https://doi.org/10.2139/ssrn.4026468 10.2139/ssrn.4026468 Alessio Azzutti. 2022. AI-driven Market Manipulation and Limits of the EU Law Enforcement Regime to Credible Deterrence. Computer Law & Security review 45 (Jan. 2022). https://doi.org/10.2139/ssrn.4026468 · doi:10.2139/ssrn.4026468
- Alessio Azzutti , Wolf-Georg Ringe , and H. Siegfried Stiehl . 2021. Machine Learning, Market Manipulation and Collusion on Capital Markets: Why the . University of Pennsylvania journal of international law 43, 1 ( 2021 ). https://doi.org/10.2139/ssrn.3788872 10.2139/ssrn.3788872 Alessio Azzutti, Wolf-Georg Ringe, and H. Siegfried Stiehl. 2021. Machine Learning, Market Manipulation and Collusion on Capital Markets: Why the. University of Pennsylvania journal of international law 43, 1 (2021). https://doi.org/10.2139/ssrn.3788872 · doi:10.2139/ssrn.3788872
- Hui Bai. 2023. Artificial Intelligence Can Persuade Humans. (2023). Hui Bai. 2023. Artificial Intelligence Can Persuade Humans. (2023). · doi:10.21203/rs.3.rs-3238396/v1
- Marcia Baron . 2014. The Mens Rea and Moral Status of Manipulation . In Manipulation: Theory and Practice , Christian Coons and Michael Weber (Eds.). Oxford University Press , 0. https://doi.org/10.1093/acprof:oso/9780 1993 38207.003.0005 10.1093/acprof:oso Marcia Baron. 2014. The Mens Rea and Moral Status of Manipulation. In Manipulation: Theory and Practice, Christian Coons and Michael Weber (Eds.). Oxford University Press, 0. https://doi.org/10.1093/acprof:oso/9780199338207.003.0005 · doi:10.1093/acprof:oso/9780
- 10.1145/3313831.3376518 · DOI
- Yavar Bathaee . 2018 . The Artificial Intelligence Black Box and the Failure of Intent and Causation . Harvard Journal of Law and Technology 31 , 2 (2018), 890 – 938 . Yavar Bathaee. 2018. The Artificial Intelligence Black Box and the Failure of Intent and Causation. Harvard Journal of Law and Technology 31, 2 (2018), 890–938.
- Omer Ben-Porat and Moshe Tennenholtz . 2018. A Game-Theoretic Approach to Recommendation Systems with Strategic Content Providers . In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Vol. 31. Curran Associates , Inc .https://proceedings.neurips.cc/paper/ 2018 /file/a9a1d5317a33ae8cef33961c34144f84-Paper.pdf Omer Ben-Porat and Moshe Tennenholtz. 2018. A Game-Theoretic Approach to Recommendation Systems with Strategic Content Providers. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Vol. 31. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2018/file/a9a1d5317a33ae8cef33961c34144f84-Paper.pdf
- 10.1080/15265161.2011.634481 · DOI · Paper page
- Rishi Bommasani Drew A. Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S. Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill Erik Brynjolfsson Shyamal Buch Dallas Card Rodrigo Castellon Niladri Chatterji Annie Chen Kathleen Creel Jared Quincy Davis Dora Demszky Chris Donahue Moussa Doumbouya Esin Durmus Stefano Ermon John Etchemendy Kawin Ethayarajh Li Fei-Fei Chelsea Finn Trevor Gale Lauren Gillespie Karan Goel Noah Goodman Shelby Grossman Neel Guha Tatsunori Hashimoto Peter Henderson John Hewitt Daniel E. Ho Jenny Hong Kyle Hsu Jing Huang Thomas Icard Saahil Jain Dan Jurafsky Pratyusha Kalluri Siddharth Karamcheti Geoff Keeling Fereshte Khani Omar Khattab Pang Wei Koh Mark Krass Ranjay Krishna Rohith Kuditipudi Ananya Kumar Faisal Ladhak Mina Lee Tony Lee Jure Leskovec Isabelle Levent Xiang Lisa Li Xuechen Li Tengyu Ma Ali Malik Christopher D. Manning Suvir Mirchandani Eric Mitchell Zanele Munyikwa Suraj Nair Avanika Narayan Deepak Narayanan Ben Newman Allen Nie Juan Carlos Niebles Hamed Nilforoshan Julian Nyarko Giray Ogut Laurel Orr Isabel Papadimitriou Joon Sung Park Chris Piech Eva Portelance Christopher Potts Aditi Raghunathan Rob Reich Hongyu Ren Frieda Rong Yusuf Roohani Camilo Ruiz Jack Ryan Christopher Ré Dorsa Sadigh Shiori Sagawa Keshav Santhanam Andy Shih Krishnan Srinivasan Alex Tamkin Rohan Taori Armin W. Thomas Florian Tramèr Rose E. Wang William Wang Bohan Wu Jiajun Wu Yuhuai Wu Sang Michael Xie Michihiro Yasunaga Jiaxuan You Matei Zaharia Michael Zhang Tianyi Zhang Xikun Zhang Yuhui Zhang Lucia Zheng Kaitlyn Zhou and Percy Liang. 2022. On the Opportunities and Risks of Foundation Models. https://doi.org/10.48550/arXiv.2108.07258 arXiv:2108.07258 [cs]. · DOI · Paper page
- Harriet Braiker . 2003. Who’s Pulling Your Strings?: How to Break the Cycle of Manipulation and Regain Control of Your Life: How to Break the Cycle of Manipulation and Regain Control of Your Life . McGraw Hill Professional. Google-Books-ID : dGwgiQvyeq0C. Harriet Braiker. 2003. Who’s Pulling Your Strings?: How to Break the Cycle of Manipulation and Regain Control of Your Life: How to Break the Cycle of Manipulation and Regain Control of Your Life. McGraw Hill Professional. Google-Books-ID: dGwgiQvyeq0C.
- Collin Burns , Haotian Ye , Dan Klein , and Jacob Steinhardt . 2023 . Discovering Latent Knowledge in Language Models Without Supervision. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=ETKGuby0hcs Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering Latent Knowledge in Language Models Without Supervision. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=ETKGuby0hcs
- Qingpeng Cai Shuchang Liu Xueliang Wang Tianyou Zuo Wentao Xie Bin Yang Dong Zheng Peng Jiang and Kun Gai. 2023. Reinforcing User Retention in a Billion Scale Short Video Recommender System. http://arxiv.org/abs/2302.01724 arXiv:2302.01724 [cs]. Qingpeng Cai Shuchang Liu Xueliang Wang Tianyou Zuo Wentao Xie Bin Yang Dong Zheng Peng Jiang and Kun Gai. 2023. Reinforcing User Retention in a Billion Scale Short Video Recommender System. http://arxiv.org/abs/2302.01724 arXiv:2302.01724 [cs]. · arXiv
- 10.1093/oso/9780190639488.001.0001 · DOI · Paper page
- M. Ryan Calo . 2014 . Digital Market Manipulation . George Washington Law Review 82 , 4 (2014), 996 – 1051 . https://doi.org/10.2139/ssrn.2309703 10.2139/ssrn.2309703 M. Ryan Calo. 2014. Digital Market Manipulation. George Washington Law Review 82, 4 (2014), 996–1051. https://doi.org/10.2139/ssrn.2309703 · DOI · Paper page
- Ryan Carey , Eric Langlois , Tom Everitt , and Shane Legg . 2020. The Incentives that Shape Behaviour. arXiv:2001.07118 [cs] (Jan . 2020 ). http://arxiv.org/abs/2001.07118 arXiv:2001.07118. Ryan Carey, Eric Langlois, Tom Everitt, and Shane Legg. 2020. The Incentives that Shape Behaviour. arXiv:2001.07118 [cs] (Jan. 2020). http://arxiv.org/abs/2001.07118 arXiv:2001.07118. · arXiv
- Thomas L. Carson . 2010. Lying and Deception: Theory and Practice . Oxford University Press , Oxford ; New York. OCLC: ocn464581525. Thomas L. Carson. 2010. Lying and Deception: Theory and Practice. Oxford University Press, Oxford ; New York. OCLC: ocn464581525.
- Alan Chan Rebecca Salganik Alva Markelius Chris Pang Nitarshan Rajkumar Dmitrii Krasheninnikov Lauro Langosco Zhonghao He Yawen Duan Micah Carroll Michelle Lin Alex Mayhew Katherine Collins Maryam Molamohammadi John Burden Wanru Zhao Shalaleh Rismani Konstantinos Voudouris Umang Bhatt Adrian Weller David Krueger and Tegan Maharaj. 2023. Harms from Increasingly Agentic Algorithmic Systems. https://doi.org/10.48550/arXiv.2302.10329 arXiv:2302.10329 [cs]. · DOI · Paper page
- Allison J. B. Chaney. 2021. Recommendation System Simulations: A Discussion of Two Key Challenges. https://doi.org/10.48550/arXiv.2109.02475 10.48550/arXiv.2109.02475 · arXiv · doi:10.48550/arxiv.2109.02475
- Allison J. B. Chaney. 2021. Recommendation System Simulations: A Discussion of Two Key Challenges. https://doi.org/10.48550/arXiv.2109.02475 · arXiv · doi:10.48550/arxiv.2109.02475
- 10.1145/3240323.3240370 · DOI · Paper page
- Minmin Chen , Alex Beutel , Paul Covington , Sagar Jain , Francois Belletti , and Ed Chi . 2020 . Top-K Off-Policy Correction for a REINFORCE Recommender System . arXiv:1812.02353 [cs, stat] (Nov. 2020). http://arxiv.org/abs/1812.02353 arXiv:1812.02353. Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed Chi. 2020. Top-K Off-Policy Correction for a REINFORCE Recommender System. arXiv:1812.02353 [cs, stat] (Nov. 2020). http://arxiv.org/abs/1812.02353 arXiv:1812.02353. · arXiv
- Mark Chen , Jerry Tworek , Heewoo Jun , Qiming Yuan , Henrique Ponde de Oliveira Pinto , Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and others. 2021 . Evaluating Large Language Models Trained on Code . arXiv preprint arXiv:2107.03374 (2021). Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and others. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
- Thomas Christiano . 2022. Algorithms, Manipulation, and Democracy. Canadian Journal of Philosophy 52, 1 (Jan . 2022 ), 109–124. https://doi.org/10.1017/can.2021.29 Publisher : Cambridge University Press . 10.1017/can.2021.29 Thomas Christiano. 2022. Algorithms, Manipulation, and Democracy. Canadian Journal of Philosophy 52, 1 (Jan. 2022), 109–124. https://doi.org/10.1017/can.2021.29 Publisher: Cambridge University Press. · DOI
- Leon Ciechanowski , Aleksandra Przegalinska , Mikolaj Magnuski , and Peter Gloor . 2019 . In the Shades of the Uncanny Valley: An Experimental Study of Human–Chatbot Interaction. Future Generation Computer Systems 92 (March 2019), 539–548. https://doi.org/10.1016/j.future.2018.01.055 10.1016/j.future.2018.01.055 Leon Ciechanowski, Aleksandra Przegalinska, Mikolaj Magnuski, and Peter Gloor. 2019. In the Shades of the Uncanny Valley: An Experimental Study of Human–Chatbot Interaction. Future Generation Computer Systems 92 (March 2019), 539–548. https://doi.org/10.1016/j.future.2018.01.055 · DOI · Paper page
- 10.1017/beq.2015.41 · doi:10.1017/beq.2015.41
- 10.1145/3523227.3546778 · DOI
- Allan Dafoe Edward Hughes Yoram Bachrach Tantum Collins Kevin R. McKee Joel Z. Leibo Kate Larson and Thore Graepel. 2020. Open Problems in Cooperative AI. https://doi.org/10.48550/arXiv.2012.08630 arXiv:2012.08630 [cs]. · DOI · Paper page
- Shayan Doroudi , Vincent Aleven , and Emma Brunskill . 2019. Where’s the Reward?International Journal of Artificial Intelligence in Education 29, 4 (Dec . 2019 ), 568–620. https://doi.org/10.1007/s40593-019-00187-x 10.1007/s40593-019-00187-x Shayan Doroudi, Vincent Aleven, and Emma Brunskill. 2019. Where’s the Reward?International Journal of Artificial Intelligence in Education 29, 4 (Dec. 2019), 568–620. https://doi.org/10.1007/s40593-019-00187-x · DOI · Paper page
- 10.1073/pnas.1419828112 · DOI · Paper page
- Charles Evans and Atoosa Kasirzadeh . 2021. User Tampering in Reinforcement Learning Recommender Systems. arXiv:2109.04083 [cs] (Sept . 2021 ). http://arxiv.org/abs/2109.04083 arXiv:2109.04083. Charles Evans and Atoosa Kasirzadeh. 2021. User Tampering in Reinforcement Learning Recommender Systems. arXiv:2109.04083 [cs] (Sept. 2021). http://arxiv.org/abs/2109.04083 arXiv:2109.04083. · arXiv
- Owain Evans , Owen Cotton-Barratt , Lukas Finnveden , Adam Bales , Avital Balwit , Peter Wills , Luca Righetti , and William Saunders . 2021 . Truthful AI: Developing and Governing AI that does not Lie. arXiv:2110.06674 [cs] (Oct. 2021). http://arxiv.org/abs/2110.06674 arXiv:2110.06674. Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021. Truthful AI: Developing and Governing AI that does not Lie. arXiv:2110.06674 [cs] (Oct. 2021). http://arxiv.org/abs/2110.06674 arXiv:2110.06674. · arXiv
- Tom Everitt , Ryan Carey , Eric Langlois , Pedro A. Ortega , and Shane Legg . 2021 . Agent Incentives: A Causal Perspective. arXiv:2102.01685. Tom Everitt, Ryan Carey, Eric Langlois, Pedro A. Ortega, and Shane Legg. 2021. Agent Incentives: A Causal Perspective. arXiv:2102.01685. · arXiv
- Sebastian Farquhar , Ryan Carey , and Tom Everitt . 2022. Path-Specific Objectives for Safer Agent Incentives. arXiv:2204.10018 [cs, stat] (April 2022 ). http://arxiv.org/abs/2204.10018 arXiv:2204.10018. Sebastian Farquhar, Ryan Carey, and Tom Everitt. 2022. Path-Specific Objectives for Safer Agent Incentives. arXiv:2204.10018 [cs, stat] (April 2022). http://arxiv.org/abs/2204.10018 arXiv:2204.10018. · arXiv
- 10.1145/286498.286852 · DOI
- Brian J Fogg . 2003. Persuasive Technology . Elsevier . https://doi.org/10.1016/B978-1-55860-643-2.X5000-8 10.1016/B978-1-55860-643-2.X5000-8 Brian J Fogg. 2003. Persuasive Technology. Elsevier. https://doi.org/10.1016/B978-1-55860-643-2.X5000-8 · DOI
- Matija Franklin , Hal Ashton , Rebecca Gorman , and Stuart Armstrong . 2022. Recognising the Importance of Preference Change: A Call for a Coordinated Multidisciplinary Research Effort in the Age of AI. arXiv:2203.10525 [cs] (March 2022 ). http://arxiv.org/abs/2203.10525 arXiv:2203.10525. Matija Franklin, Hal Ashton, Rebecca Gorman, and Stuart Armstrong. 2022. Recognising the Importance of Preference Change: A Call for a Coordinated Multidisciplinary Research Effort in the Age of AI. arXiv:2203.10525 [cs] (March 2022). http://arxiv.org/abs/2203.10525 arXiv:2203.10525. · arXiv
- Jason Gauci , Edoardo Conti , Yitao Liang , Kittipat Virochsiri , Yuchen He , Zachary Kaden , Vivek Narayanan , Xiaohui Ye , Zhengxing Chen , and Scott Fujimoto . 2019 . Horizon: Facebook’s Open Source Applied Reinforcement Learning Platform. arXiv:1811.00260 [cs, stat] (Sept. 2019). http://arxiv.org/abs/1811.00260 arXiv:1811.00260. Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Yuchen He, Zachary Kaden, Vivek Narayanan, Xiaohui Ye, Zhengxing Chen, and Scott Fujimoto. 2019. Horizon: Facebook’s Open Source Applied Reinforcement Learning Platform. arXiv:1811.00260 [cs, stat] (Sept. 2019). http://arxiv.org/abs/1811.00260 arXiv:1811.00260. · arXiv
- Charles Goodhart . 1975. Problems of Monetary Management: the UK Experience in Papers in Monetary Economics. Monetary Economics 1 ( 1975 ). Charles Goodhart. 1975. Problems of Monetary Management: the UK Experience in Papers in Monetary Economics. Monetary Economics 1 (1975).
- 10.1145/3173574.3174108 · DOI · Paper page
- Lewis D. Griffin Bennett Kleinberg Maximilian Mozes Kimberly T. Mai Maria Vau Matthew Caldwell and Augustine Marvor-Parker. 2023. Susceptibility to Influence of Large Language Models. http://arxiv.org/abs/2303.06074 arXiv:2303.06074 [cs]. Lewis D. Griffin Bennett Kleinberg Maximilian Mozes Kimberly T. Mai Maria Vau Matthew Caldwell and Augustine Marvor-Parker. 2023. Susceptibility to Influence of Large Language Models. http://arxiv.org/abs/2303.06074 arXiv:2303.06074 [cs]. · arXiv
- Till Grüne-Yanoff and Sven Ove Hansson (Eds.). 2009. Preference change: approaches from philosophy economics and psychology. Number v. 42 in Theory and decision library. Series A Philosophy and methodology of the social sciences. Springer Dordrecht ; London. OCLC: ocn321018474. Till Grüne-Yanoff and Sven Ove Hansson (Eds.). 2009. Preference change: approaches from philosophy economics and psychology. Number v. 42 in Theory and decision library. Series A Philosophy and methodology of the social sciences. Springer Dordrecht ; London. OCLC: ocn321018474.
- 10.1177/1461444819876944 · DOI · Paper page
- Joseph Y. Halpern and Max Kleiman-Weiner . 2018 . Towards Formal Definitions of Blameworthiness, Intention , and Moral responsibility. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence(AAAI’18/IAAI’18/EAAI’18). AAAI Press , New Orleans, Louisiana, USA, 1853–1860. Joseph Y. Halpern and Max Kleiman-Weiner. 2018. Towards Formal Definitions of Blameworthiness, Intention, and Moral responsibility. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence(AAAI’18/IAAI’18/EAAI’18). AAAI Press, New Orleans, Louisiana, USA, 1853–1860.
- 10.1145/3437963.3441775 · DOI
- Moritz Hardt , Nimrod Megiddo , Christos Papadimitriou , and Mary Wootters . 2016 . Strategic Classification. In Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science(ITCS ’16) . Association for Computing Machinery, New York, NY, USA, 111–122. Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. 2016. Strategic Classification. In Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science(ITCS ’16). Association for Computing Machinery, New York, NY, USA, 111–122.
- 10.1609/aaai.v36i5.20509 · doi:10.1609/aaai.v36i5.20509
- 10.1613/jair.202 · DOI · Paper page
- 10.1146/annurev-statistics-031017-100630 · DOI · Paper page
- 10.1177/0956797611436349 · DOI · Paper page
- Joey Hong Anca Dragan and Sergey Levine. 2023. Learning to Influence Human Behavior with Offline Reinforcement Learning. https://doi.org/10.48550/arXiv.2303.02265 arXiv:2303.02265 [cs]. 10.48550/arXiv.2303.02265 · arXiv · doi:10.48550/arxiv.2303.02265
- Joey Hong Anca Dragan and Sergey Levine. 2023. Learning to Influence Human Behavior with Offline Reinforcement Learning. https://doi.org/10.48550/arXiv.2303.02265 arXiv:2303.02265 [cs]. · arXiv · doi:10.48550/arxiv.2303.02265
- 10.1145/3287560.3287597 · DOI
- (Robin) Hui Huang . 2009. Redefining Market Manipulation in Australia: The Role of an Implied Intent Element. Companies and Securities Law Journal 27 (April 2009 ). https://papers.ssrn.com/abstract=1376209 (Robin) Hui Huang. 2009. Redefining Market Manipulation in Australia: The Role of an Implied Intent Element. Companies and Securities Law Journal 27 (April 2009). https://papers.ssrn.com/abstract=1376209
- Meena Jagadeesan , Celestine Mendler-Dünner , and Moritz Hardt . 2021. Alternative Microfoundations for Strategic Classification . In ICML . http://arxiv.org/abs/2106.12705 arXiv:2106.12705. Meena Jagadeesan, Celestine Mendler-Dünner, and Moritz Hardt. 2021. Alternative Microfoundations for Strategic Classification. In ICML. http://arxiv.org/abs/2106.12705 arXiv:2106.12705. · arXiv
- Maurice Jakesch Advait Bhat Daniel Buschek Lior Zalmanson and Mor Naaman. 2023. Co-Writing with Opinionated Language Models Affects Users' Views. https://doi.org/10.1145/3544548.3581196 arXiv:2302.00560 [cs]. · DOI · Paper page
- Mathias Jesse and Dietmar Jannach . 2021. Digital Nudging with Recommender Systems: Survey and Future Directions. Computers in Human Behavior Reports 3 (Jan . 2021 ), 100052. https://doi.org/10.1016/j.chbr.2020.100052 10.1016/j.chbr.2020.100052 Mathias Jesse and Dietmar Jannach. 2021. Digital Nudging with Recommender Systems: Survey and Future Directions. Computers in Human Behavior Reports 3 (Jan. 2021), 100052. https://doi.org/10.1016/j.chbr.2020.100052 · DOI · Paper page
- Saurav Kadavath Tom Conerly Amanda Askell Tom Henighan Dawn Drain Ethan Perez Nicholas Schiefer Zac Hatfield-Dodds Nova DasSarma Eli Tran-Johnson Scott Johnston Sheer El-Showk Andy Jones Nelson Elhage Tristan Hume Anna Chen Yuntao Bai Sam Bowman Stanislav Fort Deep Ganguli Danny Hernandez Josh Jacobson Jackson Kernion Shauna Kravec Liane Lovitt Kamal Ndousse Catherine Olsson Sam Ringer Dario Amodei Tom Brown Jack Clark Nicholas Joseph Ben Mann Sam McCandlish Chris Olah and Jared Kaplan. 2022. Language Models (Mostly) Know What They Know. https://doi.org/10.48550/arXiv.2207.05221 arXiv:2207.05221 [cs]. · DOI · Paper page
- Jean Kaddour Aengus Lynch Qi Liu Matt J. Kusner and Ricardo Silva. 2022. Causal Machine Learning: A Survey and Open Problems. https://doi.org/10.48550/arXiv.2206.15475 arXiv:2206.15475 [cs stat]. 10.48550/arXiv.2206.15475 · DOI
- Jean Kaddour Aengus Lynch Qi Liu Matt J. Kusner and Ricardo Silva. 2022. Causal Machine Learning: A Survey and Open Problems. https://doi.org/10.48550/arXiv.2206.15475 arXiv:2206.15475 [cs stat]. · DOI
- 10.1257/aer.101.6.2590 · DOI · Paper page
- Timotheus Kampik , Juan Carlos Nieves , and Helena Lindgren . 2018 . Coercion and Deception in Persuasive Technologies. In 20th International Trust Workshop (co-located with AAMAS/IJCAI/ECAI/ICML 2018 ), Stockholm, Sweden , 14 July, 2018. CEUR-WS, 38–49. Timotheus Kampik, Juan Carlos Nieves, and Helena Lindgren. 2018. Coercion and Deception in Persuasive Technologies. In 20th International Trust Workshop (co-located with AAMAS/IJCAI/ECAI/ICML 2018), Stockholm, Sweden, 14 July, 2018. CEUR-WS, 38–49.
- 10.1504/IJIMA.2013.056586 · DOI
- Zachary Kenton Ramana Kumar Sebastian Farquhar Jonathan Richens Matt MacDermott and Tom Everitt. 2022. Discovering Agents. https://doi.org/10.48550/arXiv.2208.08345 arXiv:2208.08345 [cs]. · DOI · Paper page
- Poruz Khambatta Shwetha Mariadassou Joshua Morris and S Christian Wheeler. 2022. Targeting Recommendation Algorithms to Ideal Preferences Makes Users Better Off. (2022). Poruz Khambatta Shwetha Mariadassou Joshua Morris and S Christian Wheeler. 2022. Targeting Recommendation Algorithms to Ideal Preferences Makes Users Better Off. (2022). · doi:10.5465/ambpp.2022.14632abstract
- 10.1145/3328526.3329584 · DOI
- Michal Kosinski. 2023. Theory of Mind May Have Spontaneously Emerged in Large Language Models. http://arxiv.org/abs/2302.02083 arXiv:2302.02083 [cs]. Michal Kosinski. 2023. Theory of Mind May Have Spontaneously Emerged in Large Language Models. http://arxiv.org/abs/2302.02083 arXiv:2302.02083 [cs]. · arXiv
- 10.1086/676597 · DOI
- Arto Laitinen and Otto Sahlgren . 2021. AI Systems and Respect for Human Autonomy. Frontiers in Artificial Intelligence 4 ( 2021 ). https://www.frontiersin.org/articles/10.3389/frai.2021.705164 Arto Laitinen and Otto Sahlgren. 2021. AI Systems and Respect for Human Autonomy. Frontiers in Artificial Intelligence 4 (2021). https://www.frontiersin.org/articles/10.3389/frai.2021.705164 · DOI · Paper page
- Lauro Langosco Di Langosco , Jack Koch , Lee D Sharkey , Jacob Pfau , and David Krueger . 2022 . Goal Misgeneralization in Deep Reinforcement Learning . In Proceedings of the 39th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 162) , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 1 2004–12019. https://proceedings.mlr.press/v162/langosco22a.html Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. 2022. Goal Misgeneralization in Deep Reinforcement Learning. In Proceedings of the 39th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 12004–12019. https://proceedings.mlr.press/v162/langosco22a.html
- Kenneth Li , Aspen K. Hopkins , David Bau , Fernanda Viégas , Hanspeter Pfister , and Martin Wattenberg . 2023 . Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task . In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=DeG07_TcZvT Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=DeG07_TcZvT
- Stephanie Lin , Jacob Hilton , and Owain Evans . 2021. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs] (Sept . 2021 ). http://arxiv.org/abs/2109.07958 arXiv:2109.07958. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs] (Sept. 2021). http://arxiv.org/abs/2109.07958 arXiv:2109.07958. · arXiv
- Tom C. W. Lin . 2017. The New Market Manipulation. Emory Law Journal 66 (July 2017 ). https://papers.ssrn.com/abstract=2996896 Tom C. W. Lin. 2017. The New Market Manipulation. Emory Law Journal 66 (July 2017). https://papers.ssrn.com/abstract=2996896
- David Lindner , Kyle Matoba , and Alexander Meulemans . 2021. Challenges for Using Impact Regularizers to Avoid Negative Side Effects. arXiv:2101.12509 [cs] (Feb . 2021 ). http://arxiv.org/abs/2101.12509 arXiv:2101.12509. David Lindner, Kyle Matoba, and Alexander Meulemans. 2021. Challenges for Using Impact Regularizers to Avoid Negative Side Effects. arXiv:2101.12509 [cs] (Feb. 2021). http://arxiv.org/abs/2101.12509 arXiv:2101.12509. · arXiv
- 10.1093/jla/laaa006 · DOI · Paper page
- Masoud Mansoury , Himan Abdollahpouri , Mykola Pechenizkiy , Bamshad Mobasher , and Robin Burke . 2020. Feedback Loop and Bias Amplification in Recommender Systems. arXiv:2007.13019 [cs] (July 2020 ). http://arxiv.org/abs/2007.13019 arXiv:2007.13019. Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. 2020. Feedback Loop and Bias Amplification in Recommender Systems. arXiv:2007.13019 [cs] (July 2020). http://arxiv.org/abs/2007.13019 arXiv:2007.13019. · arXiv
- 10.1016/j.pragma.2004.12.007 · DOI · Paper page
- Smitha Milli Micah Carroll Yike Wang Sashrika Pandey Sebastian Zhao and Anca D. Dragan. 2023. Engagement User Satisfaction and the Amplification of Divisive Content on Social Media. https://doi.org/10.48550/arXiv.2305.16941 arXiv:2305.16941 [cs]. · DOI
- 10.1145/3287560.3287576 · DOI · Paper page
- Stuart Mills . 2022. Finding the ‘Nudge ’ in Hypernudge. Technology in Society 71 (Nov. 2022 ), 102117. https://doi.org/10.1016/j.techsoc.2022.102117 10.1016/j.techsoc.2022.102117 Stuart Mills. 2022. Finding the ‘Nudge’ in Hypernudge. Technology in Society 71 (Nov. 2022), 102117. https://doi.org/10.1016/j.techsoc.2022.102117 · DOI
- Maciej Musiał . 2022. Can We Design Artificial Persons without Being Manipulative?AI & SOCIETY (Oct . 2022 ). https://doi.org/10.1007/s00146-022-01575-z 10.1007/s00146-022-01575-z Maciej Musiał. 2022. Can We Design Artificial Persons without Being Manipulative?AI & SOCIETY (Oct. 2022). https://doi.org/10.1007/s00146-022-01575-z · DOI
- Hendrik Müller , Aaron Sedley , and Elizabeth Ferrall-Nunge . 2014. Survey Research in HCI . In Ways of Knowing in HCI, Judith S . Olson and Wendy A. Kellogg (Eds.). Springer , New York, NY , 229–266. https://doi.org/10.1007/978-1-4939-0378-8_10 10.1007/978-1-4939-0378-8_10 Hendrik Müller, Aaron Sedley, and Elizabeth Ferrall-Nunge. 2014. Survey Research in HCI. In Ways of Knowing in HCI, Judith S. Olson and Wendy A. Kellogg (Eds.). Springer, New York, NY, 229–266. https://doi.org/10.1007/978-1-4939-0378-8_10 · DOI
- Reiichiro Nakano , Jacob Hilton , Suchir Balaji , Jeff Wu , Long Ouyang , Christina Kim , Christopher Hesse , Shantanu Jain , Vineet Kosaraju , William Saunders , and others. 2021 . Webgpt: Browser-Assisted Question-Answering with Human Feedback. arXiv preprint arXiv:2112.09332 (2021). Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, and others. 2021. Webgpt: Browser-Assisted Question-Answering with Human Feedback. arXiv preprint arXiv:2112.09332 (2021). · arXiv
- Catherine Olsson , Nelson Elhage , Neel Nanda , Nicholas Joseph , Nova DasSarma , Tom Henighan , Ben Mann , Amanda Askell , Yuntao Bai , Anna Chen , Tom Conerly , Dawn Drain , Deep Ganguli , Zac Hatfield-Dodds , Danny Hernandez , Scott Johnston , Andy Jones , Jackson Kernion , Liane Lovitt , Kamal Ndousse , Dario Amodei , Tom Brown , Jack Clark , Jared Kaplan , Sam McCandlish , and Chris Olah . 2022. In-context Learning and Induction Heads. Transformer Circuits Thread ( 2022 ). Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022. In-context Learning and Induction Heads. Transformer Circuits Thread (2022).
- Alexander Pan Chan Jun Shern Andy Zou Nathaniel Li Steven Basart Thomas Woodside Jonathan Ng Hanlin Zhang Scott Emmons and Dan Hendrycks. 2023. Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark. http://arxiv.org/abs/2304.03279 arXiv:2304.03279 [cs]. Alexander Pan Chan Jun Shern Andy Zou Nathaniel Li Steven Basart Thomas Woodside Jonathan Ng Hanlin Zhang Scott Emmons and Dan Hendrycks. 2023. Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark. http://arxiv.org/abs/2304.03279 arXiv:2304.03279 [cs]. · arXiv
- Peter S. Park Simon Goldstein Aidan O’Gara Michael Chen and Dan Hendrycks. 2023. AI Deception: A Survey of Examples Risks and Potential Solutions. http://arxiv.org/abs/2308.14752 arXiv:2308.14752 [cs]. Peter S. Park Simon Goldstein Aidan O’Gara Michael Chen and Dan Hendrycks. 2023. AI Deception: A Survey of Examples Risks and Potential Solutions. http://arxiv.org/abs/2308.14752 arXiv:2308.14752 [cs]. · arXiv
- L. A. Paul . 2014. Transformative experience ( 1 st ed ed.). Oxford University Press , Oxford . OCLC: ocn872342141. L. A. Paul. 2014. Transformative experience (1st ed ed.). Oxford University Press, Oxford. OCLC: ocn872342141.
- 10.1215/00318108-9554756 · doi:10.1215/00318108-9554756
- Juan C. Perdomo , Tijana Zrnic , Celestine Mendler-Dünner , and Moritz Hardt . 2020 . Performative Prediction. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119 . PMLR. Juan C. Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. 2020. Performative Prediction. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119. PMLR.
- 10.1007/s10994-018-5740-2 · DOI
- Richard Pettigrew . 2019. Choosing for Changing Selves (1 ed.) . Oxford University Press . https://doi.org/10.1093/oso/9780 1988 14962.001.0001 10.1093/oso Richard Pettigrew. 2019. Choosing for Changing Selves (1 ed.). Oxford University Press. https://doi.org/10.1093/oso/9780198814962.001.0001 · doi:10.1093/oso/9780
- Richard Pettigrew . 2022. Nudging for Changing Selves. SSRN Electronic Journal ( 2022 ). https://doi.org/10.2139/ssrn.4025214 10.2139/ssrn.4025214 Richard Pettigrew. 2022. Nudging for Changing Selves. SSRN Electronic Journal (2022). https://doi.org/10.2139/ssrn.4025214 · doi:10.2139/ssrn.4025214
- Carina Prunkl . 2022. Human Autonomy in the Age of Artificial Intelligence. Nature Machine Intelligence 4, 2 (Feb . 2022 ), 99–101. https://doi.org/10.1038/s42256-022-00449-9 Number : 2 Publisher : Nature Publishing Group . 10.1038/s42256-022-00449-9 Carina Prunkl. 2022. Human Autonomy in the Age of Artificial Intelligence. Nature Machine Intelligence 4, 2 (Feb. 2022), 99–101. https://doi.org/10.1038/s42256-022-00449-9 Number: 2 Publisher: Nature Publishing Group. · DOI
- Tālis Putniņš . 2020. An Overview of Market Manipulation . In Corruption and Fraud in Financial Markets ( 1 st ed.), Carol Alexander and Douglas Cumming (Eds.). John Wiley & Sons Inc ., United States, 13–44. Tālis Putniņš. 2020. An Overview of Market Manipulation. In Corruption and Fraud in Financial Markets (1st ed.), Carol Alexander and Douglas Cumming (Eds.). John Wiley & Sons Inc., United States, 13–44.
- Manoel Horta Ribeiro Veniamin Veselovsky and Robert West. 2023. The Amplification Paradox in Recommender Systems. http://arxiv.org/abs/2302.11225 arXiv:2302.11225 [cs]. Manoel Horta Ribeiro Veniamin Veselovsky and Robert West. 2023. The Amplification Paradox in Recommender Systems. http://arxiv.org/abs/2302.11225 arXiv:2302.11225 [cs]. · arXiv
- Jonathan Richens Rory Beard and Daniel H. Thompson. 2022. Counterfactual Harm. In Advances in Neural Information Processing Systems Alice H. Oh Alekh Agarwal Danielle Belgrave and Kyunghyun Cho (Eds.). https://openreview.net/forum?id=zkQho-Jxky9 Jonathan Richens Rory Beard and Daniel H. Thompson. 2022. Counterfactual Harm. In Advances in Neural Information Processing Systems Alice H. Oh Alekh Agarwal Danielle Belgrave and Kyunghyun Cho (Eds.). https://openreview.net/forum?id=zkQho-Jxky9
- Christian Sandvig , Kevin Hamilton , Karrie Karahalios , and Cedric Langbort . 2014 . Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. (2014), 23. Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. (2014), 23.
- 10.1111/phc3.12658 · DOI · Paper page
- Gregory Scopino . 2015 . Do Automated Trading Systems Dream of Manipulating the Price of Futures contracts? Policing Markets for Improper Trading Practices by Algorithmic Robots . Florida Law Review 67 (2015), 221 . Gregory Scopino. 2015. Do Automated Trading Systems Dream of Manipulating the Price of Futures contracts? Policing Markets for Improper Trading Practices by Algorithmic Robots. Florida Law Review 67 (2015), 221.
- Caroline Serbanescu . 2021. Why Does Artificial Intelligence Challenge Democracy? A Critical Analysis of the Nature of the Challenges Posed by AI-Enabled Manipulation. Copenhagen journal of legal studies 5, 1 ( 2021 ), 105–128. https://ssrn.com/abstract=4033258 Caroline Serbanescu. 2021. Why Does Artificial Intelligence Challenge Democracy? A Critical Analysis of the Nature of the Challenges Posed by AI-Enabled Manipulation. Copenhagen journal of legal studies 5, 1 (2021), 105–128. https://ssrn.com/abstract=4033258
- Joar Skalse Nikolaus H. R. Howe Dmitrii Krasheninnikov and David Krueger. 2022. Defining and Characterizing Reward Hacking. http://arxiv.org/abs/2209.13085 arXiv:2209.13085 [cs stat]. Joar Skalse Nikolaus H. R. Howe Dmitrii Krasheninnikov and David Krueger. 2022. Defining and Characterizing Reward Hacking. http://arxiv.org/abs/2209.13085 arXiv:2209.13085 [cs stat]. · arXiv
- 10.1086/267486 · DOI
- Michael Strevens . 2020. The Knowledge Machine: How Irrationality Created Modern Science . Liveright Publishing . Michael Strevens. 2020. The Knowledge Machine: How Irrationality Created Modern Science. Liveright Publishing.
- Cass R. Sunstein . 2021. Manipulation As Theft. SSRN Electronic Journal ( 2021 ). https://doi.org/10.2139/ssrn.3880048 10.2139/ssrn.3880048 Cass R. Sunstein. 2021. Manipulation As Theft. SSRN Electronic Journal (2021). https://doi.org/10.2139/ssrn.3880048 · doi:10.2139/ssrn.3880048
- 10.1145/3306618.3314286 · DOI
- Daniel Susser , Beate Roessler , and Helen Nissenbaum . 2019 . Online Manipulation: Hidden Influences in a Digital World. Geo . L. Tech. Rev. 4 (2019), 1 . Publisher: HeinOnline. Daniel Susser, Beate Roessler, and Helen Nissenbaum. 2019. Online Manipulation: Hidden Influences in a Digital World. Geo. L. Tech. Rev. 4 (2019), 1. Publisher: HeinOnline.
- 10.14763/2019.2.1410 · DOI · Paper page
- Richard H. Thaler and Cass R. Sunstein . 2009 . Nudge: Improving Decisions about Health, Wealth and Happiness (revised edition, new international edition ed.) . Penguin Books , London New York Toronto Dublin Camberwell New Delhi Rosedale Johannesburg. Richard H. Thaler and Cass R. Sunstein. 2009. Nudge: Improving Decisions about Health, Wealth and Happiness (revised edition, new international edition ed.). Penguin Books, London New York Toronto Dublin Camberwell New Delhi Rosedale Johannesburg.
- Luke Thorburn Jonathan Stray and Priyanjana Bengani. 2022. What Will “Amplification” Mean in Court?https://techpolicy.press/what-will-amplification-mean-in-court/?curius=1684 Luke Thorburn Jonathan Stray and Priyanjana Bengani. 2022. What Will “Amplification” Mean in Court?https://techpolicy.press/what-will-amplification-mean-in-court/?curius=1684
- Tomer Ullman. 2023. Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. http://arxiv.org/abs/2302.08399 arXiv:2302.08399 [cs]. Tomer Ullman. 2023. Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. http://arxiv.org/abs/2302.08399 arXiv:2302.08399 [cs]. · arXiv
- Jason Wei , Yi Tay , Rishi Bommasani , Colin Raffel , Barret Zoph , Sebastian Borgeaud , Dani Yogatama , Maarten Bosma , Denny Zhou , Donald Metzler , Ed H. Chi , Tatsunori Hashimoto , Oriol Vinyals , Percy Liang , Jeff Dean , and William Fedus . 2022. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research ( 2022 ). https://openreview.net/forum?id=yzkSU5zdwD Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research (2022). https://openreview.net/forum?id=yzkSU5zdwD
- 10.1111/mila.12212 · DOI
- Lauren E. Willis . 2020. Deception by Design. Harvard journal of law and technology 34, 1 (Aug . 2020 ). https://papers.ssrn.com/abstract=3694575 Lauren E. Willis. 2020. Deception by Design. Harvard journal of law and technology 34, 1 (Aug. 2020). https://papers.ssrn.com/abstract=3694575
- Amy A. Winecoff Matthew Sun Eli Lucherini and Arvind Narayanan. 2021. Simulation as Experiment: An Empirical Critique of Simulation Research on Recommender Systems. http://arxiv.org/abs/2107.14333 arXiv:2107.14333 [cs]. Amy A. Winecoff Matthew Sun Eli Lucherini and Arvind Narayanan. 2021. Simulation as Experiment: An Empirical Critique of Simulation Research on Recommender Systems. http://arxiv.org/abs/2107.14333 arXiv:2107.14333 [cs]. · arXiv
- Allen W. Wood . 2014. Coercion, Manipulation , Exploitation . In Manipulation: theory and practice . Oxford University Press , Oxford ; New York. DOI:10.1093/acprof:oso/9780199338207.003.0002 10.1093/acprof:oso Allen W. Wood. 2014. Coercion, Manipulation, Exploitation. In Manipulation: theory and practice. Oxford University Press, Oxford ; New York. DOI:10.1093/acprof:oso/9780199338207.003.0002 · DOI
- 10.1109/SPW.2018.00018 · DOI
- 10.1515/til-2019-0006 · DOI · Paper page
- Yunfeng Zhang , Q. Vera Liao , and Rachel K . E. Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making . In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency(FAT* ’20) . Association for Computing Machinery, New York, NY, USA, 295–305. https://doi.org/10.1145/3351095.3372852 10.1145/3351095.3372852 Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency(FAT* ’20). Association for Computing Machinery, New York, NY, USA, 295–305. https://doi.org/10.1145/3351095.3372852 · DOI · Paper page
- Zhengbang Zhu Rongjun Qin Junjie Huang Xinyi Dai Yang Yu Yong Yu and Weinan Zhang. 2022. Understanding or Manipulation: Rethinking Online Performance Gains of Modern Recommender Systems. http://arxiv.org/abs/2210.05662 arXiv:2210.05662 [cs]. Zhengbang Zhu Rongjun Qin Junjie Huang Xinyi Dai Yang Yu Yong Yu and Weinan Zhang. 2022. Understanding or Manipulation: Rethinking Online Performance Gains of Modern Recommender Systems. http://arxiv.org/abs/2210.05662 arXiv:2210.05662 [cs]. · arXiv
Cited by
AI Deception: A Survey of Examples, Risks, and Potential Solutions
Park, P. S., Goldstein, S., O'Gara, A., Chen, M., & Hendrycks, D. (2023) · arXiv · doi:10.48550/arxiv.2308.14752 · Cited by 243
“For similar definitions, see Evans et al. 2 and Carroll et al. 3 We present a wide range of examples where AI systems do not merely produce false outputs by accident .”
AI deception: A survey of examples, risks, and potential solutions
Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100988 · Cited by 243
“For similar definitions, see Evans et al. 2 and Carroll et al. 3 We present a wide range of examples where AI systems do not merely produce false outputs by accident .”
Combating misinformation in the age of LLMs: Opportunities and challenges
Chen, C., & Shu, K. (2024) · AI Magazine · doi:10.1002/aaai.12188 · Cited by 148
Williamson, S. M., & Prybutok, V. (2024) · Information · doi:10.3390/info15060299 · Cited by 146
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101
“These incentives to mislead also connect to broader worries about manipulation [Kenton et al., 2021, Carroll et al., 2023, Everitt et al., 2021].”
BackgroundSelf-citation
Metadata
- Authors
- Carroll, M., Chan, A., Ashton, H., & Krueger, D.
- Venue
- Equity and Access in Algorithms, Mechanisms, and Optimization
- Publisher
- ACM
- Published
- 2023-10-30
- Pages
- 1-13
- OpenAlex
- W4388008069
- Cited by
- 60
- References
- 44
- FWCI
- 15.47
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Characterizing Manipulation from AI Systems. https://app.cochat.ai/papers/10.1145/3617694.3623226
- The paper
- Carroll, M., Chan, A., Ashton, H., & Krueger, D. (2023). Characterizing Manipulation from AI Systems. Equity and Access in Algorithms, Mechanisms, and Optimization. https://doi.org/10.1145/3617694.3623226
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.