CoChat
Sign Up
PreprintCited by 8Ethics and Social Impacts of AIReinforcement Learning in RoboticsPsychiatry, Mental Health, Neuroscience

AI Alignment with Changing and Influenceable Reward Functions

Open access
Carroll, Micah, Foote, Davis, Siththaranjan, Anand, Russell, Stuart, Dragan, Anca·2024·arXiv
DOI 10.48550/arxiv.2405.17713arXiv 2405.17713
Open this paper in CoChat — free accountOpen PDF

Abstract

Existing AI alignment approaches assume that preferences are static, which is unrealistic: our preferences change, and may even be influenced by our interactions with AI systems themselves. To clarify the consequences of incorrectly assuming static preferences, we introduce Dynamic Reward Markov Decision Processes (DR-MDPs), which explicitly model preference changes and the AI's influence on them. We show that despite its convenience, the static-preference assumption may undermine the soundness of existing alignment techniques, leading them to implicitly reward AI systems for influencing user preferences in ways users may not truly want. We then explore potential solutions. First, we offer a unifying perspective on how an agent's optimization horizon may partially help reduce undesirable AI influence. Then, we formalize different notions of AI alignment that account for preference change from the outset. Comparing the strengths and limitations of 8 such notions of alignment, we find that they all either err towards causing undesirable AI influence, or are overly risk-averse, suggesting that a straightforward solution to the problems of changing preferences may not exist. As there is no avoiding grappling with changing preferences in real-world settings, this makes it all the more important to handle these issues with care, balancing risks and capabilities. We hope our work can provide conceptual clarity and constitute a first step towards AI alignment practices which explicitly account for (and contend with) the changing and influenceable nature of human preferences.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2405.17713

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some safety constraints are difficult to specify in natural language.

    “Moreover, some safety constraints are difficult to specify in natural language. For example, if we are interested in preventing LLM chatbots from manipulating humans, we would need to precisely specify how to 'not change the user's beliefs' (Carroll et al., 2024).”

    From Feedback Loops With Language Models Drive In-Context Reward Hacking (2024) · cites this paper as Carroll et al., 2024

    Appendix A.2 (page unavailable) says modeling belief changes requires normative choices, including how to prevent an AI from worsening a user's beliefs through deception. Section 7 (page unavailable) also describes practical difficulty in operationalizing a measure of influence. These passages support the difficulty of specifying an anti-manipulation constraint, but they do not discuss specifying it in natural language or equate avoiding manipulation with not changing a user's beliefs at all. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 120

  1. Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained Policy Optimization, May 2017. URL http://arxiv. org/abs/1705.10528. arXiv:1705.10528 [cs. · arXiv
  2. Afsar, M. M., Crump, T., and Far, B. Reinforce- ment learning based recommender systems: A sur- vey. arXiv:2101.06286 [cs], January 2021. URL http://arxiv.org/abs/2101.06286. arXiv: 2101.06286. · arXiv
  3. Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs, Febru- ary 2024. URL http://arxiv.org/abs/2402. 14740. arXiv:2402.14740 [cs. · arXiv
  4. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul- man, J., and Mané, D. Concrete Problems in AI Safety. arXiv:1606.06565 [cs], July 2016. URL http://arxiv.org/abs/1606.06565. arXiv: 1606.06565. · arXiv
  5. Armstrong, S. and Mindermann, S. Occam's razor is insufficient to infer the preferences of irrational agents. arXiv:1712.05812 [cs], January 2019. URL http://arxiv.org/abs/1712.05812. arXiv: 1712.05812. · arXiv
  6. Ayanwale, A. B., Alimi, T., and Ayanbimipe, M. A. The Influence of Advertising on Consumer Brand Prefer- ence. Journal of Social Sciences, 10(1):9-16, January 2005. ISSN 0971-8923. doi: 10.1080/09718923.2005. 11892453. · DOI
  7. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McK- innon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., and Kaplan, J. Consti- tutional AI: Harmlessness from AI Feedback, Decem- ber 2022. URL http://arxiv.org/abs/2212. 08073. arXiv:2212.08073 [cs]. · arXiv
  8. Benn, C. and Lazar, S. What's Wrong with Au- tomated Influence. Canadian Journal of Phi- losophy, 52(1):125-148, January 2022. ISSN 0045-5091, 1911-0820. doi: 10.1017/can.2021. · DOI
  9. Benthall, S. and Shekman, D. Designing Fiduciary Artificial Intelligence, July 2023. URL http://arxiv.org/ abs/2308.02435. arXiv:2308.02435 [cs. · arXiv
  10. Benzion, U., Rapoport, A., and Yagil, J. Discount Rates Inferred from Decisions: An Experimental Study. Man- agement Science, 35(3):270-284, 1989. ISSN 0025- 1909. URL https://www.jstor.org/stable/ 2631972. Publisher: INFORMS.
  11. Brandt, F., Conitzer, V., and Endriss, U. Computational Social Choice. pp. 84, 2012.
  12. Bratman, M. Intention, Plans, and Practical Reason. Cam- bridge, MA: Harvard University Press, Cambridge, 1987.
  13. Bruckner, D. W. In Defense of Adaptive Preferences. Philo- sophical Studies: An International Journal for Philos- ophy in the Analytic Tradition, 142(3):307-324, 2009. ISSN 0031-8116. URL https://www.jstor.org/ stable/27734372. Publisher: Springer.
  14. Burtell, M. and Woodside, T. Artificial Influ- ence: An Analysis Of AI-Driven Persuasion, March 2023. URL http://arxiv.org/abs/2303. 08721. arXiv:2303.08721 [cs. · arXiv
  15. Bykvist, K. Prudence for Changing Selves. Utilitas, 18(3): 264-283, September 2006. ISSN 0953-8208, 1741-6183. doi: 10.1017/S0953820806002032. · DOI
  16. Cai, Q., Liu, S., Wang, X., Zuo, T., Xie, W., Yang, B., Zheng, D., Jiang, P., and Gai, K. Reinforcing User Re- tention in a Billion Scale Short Video Recommender System, February 2023. URL http://arxiv.org/ abs/2302.01724. arXiv:2302.01724 [cs. · arXiv
  17. Callard, A. Aspiration: The Agency of Becoming. Oxford University Press, Oxford, New York, April 2018. ISBN 978-0-19-063948-8.
  18. Carroll, M., Dragan, A., Russell, S., and Hadfield-Menell, D. Estimating and Penalizing Induced Preference Shifts in Recommender Systems, July 2022. URL http:// arxiv.org/abs/2204.11966. arXiv:2204.11966 · arXiv
  19. Carroll, M., Chan, A., Ashton, H., and Krueger, D. Characterizing Manipulation from AI Systems, March 2023. URL http://arxiv.org/abs/ 2303.09387. arXiv:2303.09387 · arXiv
  20. Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T., Marks, S., Segerie, C.-R., Carroll, M., Peng, A., Christoffersen, P., Damani, M., Slocum, S., Anwar, U., Siththaranjan, A., Nadeau, M., Michaud, E. J., Pfau, J., Krasheninnikov, D., Chen, X., Langosco, L., Hase, P., Bıyık, E., Dragan, A., Krueger, D., Sadigh, D., and Hadfield-Menell, D. Open Problems and Fun- damental Limitations of Reinforcement Learning from Human Feedback, July 2023. URL http://arxiv. org/abs/2307.15217. arXiv:2307.15217 [cs]. · arXiv
  21. Chan, L., Hadfield-Menell, D., Srinivasa, S., and Dragan, A. The Assistive Multi-Armed Bandit. arXiv:1901.08654 [cs, stat], January 2019. URL http://arxiv.org/ abs/1901.08654. arXiv: 1901.08654. · arXiv
  22. Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. arXiv:1706.03741 [cs, stat], July 2017. URL http://arxiv.org/abs/1706. 03741. arXiv: 1706.03741. · arXiv
  23. Conitzer, V., Freedman, R., Heitzig, J., Holliday, W. H., Jacobs, B. M., Lambert, N., Mossé, M., Pacuit, E., Rus- sell, S., Schoelkopf, H., Tewolde, E., and Zwicker, W. S. Social Choice for AI Alignment: Dealing with Diverse Human Feedback, April 2024. URL http://arxiv. org/abs/2404.10271. arXiv:2404.10271 [cs. · arXiv
  24. Critch, A. and Krueger, D. AI Research Consid- erations for Human Existential Safety (ARCHES), May 2020. URL http://arxiv.org/abs/2006. 04948. arXiv:2006.04948 · arXiv
  25. Cui, Y., Zhang, Q., Allievi, A., Stone, P., Niekum, S., and Knox, W. B. The EMPATHIC Framework for Task Learning from Implicit Human Feedback, Decem- ber 2020. URL http://arxiv.org/abs/2009. 13649. arXiv:2009.13649 [cs. · arXiv
  26. Cunningham, T., Pandey, S., Sigerson, L., Stray, J., Allen, J., Barrilleaux, B., Iyer, R., Milli, S., Kothari, M., and Rezaei, B. What We Know About Using Non-Engagement Signals in Content Ranking, Febru- ary 2024. URL http://arxiv.org/abs/2402. 06831. arXiv:2402.06831 [cs. · arXiv
  27. Curmei, M., Haupt, A. A., Recht, B., and Hadfield-Menell, D. Towards Psychologically-Grounded Dynamic Prefer- ence Models. In Proceedings of the 16th ACM Conference on Recommender Systems, pp. 35-48, 2022.
  28. Dobbe, R., Gilbert, T. K., and Mintz, Y. Hard Choices in Artificial Intelligence, June 2021. URL http:// arxiv.org/abs/2106.11022. arXiv:2106.11022 [cs, eess]. · arXiv
  29. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. Fairness Through Awareness. arXiv:1104.3913 [cs], November 2011. URL http://arxiv.org/abs/ 1104.3913. arXiv: 1104.3913. · arXiv
  30. Elster, J. (ed.). Ulysses and the Sirens: Studies in Rationality and Irrationality. Editions de la Maison des sciences de l'homme, Paris, 1979.
  31. Elster, J. Sour grapes: studies in the subversion of rational- ity. Cambridge philosophy classics. Cambridge university press, New York (N.Y.), 1983. ISBN 978-1-107-14202-2 978-1-316-50700-1.
  32. Evans, O., Stuhlmueller, A., and Goodman, N. D. Learning the Preferences of Ignorant, Inconsistent Agents. arXiv:1512.05832 [cs], December 2015. URL http://arxiv.org/abs/1512.05832. arXiv: 1512.05832. · arXiv
  33. Everitt, T., Hutter, M., Kumar, R., and Krakovna, V. Re- ward Tampering Problems and Solutions in Reinforce- ment Learning: A Causal Influence Diagram Perspec- tive. arXiv:1908.04734 [cs], March 2021b. URL http://arxiv.org/abs/1908.04734. arXiv: 1908.04734. · arXiv
  34. Farquhar, S., Carey, R., and Everitt, T. Path-Specific Objec- tives for Safer Agent Incentives. arXiv:2204.10018 [cs, stat], April 2022. URL http://arxiv.org/abs/ 2204.10018. arXiv: 2204.10018. · arXiv
  35. Franklin, M., Ashton, H., Gorman, R., and Armstrong, S. Recognising the importance of preference change: A call for a coordinated multidisciplinary research effort in the age of AI. arXiv:2203.10525 [cs], March 2022. URL http://arxiv.org/abs/2203.10525. arXiv: 2203.10525. · arXiv
  36. Frederick, S., Loewenstein, G., and O'Donoghue, T. Time Discounting and Time Preference: A Critical Review. Journal of Economic Litera- ture, 40(2):351-401, June 2002. ISSN 0022- 0515. doi: 10.1257/002205102320161311. URL https://www.aeaweb.org/articles?id=10. 1257/002205102320161311. · DOI
  37. Freeman, R., Zahedi, S. M., and Conitzer, V. Fair and Efficient Social Choice in Dynamic Settings. In Pro- ceedings of the Twenty-Sixth International Joint Con- ference on Artificial Intelligence, pp. 4580-4587, Mel- bourne, Australia, August 2017. International Joint Con- ferences on Artificial Intelligence Organization. ISBN 978-0-9992411-0-3. doi: 10.24963/ijcai.2017/639. URL https://www.ijcai.org/proceedings/ 2017/639. · DOI
  38. Gabriel, I. Artificial Intelligence, Values and Align- ment. arXiv:2001.09768 [cs], January 2020. URL http://arxiv.org/abs/2001.09768. arXiv: 2001.09768. · arXiv
  39. Gabriel, I., Manzini, A., Keeling, G., Hendricks, L. A., Rieser, V., Iqbal, H., Tomašev, N., Ktena, I., Kenton, Z., Rodriguez, M., El-Sayed, S., Brown, S., Akbulut, C., Trask, A., Hughes, E., Bergman, A. S., Shelby, R., Mar- chal, N., Griffin, C., Mateos-Garcia, J., Weidinger, L., Street, W., Lange, B., Ingerman, A., Lentz, A., Enger, R., Barakat, A., Krakovna, V., Siy, J. O., Kurth-Nelson, Z., McCroskery, A., Bolina, V., Law, H., Shanahan, M., Alberts, L., Balle, B., de Haas, S., Ibitoye, Y., Dafoe, A., Goldberg, B., Krier, S., Reese, A., Witherspoon, S., Hawkins, W., Rauh, M., Wallace, D., Franklin, M., Gold- stein, J. A., Lehman, J., Klenk, M., Vallor, S., Biles, C., Morris, M. R., King, H., Arcas, B. A. y., Isaac, W., and Manyika, J. The Ethics of Advanced AI Assistants, April 2024. URL http://arxiv.org/abs/2404. 16244. arXiv:2404.16244 [cs]. · arXiv
  40. Gauci, J., Conti, E., Liang, Y., Virochsiri, K., He, Y., Kaden, Z., Narayanan, V., Ye, X., Chen, Z., and Fujimoto, S. Horizon: Facebook's Open Source Applied Reinforce- ment Learning Platform. arXiv:1811.00260 [cs, stat], September 2019. URL http://arxiv.org/abs/ 1811.00260. arXiv: 1811.00260. · arXiv
  41. George, D. Preference pollution: how markets create the desires we dislike. University of Michigan Press, Ann Arbor, 2001. ISBN 978-0-472-11220-3.
  42. Griffin, J. Well-Being: Its Meaning, Measurement, and Moral Importance. Oxford, GB: Clarendon Press, 1986.
  43. Hadfield-Menell, D. and Hadfield, G. Incomplete Con- tracting and AI Alignment. arXiv:1804.04268 [cs], April 2018. URL http://arxiv.org/abs/1804. 04268. arXiv: 1804.04268. · arXiv
  44. Halpern, D. and Sanders, M. Nudging by Gov- ernment: Progress, Impact, & Lessons Learned. Behavioral Science & Policy, 2(2):53-65, Octo- ber 2016. ISSN 2379-4607. doi: 10.1177/ 237946151600200206. URL https://doi.org/10. 1177/237946151600200206. Publisher: SAGE Publications. · DOI
  45. Halpern, J. Y. and Kleiman-Weiner, M. Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility, October 2018. URL http://arxiv. org/abs/1810.05903. arXiv:1810.05903 [cs]. · arXiv
  46. Hammond, L., Fox, J., Everitt, T., Carey, R., Abate, A., and Wooldridge, M. Reasoning about Causality in Games, January 2023. URL http://arxiv.org/ abs/2301.02324. arXiv:2301.02324 [cs. · arXiv
  47. Hardt, M., Jagadeesan, M., and Mendler-Dünner, C. Performative Power. arXiv:2203.17232 [cs, econ], March 2022. URL http://arxiv.org/abs/ 2203.17232. arXiv: 2203.17232. · arXiv
  48. Harsanyi, J. C. Welfare Economics of Variable Tastes. The Review of Economic Studies, 21(3):204-213, 1953. ISSN 0034-6527. doi: 10.2307/2295773. URL https: //www.jstor.org/stable/2295773. Publisher: [Oxford University Press, Review of Economic Studies, Ltd.]. · DOI
  49. Hausman, D. and Welch, B. Debate: To Nudge or Not to Nudge*. Journal of Political Philosophy, 18:123-136, November 2009. doi: 10.1111/j.1467-9760.2009.00351. · DOI
  50. Hazrati, N. and Ricci, F. Recommender systems effect on the evolution of users' choices distribution. Infor- mation Processing & Management, 59(1):102766, Jan- uary 2022. ISSN 03064573. doi: 10.1016/j.ipm.2021. 102766. URL https://linkinghub.elsevier. com/retrieve/pii/S0306457321002466. · DOI
  51. Hendrycks, D., Mazeika, M., and Woodside, T. An Overview of Catastrophic AI Risks, October 2023. URL http://arxiv.org/abs/2306. 12001. arXiv:2306.12001 · arXiv
  52. Hong, J., Bhatia, K., and Dragan, A. On the Sensitivity of Reward Inference to Misspecified Human Models, December 2022. URL http://arxiv.org/abs/ 2212.04717. arXiv:2212.04717 [cs]. · arXiv
  53. Hong, J., Dragan, A., and Levine, S. Learning to Influence Human Behavior with Offline Reinforcement Learning, June 2023. URL http://arxiv.org/abs/2303. 02265. arXiv:2303.02265 [cs. · arXiv
  54. Huszár, F., Ktena, S. I., O'Brien, C., Belli, L., Schlaikjer, A., and Hardt, M. Algorithmic Amplification of Politics on Twitter. arXiv:2110.11010 [cs], October 2021. URL http://arxiv.org/abs/2110.11010. arXiv: 2110.11010. · arXiv
  55. Hyman, H. H. and Wright, C. R. Education's Lasting In- fluence on Values. Marketing Department, University of Chicago Press, 5801 S, 1979. ERIC Number: ED187642.
  56. Irvine, R., Boubert, D., Raina, V., Liusie, A., Mudupalli, V., Korshuk, A., Liu, Z., Cremer, F., Assassi, V., Beauchamp, C.-C., Lu, X., Rialan, T., and Beauchamp, W. Reward- ing Chatbots for Real-World Engagement with Millions of Users, March 2023. URL http://arxiv.org/ abs/2303.06135. arXiv:2303.06135 [cs]. · arXiv
  57. Jackson, M. O. Mechanism Theory. SSRN Elec- tronic Journal, 2014. ISSN 1556-5068. doi: 10. 2139/ssrn.2542983. URL http://www.ssrn.com/ abstract=2542983. · DOI
  58. Jeon, H. J., Milli, S., and Dragan, A. D. Reward-rational (im- plicit) choice: A unifying formalism for reward learning, December 2020. URL http://arxiv.org/abs/ 2002.04833. arXiv:2002.04833 [cs. · arXiv
  59. Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Miku- lik, V., and Irving, G. Alignment of Language Agents, March 2021. URL http://arxiv.org/ abs/2103.14659. arXiv:2103.14659 · arXiv
  60. Kim, D.-K., Riemer, M., Liu, M., Foerster, J. N., Everett, M., Sun, C., Tesauro, G., and How, J. P. Influencing Long-Term Behavior in Multiagent Reinforcement Learn- ing, October 2022. URL http://arxiv.org/abs/ 2203.03535. arXiv:2203.03535 [cs. · arXiv
  61. Kirk, H. R., Vidgen, B., Röttger, P., and Hale, S. A. The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models, November 2023. URL http://arxiv.org/ abs/2310.02457. arXiv:2310.02457 [cs. · arXiv
  62. Kleinberg, J., Mullainathan, S., and Raghavan, M. The Challenge of Understanding What Users Want: Inconsistent Preferences and Engagement Optimiza- tion. arXiv:2202.11776 [cs], February 2022. URL http://arxiv.org/abs/2202.11776. arXiv: 2202.11776. · arXiv
  63. Krakovna, V., Orseau, L., Kumar, R., Martic, M., and Legg, S. Penalizing side effects using stepwise relative reach- ability. arXiv:1806.01186 [cs, stat], March 2019. URL http://arxiv.org/abs/1806.01186. arXiv: 1806.01186. · arXiv
  64. Krueger, D., Maharaj, T., and Leike, J. Hidden Incentives for Auto-Induced Distributional Shift. arXiv:2009.09153 [cs, stat], September 2020. URL http://arxiv.org/ abs/2009.09153. arXiv: 2009.09153. · arXiv
  65. Kulkarni, K. and Neth, S. Social Choice with Changing Pref- erences: Representation Theorems and Long-Run Poli- cies, November 2020. URL http://arxiv.org/ abs/2011.02544. arXiv:2011.02544 [cs. · arXiv
  66. Lang, L., Foote, D., Russell, S., Dragan, A., Jenner, E., and Emmons, S. When Your AIs Deceive You: Challenges with Partial Observability of Human Evaluators in Re- ward Learning, March 2024. URL http://arxiv. org/abs/2402.17747. arXiv:2402.17747 [cs, stat]. · arXiv
  67. Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. Scalable agent alignment via reward mod- eling: a research direction. arXiv:1811.07871 [cs, stat], November 2018. URL http://arxiv.org/abs/ 1811.07871. arXiv: 1811.07871. · arXiv
  68. Li, B. Z., Tamkin, A., Goodman, N., and Andreas, J. Elic- iting Human Preferences with Language Models, Oc- tober 2023. URL http://arxiv.org/abs/2310. 11589. arXiv:2310.11589 [cs. · arXiv
  69. Lindner, D. and El-Assady, M. Humans are not Boltz- mann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Rein- forcement Learning, June 2022. URL http://arxiv. org/abs/2206.13316. arXiv:2206.13316 [cs, stat]. · arXiv
  70. Locke, J. An Essay Concerning Human Understanding. Oxford University Press, New York, 1689.
  71. Loewenstein, G. and Angner, E. Predicting and indulging changing preferences. In Time and decision: Economic and psychological perspectives on intertemporal choice, pp. 351-391.
  72. Loewenstein, G., Read, D., and Baumeister, R. (eds.). Time and decision: Economic and psychological perspectives on intertemporal choice. Time and decision: Economic and psychological perspectives on intertemporal choice. Russell Sage Foundation, New York, NY, US, 2003. ISBN 978-0-87154-549-7. Pages: xiii, 569.
  73. MacAskill, W. What we owe the future. Basic Books, Ha- chette Book Group, New York, NY, first edition edition, 2022. ISBN 978-1-5416-1862-6. OCLC: 1314633519.
  74. McKinney, L., Duan, Y., Krueger, D., and Gleave, A. On The Fragility of Learned Reward Functions, Jan- uary 2023. URL http://arxiv.org/abs/2301. 03652. arXiv:2301.03652 [cs. · arXiv
  75. Milli, S., Carroll, M., Wang, Y., Pandey, S., Zhao, S., and Dragan, A. D. Engagement, User Satisfaction, and the Amplification of Divisive Content on Social Media, September 2023. URL http://arxiv.org/abs/ 2305.16941. arXiv:2305.16941 [cs. · arXiv
  76. Mishra, A. AI Alignment and Social Choice: Fun- damental Limitations and Policy Implications, Octo- ber 2023. URL http://arxiv.org/abs/2310. 16048. arXiv:2310.16048 [cs. · arXiv
  77. Moshirnia, A. No Security through Obscurity: Changing Circumvention Law to Protect Our Democracy against Cyberattacks. Brooklyn Law Review, 83(4):1279-1344, 2017. URL https://heinonline.org/HOL/P? h=hein.journals/brklr83&i=1317.
  78. Ng, A. Y. and Russell, S. J. Algorithms for Inverse Rein- forcement Learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, pp. 663-670, San Francisco, CA, USA, June 2000.
  79. Ord, T. The precipice: existential risk and the future of humanity. Bloomsbury Publishing, London, 2021. ISBN 978-1-5266-0023-3. OCLC: 1252948575.
  80. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. pp. 68, 2022.
  81. Parfit, D. Personal Identity and Rationality. Synthese, 53(2): 227-241, 1982. ISSN 0039-7857. URL https:// www.jstor.org/stable/20115799. Publisher: Springer.
  82. Parfit, D. Reasons and persons. Clarendon Press, Oxford, 1. issued in paperback (with corr.), reprinted with further corr edition, 1984. ISBN 978-0-19-824908-5.
  83. Paul, L. A. Transformative experience. Oxford University Press, Oxford, 1st ed edition, 2014. ISBN 978-0-19- 871795-9. OCLC: ocn872342141.
  84. Paul, L. A. Choosing for Changing Selves. The Philosoph- ical Review, 131(2):230-235, April 2022. ISSN 0031- 8108. doi: 10.1215/00318108-9554756. URL https: //doi.org/10.1215/00318108-9554756. · DOI
  85. Paul, L. A. and Sunstein, C. R. 'As Judged By Themselves': Transformative Experiences and Endogenous Preferences, September 2019. URL https://papers.ssrn. com/abstract=3455421.
  86. Perdomo, J. C., Zrnic, T., Mendler-Dünner, C., and Hardt, M. Performative Prediction. arXiv:2002.06673 [cs, stat], April 2020. URL http://arxiv.org/abs/2002. 06673. arXiv: 2002.06673. · arXiv
  87. Peston, M. H. Chapter 17. Changing Utility Functions. In Chapter 17. Changing Utility Functions, pp. 233-236. Princeton University Press, 1967. ISBN 978-1-4008- 7738-6. doi: 10.1515/9781400877386-019. URL https://www.degruyter.com/document/ doi/10.1515/9781400877386-019/html. · DOI
  88. Pettigrew, R. Choosing for Changing Selves. Oxford Uni- versity Press, 1 edition, December 2019. ISBN 978- 0-19-881496-2 978-0-19-185280-0. doi: 10.1093/oso/ 9780198814962.001.0001. · DOI
  89. Pettigrew, R. Nudging for Changing Selves. SSRN Electronic Journal, 2022. ISSN 1556-5068. doi: 10.2139/ssrn.4025214. URL https://www.ssrn. com/abstract=4025214. · DOI
  90. Pollak, R. A. Consistent Planning. The Review of Eco- nomic Studies, 35(2):201-208, 1968. ISSN 0034-6527. doi: 10.2307/2296548. URL https://www.jstor. org/stable/2296548. Publisher: [Oxford Univer- sity Press, Review of Economic Studies, Ltd.]. · DOI
  91. Pollak, R. A. Endogenous Tastes in Demand and Wel- fare Analysis. The American Economic Review, 68 (2):374-379, 1978. ISSN 0002-8282. URL https: //www.jstor.org/stable/1816724. Publisher: American Economic Association.
  92. Rafailidis, D. and Nanopoulos, A. Modeling Users Prefer- ence Dynamics and Side Information in Recommender Systems. IEEE Transactions on Systems, Man, and Cy- bernetics: Systems, 46(6):782-792, June 2016. ISSN 2168-2216, 2168-2232. doi: 10.1109/TSMC.2015. 2460691. URL http://ieeexplore.ieee.org/ document/7194815/. · DOI
  93. Regan, K. and Boutilier, C. Robust Policy Compu- tation in Reward-Uncertain MDPs Using Nondomi- nated Policies. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 24(1):1127-1133, July 2010. ISSN 2374-3468, 2159-5399. doi: 10.1609/ aaai.v24i1.7740. URL https://ojs.aaai.org/ index.php/AAAI/article/view/7740. · DOI
  94. Ribeiro, M. H., Veselovsky, V., and West, R. The Am- plification Paradox in Recommender Systems, Febru- ary 2023. URL http://arxiv.org/abs/2302. 11225. arXiv:2302.11225 [cs. · arXiv
  95. Rosati, C. S. The Story of a Life. Social Philosophy and Policy, 30(1-2):21-50, January 2013. ISSN 0265-0525, 1471-6437. doi: 10.1017/S0265052513000022. · DOI
  96. Schelling, T. C. Egonomics, or the Art of Self- Management. The American Economic Review, 68(2,): 290-294, 1978. URL http://www.jstor.org/ stable/1816707.
  97. Schelling, T. C. Self-Command in Practice, in Policy, and in a Theory of Rational Choice. American Economic Review, 74(2):1-11, 1984. URL https://ideas.repec.org//a/aea/ aecrev/v74y1984i2p1-11.html. Publisher: American Economic Association.
  98. Schelling, T. C. Enforcing Rules on Oneself. The Journal of Law, Economics, and Organization, 1(2):357-374, Octo- ber 1985. ISSN 8756-6222. doi: 10.1093/oxfordjournals. jleo.a036896. URL https://doi.org/10.1093/ oxfordjournals.jleo.a036896. · DOI
  99. Shah, R., Gundotra, N., Abbeel, P., and Dragan, A. D. On the Feasibility of Learning, Rather than Assuming, Hu- man Biases for Reward Inference. arXiv:1906.09624 [cs, stat], June 2019a. URL http://arxiv.org/abs/ 1906.09624. arXiv: 1906.09624. · arXiv
  100. Shah, R., Krasheninnikov, D., Alexander, J., Abbeel, P., and Dragan, A. Preferences Implicit in the State of the World. arXiv:1902.04198 [cs, stat], April 2019b. URL http://arxiv.org/abs/1902.04198. arXiv: 1902.04198. · arXiv
  101. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield- Dodds, Z., Johnston, S. R., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., and Perez, E. Towards Un- derstanding Sycophancy in Language Models, Octo- ber 2023. URL http://arxiv.org/abs/2310. 13548. arXiv:2310.13548 [cs, stat]. · arXiv
  102. Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF, Decem- ber 2023. URL http://arxiv.org/abs/2312. 08358. arXiv:2312.08358 [cs, stat]. · arXiv
  103. Stigler, G. J. and Becker, G. S. De Gustibus Non Est Dis- putandum. pp. 16, 1977.
  104. Stray, J., Adler, S., and Hadfield-Menell, D. What are you optimizing for? Aligning Recommender Systems with Human Values. pp. 7, 2021.
  105. Strohmaier, D. and Messerli, M. Preference Change. Cambridge University Press, 1 edition, January 2024. ISBN 978-1-00-918186-0 978-1-00-947579-2 978-1-00- 918185-3. doi: 10.1017/9781009181860. · DOI
  106. Subramani, R., Williams, M., Heitmann, M., Holm, H., Griffin, C., and Skalse, J. On The Expressivity of Objective-Specification Formalisms in Reinforcement Learning, February 2024. URL http://arxiv.org/ abs/2310.11840. arXiv:2310.11840 [cs]. · arXiv
  107. Susser, D., Roessler, B., and Nissenbaum, H. Online Ma- nipulation: Hidden Influences in a Digital World, De- cember 2018. URL https://papers.ssrn.com/ abstract=3306006.
  108. Sutton, R. S. and Barto, A. Reinforcement learning: an in- troduction. Adaptive computation and machine learning. The MIT Press, Cambridge, Massachusetts, nachdruck edition, 2018. ISBN 978-0-262-19398-6.
  109. Taylor, J., Yudkowsky, E., LaVictoire, P., and Critch, A. Alignment for Advanced Machine Learning Systems. pp. 25, 2016.
  110. Thaler, R. H. and Sunstein, C. R. Nudge: Improving de- cisions about health, wealth, and happiness. Nudge: Improving decisions about health, wealth, and happiness. Yale University Press, New Haven, CT, US, 2008. ISBN 978-0-300-12223-7. Pages: x, 293.
  111. Thorburn, L., Stray, J., and Bengani, P. What Will "Amplification" Mean in Court?, 2022. URL https://techpolicy.press/ what-will-amplification-mean-in-court/ ?curius=1684.
  112. Tien, J., He, J. Z.-Y., Erickson, Z., Dragan, A. D., and Brown, D. S. Causal Confusion and Reward Misidentification in Preference-Based Reward Learn- ing, March 2023. URL http://arxiv.org/abs/ 2204.06601. arXiv:2204.06601 [cs. · arXiv
  113. Ullmann-Margalit, E. Big Decisions: Opting, Converting, Drifting. pp. 16, 2006.
  114. Velleman, J. D. Well-Being And Time. Pacific Philo- sophical Quarterly, 72(1):48-77, 1991. doi: 10.1111/ j.1468-0114.1991.tb00410.x. Publisher: Wiley Periodi- cals. · DOI
  115. von Weizsäcker, C. C. Notes on endogenous change of tastes. Journal of Economic Theory, 3(4):345-372, December 1971. ISSN 0022- 0531. doi: 10.1016/0022-0531(71)90037-8. URL https://www.sciencedirect.com/ science/article/pii/0022053171900378. · DOI
  116. Ward, F. R., MacDermott, M., Belardinelli, F., Toni, F., and Everitt, T. The Reasons that Agents Act: Intention and Instrumental Goals, February 2024. URL http:// arxiv.org/abs/2402.07221. arXiv:2402.07221 · arXiv
  117. Warnell, G., Waytowich, N., Lawhern, V., and Stone, P. Deep TAMER: Interactive Agent Shaping in High- Dimensional State Spaces. arXiv:1709.10163 [cs], Jan- uary 2018. URL http://arxiv.org/abs/1709. 10163. arXiv: 1709.10163. · arXiv
  118. Wylie, C. Mindf*ck, Cambridge Analytica and the Plot to Break America. Penguin Ran- dom House, 2020. URL https://www. penguinrandomhouse.com/books/604375/ mindfck-by-christopher-wylie/.
  119. Xie, A., Losey, D. P., Tolsma, R., Finn, C., and Sadigh, D. Learning Latent Representations to Influence Multi- Agent Interaction. pp. 14, 2020.
  120. Ziebart, B. D., Bagnell, J. A., and Dey, A. K. Modeling Interaction via the Principle of Maximum Causal Entropy. pp. 8, 2010.

Cited by

Sorted by citations

Related

Authors
Carroll, M., Foote, D., Siththaranjan, A., Russell, S., & Dragan, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2405.17713
arXiv
2405.17713
OpenAlex
W4399151482
Semantic Scholar
e4eee21ef4e11d3d0ad444b049c6c758b5240187
Cited by
7
References
0
Citations by year
2026: 4 · 2025: 2 · 2024: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
Keywords
AI alignment, changing preferences, preference change, reward function, human-AI interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review