Estimating and Penalizing Preference Shift in Recommender Systems
Abstract
Recommender systems trained via long-horizon optimization (e.g., reinforcement learning) will have incentives to actively manipulate user preferences through the recommended content. While some work has argued for making systems myopic to avoid this issue, even such systems can induce systematic undesirable preference shifts. Thus, rather than artificially stifling the capabilities of the system, in this work we explore how we can make capable systems that explicitly avoid undesirable shifts. We advocate for (1) estimating the preference shifts that would be induced by recommender system policies, and (2) explicitly characterizing what unwanted shifts are and assessing before deployment whether such policies will produce them – ideally even actively optimizing to avoid them. These steps involve two challenging ingredients: (1) requires the ability to anticipate how hypothetical policies would influence user preferences if deployed; instead, (2) requires metrics to assess whether such influences are manipulative or otherwise unwanted. We study how to do (1) from historical user interaction data by building a user predictive model that implicitly contains their preference dynamics; to address (2), we introduce the notion of a “safe policy”, which defines a trust region within which behavior is believed to be safe. We show that recommender systems that optimize for staying in the trust region avoid manipulative behaviors (e.g., changing preferences in ways that make users more predictable), while still generating engagement.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Research on 'filter bubbles' has moved from considering algorithmic curation as decisive force shaping online engagement towards acknowledging the influence of users' own choices on social media ranking algorithms and search engines, thus highlighting the intricate feedback loops between machine and human decisions.
“This challenge is reflected in research on”
From Machine culture (2023) · cites this paper as [175]
Cites
Artificial intelligence: a modern approach
Kumar, D. A., Balasubramanian, S., Byeon, D. H., & Manerkar, P. G. V. (1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 22,182 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Louviere, J. J., Hensher, D. A., Swait́, J., & Adamowicz, W. (2000) · Cambridge University Press eBooks · doi:10.1017/cbo9780511753831 · Cited by 6,915 · see citing works
An introduction to hidden Markov models
Rabiner, L., & Juang, B. (1986) · IEEE ASSP Magazine · doi:10.1109/massp.1986.1165342 · Cited by 4,775 · see citing works
Experimental evidence of massive-scale emotional contagion through social networks
Kramer, A. D. I., Guillory, J. E., & Hancock, J. T. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1320040111 · Cited by 3,687 · see citing works
Deep Neural Networks for YouTube Recommendations
Covington, P., Adams, J., & Sargin, E. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959190 · Cited by 3,539 · see citing works
Deep Learning Based Recommender System
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2019) · ACM Computing Surveys · doi:10.1145/3285029 · Cited by 1,523 · see citing works
Online Social Network Site Addiction: A Comprehensive Review
Andreassen, C. S. (2015) · Current Addiction Reports · doi:10.1007/s40429-015-0056-9 · Cited by 1,212 · see citing works
Psychological targeting as an effective approach to digital mass persuasion
Matz, S. C., Kosinski, M., Nave, G., & Stillwell, D. J. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1710966114 · Cited by 892 · see citing works
Recurrent Recommender Networks
Wu, C.-Y., Ahmed, A. A., Beutel, A., Smola, A. J., & Jing, H. (2017) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3018661.3018689 · Cited by 733 · see citing works
Deep Learning based Recommender System: A Survey and New PerspectivesPreprint
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.07435 · Cited by 719 · see citing works
The truth about the truth: a meta-analytic review of the truth effect
Dechêne, A., Stahl, C., Hansen, J., & Wänke, M. (2010) · Personality and Social Psychology Review · doi:10.1177/1088868309352251 · Cited by 592 · see citing works
Nguyen, T. T., Hui, P.-M., Harper, F. M., Terveen, L., & Konstan, J. A. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568012 · Cited by 584 · see citing works
Reinforcement Learning based Recommender Systems: A Survey
Afsar, M. M., Crump, T., & Far, B. H. (2022) · ACM Computing Surveys · doi:10.1145/3543846 · Cited by 560 · see citing works
Embedding-based News Recommendation for Millions of Users
Okura, S., Tagami, Y., Ono, S., & Tajima, A. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098108 · Cited by 511 · see citing works
Cosley, D., Lam, S. K., Albert, I., Konstan, J. A., & Riedl, J. (2003) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642713 · Cited by 466 · see citing works
Douglas, C. (2015) · Choice Reviews Online · doi:10.5860/choice.190825 · Cited by 465 · see citing works
A Non-Homogeneous Hidden Markov Model for Precipitation Occurrence
Hughes, J. P., Guttorp, P. M., & Charles, S. P. (1999) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/1467-9876.00136 · Cited by 404 · see citing works
How algorithmic confounding in recommendation systems increases homogeneity and decreases utility
Chaney, A. J. B., Stewart, B., & Engelhardt, B. E. (2018) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3240323.3240370 · Cited by 345 · see citing works
Degenerate Feedback Loops in Recommender Systems
Jiang, R., Chiappa, S., Lattimore, T., György, A., & Kohli, P. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314288 · Cited by 232 · see citing works
Häubl, G., & Murray, K. B. (2003) · Journal of Consumer Psychology · doi:10.1207/153276603768344807 · Cited by 226 · see citing works
Modeling and broadening temporal user interest in personalized news recommendation
Li, L., Li, Z., Yang, F., & Li, T. (2013) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.11.020 · Cited by 199 · see citing works
MacLeod, C., & Campbell, L. (1992) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.63.6.890 · Cited by 153 · see citing works
WWW 2014 - Proceedings of the 23rd International Conference on World Wide Web
Nguyen, T. T., Hui, P.-M., Harper, M., Terveen, L., & Konstan, J. A. (2014) · Cited by 146 · see citing works
Persuasion in Recommender Systems
Ulrike Gretzel, B., & Fesenmaier, D. R. (2006) · International Journal of Electronic Commerce · doi:10.2753/jec1086-4415110204 · Cited by 145 · see citing works
Hasan, R., Jha, A. K., & Liu, Y. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.11.020 · Cited by 144 · see citing works
Behaviorism is Not Enough: Better Recommendations through Listening to Users
Ekstrand, M. D., & Willemsen, M. C. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959179 · Cited by 101 · see citing works
Bountouridis, D., Harambam, J., Makhortykh, M., Marrero, M., Tintarev, N., & Hauff, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287583 · Cited by 92 · see citing works
A Theory of Chosen Preferences
Bernheim, B. D., Braghieri, L., Martínez-Marquina, A., & Zuckerman, D. M. (2019) · American Economic Review · doi:10.1257/aer.20190390 · Cited by 78 · see citing works
Auditing E-Commerce Platforms for Algorithmically Curated Vaccine Misinformation
Juneja, P., & Mitra, T. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445250 · Cited by 48 · see citing works
Toward Simulating Environments in Reinforcement Learning Based RecommendationsPreprint
Zhao, X., Xia, L., Ding, Z.-Y., Yin, D.-W., & Tang, J.-L. (2019) · arXiv.org · doi:10.48550/arxiv.1906.11462 · Cited by 26 · see citing works
Aspiration: The Agency of Becoming
Callard, A. (2019) · Cited by 23 · see citing works
Excessive Use of Online Video Streaming Services Survey
Hasan, R., Jha, A. K., & Liu, Y. (2018) · PsycTESTS Dataset · doi:10.1037/t65990-000 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Brinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller, T. F., Nussberger, A.-M., Czaplicka, A., Acerbi, A., Griffiths, T. L., Henrich, J., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01742-2 · Cited by 132 · see citing works
Feedback Loops With Language Models Drive In-Context Reward Hacking
Pan, A., Jones, E., Jagadeesan, M., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06627 · Cited by 3 · see citing works
- Authors
- Carroll, M., Hadfield-Menell, D., Russell, S., & Dragan, A.
- Affiliations
- University of California, Berkeley (US)
- Venue
- Fifteenth ACM Conference on Recommender Systems
- Publisher
- ACM
- Published
- 2021-09-13
- Pages
- 661-667
- OpenAlex
- W3199782056
- Cited by
- 20
- References
- 34
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 5 · 2023: 8 · 2022: 3 · 2021: 1
- FWCI
- 3.52
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Recommender Systems (RecSys) (publishedVersion)
- Funders
- ONR YIP
- Topics
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Keywords
- preference change, recommender systems, preference manipulation, reinforcement learning, user engagement, trust region, dynamic user preferences, manipulative behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
