Multi-Principal Assistance Games
Open accessAbstract
Assistance games (also known as cooperative inverse reinforcement learning games) have been proposed as a model for beneficial AI, wherein a robotic agent must act on behalf of a human principal but is initially uncertain about the humans payoff function. This paper studies multi-principal assistance games, which cover the more general case in which the robot acts on behalf of N humans who may have widely differing payoffs. Impossibility theorems in social choice theory and voting theory can be applied to such games, suggesting that strategic behavior by the human principals may complicate the robots task in learning their payoffs. We analyze in particular a bandit apprentice game in which the humans act first to demonstrate their individual preferences for the arms and then the robot acts to maximize the sum of human payoffs. We explore the extent to which the cost of choosing suboptimal arms reduces the incentive to mislead, a form of natural mechanism design. In this context we propose a social choice method that uses shared control of a system to combine preference inference with social welfare optimization.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A partial solution to some of the problems with alignment can be found in cases where a single AI agent can be used to act on behalf of multiple principals.
“A partial solution to some of the problems with alignment described in Section 4.1 can be found in cases where a single AI agent can be used to act on behalf of multiple principals (Fickinger et al., 2020).”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Fickinger et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037 · see citing works
Manipulation of Voting Schemes: A General Result
Gibbard, A. (1973) · Econometrica · doi:10.2307/1914083 · Cited by 3,271 · see citing works
Apprenticeship learning via inverse reinforcement learning
Abbeel, P., & Ng, A. Y. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015430 · Cited by 2,855 · see citing works
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,868 · see citing works
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621 · see citing works
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314 · see citing works
Interactive Robots as Social Partners and Peer Tutors for Children: A Field Trial
Kanda, T., Hirano, T., Eaton, D., & Ishiguro, H. (2004) · Human-Computer Interaction · doi:10.1207/s15327051hci1901&2_4 · Cited by 1,013 · see citing works
Bayesian Inverse Reinforcement Learning
Krishnamurthy, V. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009449441.027 · Cited by 443 · see citing works
Social Choice TheoryPreprint
Sen, A. K., Arrow, K. J., & Intriligator, M. D. (1986) · RePEc: Research Papers in Economics · Cited by 418 · see citing works
Cooperative Inverse Reinforcement Learning
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03137 · Cited by 413 · see citing works
Self-citationAn Introduction to the Theory of Mechanism Design
Börgers, T., Krähmer, D., & Strausz, R. (2015) · doi:10.1093/acprof:oso/9780199734023.001.0001 · Cited by 380 · see citing works
Multiobjective Reinforcement Learning: A Comprehensive Overview
Liu, C., Xu, X., & Hu, D. (2015) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2014.2358639 · Cited by 361 · see citing works
Nace, D., & Pióro, M. (2008) · IEEE Communications Surveys & Tutorials · doi:10.1109/surv.2008.080403 · Cited by 202 · see citing works
Effects of Repeated Exposure to a Humanoid Robot on Children with Autism
Robins, B., Dautenhahn, K., Boekhorst, R. T., & Billard, A. (2004) · Springer eBooks · doi:10.1007/978-0-85729-372-5_23 · Cited by 183 · see citing works
The Distortion of Cardinal Preferences in Voting
Procaccia, A. D., & Rosenschein, J. S. (2006) · Lecture notes in computer science · doi:10.1007/11839354_23 · Cited by 151 · see citing works
Optimal social choice functions: A utilitarian view
Boutilier, C. E., Caragiannis, I., Haber, S., Lu, T., Procaccia, A. D., & Sheffet, O. (2015) · Artificial Intelligence · doi:10.1016/j.artint.2015.06.003 · Cited by 149 · see citing works
Goal Inference as Inverse Planning
Baker, C. L., Tenenbaum, J. B., & Saxe, R. R. (2007) · eScholarship (California Digital Library) · Cited by 119 · see citing works
Optimal social choice functions
Boutilier, C. E., Caragiannis, I., Haber, S., Lu, T., Procaccia, A. D., & Sheffet, O. (2012) · ACM Conference on Electronic Commerce (EC) · doi:10.1145/2229012.2229030 · Cited by 117 · see citing works
Straightforwardness of Game Forms with Lotteries as Outcomes
Gibbard, A. (1978) · Econometrica · doi:10.2307/1914235 · Cited by 112 · see citing works
Deriving consensus in multiagent systems
Ephrati, E., & Rosenschein, J. S. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(95)00105-0 · Cited by 88 · see citing works
Showing versus Doing: Teaching by DemonstrationPreprint
Ho, M. K., Littman, M. L., MacGlashan, J., Cushman, F. A., & Austerweil, J. L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14004 · Cited by 71 · see citing works
Multi-Armed Bandits with Fairness Constraints for Distributing Resources to Human Teammates
Claure, H., Chen, Y., Modi, J., Jung, M. F., & Nikolaidis, S. (2020) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3319502.3374806 · Cited by 50 · see citing works
Fairness in Multi-Agent Sequential Decision-Making
Zhang, C., & Shah, J. (2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 36 · see citing works
The Assistive Multi-Armed Bandit
Chan, L., Hadfield-Menell, D., Srinivasa, S. S., & Dragan, A. D. (2019) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2019.8673234 · Cited by 31 · see citing works
Self-citationRobot Planning with Mathematical Models of Human State and ActionPreprint
Dragan, A. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.04226 · Cited by 29 · see citing works
Reinforcement Learning with Fairness Constraints for Resource Distribution in Human-Robot TeamsPreprint
Claure, H., Chen, Y., Modi, J., Jung, M. F., & Nikolaidis, S. (2019) · arXiv (Cornell University) · Cited by 15 · see citing works
Learning to Interactively Learn and Assist
Woodward, M. P., Finn, C., & Hausman, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i03.5636 · Cited by 5 · see citing works
Multi-Armed Bandits with Fairness Constraints for Distributing Resources to Human TeammatesPreprint
Claure, H., Chen, Y., Modi, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00313 · Cited by 4 · see citing works
The Assistive Multi-Armed BanditPreprint
Chan, L., Hadfield-Menell, D., Srinivasa, S. S., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08654
Self-citationLearning to Interactively Learn and Assist
Woodward, M., Finn, C., & Hausman, K. (2019) · arXiv · doi:10.48550/arxiv.1906.10187
Flagged first, then by citations
Research works printed but not linked: 2
- Advances in neural information processing systems , pp. 3909-3917, 2016.
- Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J., and Dragan, A. Inverse reward design. In Advances in neural information processing systems , pp. 6765-6774, 2017.
Cited by
Sorted by citations
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications
Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, Y.-X., & Knoll, A. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3457538 · Cited by 341 · see citing works
Trusting Automation: Designing for Responsivity and Resilience
Chiou, E. K., & Lee, J. D. (2023) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208211009995 · Cited by 301 · see citing works
Multi-Agent Risks from Advanced AI
Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.14143 · Cited by 10 · see citing works
Value of Information in Games With Multiple Strategic Information Providers
Velicheti, R. K., Baştopçu, M., & Başar, T. (2025) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2025.3528302 · Cited by 4 · see citing works
Perrier, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.04234 · Cited by 2 · see citing works
Understanding Learned Reward Functions
Michaud, E. J., Gleave, A., & Russell, S. (2020) · arXiv · doi:10.48550/arxiv.2012.05862 · Cited by 7 · see citing works
Self-citation
- Authors
- Fickinger, A., Zhuang, S., Hadfield-Menell, D., & Russell, S.
- Affiliations
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2007.09540
- OpenAlex
- W3043674179
- Cited by
- 5
- References
- 27
- Citations by year
- 2025: 1 · 2024: 1 · 2021: 2 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
- Keywords
- human-robot interaction, strategic behavior, mechanism design, social choice, voting theory, shared control, preference inference, social welfare maximization, impossibility theorem
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
