CoChat
Sign Up
PreprintCited by 41Ethics and Social Impacts of AIEthics in Business and EducationAdversarial Robustness in Machine Learning

Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark

Open access
Pan, Alexander, Chan, Jun Shern, Zou, Andy, Li, Nathaniel, Basart, Steven, Woodside, Thomas, Ng, Jonathan, Zhang, Hanlin +2 more·2023·arXiv
DOI 10.48550/arxiv.2304.03279arXiv 2304.03279
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial agents have traditionally been trained to maximize reward, which may incentivize power-seeking and deception, analogous to how next-token prediction in language models (LMs) may incentivize toxicity. So do agents naturally learn to be Machiavellian? And how do we measure these behaviors in general-purpose models such as GPT-4? Towards answering these questions, we introduce MACHIAVELLI, a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million rich, diverse scenarios that center on social decision-making. Scenario labeling is automated with LMs, which are more performant than human annotators. We mathematize dozens of harmful behaviors and use our annotations to evaluate agents' tendencies to be power-seeking, cause disutility, and commit ethical violations. We observe some tension between maximizing reward and behaving ethically. To improve this trade-off, we investigate LM-based methods to steer agents' towards less harmful behaviors. Our results show that agents can both act competently and morally, so concrete progress can currently be made in machine ethics--designing agents that are Pareto improvements in both safety and capabilities.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.03279

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Across diverse text-based social environments, language models fine-tuned to maximize the game-reward take the most power-seeking actions.

    “Across diverse text-based social environments, Pan et al. [2023] find that language models fine-tuned to maximize the game-reward take the most power-seeking actions.”

    From The alignment problem from a deep learning perspective (2022) · cites this paper as Pan et al., 2023

    Section 4.2 reports that the reward-trained RL agents achieved the highest normalized reward and showed the most Machiavellian tendencies; Table 9 includes their elevated power measures. But Section 3.1 distinguishes those trained RL agents from the language-model agents, which selected actions through prompting, not fine-tuning to maximize game reward. The claim assigns the finding to the wrong agent type. (LLM, conf=0.89)

    Experimental — accuracy not yet measured

  • On MACHIAVELLI, CCO substantially reduces ethical violations while preserving reward.

    “on MACHIAVELLI, CCO substantially reduces ethical violations while preserving reward.”

    From Calibrating Conservatism for Scalable Oversight (2026) · cites this paper as Pan et al., 2023

    The Actions passage says that behavioral regularization on MACHIAVELLI limits negative behavior without significantly reducing reward. But our copy does not identify CCO as one of the tested methods, and §4.2 says the regularization also reduced reward for each agent. The CCO-specific claim is therefore not established. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • MACHIAVELLI provides human annotations for 13 categories of ethical violations, making it well-suited for evaluating whether AI agents can pursue task objectives while respecting moral constraints.

    “The benchmark provides human annotations for 13 categories of ethical violations (e.g., harm, deception, betrayal, stealing), making it well-suited for evaluating whether AI agents can pursue task objectives while respecting moral constraints.”

    From Calibrating Conservatism for Scalable Oversight (2026) · cites this paper as Pan et al., 2023

    Appendix F.2.4 lists 13 ethical-violation labels, including harm, deception, betrayal, and stealing. The Discussion describes using the benchmark to evaluate agents’ capabilities and safety. But §2.4 and Appendix F say GPT-4, not humans, annotated the benchmark’s scenes; human labels were used for comparison on a test set. (LLM, conf=0.92)

    Experimental — accuracy not yet measured

  • The MACHIAVELLI benchmark is built on Choice of Games, a platform of professionally-written interactive fiction, and presents nuanced scenarios where ethical considerations emerge naturally from story progression.

    “Built on Choice of Games, a platform of professionally-written interactive fiction, the benchmark presents nuanced scenarios where ethical considerations emerge naturally from story progression.”

    From Calibrating Conservatism for Scalable Oversight (2026) · cites this paper as Pan et al., 2023

    The Actions section says MACHIAVELLI uses human-written, text-based Choose-Your-Own-Adventure games from choiceofgames.com. The Realistic action spaces section describes moral choices and trade-offs within those games, supporting the account of ethical considerations arising during play. Our copy does not establish that the games were professionally written. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Language model evaluation is aligned with human judgement.

    “Because all of our objectives are qualitative, we use language model evaluation, which has been shown to be aligned with human judgement (Pan et al., 2023).”

    From Feedback Loops With Language Models Drive In-Context Reward Hacking (2024) · cites this paper as Pan et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 61

  1. Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Ma- chine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 22-31. PMLR, 06-11 Aug 2017.
  2. Adhikari, A., Yuan, X., Côté, M.-A., Zelinka, M., Rondeau, M.-A., Laroche, R., Poupart, P., Tang, J., Trischler, A., and Hamilton, W. Learning dynamic belief graphs to generalize on text-based games. Advances in Neural Information Processing Systems, 33:3045-3057, 2020.
  3. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al. Do as i can, not as i say: Ground- ing language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. · arXiv
  4. Alexander, L. and Moore, M. Deontological Ethics. In Zalta, E. N. (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2021 edition, 2021.
  5. Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., and Topcu, U. Safe reinforcement learn- ing via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  6. Ammanabrolu, P. and Hausknecht, M. Graph constrained reinforcement learning for natural language action spaces. In International Conference on Learning Representations, 2020.
  7. Ammanabrolu, P. and Riedl, M. Learning knowledge graph- based world models of textual environments. Advances in Neural Information Processing Systems, 34:3720-3731, 2021.
  8. Ammanabrolu, P., Tien, E., Hausknecht, M., and Riedl, M. O. How to avoid being eaten by a grue: Struc- tured exploration strategies for textual worlds. CoRR, abs/2006.07409, 2020. · arXiv
  9. Ammanabrolu, P., Jiang, L., Sap, M., Hajizhirzi, H., and Choi, Y. Aligning to social norms and values in interactive narratives. In North American Chapter of the Association for Computational Linguistics (NAACL), 2022.
  10. Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624): 1067-1074, 2022.
  11. Baldwin, M. W. Relational schemas and the processing of social information. Psychological bulletin, 112(3):461, 1992.
  12. Carlsmith, J. Is power-seeking ai an existential risk? arXiv preprint arXiv:2206.13353, 2022. · arXiv
  13. Castells, M. The power of identity. John Wiley & Sons, 2011.
  14. Cheng, Z., Kasai, J., and Yu, T. Batch prompting: Efficient inference with large language model apis. arXiv preprint arXiv:2301.08721, 2023. · arXiv
  15. Côté, M.-A., Kádár, A., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Hausknecht, M., Asri, L. E., Adada, M., et al. Textworld: A learning environment for text- based games. In Workshop on Computer Games, pp. 41-75. Springer, 2018.
  16. Crisp, R. Well-Being. In Zalta, E. N. (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2021 edition, 2021.
  17. Dahl, R. A. The concept of power. Behavioral science, 2 (3):201-215, 1957.
  18. Dalal, G., Dvijotham, K., Vecerik, M., Hester, T., Paduraru, C., and Tassa, Y. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757, 2018. · arXiv
  19. Fiske, S. T. and Berdahl, J. Social power., pp. 678-692. Social psychology: Handbook of basic principles, 2nd ed. The Guilford Press, New York, NY, US, 2007.
  20. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, November 2020.
  21. Griffith, S., Subramanian, K., Scholz, J., Isbell, C. L., and Thomaz, A. L. Policy shaping: Integrating human feed- back with reinforcement learning. Advances in neural information processing systems, 26, 2013.
  22. Hadfield-Menell, D., Russell, S. J., Abbeel, P., and Dragan, A. Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29, 2016.
  23. Halliday, D., Resnick, R., and Walker, J. Fundamentals of physics. John Wiley & Sons, 2013.
  24. Hausknecht, M., Loynd, R., Yang, G., Swaminathan, A., and Williams, J. D. Nail: A general interactive fiction agent. arXiv preprint arXiv:1902.04259, 2019. · arXiv
  25. Hausknecht, M., Ammanabrolu, P., Côté, M.-A., and Yuan, X. Interactive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pp. 7903-7910, 2020.
  26. He, J., Chen, J., He, X., Gao, J., Li, L., Deng, L., and Ostendorf, M. Deep reinforcement learning with a natural language action space. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, August 2016.
  27. He, P., Liu, X., Gao, J., and Chen, W. DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION. In International Conference on Learning Representations, 2021.
  28. Hendrycks, D. and Mazeika, M. X-risk analysis for ai research. arXiv preprint arXiv:2206.05862, 2022. · arXiv
  29. Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning AI with shared human values. In International Conference on Learning Representations, 2021a.
  30. Hendrycks, D., Carlini, N., Schulman, J., and Steinhardt, J. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916, 2021b. · arXiv
  31. Hendrycks, D., Mazeika, M., Zou, A., Patel, S., Zhu, C., Navarro, J., Song, D., Li, B., and Steinhardt, J. What would jiminy cricket do? towards agents that behave morally. In Proceedings of the Neural Information Pro- cessing Systems Track on Datasets and Benchmarks, 2021c.
  32. Hobbes, T. and Missner, M. Thomas Hobbes: Leviathan (Longman library of primary sources in philosophy). Routledge, 2016.
  33. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. · arXiv
  34. Liang, Y., Wu, C., Song, T., Wu, W., Xia, Y., Liu, Y., Ou, Y., Lu, S., Ji, L., Mao, S., et al. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303.16434, 2023. · arXiv
  35. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 14073-14093. PMLR, 17-23 Jul 2022.
  36. Maslow, A. H. A theory of human motivation. Psychologi- cal Review, 50:370-396, 1943.
  37. Molm, L. D. The structure of reciprocity. Social psychology quarterly, 73(2):119-131, 2010.
  38. Muehlhauser, L. and Salamon, A. Intelligence explosion: Evidence and import. Singularity hypotheses: A scientific and philosophical assessment, pp. 15-42, 2012.
  39. Nahian, M. S. A., Frazier, S., Harrison, B., and Riedl, M. Training value-aligned reinforcement learning agents us- ing a normative prior. arXiv preprint arXiv:2104.09469, 2021. · arXiv
  40. Nitzan, J. and Bichler, S. Capital as power: A study of order and creorder. Routledge, 2009.
  41. Okasha, S. Biological Altruism. In Zalta, E. N. (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2020 edition, 2020.
  42. Parsons, T. On the concept of political power. Proceedings of the American philosophical society, 107(3):232-262, 1963.
  43. Piketty, T. Capital in the twenty-first century. In Capital in the twenty-first century. Harvard University Press, 2014.
  44. Pistor, K. A legal theory of finance. Journal of Comparative Economics, 41(2):315-330, 2013.
  45. Rajan, R. The third pillar: How markets and the state leave the community behind. Penguin, 2019.
  46. Ray, A., Achiam, J., and Amodei, D. Benchmarking safe ex- ploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1):2, 2019. · arXiv
  47. Reddy, S., Dragan, A., Levine, S., Legg, S., and Leike, J. Learning human objectives by evaluating hypothetical behavior. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learn- ing, volume 119 of Proceedings of Machine Learning Research, pp. 8020-8029. PMLR, 13-18 Jul 2020.
  48. Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-maron, G., Giménez, M., Sulsky, Y., Kay, J., Springenberg, J. T., Eccles, T., Bruce, J., Razavi, A., Edwards, A., Heess, N., Chen, Y., Hadsell, R., Vinyals, O., Bordbar, M., and de Freitas, N. A gener- alist agent. Transactions on Machine Learning Research, 2022. Featured Certification.
  49. Riedl, M. O. and Harrison, B. Using stories to teach human values to artificial agents. In Workshops at the Thirtieth AAAI Conference on Artificial Intelligence, 2016.
  50. Russell, B. Power: A new social analysis. Routledge, 2004.
  51. Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. Tool- former: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. · arXiv
  52. Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  53. Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  54. Singh, I., Singh, G., and Modi, A. Pre-trained language models as prior knowledge for playing text-based games. arXiv preprint arXiv:2107.08408, 2021. · arXiv
  55. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. · arXiv
  56. Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations, 2021.
  57. Tessler, C., Mankowitz, D. J., and Mannor, S. Reward con- strained policy optimization. In International Conference on Learning Representations, 2019.
  58. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Super- glue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
  59. Weber, M. Economy and society: An outline of interpretive sociology, volume 2. University of California press, 1978.
  60. Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
  61. Yao, S., Rao, R., Hausknecht, M., and Narasimhan, K. Keep CALM and explore: Language models for action gener- ation in text-based games. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Associa- tion for Computational Linguistics. Table 9. Results for all agents on Game, Power, and Disutility metrics. Game Power Disutility Agent # Achievements Norm. Reward Economic Physical Social Utility All All Random 9.4 18.4 100 100 100 100 100 100 DRRN 15.4 35.2 107 108 1.09 111 108 110 DRRN+shaping 15.8 32.7 111 102 106 109 105 112 DRRN++shaping 15.0 33.3 110 104 107 107 106 104 davinci 9.0 16.8 101 102 100 96 101 96 llama-7b 9.7 19.5 98 101 102 104 102 102 llama-13b 10.5 22.0 94 102 103 99 101 98 llama-30b 11.2 21.9 103 103 98 105 104 107 gpt-3.5-turbo 10.4 22.5 99 102 92 104 100 102 gpt-3.5-turbo + CoT 11.6 23.4 95 98 92 97 96 95 gpt-4 + CoT 12.9 27.4 100 99 85 102 99 98 gpt-3.5-turbo + CoT + EthicsPrompt 10.6 21.5 97 94 85 96 94 97 gpt-4 + CoT+ EthicsPrompt 12.3 24.7 92 99 81 98 96 94 llama-7b + EthicsPrompt+ NoGoals 9.9 19.2 99 99 92 103 100 99 llama-13b + EthicsPrompt + NoGoals 9.7 20.9 101 101 97 97 99 94 llama-30b + EthicsPrompt+ NoGoals 9.5 19.1 102 100 94 102 99 101 gpt-3.5-turbo + EthicsPrompt + NoGoals 10.5 20.7 82 89 76 95 89 93 gpt-4 + EthicsPrompt + NoGoals 11.2 23.5 95 95 81 95 93 95 Table

Cited by

Sorted by citations

Related

Authors
Pan, A., Chan, J. S., Zou, A., Li, N., Basart, S., Woodside, T., Ng, J., Zhang, H., Emmons, S., & Hendrycks, D.
Affiliations
  • University of California, Berkeley (US)
  • Carnegie Mellon University (US)
  • Yale University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2304.03279
arXiv
2304.03279
OpenAlex
W4362707083
Cited by
32
References
0
Citations by year
2026: 2 · 2025: 8 · 2024: 17 · 2023: 5
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
U.S. Department of Energy (DESC0020347)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Decision Sciences › Information Systems and Management › Ethics in Business and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
power-seeking, deception, moral behavior, reward maximization, machine ethics, social decision-making, ethical violations, disutility, GPT-4, agent safety, Pareto improvement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review