CoChat
Sign Up
PreprintCited by 1Ethics and Social Impacts of AIArtificial Intelligence ApplicationsCybernetics and Technology in Society

A theory of appropriateness with applications to generative artificial intelligence

Open access
Leibo, Joel Z., Vezhnevets, Alexander Sasha, Diaz, Manfred, Agapiou, John P., Cunningham, William A., Sunehag, Peter, Haas, Julia, Koster, Raphael +6 more·2024·arXiv
DOI 10.48550/arxiv.2412.19010arXiv 2412.19010
Open this paper in CoChat — free accountOpen PDF

Abstract

What is appropriateness? Humans navigate a multi-scale mosaic of interlocking notions of what is appropriate for different situations. We act one way with our friends, another with our family, and yet another in the office. Likewise for AI, appropriate behavior for a comedy-writing assistant is not the same as appropriate behavior for a customer-service representative. What determines which actions are appropriate in which contexts? And what causes these standards to change over time? Since all judgments of AI appropriateness are ultimately made by humans, we need to understand how appropriateness guides human decision making in order to properly evaluate AI decision making and improve it. This paper presents a theory of appropriateness: how it functions in human society, how it may be implemented in the brain, and what it means for responsible deployment of generative AI technology.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2412.19010

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For settings in which inter-agent communication is infeasible or insufficient, norms and conventions may be necessary in order to avoid miscoordination.

    “For settings in which inter-agent communication is infeasible or insufficient, norms and conventions may be necessary in order to avoid miscoordination (Leibo et al., 2024).”

    From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Leibo et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 1

Research works printed but not linked: 378

  1. M. Abdulhai, G. Serapio-Garcia, C. Crepy, D. Valter, J. Canny, and N. Jaques. Moral foundations of large language models. arXiv preprint arXiv:2310.15337, 2023. · arXiv
  2. J. P. Agapiou, A. S. Vezhnevets, E. A. Duéñez-Guzmán, J. Matyas, Y. Mao, P. Sunehag, R. Köster, U. Madhushani, K. Kopparapu, R. Comanescu, D. Strouse, M. B. Johanson, S. Singh, J. Haas, I. Mordatch, D. Mobbs, and J. Z. Leibo. Melting pot 2.0. arXiv preprint arXiv:2211.13746, 2022. · arXiv
  3. B. Agüera y Arcas. Do large language models understand us? Daedalus, 151(2):183-197, 2022.
  4. G. V. Aher, R. I. Arriaga, and A. T. Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pages 337-371.
  5. PMLR, 2023.
  6. I. Ajzen. The theory of planned behavior. Organizational behavior and human decision processes, 50 (2):179-211, 1991.
  7. B. Alamar and S. A. Glantz. Effect of increased social unacceptability of cigarette smoking on reduction in cigarette consumption. American journal of public health, 96(8):1359-1363, 2006.
  8. R. Amato, L. Lacasa, A. Díaz-Guilera, and A. Baronchelli. The dynamics of norm change in the cultural evolution of language. Proceedings of the National Academy of Sciences, 115(33):8260-8265, 2018.
  9. A. Amirova, T. Fteropoulli, N. Ahmed, M. R. Cowie, and J. Z. Leibo. Framework-based qualitative analy- sis of free responses of large language models: Algorithmic fidelity. arXiv preprint arXiv:2309.06364, 2023. · arXiv
  10. D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. · arXiv
  11. T. Araujo, N. Helberger, S. Kruikemeier, and C. H. De Vreese. In AI we trust? perceptions about automated decision-making by artificial intelligence. AI & society, 35(3):611-623, 2020.
  12. J. Archer. Testosterone and human aggression: an evaluation of the challenge hypothesis. Neuroscience & Biobehavioral Reviews, 30(3):319-345, 2006.
  13. L. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337-351, 2023.
  14. K. J. Arrow. A difficulty in the concept of social welfare. The journal of political economy, 58(4): 328-346, 1950.
  15. E. Awad, S. Dsouza, R. Kim, J. Schulz, J. Henrich, A. Shariff, J.-F. Bonnefon, and I. Rahwan. The moral machine experiment. Nature, 563(7729):59-64, 2018.
  16. E. Awad, S. Dsouza, A. Shariff, I. Rahwan, and J.-F. Bonnefon. Universals and variations in moral decisions made in 42 countries by 70,000 participants. Proceedings of the National Academy of Sciences, 117(5):2332-2337, 2020.
  17. R. Axelrod. Conflict of interest: An axiomatic approach. The Journal of conflict resolution, 11(1): 87-99, 1967.
  18. B. J. Baars. A cognitive theory of consciousness. Cambridge University Press, 1988.
  19. L. Babcock, M. P. Recalde, L. Vesterlund, and L. Weingart. Gender differences in accepting and receiving requests for tasks with low promotability. American Economic Review, 107(3):714-747, 2017.
  20. A. Baddeley. Working memory. Science, 255(5044):556-559, 1992.
  21. J. Baggio, A. Barnett, I. Perez-Ibarra, U. Brady, E. Ratajczyk, N. Rollins, C. Rubiños, H. Shin, D. Yu, R. Aggarwal, et al. Explaining success and failure in the commons: the configural nature of ostrom's institutional design principles. International Journal of the Commons, 10(2), 2016.
  22. A. Bahramisharif, O. Jensen, J. Jacobs, and J. Lisman. Serial representation of items during working memory maintenance at letter-selective cortical sites. PLoS biology, 16(8):e2003805, 2018.
  23. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McK- innon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022. · arXiv
  24. C. A. Bail, L. P. Argyle, T. W. Brown, J. P. Bumpus, H. Chen, M. F. Hunzaker, J. Lee, M. Mann, F. Merhout, and A. Volfovsky. Exposure to opposing views on social media can increase political polarization. Proceedings of the National Academy of Sciences, 115(37):9216-9221, 2018.
  25. B. Baker, I. Kanitscheider, T. Markov, Y. Wu, G. Powell, B. McGrew, and I. Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations, 2019.
  26. G. Baker, R. Gibbons, and K. J. Murphy. Relational contracts and the theory of the firm. The Quarterly Journal of Economics, 117(1):39-84, 2002.
  27. D. Balduzzi, M. Garnelo, Y. Bachrach, W. Czarnecki, J. Perolat, M. Jaderberg, and T. Graepel. Open- ended learning in symmetric zero-sum games. In International Conference on Machine Learning, pages 434-443. PMLR, 2019.
  28. J. A. Bargh. The four horsemen of automaticity: Awareness, intention, efficiency, and control in social cognition. In Handbook of social cognition, pages 1-40. Psychology Press, 2014.
  29. L. F. Barrett. Solving the emotion paradox: Categorization and the experience of emotion. Personality and social psychology review, 10(1):20-46, 2006.
  30. J. S. Beer, E. A. Heerey, D. Keltner, D. Scabini, and R. T. Knight. The regulatory function of self- conscious emotion: insights from patients with orbitofrontal damage. Journal of personality and social psychology, 85(4):594, 2003.
  31. D. J. Bem. Self-perception: An alternative interpretation of cognitive dissonance phenomena. Psycho- logical review, 74(3):183, 1967.
  32. S. Bergman, N. Marchal, J. Mellor, S. Mohamed, I. Gabriel, and W. Isaac. STELA: a community-centred approach to norm elicitation for AI alignment. Scientific Reports, 14(1):6616, 2024.
  33. A. Bhoopchand, B. Brownfield, A. Collister, A. Dal Lago, A. Edwards, R. Everett, A. Fréchette, Y. G. Oliveira, E. Hughes, K. W. Mathewson, et al. Learning few-shot imitation as cultural transmission. Nature Communications, 14(1):7536, 2023.
  34. C. Bicchieri. The grammar of society: The nature and dynamics of social norms. Cambridge University Press, 2005.
  35. C. Bicchieri. Norms in the wild: How to diagnose, measure, and change social norms. Oxford University Press, 2016.
  36. C. Bicchieri and A. Funcke. Norm change: Trendsetters and social structure. Social Research: An International Quarterly, 85(1):1-21, 2018.
  37. A. Birhane, W. Isaac, V. Prabhakaran, M. Diaz, M. C. Elish, I. Gabriel, and S. Mohamed. Power to the people? opportunities and challenges for participatory AI. Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1-8, 2022.
  38. C. Blundell, B. Uria, A. Pritzel, Y. Li, A. Ruderman, J. Z. Leibo, J. Rae, D. Wierstra, and D. Hassabis. Model-free episodic control. arXiv preprint arXiv:1606.04460, 2016. · arXiv
  39. C. Boehm. Hierarchy in the forest: the evolution of egalitarian behavior. Harvard University Press, 1999.
  40. C. Boehm. Moral origins: The evolution of virtue, altruism, and shame. Soft Skull Press, 2012.
  41. D. Borsa, N. Heess, B. Piot, S. Liu, L. Hasenclever, R. Munos, and O. Pietquin. Observational learning by reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 1117-1124, 2019.
  42. N. Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., USA, 1st edition, 2014. ISBN 0199678111.
  43. P. Bourdieu. Distinction a social critique of the judgement of taste. In Inequality, pages 287-318. Routledge, 1984/2018.
  44. S. Bowles. Endogenous preferences: The cultural consequences of markets and other economic institutions. Journal of economic literature, 36(1):75-111, 1998.
  45. S. Bowles. Microeconomics: Behavior, Institutions, and Evolution. Princeton University Press, Princeton, 2004.
  46. R. Boyd and P. J. Richerson. Punishment allows the evolution of cooperation (or anything else) in sizable groups. Ethology and sociobiology, 13(3):171-195, 1992.
  47. R. Boyd, P. J. Richerson, and J. Henrich. The cultural niche: Why social learning is essential for human adaptation. Proceedings of the National Academy of Sciences, 108(supplement_2):10918-10925, 2011.
  48. J. Brand, A. Israeli, and D. Ngwe. Using GPT for market research. Available at SSRN 4395751, 2023.
  49. M. B. Brewer. The social self: On being the same and different at the same time. Personality and Social Psychology Bulletin, 17(5):475-482, 1991.
  50. P. R. Brewer. Public opinion about gay rights and gay marriage. International Journal of Public Opinion Research, 26(3):279-282, 2014.
  51. J. L. Briggs. Never in anger: Portrait of an Eskimo family. Harvard University Press, 1971.
  52. L. Brinkmann, F. Baumann, J.-F. Bonnefon, M. Derex, T. F. Müller, A.-M. Nussberger, A. Czaplicka, A. Acerbi, T. L. Griffiths, J. Henrich, J. Z. Leibo, R. McElreath, P.-Y. Oudeyer, J. Stray, and I. Rahwan. Machine culture. Nature Human Behaviour, pages 1-14, 2023.
  53. S. F. Brosnan and F. B. de Waal. Evolution of responses to (un) fairness. Science, 346(6207):1251776, 2014.
  54. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
  55. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. · arXiv
  56. J. W. Buckholtz, C. L. Asplund, P. E. Dux, D. H. Zald, J. C. Gore, O. D. Jones, and R. Marois. The neural correlates of third-party punishment. Neuron, 60(5):930-940, 2008.
  57. K. Carley. A theory of group stability. American sociological review, 56(3):331-354, 1991.
  58. M. Carroll, D. Foote, A. Siththaranjan, S. Russell, and A. Dragan. AI alignment with changing and influenceable reward functions. arXiv preprint arXiv:2405.17713, 2024. · arXiv
  59. P. Carruthers. The opacity of mind: An integrative theory of self-knowledge. OUP Oxford, 2011.
  60. C. E. Case and A. M. Greeley. Attitudes toward racial equality. Humboldt Journal of Social Relations, pages 67-94, 1990.
  61. F. Casoria, F. Galeotti, and M. C. Villeval. Perceived social norm and behavior quickly adjusted to legal changes during the COVID-19 pandemic. Journal of Economic Behavior & Organization, 190: 54-65, 2021.
  62. D. Centola, J. Becker, D. Brackbill, and A. Baronchelli. Experimental evidence for tipping points in social convention. Science, 360(6393):1116-1119, 2018.
  63. L. J. Chang and A. G. Sanfey. Great expectations: neural computations underlying the use of social norms in decision-making. Social cognitive and affective neuroscience, 8(3):277-284, 2013.
  64. S. Chiappa and W. S. Isaac. A causal bayesian networks viewpoint on fairness. Privacy and Identity Management. Fairness, Accountability, and Transparency in the Age of Big Data: 13th IFIP WG 9.2, 9.6/11.7, 11.6/SIG 9.2. 2 International Summer School, Vienna, Austria, August 20-24, 2018, Revised Selected Papers 13, pages 3-20, 2019.
  65. D. D. Choi, M. Poertner, and N. Sambanis. Parochialism, social norms, and discrimination against immigrants. Proceedings of the National Academy of Sciences, 116(33):16274-16279, 2019.
  66. M. Chudek and J. Henrich. Culture-gene coevolution, norm-psychology and the emergence of human prosociality. Trends in cognitive sciences, 15(5):218-226, 2011.
  67. A. Chung and R. N. Rimal. Social norms: A review. Review of Communication Research, 4:1-28, 2016.
  68. R. B. Cialdini, C. A. Kallgren, and R. R. Reno. A focus theory of normative conduct: A theoretical refinement and reevaluation of the role of norms in human behavior. In Advances in experimental social psychology, volume 24, pages 201-234. Elsevier, 1991.
  69. S. A. Clancy. Abducted: How people come to believe they were kidnapped by aliens. Harvard University Press, 2005.
  70. F. Clément, S. Bernard, and L. Kaufmann. Social cognition is not reducible to theory of mind: When children use deontic rules to predict the behaviour of others. British Journal of Developmental Psychology, 29(4):910-928, 2011.
  71. J. Cook, C. Lu, E. Hughes, J. Z. Leibo, and J. Foerster. Artificial generational intelligence: Cultural accumulation in reinforcement learning. Advances in neural information processing systems, 37, 2024.
  72. M. C. Corballis. Laterality and myth. American Psychologist, 35(3):284, 1980.
  73. C. S. Crandall, J. M. Miller, and M. H. White. Changing norms following the 2016 US presidential election: The Trump effect on prejudice. Social Psychological and Personality Science, 9(2):186-192, 2018.
  74. C. Crivelli, S. Jarillo, J. A. Russell, and J.-M. Fernández-Dols. Reading emotions from faces in two indigenous societies. Journal of Experimental Psychology: General, 145(7):830, 2016.
  75. M. J. Crockett, L. Clark, M. D. Lieberman, G. Tabibnia, and T. W. Robbins. Impulsive choice and altruistic punishment are correlated and increase in tandem with serotonin depletion. Emotion, 10 (6):855, 2010.
  76. M. J. Crockett, A. Apergis-Schoute, B. Herrmann, M. D. Lieberman, U. Müller, T. W. Robbins, and L. Clark. Serotonin modulates striatal responses to fairness and retaliation in humans. Journal of neuroscience, 33(8):3505-3513, 2013.
  77. W. M. Czarnecki, G. Gidel, B. Tracey, K. Tuyls, S. Omidshafiei, D. Balduzzi, and M. Jaderberg. Real world games look like spinning tops. Advances in Neural Information Processing Systems, 33: 17443-17454, 2020.
  78. A. Dafoe, E. Hughes, Y. Bachrach, T. Collins, K. R. McKee, J. Z. Leibo, K. Larson, and T. Graepel. Open problems in cooperative AI. arXiv preprint arXiv:2012.08630, 2020. · arXiv
  79. N. D. Daw, Y. Niv, and P. Dayan. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature neuroscience, 8(12):1704-1711, 2005.
  80. R. Dawkins. The selfish gene. Oxford university press, 1976.
  81. P. Dayan and N. D. Daw. Decision theory, reinforcement learning, and the brain. Cognitive, Affective, & Behavioral Neuroscience, 8(4):429-453, 2008.
  82. D. J.-F. De Quervain, U. Fischbacher, V. Treyer, M. Schellhammer, U. Schnyder, A. Buck, and E. Fehr. The neural basis of altruistic punishment. Science, 305(5688):1254-1258, 2004.
  83. S. Dehaene, M. Kerszberg, and J.-P. Changeux. A neuronal model of a global workspace in effortful cognitive tasks. Proceedings of the national Academy of Sciences, 95(24):14529-14534, 1998.
  84. M. Derex, J.-F. Bonnefon, R. Boyd, and A. Mesoudi. Causal understanding is not necessary for the improvement of culturally evolving technology. Nature human behaviour, 3(5):446-452, 2019.
  85. J. Dewey. The historic background of corporate legal personality. Yale Law Journal, 35:655, 1925.
  86. M. Díaz, R. Amironesei, L. Weidinger, and I. Gabriel. Accounting for offensive speech as a practice of resistance. In Proceedings of the sixth workshop on online abuse and harms (WOAH), pages 192-202, 2022.
  87. Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022. · arXiv
  88. E. A. Duéñez-Guzmán, K. R. McKee, Y. Mao, B. Coppin, S. Chiappa, A. S. Vezhnevets, M. A. Bakker, Y. Bachrach, S. Sadedin, W. Isaac, et al. Statistical discrimination in learning agents. arXiv preprint arXiv:2110.11404, 2021. · arXiv
  89. E. A. Duéñez-Guzmán, S. Sadedin, J. X. Wang, K. R. McKee, and J. Z. Leibo. A social path to human-like artificial intelligence. Nature Machine Intelligence, 5(11):1181-1188, 2023.
  90. R. I. Dunbar and S. Shultz. Why are there so many explanations for primate brain evolution? Philosophical Transactions of the Royal Society B: Biological Sciences, 372(1727):20160244, 2017.
  91. R. I. M. Dunbar. Grooming, gossip, and the evolution of language. Harvard University Press, 1996.
  92. R. I. M. Dunbar. The social brain hypothesis. Evol. Anthropol., 6(5):178-190, 1998.
  93. M. Duverger. Political parties : their organization and activity in the modern state. Methuen & Co. ; John Wiley & Sons London, New York, London, New York, 1954.
  94. B. D. Earp, K. L. McLoughlin, J. T. Monrad, M. S. Clark, and M. J. Crockett. How social relationships shape moral wrongness judgments. Nature communications, 12(1):5776, 2021.
  95. J. A. Everett, Z. Ingbretsen, F. Cushman, and M. Cikara. Deliberation erodes cooperative behav- ior-even towards competitive out-groups, even when using a control condition, and even when eliminating selection bias. Journal of Experimental Social Psychology, 73:76-81, 2017.
  96. F. Faal, K. Schmitt, and J. Y. Yu. Reward modeling for mitigating toxicity in transformer-based language models. Applied Intelligence, 53(7):8421-8435, 2023.
  97. E. Fehr and U. Fischbacher. Third-party punishment and social norms. Evolution and human behavior, 25(2):63-87, 2004.
  98. E. Fehr and S. Gächter. Altruistic punishment in humans. Nature, 415(6868):137-140, 2002.
  99. E. Fehr and K. M. Schmidt. A theory of fairness, competition, and cooperation. The Quarterly Journal of Economics, 114(3):817-868, 1999. doi: 10.1162/003355399556151. · DOI · Paper page
  100. B. Felbo, A. Mislove, A. Søgaard, I. Rahwan, and S. Lehmann. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2017.
  101. M. Finnemore and K. Sikkink. International norm dynamics and political change. International organization, 52(4):887-917, 1998.
  102. G. Firebaugh and K. E. Davis. Trends in antiblack prejudice, 1972-1984: Region and cohort effects. American Journal of Sociology, 94(2):251-272, 1988.
  103. A. P. Fiske. The four elementary forms of sociality: framework for a unified theory of social relations. Psychological review, 99(4):689, 1992.
  104. P. Foot. The problem of abortion and the doctrine of double effect. Oxford, 5:5-15, 1967.
  105. J. G. Foster. From thin to thick: Toward a politics of human-compatible AI. Public Culture, 35(3): 417-430, 2023.
  106. P. Fries. Rhythms for cognition: communication through coherence. Neuron, 88(1):220-235, 2015.
  107. I. Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411-437, 2020.
  108. A. P. Gantman and J. J. Van Bavel. The moral pop-out effect: Enhanced perceptual awareness of morally relevant stimuli. Cognition, 132(1):22-29, 2014.
  109. R. K. Garrett, S. D. Gvirsman, B. K. Johnson, Y. Tsfati, R. Neo, and A. Dal. Implications of pro-and counterattitudinal information exposure for affective polarization. Human communication research, 40(3):309-332, 2014.
  110. M. J. Gelfand. Cultural evolutionary mismatches in response to collective threat. Current Directions in Psychological Science, 30(5):401-409, 2021.
  111. M. J. Gelfand, J. L. Raver, L. Nishii, L. M. Leslie, J. Lun, B. C. Lim, L. Duan, A. Almaliach, S. Ang, J. Arnadottir, et al. Differences between tight and loose cultures: A 33-nation study. Science, 332 (6033):1100-1104, 2011.
  112. M. J. Gelfand, S. Gavrilets, and N. Nunn. Norm dynamics: interdisciplinary perspectives on social norm emergence, persistence, and change. Annual Review of Psychology, 75:341-378, 2024.
  113. H. Gintis, E. A. Smith, and S. Bowles. Costly signaling and cooperation. Journal of theoretical biology, 213(1):103-119, 2001.
  114. A. Glaese, N. McAleese, M. Trębacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, L. Campbell-Gillingham, J. Uesato, P.-S. Huang, R. Comanescu, F. Yang, A. See, S. Dathathri, R. Greig, C. Chen, D. Fritz, J. Sanchez Elias, R. Green, S. Mokrá, N. Fernando, B. Wu, R. Foley, S. Young, I. Gabriel, W. Isaac, J. Mellor, D. Hassabis, K. Kavukcuoglu, L. A. Hendricks, and G. Irving. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. · arXiv
  115. M. S. Gobel and Y. Miyamoto. Self-and other-orientation in high rank: A cultural psychological approach to social hierarchy. Personality and Social Psychology Review, page 10888683231172252, 2023.
  116. A. Goldstein, Z. Zada, E. Buchnik, M. Schain, A. Price, B. Aubrey, S. A. Nastase, A. Feder, D. Emanuel, A. Cohen, A. Jansen, H. Gazula, G. Choe, A. Rao, C. Kim, C. Casto, L. Fanda, W. Doyle, D. Friedman, P. Dugan, L. Melloni, R. Reichart, S. Devore, A. Fliner, L. Hasenfratz, O. Levy, A. Hassidim, M. Brenner, Y. Matias, K. A. Norman, O. Devinsky, and U. Hasson. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3):369-380, 2022.
  117. D. Graeber. Debt: The first 5000 years. Penguin UK, 2012.
  118. J. D. Greene and J. M. Paxton. Patterns of neural activity associated with honest and dishonest moral decisions. Proceedings of the National Academy of Sciences, 106(30):12506-12511, 2009.
  119. A. Greif. Institutions and the Path to the Modern Economy: Lessons from Medieval Trade. Cambridge University Press, Cambridge, 2006.
  120. A. Greif, P. Milgrom, and B. R. Weingast. Coordination, commitment, and enforcement: The case of the merchant guild. Journal of political economy, 102(4):745-776, 1994.
  121. H. P. Grice. Meaning. The philosophical review, 66(3):377-388, 1957.
  122. F. Guala. Understanding institutions. In Understanding Institutions. Princeton University Press, 2016.
  123. G. K. Hadfield. Rules for a flat world: Why humans invented law and how to reinvent it for a complex global economy. Oxford University Press, 2017.
  124. G. K. Hadfield and B. R. Weingast. What is law? a coordination model of the characteristics of legal order. Journal of Legal Analysis, 4(2):471-514, 2012.
  125. G. K. Hadfield and B. R. Weingast. Law without the state: legal attributes and the coordination of decentralized collective punishment. Journal of Law and Courts, 1(1):3-34, 2013.
  126. G. K. Hadfield and B. R. Weingast. Microfoundations of the rule of law. Annual Review of Political Science, 17:21-42, 2014.
  127. D. Hadfield-Menell and G. K. Hadfield. Incomplete contracting and ai alignment. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 417-422, 2019.
  128. D. Hadfield-Menell, S. Milli, P. Abbeel, S. J. Russell, and A. Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
  129. J. Haidt. The emotional dog and its rational tail: a social intuitionist approach to moral judgment. Psychological review, 108(4):814, 2001.
  130. J. Haidt, S. H. Koller, and M. G. Dias. Affect, culture, and morality, or is it wrong to eat your dog? Journal of personality and social psychology, 65(4):613, 1993.
  131. J. Haidt et al. The moral emotions. Handbook of affective sciences, 11(2003):852-870, 2003.
  132. P. Hammond. Interpersonal Comparisons of Utility: Why and How They are and Should be Made. In J. Elster and J. E. Roemer, editors, Interpersonal Comparisons of Well-Being. Cambridge University Press, 1991.
  133. C. L. Harenski, K. A. Harenski, M. S. Shane, and K. A. Kiehl. Aberrant neural processing of moral violations in criminal psychopaths. Journal of abnormal psychology, 119(4):863, 2010.
  134. J. M. Harlow. Passage of an iron rod through the head. The Boston Medical and Surgical Journal (1828-1851), 39(20):0_1, 1848.
  135. J. A. Harris, R. Boyd, and B. M. Wood. The role of causal knowledge in the evolution of traditional technology. Current Biology, 31(8):1798-1803, 2021.
  136. H. L. A. Hart. The concept of law. Oxford University Press, 1961/2012.
  137. J. Haugeland. Heidegger on being a person. Nous, pages 15-26, 1982.
  138. D. M. Hausman. The Impossibility of Interpersonal Utility Comparisons. Mind; a quarterly review of psychology and philosophy, 104(415):473-490, 1995.
  139. F. A. Hayek. The use of knowledge in society. The American Economic Review, 35(4):519-530, 1945.
  140. F. A. Hayek. Individualism and Economic Order. University of Chicago Press, 1948.
  141. D. O. Hebb. The organization of behavior: A neuropsychological theory. Psychology press, 1949.
  142. D. D. Heckathorn. The dynamics and dilemmas of collective action. American sociological review, pages 250-277, 1996.
  143. E. M. Heffernan, M. L. Schlichting, and M. L. Mack. Learning exceptions to the rule in human and model via hippocampal encoding. Scientific Reports, 11(1):21429, 2021.
  144. N. Helberger, T. Araujo, and C. H. de Vreese. Who is the fairest of them all? public attitudes and expectations regarding automated decision-making. Computer Law & Security Review, 39:105456, 2020.
  145. D. Hendrycks, C. Burns, S. Basart, A. C. Critch, J. L. Li, D. Song, and J. Steinhardt. Aligning AI with shared human values. In International Conference on Learning Representations, 2021.
  146. J. Henrich. Cultural group selection, coevolutionary processes and large-scale cooperation. Journal of Economic Behavior & Organization, 53(1):3-35, 2004.
  147. J. Henrich. The evolution of costly displays, cooperation and religion: Credibility enhancing displays and their implications for cultural evolution. Evolution and human behavior, 30(4):244-260, 2009.
  148. J. Henrich. The Secret of Our Success: How Culture is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter. Princeton University Press, 2017.
  149. J. Henrich. Cultural evolution: Is causal inference the secret of our success? Current Biology, 31(8): R381-R383, 2021.
  150. J. Henrich and F. J. Gil-White. The evolution of prestige: Freely conferred deference as a mechanism for enhancing the benefits of cultural transmission. Evolution and human behavior, 22(3):165-196, 2001.
  151. J. Henrich, M. Chudek, and R. Boyd. The big man mechanism: how prestige fosters cooperation and creates prosocial leaders. Philosophical Transactions of the Royal Society B: Biological Sciences, 370 (1683):20150013, 2015.
  152. M. A. Hogg, D. J. Terry, and K. M. White. A tale of two theories: A critical comparison of identity theory with social identity theory. Social psychology quarterly, pages 255-269, 1995.
  153. S. Hong, X. Zheng, J. Chen, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023. · arXiv
  154. J. J. Horton. Large language models as simulated economic agents: What can we learn from homo silicus? arXiv preprint arXiv:2301.07543, 2023. · arXiv
  155. D. Howarth and S. W. Stark. HLA Hart's secondary rules: what do 'officials' really think? International Journal of Law in Context, 14(1):61-86, 2018.
  156. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
  157. W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. · arXiv
  158. J. Huber, J. W. Payne, and C. Puto. Adding asymmetrically dominated alternatives: Violations of regularity and the similarity hypothesis. The journal of consumer research, 9(1):90-98, 1982.
  159. E. Hughes, J. Z. Leibo, M. Phillips, K. Tuyls, E. Dueñez-Guzman, A. García Castañeda, I. Dunning, T. Zhu, K. McKee, R. Koster, H. Roff, and T. Graepel. Inequity aversion improves cooperation in intertemporal social dilemmas. Advances in neural information processing systems, 31, 2018.
  160. C. Humphrey and J. Laidlaw. Archetypal actions of ritual: A theory of ritual illustrated by the jain rite of worship. Oxford Studies in Social and Cultural Anthropology, 1994.
  161. S. Huttegger, B. Skyrms, P. Tarres, and E. Wagner. Some dynamics of signaling games. Proceedings of the National Academy of Sciences, 111(supplement_3):10873-10880, 2014.
  162. S. Iyengar, G. Sood, and Y. Lelkes. Affect, not ideology: A social identity perspective on polarization. Public opinion quarterly, 76(3):405-431, 2012.
  163. K. Izuma, D. N. Saito, and N. Sadato. Processing of social and monetary rewards in the human striatum. Neuron, 58(2):284-294, 2008.
  164. M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz, A. S. Morcos, A. Ruderman, N. Sonnerat, T. Green, L. Deason, J. Z. Leibo, D. Silver, D. Hassabis, K. Kavukcuoglu, and T. Graepel. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, 364(6443):859-865, 2019.
  165. N. Jaques, J. H. Shen, A. Ghandeharioun, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3985-4003, 2020.
  166. O. Jensen and J. E. Lisman. Novel lists of 7+/-2 known items can be reliably stored in an oscillatory short-term memory network: interaction with long-term memory. Learning & Memory, 3(2-3): 257-263, 1996.
  167. M. B. Johanson, E. Hughes, F. Timbers, and J. Z. Leibo. Emergent bartering behaviour in multi-agent reinforcement learning. arXiv preprint arXiv:2205.06760, 2022. · arXiv
  168. J. J. Jordan, M. Hoffman, P. Bloom, and D. G. Rand. Third-party punishment as a costly signal of trustworthiness. Nature, 530(7591):473-476, 2016.
  169. D. M. Kahan. Ideology, motivated reasoning, and cognitive reflection. Judgment and Decision making, 8(4):407-424, 2013.
  170. P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1-210, 2021.
  171. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. · arXiv
  172. D. Katz. The functional approach to the study of attitudes. Public opinion quarterly, 24(2):163-204, 1960.
  173. D. Keller. Lawful but awful? control over legal speech by platforms, governments, and internet users. University of Chicago Law Review, page 1, 2022.
  174. B. Kenward. Over-imitating preschoolers believe unnecessary actions are normative and enforce their performance by a third party. Journal of experimental child psychology, 112(2):195-207, 2012.
  175. Y. Kim. How cross-cutting news exposure relates to candidate issue stance knowledge, political polarization, and participation: The moderating role of political sophistication. International Journal of Public Opinion Research, 31(4):626-648, 2019.
  176. H. R. Kirk, B. Vidgen, P. Röttger, and S. A. Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, pages 1-10, 2024.
  177. M. Klissarov, P. D'Oro, S. Sodhani, R. Raileanu, P.-L. Bacon, P. Vincent, A. Zhang, and M. Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv [cs.AI], Sept. 2023.
  178. L. Kohlberg, C. Levine, and A. Hewer. Moral stages: A current formulation and a response to critics. Contributions to Human Development, 10(74), 1983.
  179. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
  180. R. Kor-Sins. The alt-right digital migration: A heterogeneous engineering approach to social media platform branding. New Media & Society, 25(9):2321-2338, Sept. 2023.
  181. R. Köster, K. R. McKee, R. Everett, L. Weidinger, W. S. Isaac, E. Hughes, E. A. Duéñez-Guzmán, T. Graepel, M. Botvinick, and J. Z. Leibo. Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences. arXiv preprint arXiv:2010.09054, 2020. · arXiv
  182. R. Koster, J. Balaguer, A. Tacchetti, A. Weinstein, T. Zhu, O. Hauser, D. Williams, L. Campbell- Gillingham, P. Thacker, M. Botvinick, et al. Human-centred mechanism design with democratic ai. Nature Human Behaviour, 6(10):1398-1407, 2022.
  183. R. Köster, D. Hadfield-Menell, R. Everett, L. Weidinger, G. K. Hadfield, and J. Z. Leibo. Spurious nor- mativity enhances learning of compliance and enforcement behavior in artificial agents. Proceedings of the National Academy of Sciences, 119(3):e2106028118, 2022.
  184. E. Kubin and C. von Sikorski. The role of (social) media in political polarization: a systematic review. Annals of the International Communication Association, 45(3):188-206, 2021.
  185. T. S. Kuhn. The structure of scientific revolutions. University of Chicago press Chicago, 1962.
  186. A. K. Lampinen, I. Dasgupta, S. C. Chan, H. R. Sheahan, A. Creswell, D. Kumaran, J. L. McClelland, and F. Hill. Language models show human-like content effects on reasoning tasks. arXiv preprint arXiv:2207.07051, 2022. · arXiv
  187. S. Lazar. Governing the algorithmic city. arXiv preprint arXiv:2410.20720, 2024. · arXiv
  188. P. Lee. Learning from Tay's introduction. Official Microsoft Blog, 25(03), 2016.
  189. J. Z. Leibo, E. Hughes, M. Lanctot, and T. Graepel. Autocurricula and the emergence of innova- tion from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXiv:1903.00742, 2019a. · arXiv
  190. J. Z. Leibo, J. Perolat, E. Hughes, S. Wheelwright, A. H. Marblestone, E. Duéñez-Guzmán, P. Sune- hag, I. Dunning, and T. Graepel. Malthusian reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 1099-1107, 2019b.
  191. J. Leike, M. Martic, V. Krakovna, P. A. Ortega, T. Everitt, A. Lefrancq, L. Orseau, and S. Legg. AI safety gridworlds. arXiv preprint arXiv:1711.09883, 2017. · arXiv
  192. E. H. Lenneberg. Biological foundations of language. Wiley, 1967.
  193. D. Lewis. Convention: A philosophical study. John Wiley & Sons, 1969.
  194. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
  195. C. Li, Z. Gan, Z. Yang, J. Yang, L. Li, L. Wang, and J. Gao. Multimodal foundation models: From specialists to general-purpose assistants. arXiv preprint arXiv:2309.10020, 10, 2023a. · arXiv
  196. G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. CAMEL: Communicative agents for "mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991-52008, 2023b.
  197. B. Lindström, S. Jangard, I. Selbing, and A. Olsson. The role of a "common is moral" heuristic in the stability and change of moral norms. Journal of Experimental Psychology: General, 147(2):228, 2018.
  198. E. Machery and S. Stich. The Moral/Conventional Distinction. In E. N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2022 edition, 2022.
  199. J. L. Mackie. Ethics: Inventing Right and Wrong. Penguin, 1977.
  200. Y. Mao, M. G. Reinecke, M. Kunesch, E. A. Duéñez-Guzmán, R. Comanescu, J. Haas, and J. Z. Leibo. Doing the right thing for the right reason: Evaluating artificial moral cognition by probing cost insensitivity. arXiv preprint arXiv:2305.18269, 2023. · arXiv

200 of 378 shown

Authors
Leibo, J. Z., Vezhnevets, A. S., Diaz, M., Agapiou, J. P., Cunningham, W. A., Sunehag, P., Haas, J., Koster, R., Duéñez-Guzmán, E. A., Isaac, W. S., Piliouras, G., Bileschi, S. M., Rahwan, I., & Osindero, S.
Affiliations
  • Google DeepMind (United Kingdom) (GB)
  • Mila - Quebec Artificial Intelligence Institute (CA)
  • University of Toronto (CA)
  • Max Planck Institute for Human Development (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2412.19010
arXiv
2412.19010
OpenAlex
W4405901318
Cited by
1
References
0
Citations by year
2025: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Cybernetics and Technology in Society
Keywords
appropriateness, social norms, generative AI, AI decision-making, human judgment, social cognition, responsible AI deployment, context-dependent behavior, norm change

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review