Moral Machine or Tyranny of the Majority?
Open accessAbstract
With Artificial Intelligence systems increasingly applied in consequential domains, researchers have begun to ask how these systems ought to act in ethically charged situations where even humans lack consensus. In the Moral Machine project, researchers crowdsourced answers to "Trolley Problems" concerning autonomous vehicles. Subsequently, Noothigattu et al. (2018) proposed inferring linear functions that approximate each individual's preferences and aggregating these linear models by averaging parameters across the population. In this paper, we examine this averaging mechanism, focusing on fairness concerns in the presence of strategic effects. We investigate a simple setting where the population consists of two groups, with the minority constituting an α < 0.5 share of the population. To simplify the analysis, we consider the extreme case in which within-group preferences are homogeneous. Focusing on the fraction of contested cases where the minority group prevails, we make the following observations: (a) even when all parties report their preferences truthfully, the fraction of disputes where the minority prevails is less than proportionate in α; (b) the degree of sub-proportionality grows more severe as the level of disagreement between the groups increases; (c) when parties report preferences strategically, pure strategy equilibria do not always exist; and (d) whenever a pure strategy equilibrium exists, the majority group prevails 100% of the time. These findings raise concerns about stability and fairness of preference vector averaging as a mechanism for aggregating diverging voices. Finally, we discuss alternatives, including randomized dictatorship and median-based mechanisms.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Traditional alignment strategies that aggregate over diverse preferences exacerbate value monism by collapsing genuine disagreements into a single normative stance.
“Traditional alignment strategies that aggregate over diverse preferences have been shown to exacerbate this issue (Casper et al., 2023;Kaufmann et al., 2024;Feffer et al., 2023), collapsing genuine disagreements (Durmus et al., 2024;Sorensen et al., 2024a;Bakker et al., 2022;AlKhamissi et al., 2024;Ryan et al., 2024) into a single normative stance-an issue known as value monism (Gabriel, 2020).”
From Benchmarking Overton Pluralism in LLMs (2025) · cites this paper as Feffer et al., 2023
The Introduction (page unavailable) warns that averaging preference vectors can give minority preferences less than proportional influence and urges addressing normative disagreements rather than simply passing them through an aggregation mechanism. This supports the concern behind the citation, but the paper analyzes a specific participatory-ML mechanism in a stylized autonomous-vehicle setting, not traditional alignment strategies generally. The citing sentence attributes the term “value monism” to Gabriel, not Feffer et al. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 30
- Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.-F., & Rahwan, I. (2018). The moral machine experiment. Nature, 563 (7729), 59-64.
- Brill, M., & Conitzer, V. (2015). Strategic voting and strategic candidacy. In Association for the Advancement of Artificial Intelligence (AAAI).
- Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. In ACM Conference on Fairness, Accountability and Transparency (FAccT).
- Chamberlin, J. R. (1985). An investigation into the relative manipulability of four voting systems. Behavioral Science, 30 (4), 195-203.
- Conitzer, V., Brill, M., & Freeman, R. (2015). Crowdsourcing societal tradeoffs. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS).
- Conitzer, V., Freeman, R., Brill, M., & Li, Y. (2016). Rules for choosing societal tradeoffs. In Association for the Advancement of Artificial Intelligence (AAAI).
- El-Mhamdi, E.-M., Farhadkhani, S., Guerraoui, R., & Hoang, L.-N. (2021). On the strate- gyproofness of the geometric median. arXiv preprint arXiv:2106.02394 . · arXiv
- Gibbard, A. (1973). Manipulation of voting schemes: a general result. Econometrica: Journal of the Econometric Society, (pp. 587-601).
- Gibbard, A. (1977). Manipulation of schemes that mix voting with chance. Econometrica: Journal of the Econometric Society, (pp. 665-681).
- Hiranandani, G., Mathur, J., Narasimhan, H., & Koyejo, O. (2020). Quadratic metric elicitation for fairness and beyond. arXiv preprint arXiv:2011.01516 . · arXiv
- Hiranandani, G., Narasimhan, H., & Koyejo, S. (2020). Fair performance metric elicitation. In Advances in Neural Information Processing Systems (NeurIPS).
- Hurley, W., & Lior, D. (2002). Combining expert judgment: On the performance of trimmed mean vote aggregation procedures in the presence of strategic voting. European Journal of Operational Research, 140 (1), 142-147.
- Johnston, C. M., Blessenohl, S., & Vayanos, P. (2020). Preference elicitation and aggregation to aid with patient triage during the covid-19 pandemic. In ICML Workshop on Participatory Approaches to Machine Learning.
- Kemmer, R., Yoo, Y., Escobedo, A., & Maciejewski, R. (2020). Enhancing collective estimates by aggregating cardinal and ordinal inputs. In AAAI Conference on Human Computation and Crowdsourcing (HCOMP).
- Krishna, V., & Morgan, J. (2012). Voluntary voting: Costs and benefits. Journal of Economic Theory, 147 (6), 2083-2123.
- Landemore, H., & Page, S. E. (2015). Deliberation and disagreement: Problem solving, prediction, and positive dissensus. Politics, philosophy & economics, 14 (3), 229-254.
- Lee, M. K., Kusbit, D., Kahng, A., Kim, J. T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al. (2019). Webuildai: Participatory framework for algorithmic governance. In ACM Conference on Computer-Supported Cooperative Work And Social Computing (CSCW).
- Marchese, C., & Montefiori, M. (2011). Strategy versus sincerity in mean voting. Journal of Economic Psychology, 32 (1), 93-102.
- Moulin, H. (1980). On strategy-proofness and single peakedness. Public Choice, 35 (4), 437-455.
- Noothigattu, R., Gaikwad, S., Awad, E., Dsouza, S., Rahwan, I., Ravikumar, P., & Procaccia, A. (2018). A voting-based system for ethical decision making. In Association for the Advancement of Artificial Intelligence (AAAI).
- Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366 (6464), 447-453.
- Pierson, E. (2017). Demographics and discussion influence views on algorithmic fairness. arXiv preprint arXiv:1712.09124 . · arXiv
- Pugnetti, C., & Schläpfer, R. (2018). Customer preferences and implicit tradeoffs in accident scenarios for self-driving vehicle algorithms. Journal of Risk and Financial Management, 11 (2), 28.
- Renault, R., & Trannoy, A. (2005). Protecting minorities through the average voting rule. Journal of Public Economic Theory, 7 (2), 169-199.
- Renault, R., & Trannoy, A. (2011). Assessing the extent of strategic manipulation: the average vote example. SERIEs, 2 (4), 497-513.
- Sui, X., & Boutilier, C. (2015). Approximately strategy-proof mechanisms for (constrained) facility location. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS).
- Thomson, J. J. (1985). The trolley problem. The Yale Law Journal , 94 (6), 1395-1415.
- Wang, T., Zhao, J., Yu, H., Liu, J., Yang, X., Ren, X., & Shi, S. (2019). Privacy-preserving crowd-guided ai decision-making in ethical dilemmas. In ACM International Conference on Information and Knowledge Management (CIKM).
- Zeckhauser, R. (1973). Voting systems, honest preferences and pareto optimality. American Political Science Review , 67 (3), 934-946.
- Zhang, H., Cheng, Y., & Conitzer, V. (2019). A better algorithm for societal tradeoffs. In Association for the Advancement of Artificial Intelligence (AAAI).
Cited by
Sorted by citations
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
“As a result, when preferences differ, the majority wins, potentially disadvantaging under-represented groups [Prabhakaran et al., 2021, Feffer et al., 2023, Kirk et al., 2023].”
BackgroundBenchmarking Overton Pluralism in LLMs
Poole-Dayan, E., Wu, J., Sorensen, T., Pei, J., & Bakker, M. A. (2025) · arXiv · doi:10.48550/arxiv.2512.01351
“…alignment strategies that aggregate over diverse preferences have been shown to exacerbate this issue (Casper et al., 2023; Kaufmann et al., 2024; Feffer et al., 2023), collapsing genuine disagreements (Durmus et al., 2024; Sorensen et al., 2024a; Bakker et al., 2022; AlKhamissi et al., 2024;…”
- Authors
- Feffer, M., Heidari, H., & Lipton, Z. C.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Volume
- 37
- Issue
- 5
- Pages
- 5974-5982
- ISSN
- 2159-5399 (ISSN-L)
- arXiv
- 2305.17319
- OpenAlex
- W4382239409
- Cited by
- 11
- References
- 0
- Citations by year
- 2025: 6 · 2024: 4 · 2023: 1
- FWCI
- 12.28
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2040929)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
- Keywords
- moral machines, trolley problem, autonomous vehicles, preference aggregation, strategic reporting, pure Nash equilibrium, fairness, AI ethics, ethical dilemmas
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
