CoChat
Sign Up
Cited by 10Ethics and Social Impacts of AIFeminist Epistemology and Gender StudiesExplainable Artificial Intelligence (XAI)

The Value of Disagreement in AI Design, Evaluation, and Alignment

Open access
Fazelpour, Sina, Fleisher, Will·2025·Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency
DOI 10.1145/3715275.3732146
Open this paper in CoChat — free accountOpen PDF

Abstract

Disagreements are widespread across the design, evaluation, and alignment pipelines of artificial intelligence (AI) systems.Yet, standard practices in AI development often obscure or eliminate disagreement, resulting in an engineered homogenization that can be epistemically and ethically harmful, particularly for marginalized groups.In this paper, we characterize this risk, and develop a normative framework to guide practical reasoning about disagreement in the AI lifecycle.Our contributions are two-fold.First, we introduce the notion of perspectival homogenization, characterizing it as a coupled ethical-epistemic risk that arises when an aspect of an AI system's development unjustifiably suppresses disagreement and diversity of perspectives.We argue that perspectival homogenization is best understood as a procedural risk, which calls for targeted interventions throughout the AI development pipeline.Second, we propose a normative framework to guide such interventions, grounded in lines of research that explain why disagreement can be epistemically beneficial, and how its benefits can be realized in practice.We apply this framework to key design questions across three stages of AI development tasks: when disagreement is epistemically valuable; whose perspectives should be included and preserved; how to structure tasks and navigate trade-offs; and how disagreement should be documented and communicated.In doing so, we challenge common assumptions in AI practice, offer a principled foundation for emerging participatory and pluralistic approaches, and identify actionable pathways for future work in AI design and governance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3715275.3732146

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent work highlights risks of homogenization and loss of minority viewpoints.

    “Recent work highlights risks of homogenization and loss of minority viewpoints (Fazelpour & Fleisher, 2025).”

    From On Epistemic Diversity in Large Language Models (2026) · cites this paper as Fazelpour & Fleisher, 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 110

  1. Lama Ahmad, Sandhini Agarwal, Michael Lampe, and Pamela Mishkin. 2025. OpenAI's Approach to External Red Teaming for AI Models and Systems. arXiv preprint arXiv:2503.16431 (2025). · arXiv
  2. Elizabeth Anderson. 2006. The epistemology of democracy. Episteme 3, 1-2 (2006), 8-22.
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022). · arXiv
  4. Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2023. Fairness and machine learning: Limitations and opportunities. MIT press.
  5. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We Need to Consider Disagreement in Evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, Kenneth Church, Mark Liberman, and Valia Kordoni (Eds.). Association for Computational Linguistics, Online, 15-21. https://doi.org/10.18653/v1/2021.bppf-1.3 · DOI
  6. Julia B Bear and Anita Williams Woolley. 2011. The role of gender in team collaboration and performance. Interdisciplinary science reviews 36, 2 (2011), 146-153.
  7. Stevie Bergman, Nahema Marchal, John Mellor, Shakir Mohamed, Iason Gabriel, and William Isaac. 2024. STELA: a community-centred approach to norm elicitation for AI alignment. Scientific Reports 14, 1 (2024), 6616.
  8. Emily Black, Hadi Elzayn, Alexandra Chouldechova, Jacob Goldin, and Daniel Ho. 2022. Algorithmic fairness and vertical equity: Income fairness with irs tax audit models. In Proceedings of the 2022 ACM Conference on Fairness, Account- ability, and Transparency. 1479-1503.
  9. James Bohman. 2007. Political communication and the epistemic value of diversity: Deliberation and legitimation in media societies. Communication Theory 17, 4 (2007), 348-355.
  10. Rishi Bommasani, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. 2022. Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? Advances in Neural Information Processing Systems 35 (2022), 3663-3678.
  11. Daniel Braun. 2024. I beg to differ: how disagreement is handled in the anno- tation of legal machine learning data sets. Artificial intelligence and law 32, 3 (2024), 839-862.
  12. Liam Kofi Bright. 2024. Duboisian leadership through standpoint epistemology. The Monist 107, 1 (2024), 82-97.
  13. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a per- spectivist turn in ground truthing for predictive computing. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 6860-6868.
  14. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217 (2023). · arXiv
  15. Stephanie Castelin and Grace White. 2022. "I'ma strong independent Black woman": The strong Black woman schema and mental health in college-aged Black women. Psychology of Women Quarterly 46, 2 (2022), 196-208.
  16. David Christensen. 2009. Disagreement as Evidence: The Epistemology of Controversy. Philosophy Compass 4, 5 (2009), 756-767. https://doi.org/10.1111/ j.1747-9991.2009.00237.x · DOI
  17. Robert B Cialdini and Noah J Goldstein. 2004. Social influence: Compliance and conformity. Annu. Rev. Psychol. 55, 1 (2004), 591-621.
  18. Matteo Colombo and Stephan Hartmann. 2017. Bayesian cognitive science, unification, and explanation. The British Journal for the Philosophy of Science (2017).
  19. Amanda Coston, Anna Kawakami, Haiyi Zhu, Ken Holstein, and Hoda Heidari. 2023. A validity perspective on evaluating the justified use of data-driven decision-making algorithms. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 690-704.
  20. Helen De Cruz and Johan De Smedt. 2013. The Value of Epistemic Disagreement in Scientific Practice. The Case of Homo Floresiensis. Studies in History and Philosophy of Science Part A 44, 2 (2013), 169-177. https://doi.org/10.1016/j. shpsa.2013.02.002 · DOI
  21. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics 10 (2022), 92-110.
  22. Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2023. The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (Boston, MA, USA) (EAAMO '23). Association for Computing Machinery, New York, NY, USA, Article 37, 23 pages. https://doi.org/10.1145/3617694.3623261 · DOI · Paper page
  23. Morton Deutsch and Harold B Gerard. 1955. A study of normative and informa- tional social influences upon individual judgment. The journal of abnormal and social psychology 51, 3 (1955), 629.
  24. Mark Díaz, Ian Kivlichan, Rachel Rosen, Dylan Baker, Razvan Amironesei, Vin- odkumar Prabhakaran, and Emily Denton. 2022. Crowdworksheets: Accounting for individual and collective identities underlying crowdsourced dataset anno- tation. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 2342-2351.
  25. Xiaoni Duan, Chien-Ju Ho, and Ming Yin. 2020. Does exposure to diverse perspectives mitigate biases in crowdwork? an explorative study. In Proceedings of the aaai conference on human computation and crowdsourcing, Vol. 8. 155-158.
  26. Joann G Elmore, Gary M Longton, Patricia A Carney, Berta M Geller, Tracy Onega, Anna NA Tosteson, Heidi D Nelson, Margaret S Pepe, Kimberly H Allison, Stuart J Schnitt, et al. 2015. Diagnostic concordance among pathologists interpreting breast biopsy specimens. Jama 313, 11 (2015), 1122-1132.
  27. Sina Fazelpour and David Danks. 2021. Algorithmic bias: Senses, sources, solutions. Philosophy Compass 16, 8 (2021), e12760.
  28. Sina Fazelpour and Maria De-Arteaga. 2022. Diversity in sociotechnical machine learning systems. Big Data & Society 9, 1 (2022), 20539517221082027.
  29. Sina Fazelpour, Zachary C Lipton, and David Danks. 2022. Algorithmic fairness and the situated dynamics of justice. Canadian Journal of Philosophy 52, 1 (2022), 44-60.
  30. Sina Fazelpour and Daniel Steel. 2022. Diversity, trust, and conformity: A simulation study. Philosophy of Science 89, 2 (2022), 209-231.
  31. Will Fleisher. 2018. Rational Endorsement. Philosophical Studies 175, 10 (2018), 2649-2675. https://doi.org/10.1007/s11098-017-0976-4 · DOI
  32. Will Fleisher. 2019. Endorsement and Assertion. Noûs 55, 2 (2019), 363-384. https://doi.org/10.1111/nous.12315 · DOI
  33. Will Fleisher. 2020. How to Endorse Conciliationism. Synthese 198, 10 (2020), 9913-9939. https://doi.org/10.1007/s11229-020-02695-z · DOI
  34. Will Fleisher. 2022. Understanding, idealization, and explainable AI. Episteme 19, 4 (2022), 534-560.
  35. Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagreement for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 6715-6726.
  36. Eve Fleisig, Su Lin Blodgett, Dan Klein, and Zeerak Talat. 2024. The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels. arXiv preprint arXiv:2405.05860 (2024). · arXiv
  37. Bryan Frances and Jonathan Matheson. 2024. Disagreement. In The Stanford Encyclopedia of Philosophy (Winter 2024 ed.), Edward N. Zalta and Uri Nodelman (Eds.). Metaphysics Research Lab, Stanford University.
  38. Phoebe Friesen and Jordan Goldstein. 2022. Standpoint theory and the psy sciences: Can marginalization and critical engagement lead to an epistemic advantage? Hypatia 37, 4 (2022), 659-687.
  39. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 (2022). · arXiv
  40. Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1-19.
  41. Sandra Harding. 2004. A Socially Relevant Philosophy of Science? Resources From Standpoint Theory's Controversiality. Hypatia 19, 1 (2004), 25-47. https: //doi.org/10.1111/j.1527-2001.2004.tb01267.x · DOI
  42. Sandra Harding. 2009. Standpoint theories: Productively controversial. Hypatia 24, 4 (2009), 192-200.
  43. Sandra Harding. 2019. Objectivity and diversity: Another logic of scientific research. University of Chicago Press.
  44. Sandra G Harding. 2004. The feminist standpoint theory reader: Intellectual and political controversies. Psychology Press.
  45. Remco Heesen, Liam Kofi Bright, and Andrew Zucker. 2019. Vindicating method- ological triangulation. Synthese 196 (2019), 3067-3081.
  46. Babak Heydari, Pedram Heydari, and Mohsen Mosleh. 2020. Not all bridges connect: integration in multi-community networks. The Journal of Mathematical Sociology 44, 4 (2020), 199-220.
  47. Babak Heydari, Mohsen Mosleh, and Kia Dalili. 2015. Efficient network struc- tures with separable heterogeneous connection costs. Economics Letters 134 (2015), 82-85.
  48. Chris Jay Hoofnagle, Ashkan Soltani, Nathaniel Good, and Dietrich J Wambach. 2012. Behavioral advertising: The offer you can't refuse. Harv. L. & Pol'y Rev. 6 (2012), 273.
  49. Sophie Horowitz. 2025. Higher-Order Evidence. In The Stanford Encyclopedia of Philosophy (Spring 2025 ed.), Edward N. Zalta and Uri Nodelman (Eds.). Metaphysics Research Lab, Stanford University.
  50. Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. 2024. Collective constitutional ai: Aligning a language model with public input. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. 1395-1417.
  51. Kristen Intemann. 2010. 25 years of feminist empiricism and standpoint theory: Where are we now? Hypatia 25, 4 (2010), 778-796.
  52. Shomik Jain, Vinith Suriyakumar, Kathleen Creel, and Ashia Wilson. 2024. Algorithmic Pluralism: A Structural Approach To Equal Opportunity. In The 2024 ACM Conference on Fairness, Accountability, and Transparency. 197-206.
  53. Akshita Jha, Aida Davani, Chandan K Reddy, Shachi Dave, Vinodkumar Prab- hakaran, and Sunipa Dev. 2023. SeeGULL: A stereotype benchmark with broad geo-cultural coverage leveraging generative models. arXiv preprint arXiv:2305.11840 (2023). · arXiv
  54. Shivani Kapania, Alex S Taylor, and Ding Wang. 2023. A hunt for the snark: Annotator diversity in data practices. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1-15.
  55. S Vittal Katikireddi and Sean A Valles. 2015. Coupled ethical-epistemic anal- ysis of public health research and practice: Categorizing variables to improve population health and equity. American journal of public health 105, 1 (2015), e36-e42.
  56. Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. 2024. The PRISM Alignment Project: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multi- cultural Alignment of Large Language Models. arXiv preprint arXiv:2404.16019 (2024). · arXiv
  57. Philip Kitcher. 1990. The Division of Cognitive Labor. Journal of Philosophy 87, 1 (1990), 5-22. https://doi.org/10.2307/2026796 · DOI
  58. Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Cass R Sunstein. 2018. Discrimination in the Age of Algorithms. Journal of Legal Analysis 10 (2018), 113-174.
  59. Jon Kleinberg and Manish Raghavan. 2021. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences 118, 22 (2021), e2018340118.
  60. Tzu-Sheng Kuo, Quan Ze Chen, Amy X Zhang, Jane Hsieh, Haiyi Zhu, and Kenneth Holstein. 2024. PolicyCraft: Supporting Collaborative and Partici- patory Policy Design through Case-Grounded Deliberation. arXiv preprint arXiv:2409.15644 (2024). · arXiv
  61. Tzu-Sheng Kuo, Aaron Lee Halfaker, Zirui Cheng, Jiwoo Kim, Meng-Hsin Wu, Tongshuang Wu, Kenneth Holstein, and Haiyi Zhu. 2024. Wikibench: Community-driven data curation for ai evaluation on wikipedia. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1-24.
  62. Jaakko Kuorikoski and Caterina Marchionni. 2016. Evidential diversity and the triangulation of phenomena. Philosophy of Science 83, 2 (2016), 227-247.
  63. Hélène Landemore. 2017. Beyond the fact of disagreement? The epistemic turn in deliberative democracy. Social Epistemology 31, 3 (2017), 277-295.
  64. Jürgen Landes. 2021. The variety of evidence thesis and its independence of degrees of independence. Synthese 198, 11 (2021), 10611-10641.
  65. Seth Lazar. 2024. Legitimacy, Authority, and Democratic. Oxford Studies in Political Philosophy Volume 10 (2024), 28.
  66. Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, et al. 2019. WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-35.
  67. Adam Dahlgren Lindström, Leila Methnani, Lea Krause, Petter Ericson, Íñigo Martínez de Rituerto de Troya, Dimitri Coelho Mollo, and Roel Dobbe. 2024. AI Alignment through Reinforcement Learning from Human Feedback? Contradic- tions and Limitations. arXiv preprint arXiv:2406.18346 (2024). · arXiv
  68. Zachary Lipton, Julian McAuley, and Alexandra Chouldechova. 2018. Does mitigating ML's impact disparity require treatment disparity? Advances in neural information processing systems 31 (2018).
  69. Helen E Longino. 1990. Science as social knowledge: Values and objectivity in scientific inquiry. Princeton University Press.
  70. Jonathan Matheson. 2015. The Epistemology of Disagreement. Palgrave, New York.
  71. Hugo Mercier and Dan Sperber. 2011. Argumentation: Its Adaptiveness and Efficacy. Behavioral and Brain Sciences 34, 2 (2011), 94-111. https://doi.org/10. 1017/s0140525x10003031 · DOI
  72. Chantal Mouffe. 2000. Deliberative democracy or agonistic pluralism. Reihe Poli- tikwissenschaft / Institut für Höhere Studien, Abt. Politikwissenschaft, Vol. 72. Institut für Höhere Studien (IHS), Wien, Wien. 17 pages.
  73. Ryan Muldoon. 2013. Diversity and the division of cognitive labor. Philosophy Compass 8, 2 (2013), 117-125.
  74. Diana C Mutz. 2006. Hearing the other side: Deliberative versus participatory democracy. Cambridge University Press.
  75. Hasti Narimanzadeh, Arash Badie-Modiri, Iuliia G Smirnova, and Ted Hsuan Yun Chen. 2023. Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote method. Proceedings of the ACM on Human-Computer Interaction 7, CSCW2 (2023), 1-29.
  76. Helen Nissenbaum. 2019. Contextual integrity up and down the data food chain. Theoretical inquiries in law 20, 1 (2019), 221-256.
  77. Scott E Page. 2017. The diversity bonus. Princeton University Press.
  78. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. 2022. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251 (2022). · arXiv
  79. Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Rus- sakovsky. 2019. Human uncertainty makes classification more robust. In Pro- ceedings of the IEEE/CVF international conference on computer vision. 9617-9626.
  80. Katherine W Phillips. 2017. Commentary. What is the real value of diversity in organizations? Questioning our assumptions. In The diversity bonus. Princeton University Press, 223-246.
  81. Katherine W Phillips, Katie A Liljenquist, and Margaret A Neale. 2009. Is the pain worth the gain? The advantages and liabilities of agreeing with socially distinct newcomers. Personality and Social Psychology Bulletin 35, 3 (2009), 336-350.
  82. Katherine W Phillips and Denise Lewin Loyd. 2006. When surface and deep- level diversity collide: The effects on dissenting group members. Organizational behavior and human decision processes 99, 2 (2006), 143-160.
  83. Gaile Pohlhaus. 2002. Knowing communities: An investigation of Harding's standpoint epistemology. Social epistemology 16, 3 (2002), 283-293.
  84. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Díaz. 2021. On releasing Annotator-Level labels and information in datasets. arXiv preprint arXiv:2110.05699 (2021). · arXiv
  85. Quinetta M Roberson. 2019. Diversity in the workplace: A review, synthesis, and future research agenda. Annual Review of Organizational Psychology and Organizational Behavior 6 (2019), 69-88.
  86. Pratik S Sachdeva, Renata Barreto, Claudia von Vacano, and Chris J Kennedy. 2022. Assessing annotator identity sensitivity via item response theory: A case study in a hate speech corpus. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 1585-1603.
  87. Catharine Saint-Croix. 2020. Privilege and Position: Formal Tools for Standpoint Epistemology. Res Philosophica 97, 4 (2020), 489-524. https://doi.org/10.11612/ resphil.1953 · DOI · Paper page
  88. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect?. In International Conference on Machine Learning. PMLR, 29971-30004.
  89. Mike Schaekermann, Graeme Beaton, Minahz Habib, Andrew Lim, Kate Larson, and Edith Law. 2019. Understanding expert disagreement in medical data analysis through structured adjudication. Proceedings of the ACM on Human- Computer Interaction 3, CSCW (2019), 1-23.
  90. Jan Maarten Schraagen, Susan F Chipman, and Valerie L Shalin. 2000. Cognitive task analysis. Psychology Press.
  91. Christopher Small, Michael Bjorkegren, Timo Erkkilä, Lynette Shaw, and Colin Megill. 2021. Polis: Scaling deliberation by mapping high dimensional opinion spaces. Recerca: revista de pensament i anàlisi 26, 2 (2021).
  92. Miriam Solomon. 2007. Social empiricism. MIT press.
  93. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. 2024. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070 (2024). · arXiv
  94. Dominik Stammbach, Philine Widmer, Eunjung Cho, Caglar Gulcehre, and Elliott Ash. 2024. Aligning Large Language Models with Diverse Political Viewpoints. arXiv preprint arXiv:2406.14155 (2024). · arXiv
  95. Daniel Steel, Sina Fazelpour, Kinley Gillette, Bianca Crewe, and Michael Burgess. 2018. Multiple diversity concepts and their ethical-epistemic implications. European Journal for Philosophy of Science 8, 3 (2018), 761-780.
  96. Michael Strevens. 2003. The Role of the Priority Rule in Science. Journal of Philosophy 100, 2 (2003), 55-79. https://doi.org/10.5840/jphil2003100224 · DOI
  97. Harini Suresh, Emily Tseng, Meg Young, Mary Gray, Emma Pierson, and Karen Levy. 2024. Participation in the age of foundation models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. 1609-1621.
  98. Johanna Thoma. 2015. The Epistemic Division of Labor Revisited. Philosophy of Science 82, 3 (2015), 454-472. https://doi.org/10.1086/681768 · DOI
  99. Briana Toole. 2021. Recent Work in Standpoint Epistemology. Analysis 81, 2 (2021), 338-350. https://doi.org/10.1093/analys/anab026 [114 · DOI
  100. Nancy Tuana. 2010. Leading with ethics, aiming for policy: New opportunities for philosophy of science. Synthese 177 (2010), 471-492.
  101. Nancy Tuana. 2013. Embedding philosophers in the practices of science: bringing humanities to the sciences. Synthese 190, 11 (2013), 1955-1973.
  102. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research 72 (2021), 1385-1470.
  103. Hamed Valizadegan, Quang Nguyen, and Milos Hauskrecht. 2012. Learning medical diagnosis models from multiple experts. In AMIA annual symposium proceedings, Vol. 2012. 921.
  104. Kate Vredenburgh. 2022. The right to explanation. Journal of Political Philosophy 30, 2 (2022), 209-229.
  105. Zeerak Waseem. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the first workshop on NLP and computational social science. 138-142.
  106. Amy A Winecoff and Miranda Bogen. 2024. Improving governance outcomes through AI documentation: Bridging theory and practice. arXiv preprint arXiv:2409.08960 (2024). · arXiv
  107. Alison Wylie. 2003. Why Standpoint Matters. In Science and other cultures: issues in philosophies of science and technology, Robert Figueroa and Sandra G. Harding (Eds.). Routledge, 26-48.
  108. Alison Wylie. 2017. What knowers know well: standpoint theory and gender archeology. Scientiae Studia 15, 1 (2017), 13-38.
  109. Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using "annotator ratio- nales" to improve machine learning for text categorization. In Human language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of the main conference. 260-267.
  110. Weiyu Zhang. 2015. Perceived procedural fairness in deliberation: Predictors and effects. Communication Research 42, 3 (2015), 345-364.

Cited by

Sorted by citations

Authors
Fazelpour, S., & Fleisher, W.
Affiliations
  • Northeastern University (US)
  • Georgetown University (US)
ORCID
Venue
Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2025-06-23
Pages
2138-2150
DOI
10.1145/3715275.3732146
OpenAlex
W4411550467
Cited by
10
References
13
Citations by year
2026: 8 · 2025: 2
FWCI
10.16
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • PhilPapers (PhilPapers Foundation) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Schmidt Sciences AI2050 Early Career Fellowship
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Feminist Epistemology and Gender Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
disagreement, AI design, AI evaluation, AI alignment, marginalized perspectives, participatory AI, pluralism, normative framework, AI governance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review